Sep 17 19:06:22.733286 ip-10-0-131-232 systemd[1]: Starting Kubernetes Kubelet...
Sep 17 19:06:23.139766 ip-10-0-131-232 kubenswrapper[2582]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 17 19:06:23.139766 ip-10-0-131-232 kubenswrapper[2582]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Sep 17 19:06:23.139766 ip-10-0-131-232 kubenswrapper[2582]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 17 19:06:23.139766 ip-10-0-131-232 kubenswrapper[2582]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI.
Sep 17 19:06:23.139766 ip-10-0-131-232 kubenswrapper[2582]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 17 19:06:23.142598 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:06:23.142516    2582 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime"
Sep 17 19:06:23.150577 ip-10-0-131-232 kubenswrapper[2582]: W0917 19:06:23.150560    2582 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 17 19:06:23.150577 ip-10-0-131-232 kubenswrapper[2582]: W0917 19:06:23.150575    2582 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 17 19:06:23.150577 ip-10-0-131-232 kubenswrapper[2582]: W0917 19:06:23.150579    2582 feature_gate.go:328] unrecognized feature gate: Example2
Sep 17 19:06:23.150663 ip-10-0-131-232 kubenswrapper[2582]: W0917 19:06:23.150582    2582 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 17 19:06:23.150663 ip-10-0-131-232 kubenswrapper[2582]: W0917 19:06:23.150585    2582 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 17 19:06:23.150663 ip-10-0-131-232 kubenswrapper[2582]: W0917 19:06:23.150588    2582 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 17 19:06:23.150663 ip-10-0-131-232 kubenswrapper[2582]: W0917 19:06:23.150590    2582 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 17 19:06:23.150663 ip-10-0-131-232 kubenswrapper[2582]: W0917 19:06:23.150593    2582 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 17 19:06:23.150663 ip-10-0-131-232 kubenswrapper[2582]: W0917 19:06:23.150596    2582 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 17 19:06:23.150663 ip-10-0-131-232 kubenswrapper[2582]: W0917 19:06:23.150598    2582 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 17 19:06:23.150663 ip-10-0-131-232 kubenswrapper[2582]: W0917 19:06:23.150601    2582 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 17 19:06:23.150663 ip-10-0-131-232 kubenswrapper[2582]: W0917 19:06:23.150604    2582 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 17 19:06:23.150663 ip-10-0-131-232 kubenswrapper[2582]: W0917 19:06:23.150606    2582 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 17 19:06:23.150663 ip-10-0-131-232 kubenswrapper[2582]: W0917 19:06:23.150608    2582 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 17 19:06:23.150663 ip-10-0-131-232 kubenswrapper[2582]: W0917 19:06:23.150611    2582 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 17 19:06:23.150663 ip-10-0-131-232 kubenswrapper[2582]: W0917 19:06:23.150613    2582 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 17 19:06:23.150663 ip-10-0-131-232 kubenswrapper[2582]: W0917 19:06:23.150621    2582 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 17 19:06:23.150663 ip-10-0-131-232 kubenswrapper[2582]: W0917 19:06:23.150623    2582 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 17 19:06:23.150663 ip-10-0-131-232 kubenswrapper[2582]: W0917 19:06:23.150626    2582 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 17 19:06:23.150663 ip-10-0-131-232 kubenswrapper[2582]: W0917 19:06:23.150631    2582 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 17 19:06:23.150663 ip-10-0-131-232 kubenswrapper[2582]: W0917 19:06:23.150635    2582 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 17 19:06:23.150663 ip-10-0-131-232 kubenswrapper[2582]: W0917 19:06:23.150638    2582 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 17 19:06:23.150663 ip-10-0-131-232 kubenswrapper[2582]: W0917 19:06:23.150641    2582 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 17 19:06:23.151149 ip-10-0-131-232 kubenswrapper[2582]: W0917 19:06:23.150643    2582 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 17 19:06:23.151149 ip-10-0-131-232 kubenswrapper[2582]: W0917 19:06:23.150646    2582 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 17 19:06:23.151149 ip-10-0-131-232 kubenswrapper[2582]: W0917 19:06:23.150649    2582 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 17 19:06:23.151149 ip-10-0-131-232 kubenswrapper[2582]: W0917 19:06:23.150652    2582 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 17 19:06:23.151149 ip-10-0-131-232 kubenswrapper[2582]: W0917 19:06:23.150655    2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 17 19:06:23.151149 ip-10-0-131-232 kubenswrapper[2582]: W0917 19:06:23.150658    2582 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 17 19:06:23.151149 ip-10-0-131-232 kubenswrapper[2582]: W0917 19:06:23.150660    2582 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 17 19:06:23.151149 ip-10-0-131-232 kubenswrapper[2582]: W0917 19:06:23.150663    2582 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 17 19:06:23.151149 ip-10-0-131-232 kubenswrapper[2582]: W0917 19:06:23.150666    2582 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 17 19:06:23.151149 ip-10-0-131-232 kubenswrapper[2582]: W0917 19:06:23.150669    2582 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 17 19:06:23.151149 ip-10-0-131-232 kubenswrapper[2582]: W0917 19:06:23.150672    2582 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 17 19:06:23.151149 ip-10-0-131-232 kubenswrapper[2582]: W0917 19:06:23.150675    2582 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 17 19:06:23.151149 ip-10-0-131-232 kubenswrapper[2582]: W0917 19:06:23.150678    2582 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 17 19:06:23.151149 ip-10-0-131-232 kubenswrapper[2582]: W0917 19:06:23.150681    2582 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 17 19:06:23.151149 ip-10-0-131-232 kubenswrapper[2582]: W0917 19:06:23.150683    2582 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 17 19:06:23.151149 ip-10-0-131-232 kubenswrapper[2582]: W0917 19:06:23.150686    2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 17 19:06:23.151149 ip-10-0-131-232 kubenswrapper[2582]: W0917 19:06:23.150688    2582 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 17 19:06:23.151149 ip-10-0-131-232 kubenswrapper[2582]: W0917 19:06:23.150691    2582 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 17 19:06:23.151149 ip-10-0-131-232 kubenswrapper[2582]: W0917 19:06:23.150693    2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 17 19:06:23.151702 ip-10-0-131-232 kubenswrapper[2582]: W0917 19:06:23.150696    2582 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 17 19:06:23.151702 ip-10-0-131-232 kubenswrapper[2582]: W0917 19:06:23.150699    2582 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 17 19:06:23.151702 ip-10-0-131-232 kubenswrapper[2582]: W0917 19:06:23.150701    2582 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 17 19:06:23.151702 ip-10-0-131-232 kubenswrapper[2582]: W0917 19:06:23.150703    2582 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 17 19:06:23.151702 ip-10-0-131-232 kubenswrapper[2582]: W0917 19:06:23.150706    2582 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 17 19:06:23.151702 ip-10-0-131-232 kubenswrapper[2582]: W0917 19:06:23.150708    2582 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 17 19:06:23.151702 ip-10-0-131-232 kubenswrapper[2582]: W0917 19:06:23.150711    2582 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 17 19:06:23.151702 ip-10-0-131-232 kubenswrapper[2582]: W0917 19:06:23.150713    2582 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 17 19:06:23.151702 ip-10-0-131-232 kubenswrapper[2582]: W0917 19:06:23.150716    2582 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 17 19:06:23.151702 ip-10-0-131-232 kubenswrapper[2582]: W0917 19:06:23.150718    2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 17 19:06:23.151702 ip-10-0-131-232 kubenswrapper[2582]: W0917 19:06:23.150720    2582 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 17 19:06:23.151702 ip-10-0-131-232 kubenswrapper[2582]: W0917 19:06:23.150723    2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 17 19:06:23.151702 ip-10-0-131-232 kubenswrapper[2582]: W0917 19:06:23.150725    2582 feature_gate.go:328] unrecognized feature gate: Example
Sep 17 19:06:23.151702 ip-10-0-131-232 kubenswrapper[2582]: W0917 19:06:23.150728    2582 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 17 19:06:23.151702 ip-10-0-131-232 kubenswrapper[2582]: W0917 19:06:23.150731    2582 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 17 19:06:23.151702 ip-10-0-131-232 kubenswrapper[2582]: W0917 19:06:23.150734    2582 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 17 19:06:23.151702 ip-10-0-131-232 kubenswrapper[2582]: W0917 19:06:23.150736    2582 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 17 19:06:23.151702 ip-10-0-131-232 kubenswrapper[2582]: W0917 19:06:23.150739    2582 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 17 19:06:23.151702 ip-10-0-131-232 kubenswrapper[2582]: W0917 19:06:23.150741    2582 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 17 19:06:23.151702 ip-10-0-131-232 kubenswrapper[2582]: W0917 19:06:23.150744    2582 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 17 19:06:23.152305 ip-10-0-131-232 kubenswrapper[2582]: W0917 19:06:23.150746    2582 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 17 19:06:23.152305 ip-10-0-131-232 kubenswrapper[2582]: W0917 19:06:23.150748    2582 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 17 19:06:23.152305 ip-10-0-131-232 kubenswrapper[2582]: W0917 19:06:23.150751    2582 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 17 19:06:23.152305 ip-10-0-131-232 kubenswrapper[2582]: W0917 19:06:23.150754    2582 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 17 19:06:23.152305 ip-10-0-131-232 kubenswrapper[2582]: W0917 19:06:23.150758    2582 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 17 19:06:23.152305 ip-10-0-131-232 kubenswrapper[2582]: W0917 19:06:23.150761    2582 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 17 19:06:23.152305 ip-10-0-131-232 kubenswrapper[2582]: W0917 19:06:23.150763    2582 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 17 19:06:23.152305 ip-10-0-131-232 kubenswrapper[2582]: W0917 19:06:23.150766    2582 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 17 19:06:23.152305 ip-10-0-131-232 kubenswrapper[2582]: W0917 19:06:23.150769    2582 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 17 19:06:23.152305 ip-10-0-131-232 kubenswrapper[2582]: W0917 19:06:23.150771    2582 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 17 19:06:23.152305 ip-10-0-131-232 kubenswrapper[2582]: W0917 19:06:23.150774    2582 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 17 19:06:23.152305 ip-10-0-131-232 kubenswrapper[2582]: W0917 19:06:23.150776    2582 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 17 19:06:23.152305 ip-10-0-131-232 kubenswrapper[2582]: W0917 19:06:23.150779    2582 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 17 19:06:23.152305 ip-10-0-131-232 kubenswrapper[2582]: W0917 19:06:23.150782    2582 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 17 19:06:23.152305 ip-10-0-131-232 kubenswrapper[2582]: W0917 19:06:23.150784    2582 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 17 19:06:23.152305 ip-10-0-131-232 kubenswrapper[2582]: W0917 19:06:23.150787    2582 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 17 19:06:23.152305 ip-10-0-131-232 kubenswrapper[2582]: W0917 19:06:23.150789    2582 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 17 19:06:23.152305 ip-10-0-131-232 kubenswrapper[2582]: W0917 19:06:23.150792    2582 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 17 19:06:23.152305 ip-10-0-131-232 kubenswrapper[2582]: W0917 19:06:23.150794    2582 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 17 19:06:23.152305 ip-10-0-131-232 kubenswrapper[2582]: W0917 19:06:23.150797    2582 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 17 19:06:23.152868 ip-10-0-131-232 kubenswrapper[2582]: W0917 19:06:23.150799    2582 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 17 19:06:23.152868 ip-10-0-131-232 kubenswrapper[2582]: W0917 19:06:23.150802    2582 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 17 19:06:23.152868 ip-10-0-131-232 kubenswrapper[2582]: W0917 19:06:23.150804    2582 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 17 19:06:23.152868 ip-10-0-131-232 kubenswrapper[2582]: W0917 19:06:23.150807    2582 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 17 19:06:23.152868 ip-10-0-131-232 kubenswrapper[2582]: W0917 19:06:23.150809    2582 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 17 19:06:23.152868 ip-10-0-131-232 kubenswrapper[2582]: W0917 19:06:23.150812    2582 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 17 19:06:23.152868 ip-10-0-131-232 kubenswrapper[2582]: W0917 19:06:23.150816    2582 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 17 19:06:23.152868 ip-10-0-131-232 kubenswrapper[2582]: W0917 19:06:23.150818    2582 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 17 19:06:23.152868 ip-10-0-131-232 kubenswrapper[2582]: W0917 19:06:23.150822    2582 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 17 19:06:23.152868 ip-10-0-131-232 kubenswrapper[2582]: W0917 19:06:23.150824    2582 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 17 19:06:23.152868 ip-10-0-131-232 kubenswrapper[2582]: W0917 19:06:23.150827    2582 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 17 19:06:23.152868 ip-10-0-131-232 kubenswrapper[2582]: W0917 19:06:23.150829    2582 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 17 19:06:23.152868 ip-10-0-131-232 kubenswrapper[2582]: W0917 19:06:23.150831    2582 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 17 19:06:23.152868 ip-10-0-131-232 kubenswrapper[2582]: W0917 19:06:23.150835    2582 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 17 19:06:23.152868 ip-10-0-131-232 kubenswrapper[2582]: W0917 19:06:23.150839    2582 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 17 19:06:23.152868 ip-10-0-131-232 kubenswrapper[2582]: W0917 19:06:23.150842    2582 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 17 19:06:23.152868 ip-10-0-131-232 kubenswrapper[2582]: W0917 19:06:23.150845    2582 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 17 19:06:23.153514 ip-10-0-131-232 kubenswrapper[2582]: W0917 19:06:23.153496    2582 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 17 19:06:23.153514 ip-10-0-131-232 kubenswrapper[2582]: W0917 19:06:23.153512    2582 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 17 19:06:23.153514 ip-10-0-131-232 kubenswrapper[2582]: W0917 19:06:23.153516    2582 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 17 19:06:23.153607 ip-10-0-131-232 kubenswrapper[2582]: W0917 19:06:23.153519    2582 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 17 19:06:23.153607 ip-10-0-131-232 kubenswrapper[2582]: W0917 19:06:23.153522    2582 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 17 19:06:23.153607 ip-10-0-131-232 kubenswrapper[2582]: W0917 19:06:23.153524    2582 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 17 19:06:23.153607 ip-10-0-131-232 kubenswrapper[2582]: W0917 19:06:23.153529    2582 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 17 19:06:23.153607 ip-10-0-131-232 kubenswrapper[2582]: W0917 19:06:23.153531    2582 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 17 19:06:23.153607 ip-10-0-131-232 kubenswrapper[2582]: W0917 19:06:23.153534    2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 17 19:06:23.153607 ip-10-0-131-232 kubenswrapper[2582]: W0917 19:06:23.153536    2582 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 17 19:06:23.153607 ip-10-0-131-232 kubenswrapper[2582]: W0917 19:06:23.153539    2582 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 17 19:06:23.153607 ip-10-0-131-232 kubenswrapper[2582]: W0917 19:06:23.153542    2582 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 17 19:06:23.153607 ip-10-0-131-232 kubenswrapper[2582]: W0917 19:06:23.153545    2582 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 17 19:06:23.153607 ip-10-0-131-232 kubenswrapper[2582]: W0917 19:06:23.153548    2582 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 17 19:06:23.153607 ip-10-0-131-232 kubenswrapper[2582]: W0917 19:06:23.153550    2582 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 17 19:06:23.153607 ip-10-0-131-232 kubenswrapper[2582]: W0917 19:06:23.153552    2582 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 17 19:06:23.153607 ip-10-0-131-232 kubenswrapper[2582]: W0917 19:06:23.153555    2582 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 17 19:06:23.153607 ip-10-0-131-232 kubenswrapper[2582]: W0917 19:06:23.153557    2582 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 17 19:06:23.153607 ip-10-0-131-232 kubenswrapper[2582]: W0917 19:06:23.153560    2582 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 17 19:06:23.153607 ip-10-0-131-232 kubenswrapper[2582]: W0917 19:06:23.153562    2582 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 17 19:06:23.153607 ip-10-0-131-232 kubenswrapper[2582]: W0917 19:06:23.153566    2582 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 17 19:06:23.153607 ip-10-0-131-232 kubenswrapper[2582]: W0917 19:06:23.153571    2582 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 17 19:06:23.154095 ip-10-0-131-232 kubenswrapper[2582]: W0917 19:06:23.153573    2582 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 17 19:06:23.154095 ip-10-0-131-232 kubenswrapper[2582]: W0917 19:06:23.153576    2582 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 17 19:06:23.154095 ip-10-0-131-232 kubenswrapper[2582]: W0917 19:06:23.153578    2582 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 17 19:06:23.154095 ip-10-0-131-232 kubenswrapper[2582]: W0917 19:06:23.153581    2582 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 17 19:06:23.154095 ip-10-0-131-232 kubenswrapper[2582]: W0917 19:06:23.153584    2582 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 17 19:06:23.154095 ip-10-0-131-232 kubenswrapper[2582]: W0917 19:06:23.153587    2582 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 17 19:06:23.154095 ip-10-0-131-232 kubenswrapper[2582]: W0917 19:06:23.153589    2582 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 17 19:06:23.154095 ip-10-0-131-232 kubenswrapper[2582]: W0917 19:06:23.153592    2582 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 17 19:06:23.154095 ip-10-0-131-232 kubenswrapper[2582]: W0917 19:06:23.153594    2582 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 17 19:06:23.154095 ip-10-0-131-232 kubenswrapper[2582]: W0917 19:06:23.153597    2582 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 17 19:06:23.154095 ip-10-0-131-232 kubenswrapper[2582]: W0917 19:06:23.153599    2582 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 17 19:06:23.154095 ip-10-0-131-232 kubenswrapper[2582]: W0917 19:06:23.153602    2582 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 17 19:06:23.154095 ip-10-0-131-232 kubenswrapper[2582]: W0917 19:06:23.153604    2582 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 17 19:06:23.154095 ip-10-0-131-232 kubenswrapper[2582]: W0917 19:06:23.153607    2582 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 17 19:06:23.154095 ip-10-0-131-232 kubenswrapper[2582]: W0917 19:06:23.153611    2582 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 17 19:06:23.154095 ip-10-0-131-232 kubenswrapper[2582]: W0917 19:06:23.153615    2582 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 17 19:06:23.154095 ip-10-0-131-232 kubenswrapper[2582]: W0917 19:06:23.153618    2582 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 17 19:06:23.154095 ip-10-0-131-232 kubenswrapper[2582]: W0917 19:06:23.153621    2582 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 17 19:06:23.154095 ip-10-0-131-232 kubenswrapper[2582]: W0917 19:06:23.153623    2582 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 17 19:06:23.154095 ip-10-0-131-232 kubenswrapper[2582]: W0917 19:06:23.153625    2582 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 17 19:06:23.154637 ip-10-0-131-232 kubenswrapper[2582]: W0917 19:06:23.153628    2582 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 17 19:06:23.154637 ip-10-0-131-232 kubenswrapper[2582]: W0917 19:06:23.153631    2582 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 17 19:06:23.154637 ip-10-0-131-232 kubenswrapper[2582]: W0917 19:06:23.153634    2582 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 17 19:06:23.154637 ip-10-0-131-232 kubenswrapper[2582]: W0917 19:06:23.153637    2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 17 19:06:23.154637 ip-10-0-131-232 kubenswrapper[2582]: W0917 19:06:23.153639    2582 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 17 19:06:23.154637 ip-10-0-131-232 kubenswrapper[2582]: W0917 19:06:23.153641    2582 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 17 19:06:23.154637 ip-10-0-131-232 kubenswrapper[2582]: W0917 19:06:23.153644    2582 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 17 19:06:23.154637 ip-10-0-131-232 kubenswrapper[2582]: W0917 19:06:23.153646    2582 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 17 19:06:23.154637 ip-10-0-131-232 kubenswrapper[2582]: W0917 19:06:23.153649    2582 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 17 19:06:23.154637 ip-10-0-131-232 kubenswrapper[2582]: W0917 19:06:23.153651    2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 17 19:06:23.154637 ip-10-0-131-232 kubenswrapper[2582]: W0917 19:06:23.153654    2582 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 17 19:06:23.154637 ip-10-0-131-232 kubenswrapper[2582]: W0917 19:06:23.153656    2582 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 17 19:06:23.154637 ip-10-0-131-232 kubenswrapper[2582]: W0917 19:06:23.153659    2582 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 17 19:06:23.154637 ip-10-0-131-232 kubenswrapper[2582]: W0917 19:06:23.153661    2582 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 17 19:06:23.154637 ip-10-0-131-232 kubenswrapper[2582]: W0917 19:06:23.153664    2582 feature_gate.go:328] unrecognized feature gate: Example
Sep 17 19:06:23.154637 ip-10-0-131-232 kubenswrapper[2582]: W0917 19:06:23.153666    2582 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 17 19:06:23.154637 ip-10-0-131-232 kubenswrapper[2582]: W0917 19:06:23.153669    2582 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 17 19:06:23.154637 ip-10-0-131-232 kubenswrapper[2582]: W0917 19:06:23.153672    2582 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 17 19:06:23.154637 ip-10-0-131-232 kubenswrapper[2582]: W0917 19:06:23.153675    2582 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 17 19:06:23.154637 ip-10-0-131-232 kubenswrapper[2582]: W0917 19:06:23.153677    2582 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 17 19:06:23.155156 ip-10-0-131-232 kubenswrapper[2582]: W0917 19:06:23.153679    2582 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 17 19:06:23.155156 ip-10-0-131-232 kubenswrapper[2582]: W0917 19:06:23.153682    2582 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 17 19:06:23.155156 ip-10-0-131-232 kubenswrapper[2582]: W0917 19:06:23.153684    2582 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 17 19:06:23.155156 ip-10-0-131-232 kubenswrapper[2582]: W0917 19:06:23.153687    2582 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 17 19:06:23.155156 ip-10-0-131-232 kubenswrapper[2582]: W0917 19:06:23.153689    2582 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 17 19:06:23.155156 ip-10-0-131-232 kubenswrapper[2582]: W0917 19:06:23.153692    2582 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 17 19:06:23.155156 ip-10-0-131-232 kubenswrapper[2582]: W0917 19:06:23.153695    2582 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 17 19:06:23.155156 ip-10-0-131-232 kubenswrapper[2582]: W0917 19:06:23.153698    2582 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 17 19:06:23.155156 ip-10-0-131-232 kubenswrapper[2582]: W0917 19:06:23.153702    2582 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 17 19:06:23.155156 ip-10-0-131-232 kubenswrapper[2582]: W0917 19:06:23.153706    2582 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 17 19:06:23.155156 ip-10-0-131-232 kubenswrapper[2582]: W0917 19:06:23.153709    2582 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 17 19:06:23.155156 ip-10-0-131-232 kubenswrapper[2582]: W0917 19:06:23.153712    2582 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 17 19:06:23.155156 ip-10-0-131-232 kubenswrapper[2582]: W0917 19:06:23.153715    2582 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 17 19:06:23.155156 ip-10-0-131-232 kubenswrapper[2582]: W0917 19:06:23.153717    2582 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 17 19:06:23.155156 ip-10-0-131-232 kubenswrapper[2582]: W0917 19:06:23.153719    2582 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 17 19:06:23.155156 ip-10-0-131-232 kubenswrapper[2582]: W0917 19:06:23.153722    2582 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 17 19:06:23.155156 ip-10-0-131-232 kubenswrapper[2582]: W0917 19:06:23.153724    2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 17 19:06:23.155156 ip-10-0-131-232 kubenswrapper[2582]: W0917 19:06:23.153727    2582 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 17 19:06:23.155156 ip-10-0-131-232 kubenswrapper[2582]: W0917 19:06:23.153729    2582 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 17 19:06:23.155619 ip-10-0-131-232 kubenswrapper[2582]: W0917 19:06:23.153733    2582 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 17 19:06:23.155619 ip-10-0-131-232 kubenswrapper[2582]: W0917 19:06:23.153736    2582 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 17 19:06:23.155619 ip-10-0-131-232 kubenswrapper[2582]: W0917 19:06:23.153740    2582 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 17 19:06:23.155619 ip-10-0-131-232 kubenswrapper[2582]: W0917 19:06:23.153743    2582 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 17 19:06:23.155619 ip-10-0-131-232 kubenswrapper[2582]: W0917 19:06:23.153745    2582 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 17 19:06:23.155619 ip-10-0-131-232 kubenswrapper[2582]: W0917 19:06:23.153747    2582 feature_gate.go:328] unrecognized feature gate: Example2
Sep 17 19:06:23.155619 ip-10-0-131-232 kubenswrapper[2582]: W0917 19:06:23.153750    2582 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 17 19:06:23.155619 ip-10-0-131-232 kubenswrapper[2582]: W0917 19:06:23.153752    2582 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 17 19:06:23.155619 ip-10-0-131-232 kubenswrapper[2582]: W0917 19:06:23.153755    2582 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 17 19:06:23.155619 ip-10-0-131-232 kubenswrapper[2582]: W0917 19:06:23.153757    2582 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 17 19:06:23.155619 ip-10-0-131-232 kubenswrapper[2582]: W0917 19:06:23.153759    2582 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 17 19:06:23.155619 ip-10-0-131-232 kubenswrapper[2582]: W0917 19:06:23.153762    2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 17 19:06:23.155619 ip-10-0-131-232 kubenswrapper[2582]: W0917 19:06:23.153764    2582 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 17 19:06:23.155619 ip-10-0-131-232 kubenswrapper[2582]: W0917 19:06:23.153767    2582 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 17 19:06:23.155619 ip-10-0-131-232 kubenswrapper[2582]: W0917 19:06:23.153769    2582 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 17 19:06:23.155619 ip-10-0-131-232 kubenswrapper[2582]: W0917 19:06:23.153771    2582 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 17 19:06:23.155619 ip-10-0-131-232 kubenswrapper[2582]: W0917 19:06:23.153774    2582 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 17 19:06:23.155619 ip-10-0-131-232 kubenswrapper[2582]: W0917 19:06:23.153776    2582 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 17 19:06:23.155619 ip-10-0-131-232 kubenswrapper[2582]: W0917 19:06:23.154143    2582 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 17 19:06:23.155619 ip-10-0-131-232 kubenswrapper[2582]: W0917 19:06:23.154148    2582 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 17 19:06:23.156112 ip-10-0-131-232 kubenswrapper[2582]: W0917 19:06:23.154151    2582 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 17 19:06:23.156112 ip-10-0-131-232 kubenswrapper[2582]: W0917 19:06:23.154154    2582 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 17 19:06:23.156112 ip-10-0-131-232 kubenswrapper[2582]: W0917 19:06:23.154157    2582 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 17 19:06:23.156112 ip-10-0-131-232 kubenswrapper[2582]: W0917 19:06:23.154160    2582 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 17 19:06:23.156112 ip-10-0-131-232 kubenswrapper[2582]: W0917 19:06:23.154163    2582 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 17 19:06:23.156112 ip-10-0-131-232 kubenswrapper[2582]: W0917 19:06:23.154166    2582 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 17 19:06:23.156112 ip-10-0-131-232 kubenswrapper[2582]: W0917 19:06:23.154168    2582 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 17 19:06:23.156112 ip-10-0-131-232 kubenswrapper[2582]: W0917 19:06:23.154171    2582 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 17 19:06:23.156112 ip-10-0-131-232 kubenswrapper[2582]: W0917 19:06:23.154173    2582 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 17 19:06:23.156112 ip-10-0-131-232 kubenswrapper[2582]: W0917 19:06:23.154176    2582 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 17 19:06:23.156112 ip-10-0-131-232 kubenswrapper[2582]: W0917 19:06:23.154180    2582 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 17 19:06:23.156112 ip-10-0-131-232 kubenswrapper[2582]: W0917 19:06:23.154183    2582 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 17 19:06:23.156112 ip-10-0-131-232 kubenswrapper[2582]: W0917 19:06:23.154186    2582 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 17 19:06:23.156112 ip-10-0-131-232 kubenswrapper[2582]: W0917 19:06:23.154189    2582 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 17 19:06:23.156112 ip-10-0-131-232 kubenswrapper[2582]: W0917 19:06:23.154192    2582 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 17 19:06:23.156112 ip-10-0-131-232 kubenswrapper[2582]: W0917 19:06:23.154195    2582 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 17 19:06:23.156112 ip-10-0-131-232 kubenswrapper[2582]: W0917 19:06:23.154198    2582 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 17 19:06:23.156112 ip-10-0-131-232 kubenswrapper[2582]: W0917 19:06:23.154201    2582 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 17 19:06:23.156112 ip-10-0-131-232 kubenswrapper[2582]: W0917 19:06:23.154203    2582 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 17 19:06:23.156576 ip-10-0-131-232 kubenswrapper[2582]: W0917 19:06:23.154206    2582 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 17 19:06:23.156576 ip-10-0-131-232 kubenswrapper[2582]: W0917 19:06:23.154209    2582 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 17 19:06:23.156576 ip-10-0-131-232 kubenswrapper[2582]: W0917 19:06:23.154212    2582 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 17 19:06:23.156576 ip-10-0-131-232 kubenswrapper[2582]: W0917 19:06:23.154214    2582 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 17 19:06:23.156576 ip-10-0-131-232 kubenswrapper[2582]: W0917 19:06:23.154217    2582 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 17 19:06:23.156576 ip-10-0-131-232 kubenswrapper[2582]: W0917 19:06:23.154219    2582 feature_gate.go:328] unrecognized feature gate: Example
Sep 17 19:06:23.156576 ip-10-0-131-232 kubenswrapper[2582]: W0917 19:06:23.154222    2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 17 19:06:23.156576 ip-10-0-131-232 kubenswrapper[2582]: W0917 19:06:23.154224    2582 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 17 19:06:23.156576 ip-10-0-131-232 kubenswrapper[2582]: W0917 19:06:23.154227    2582 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 17 19:06:23.156576 ip-10-0-131-232 kubenswrapper[2582]: W0917 19:06:23.154230    2582 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 17 19:06:23.156576 ip-10-0-131-232 kubenswrapper[2582]: W0917 19:06:23.154233    2582 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 17 19:06:23.156576 ip-10-0-131-232 kubenswrapper[2582]: W0917 19:06:23.154236    2582 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 17 19:06:23.156576 ip-10-0-131-232 kubenswrapper[2582]: W0917 19:06:23.154240    2582 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 17 19:06:23.156576 ip-10-0-131-232 kubenswrapper[2582]: W0917 19:06:23.154243    2582 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 17 19:06:23.156576 ip-10-0-131-232 kubenswrapper[2582]: W0917 19:06:23.154246    2582 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 17 19:06:23.156576 ip-10-0-131-232 kubenswrapper[2582]: W0917 19:06:23.154249    2582 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 17 19:06:23.156576 ip-10-0-131-232 kubenswrapper[2582]: W0917 19:06:23.154252    2582 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 17 19:06:23.156576 ip-10-0-131-232 kubenswrapper[2582]: W0917 19:06:23.154254    2582 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 17 19:06:23.156576 ip-10-0-131-232 kubenswrapper[2582]: W0917 19:06:23.154257    2582 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 17 19:06:23.157045 ip-10-0-131-232 kubenswrapper[2582]: W0917 19:06:23.154260    2582 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 17 19:06:23.157045 ip-10-0-131-232 kubenswrapper[2582]: W0917 19:06:23.154262    2582 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 17 19:06:23.157045 ip-10-0-131-232 kubenswrapper[2582]: W0917 19:06:23.154265    2582 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 17 19:06:23.157045 ip-10-0-131-232 kubenswrapper[2582]: W0917 19:06:23.154267    2582 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 17 19:06:23.157045 ip-10-0-131-232 kubenswrapper[2582]: W0917 19:06:23.154269    2582 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 17 19:06:23.157045 ip-10-0-131-232 kubenswrapper[2582]: W0917 19:06:23.154272    2582 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 17 19:06:23.157045 ip-10-0-131-232 kubenswrapper[2582]: W0917 19:06:23.154275    2582 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 17 19:06:23.157045 ip-10-0-131-232 kubenswrapper[2582]: W0917 19:06:23.154277    2582 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 17 19:06:23.157045 ip-10-0-131-232 kubenswrapper[2582]: W0917 19:06:23.154280    2582 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 17 19:06:23.157045 ip-10-0-131-232 kubenswrapper[2582]: W0917 19:06:23.154282    2582 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 17 19:06:23.157045 ip-10-0-131-232 kubenswrapper[2582]: W0917 19:06:23.154285    2582 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 17 19:06:23.157045 ip-10-0-131-232 kubenswrapper[2582]: W0917 19:06:23.154289    2582 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 17 19:06:23.157045 ip-10-0-131-232 kubenswrapper[2582]: W0917 19:06:23.154292    2582 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 17 19:06:23.157045 ip-10-0-131-232 kubenswrapper[2582]: W0917 19:06:23.154295    2582 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 17 19:06:23.157045 ip-10-0-131-232 kubenswrapper[2582]: W0917 19:06:23.154298    2582 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 17 19:06:23.157045 ip-10-0-131-232 kubenswrapper[2582]: W0917 19:06:23.154300    2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 17 19:06:23.157045 ip-10-0-131-232 kubenswrapper[2582]: W0917 19:06:23.154303    2582 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 17 19:06:23.157045 ip-10-0-131-232 kubenswrapper[2582]: W0917 19:06:23.154305    2582 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 17 19:06:23.157045 ip-10-0-131-232 kubenswrapper[2582]: W0917 19:06:23.154308    2582 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 17 19:06:23.157491 ip-10-0-131-232 kubenswrapper[2582]: W0917 19:06:23.154310    2582 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 17 19:06:23.157491 ip-10-0-131-232 kubenswrapper[2582]: W0917 19:06:23.154312    2582 feature_gate.go:328] unrecognized feature gate: Example2
Sep 17 19:06:23.157491 ip-10-0-131-232 kubenswrapper[2582]: W0917 19:06:23.154315    2582 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 17 19:06:23.157491 ip-10-0-131-232 kubenswrapper[2582]: W0917 19:06:23.154317    2582 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 17 19:06:23.157491 ip-10-0-131-232 kubenswrapper[2582]: W0917 19:06:23.154320    2582 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 17 19:06:23.157491 ip-10-0-131-232 kubenswrapper[2582]: W0917 19:06:23.154322    2582 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 17 19:06:23.157491 ip-10-0-131-232 kubenswrapper[2582]: W0917 19:06:23.154325    2582 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 17 19:06:23.157491 ip-10-0-131-232 kubenswrapper[2582]: W0917 19:06:23.154328    2582 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 17 19:06:23.157491 ip-10-0-131-232 kubenswrapper[2582]: W0917 19:06:23.154330    2582 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 17 19:06:23.157491 ip-10-0-131-232 kubenswrapper[2582]: W0917 19:06:23.154333    2582 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 17 19:06:23.157491 ip-10-0-131-232 kubenswrapper[2582]: W0917 19:06:23.154335    2582 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 17 19:06:23.157491 ip-10-0-131-232 kubenswrapper[2582]: W0917 19:06:23.154338    2582 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 17 19:06:23.157491 ip-10-0-131-232 kubenswrapper[2582]: W0917 19:06:23.154340    2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 17 19:06:23.157491 ip-10-0-131-232 kubenswrapper[2582]: W0917 19:06:23.154342    2582 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 17 19:06:23.157491 ip-10-0-131-232 kubenswrapper[2582]: W0917 19:06:23.154345    2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 17 19:06:23.157491 ip-10-0-131-232 kubenswrapper[2582]: W0917 19:06:23.154348    2582 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 17 19:06:23.157491 ip-10-0-131-232 kubenswrapper[2582]: W0917 19:06:23.154351    2582 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 17 19:06:23.157491 ip-10-0-131-232 kubenswrapper[2582]: W0917 19:06:23.154353    2582 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 17 19:06:23.157491 ip-10-0-131-232 kubenswrapper[2582]: W0917 19:06:23.154356    2582 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 17 19:06:23.157491 ip-10-0-131-232 kubenswrapper[2582]: W0917 19:06:23.154358    2582 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 17 19:06:23.157974 ip-10-0-131-232 kubenswrapper[2582]: W0917 19:06:23.154360    2582 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 17 19:06:23.157974 ip-10-0-131-232 kubenswrapper[2582]: W0917 19:06:23.154363    2582 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 17 19:06:23.157974 ip-10-0-131-232 kubenswrapper[2582]: W0917 19:06:23.154365    2582 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 17 19:06:23.157974 ip-10-0-131-232 kubenswrapper[2582]: W0917 19:06:23.154368    2582 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 17 19:06:23.157974 ip-10-0-131-232 kubenswrapper[2582]: W0917 19:06:23.154371    2582 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 17 19:06:23.157974 ip-10-0-131-232 kubenswrapper[2582]: W0917 19:06:23.154373    2582 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 17 19:06:23.157974 ip-10-0-131-232 kubenswrapper[2582]: W0917 19:06:23.154376    2582 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 17 19:06:23.157974 ip-10-0-131-232 kubenswrapper[2582]: W0917 19:06:23.154379    2582 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 17 19:06:23.157974 ip-10-0-131-232 kubenswrapper[2582]: W0917 19:06:23.154381    2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 17 19:06:23.157974 ip-10-0-131-232 kubenswrapper[2582]: W0917 19:06:23.154383    2582 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 17 19:06:23.157974 ip-10-0-131-232 kubenswrapper[2582]: W0917 19:06:23.154386    2582 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 17 19:06:23.157974 ip-10-0-131-232 kubenswrapper[2582]: W0917 19:06:23.154388    2582 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 17 19:06:23.157974 ip-10-0-131-232 kubenswrapper[2582]: W0917 19:06:23.154391    2582 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 17 19:06:23.157974 ip-10-0-131-232 kubenswrapper[2582]: W0917 19:06:23.154393    2582 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 17 19:06:23.157974 ip-10-0-131-232 kubenswrapper[2582]: W0917 19:06:23.154395    2582 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 17 19:06:23.157974 ip-10-0-131-232 kubenswrapper[2582]: W0917 19:06:23.154398    2582 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 17 19:06:23.157974 ip-10-0-131-232 kubenswrapper[2582]: W0917 19:06:23.154400    2582 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 17 19:06:23.157974 ip-10-0-131-232 kubenswrapper[2582]: W0917 19:06:23.154402    2582 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 17 19:06:23.157974 ip-10-0-131-232 kubenswrapper[2582]: W0917 19:06:23.154405    2582 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 17 19:06:23.157974 ip-10-0-131-232 kubenswrapper[2582]: W0917 19:06:23.154408    2582 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 17 19:06:23.158451 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:06:23.154482    2582 flags.go:64] FLAG: --address="0.0.0.0"
Sep 17 19:06:23.158451 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:06:23.154517    2582 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Sep 17 19:06:23.158451 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:06:23.154524    2582 flags.go:64] FLAG: --anonymous-auth="true"
Sep 17 19:06:23.158451 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:06:23.154541    2582 flags.go:64] FLAG: --application-metrics-count-limit="100"
Sep 17 19:06:23.158451 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:06:23.154546    2582 flags.go:64] FLAG: --authentication-token-webhook="false"
Sep 17 19:06:23.158451 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:06:23.154550    2582 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Sep 17 19:06:23.158451 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:06:23.154555    2582 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Sep 17 19:06:23.158451 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:06:23.154559    2582 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Sep 17 19:06:23.158451 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:06:23.154562    2582 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Sep 17 19:06:23.158451 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:06:23.154565    2582 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Sep 17 19:06:23.158451 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:06:23.154568    2582 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Sep 17 19:06:23.158451 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:06:23.154572    2582 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Sep 17 19:06:23.158451 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:06:23.154575    2582 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Sep 17 19:06:23.158451 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:06:23.154578    2582 flags.go:64] FLAG: --cgroup-root=""
Sep 17 19:06:23.158451 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:06:23.154581    2582 flags.go:64] FLAG: --cgroups-per-qos="true"
Sep 17 19:06:23.158451 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:06:23.154584    2582 flags.go:64] FLAG: --client-ca-file=""
Sep 17 19:06:23.158451 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:06:23.154587    2582 flags.go:64] FLAG: --cloud-provider="external"
Sep 17 19:06:23.158451 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:06:23.154590    2582 flags.go:64] FLAG: --cluster-dns="[]"
Sep 17 19:06:23.158451 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:06:23.154594    2582 flags.go:64] FLAG: --cluster-domain=""
Sep 17 19:06:23.158451 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:06:23.154597    2582 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Sep 17 19:06:23.158451 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:06:23.154599    2582 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Sep 17 19:06:23.158451 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:06:23.154602    2582 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Sep 17 19:06:23.158451 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:06:23.154605    2582 flags.go:64] FLAG: --container-log-max-files="5"
Sep 17 19:06:23.158451 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:06:23.154613    2582 flags.go:64] FLAG: --container-log-max-size="10Mi"
Sep 17 19:06:23.159047 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:06:23.154616    2582 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Sep 17 19:06:23.159047 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:06:23.154619    2582 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Sep 17 19:06:23.159047 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:06:23.154622    2582 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Sep 17 19:06:23.159047 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:06:23.154625    2582 flags.go:64] FLAG: --contention-profiling="false"
Sep 17 19:06:23.159047 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:06:23.154628    2582 flags.go:64] FLAG: --cpu-cfs-quota="true"
Sep 17 19:06:23.159047 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:06:23.154630    2582 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Sep 17 19:06:23.159047 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:06:23.154634    2582 flags.go:64] FLAG: --cpu-manager-policy="none"
Sep 17 19:06:23.159047 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:06:23.154636    2582 flags.go:64] FLAG: --cpu-manager-policy-options=""
Sep 17 19:06:23.159047 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:06:23.154640    2582 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Sep 17 19:06:23.159047 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:06:23.154643    2582 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Sep 17 19:06:23.159047 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:06:23.154647    2582 flags.go:64] FLAG: --enable-debugging-handlers="true"
Sep 17 19:06:23.159047 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:06:23.154649    2582 flags.go:64] FLAG: --enable-load-reader="false"
Sep 17 19:06:23.159047 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:06:23.154652    2582 flags.go:64] FLAG: --enable-server="true"
Sep 17 19:06:23.159047 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:06:23.154655    2582 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Sep 17 19:06:23.159047 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:06:23.154659    2582 flags.go:64] FLAG: --event-burst="100"
Sep 17 19:06:23.159047 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:06:23.154663    2582 flags.go:64] FLAG: --event-qps="50"
Sep 17 19:06:23.159047 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:06:23.154665    2582 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Sep 17 19:06:23.159047 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:06:23.154668    2582 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Sep 17 19:06:23.159047 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:06:23.154671    2582 flags.go:64] FLAG: --eviction-hard=""
Sep 17 19:06:23.159047 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:06:23.154674    2582 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Sep 17 19:06:23.159047 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:06:23.154677    2582 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Sep 17 19:06:23.159047 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:06:23.154679    2582 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Sep 17 19:06:23.159047 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:06:23.154682    2582 flags.go:64] FLAG: --eviction-soft=""
Sep 17 19:06:23.159047 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:06:23.154685    2582 flags.go:64] FLAG: --eviction-soft-grace-period=""
Sep 17 19:06:23.159047 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:06:23.154688    2582 flags.go:64] FLAG: --exit-on-lock-contention="false"
Sep 17 19:06:23.159681 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:06:23.154691    2582 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Sep 17 19:06:23.159681 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:06:23.154694    2582 flags.go:64] FLAG: --experimental-mounter-path=""
Sep 17 19:06:23.159681 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:06:23.154696    2582 flags.go:64] FLAG: --fail-cgroupv1="false"
Sep 17 19:06:23.159681 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:06:23.154699    2582 flags.go:64] FLAG: --fail-swap-on="true"
Sep 17 19:06:23.159681 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:06:23.154702    2582 flags.go:64] FLAG: --feature-gates=""
Sep 17 19:06:23.159681 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:06:23.154705    2582 flags.go:64] FLAG: --file-check-frequency="20s"
Sep 17 19:06:23.159681 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:06:23.154708    2582 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Sep 17 19:06:23.159681 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:06:23.154711    2582 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Sep 17 19:06:23.159681 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:06:23.154714    2582 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Sep 17 19:06:23.159681 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:06:23.154717    2582 flags.go:64] FLAG: --healthz-port="10248"
Sep 17 19:06:23.159681 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:06:23.154720    2582 flags.go:64] FLAG: --help="false"
Sep 17 19:06:23.159681 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:06:23.154722    2582 flags.go:64] FLAG: --hostname-override="ip-10-0-131-232.ec2.internal"
Sep 17 19:06:23.159681 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:06:23.154725    2582 flags.go:64] FLAG: --housekeeping-interval="10s"
Sep 17 19:06:23.159681 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:06:23.154728    2582 flags.go:64] FLAG: --http-check-frequency="20s"
Sep 17 19:06:23.159681 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:06:23.154731    2582 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Sep 17 19:06:23.159681 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:06:23.154734    2582 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Sep 17 19:06:23.159681 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:06:23.154737    2582 flags.go:64] FLAG: --image-gc-high-threshold="85"
Sep 17 19:06:23.159681 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:06:23.154740    2582 flags.go:64] FLAG: --image-gc-low-threshold="80"
Sep 17 19:06:23.159681 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:06:23.154743    2582 flags.go:64] FLAG: --image-service-endpoint=""
Sep 17 19:06:23.159681 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:06:23.154745    2582 flags.go:64] FLAG: --kernel-memcg-notification="false"
Sep 17 19:06:23.159681 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:06:23.154748    2582 flags.go:64] FLAG: --kube-api-burst="100"
Sep 17 19:06:23.159681 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:06:23.154751    2582 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Sep 17 19:06:23.159681 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:06:23.154753    2582 flags.go:64] FLAG: --kube-api-qps="50"
Sep 17 19:06:23.159681 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:06:23.154756    2582 flags.go:64] FLAG: --kube-reserved=""
Sep 17 19:06:23.160349 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:06:23.154759    2582 flags.go:64] FLAG: --kube-reserved-cgroup=""
Sep 17 19:06:23.160349 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:06:23.154761    2582 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Sep 17 19:06:23.160349 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:06:23.154764    2582 flags.go:64] FLAG: --kubelet-cgroups=""
Sep 17 19:06:23.160349 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:06:23.154767    2582 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Sep 17 19:06:23.160349 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:06:23.154769    2582 flags.go:64] FLAG: --lock-file=""
Sep 17 19:06:23.160349 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:06:23.154772    2582 flags.go:64] FLAG: --log-cadvisor-usage="false"
Sep 17 19:06:23.160349 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:06:23.154777    2582 flags.go:64] FLAG: --log-flush-frequency="5s"
Sep 17 19:06:23.160349 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:06:23.154780    2582 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Sep 17 19:06:23.160349 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:06:23.154785    2582 flags.go:64] FLAG: --log-json-split-stream="false"
Sep 17 19:06:23.160349 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:06:23.154788    2582 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Sep 17 19:06:23.160349 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:06:23.154791    2582 flags.go:64] FLAG: --log-text-split-stream="false"
Sep 17 19:06:23.160349 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:06:23.154793    2582 flags.go:64] FLAG: --logging-format="text"
Sep 17 19:06:23.160349 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:06:23.154796    2582 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Sep 17 19:06:23.160349 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:06:23.154800    2582 flags.go:64] FLAG: --make-iptables-util-chains="true"
Sep 17 19:06:23.160349 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:06:23.154802    2582 flags.go:64] FLAG: --manifest-url=""
Sep 17 19:06:23.160349 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:06:23.154805    2582 flags.go:64] FLAG: --manifest-url-header=""
Sep 17 19:06:23.160349 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:06:23.154808    2582 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Sep 17 19:06:23.160349 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:06:23.154811    2582 flags.go:64] FLAG: --max-open-files="1000000"
Sep 17 19:06:23.160349 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:06:23.154815    2582 flags.go:64] FLAG: --max-pods="110"
Sep 17 19:06:23.160349 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:06:23.154818    2582 flags.go:64] FLAG: --maximum-dead-containers="-1"
Sep 17 19:06:23.160349 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:06:23.154821    2582 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Sep 17 19:06:23.160349 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:06:23.154825    2582 flags.go:64] FLAG: --memory-manager-policy="None"
Sep 17 19:06:23.160349 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:06:23.154828    2582 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Sep 17 19:06:23.160349 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:06:23.154831    2582 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Sep 17 19:06:23.160349 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:06:23.154833    2582 flags.go:64] FLAG: --node-ip="0.0.0.0"
Sep 17 19:06:23.160983 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:06:23.154836    2582 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Sep 17 19:06:23.160983 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:06:23.154843    2582 flags.go:64] FLAG: --node-status-max-images="50"
Sep 17 19:06:23.160983 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:06:23.154846    2582 flags.go:64] FLAG: --node-status-update-frequency="10s"
Sep 17 19:06:23.160983 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:06:23.154849    2582 flags.go:64] FLAG: --oom-score-adj="-999"
Sep 17 19:06:23.160983 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:06:23.154852    2582 flags.go:64] FLAG: --pod-cidr=""
Sep 17 19:06:23.160983 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:06:23.154854    2582 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:a6abe77af61c5f2e3985a83e7d43538aefc38b1b0444127da14ee0e8cf8f76fc"
Sep 17 19:06:23.160983 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:06:23.154860    2582 flags.go:64] FLAG: --pod-manifest-path=""
Sep 17 19:06:23.160983 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:06:23.154863    2582 flags.go:64] FLAG: --pod-max-pids="-1"
Sep 17 19:06:23.160983 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:06:23.154866    2582 flags.go:64] FLAG: --pods-per-core="0"
Sep 17 19:06:23.160983 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:06:23.154868    2582 flags.go:64] FLAG: --port="10250"
Sep 17 19:06:23.160983 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:06:23.154871    2582 flags.go:64] FLAG: --protect-kernel-defaults="false"
Sep 17 19:06:23.160983 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:06:23.154874    2582 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-095af372e071dbc56"
Sep 17 19:06:23.160983 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:06:23.154877    2582 flags.go:64] FLAG: --qos-reserved=""
Sep 17 19:06:23.160983 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:06:23.154881    2582 flags.go:64] FLAG: --read-only-port="10255"
Sep 17 19:06:23.160983 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:06:23.154884    2582 flags.go:64] FLAG: --register-node="true"
Sep 17 19:06:23.160983 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:06:23.154887    2582 flags.go:64] FLAG: --register-with-taints=""
Sep 17 19:06:23.160983 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:06:23.154907    2582 flags.go:64] FLAG: --registry-burst="10"
Sep 17 19:06:23.160983 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:06:23.154911    2582 flags.go:64] FLAG: --registry-qps="5"
Sep 17 19:06:23.160983 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:06:23.154914    2582 flags.go:64] FLAG: --reserved-cpus=""
Sep 17 19:06:23.160983 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:06:23.154916    2582 flags.go:64] FLAG: --reserved-memory=""
Sep 17 19:06:23.160983 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:06:23.154920    2582 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Sep 17 19:06:23.160983 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:06:23.154923    2582 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Sep 17 19:06:23.160983 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:06:23.154926    2582 flags.go:64] FLAG: --rotate-certificates="false"
Sep 17 19:06:23.160983 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:06:23.154928    2582 flags.go:64] FLAG: --rotate-server-certificates="false"
Sep 17 19:06:23.160983 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:06:23.154931    2582 flags.go:64] FLAG: --runonce="false"
Sep 17 19:06:23.161595 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:06:23.154934    2582 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Sep 17 19:06:23.161595 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:06:23.154937    2582 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Sep 17 19:06:23.161595 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:06:23.154940    2582 flags.go:64] FLAG: --seccomp-default="false"
Sep 17 19:06:23.161595 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:06:23.154942    2582 flags.go:64] FLAG: --serialize-image-pulls="true"
Sep 17 19:06:23.161595 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:06:23.154946    2582 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Sep 17 19:06:23.161595 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:06:23.154949    2582 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Sep 17 19:06:23.161595 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:06:23.154952    2582 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Sep 17 19:06:23.161595 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:06:23.154955    2582 flags.go:64] FLAG: --storage-driver-password="root"
Sep 17 19:06:23.161595 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:06:23.154957    2582 flags.go:64] FLAG: --storage-driver-secure="false"
Sep 17 19:06:23.161595 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:06:23.154960    2582 flags.go:64] FLAG: --storage-driver-table="stats"
Sep 17 19:06:23.161595 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:06:23.154963    2582 flags.go:64] FLAG: --storage-driver-user="root"
Sep 17 19:06:23.161595 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:06:23.154966    2582 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Sep 17 19:06:23.161595 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:06:23.154968    2582 flags.go:64] FLAG: --sync-frequency="1m0s"
Sep 17 19:06:23.161595 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:06:23.154971    2582 flags.go:64] FLAG: --system-cgroups=""
Sep 17 19:06:23.161595 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:06:23.154974    2582 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Sep 17 19:06:23.161595 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:06:23.154979    2582 flags.go:64] FLAG: --system-reserved-cgroup=""
Sep 17 19:06:23.161595 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:06:23.154981    2582 flags.go:64] FLAG: --tls-cert-file=""
Sep 17 19:06:23.161595 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:06:23.154984    2582 flags.go:64] FLAG: --tls-cipher-suites="[]"
Sep 17 19:06:23.161595 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:06:23.154987    2582 flags.go:64] FLAG: --tls-min-version=""
Sep 17 19:06:23.161595 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:06:23.154990    2582 flags.go:64] FLAG: --tls-private-key-file=""
Sep 17 19:06:23.161595 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:06:23.154994    2582 flags.go:64] FLAG: --topology-manager-policy="none"
Sep 17 19:06:23.161595 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:06:23.154997    2582 flags.go:64] FLAG: --topology-manager-policy-options=""
Sep 17 19:06:23.161595 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:06:23.154999    2582 flags.go:64] FLAG: --topology-manager-scope="container"
Sep 17 19:06:23.161595 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:06:23.155002    2582 flags.go:64] FLAG: --v="2"
Sep 17 19:06:23.161595 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:06:23.155006    2582 flags.go:64] FLAG: --version="false"
Sep 17 19:06:23.162217 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:06:23.155010    2582 flags.go:64] FLAG: --vmodule=""
Sep 17 19:06:23.162217 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:06:23.155014    2582 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Sep 17 19:06:23.162217 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:06:23.155018    2582 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Sep 17 19:06:23.162217 ip-10-0-131-232 kubenswrapper[2582]: W0917 19:06:23.155147    2582 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 17 19:06:23.162217 ip-10-0-131-232 kubenswrapper[2582]: W0917 19:06:23.155151    2582 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 17 19:06:23.162217 ip-10-0-131-232 kubenswrapper[2582]: W0917 19:06:23.155154    2582 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 17 19:06:23.162217 ip-10-0-131-232 kubenswrapper[2582]: W0917 19:06:23.155157    2582 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 17 19:06:23.162217 ip-10-0-131-232 kubenswrapper[2582]: W0917 19:06:23.155160    2582 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 17 19:06:23.162217 ip-10-0-131-232 kubenswrapper[2582]: W0917 19:06:23.155163    2582 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 17 19:06:23.162217 ip-10-0-131-232 kubenswrapper[2582]: W0917 19:06:23.155165    2582 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 17 19:06:23.162217 ip-10-0-131-232 kubenswrapper[2582]: W0917 19:06:23.155168    2582 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 17 19:06:23.162217 ip-10-0-131-232 kubenswrapper[2582]: W0917 19:06:23.155172    2582 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 17 19:06:23.162217 ip-10-0-131-232 kubenswrapper[2582]: W0917 19:06:23.155175    2582 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 17 19:06:23.162217 ip-10-0-131-232 kubenswrapper[2582]: W0917 19:06:23.155177    2582 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 17 19:06:23.162217 ip-10-0-131-232 kubenswrapper[2582]: W0917 19:06:23.155180    2582 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 17 19:06:23.162217 ip-10-0-131-232 kubenswrapper[2582]: W0917 19:06:23.155183    2582 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 17 19:06:23.162217 ip-10-0-131-232 kubenswrapper[2582]: W0917 19:06:23.155185    2582 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 17 19:06:23.162217 ip-10-0-131-232 kubenswrapper[2582]: W0917 19:06:23.155188    2582 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 17 19:06:23.162217 ip-10-0-131-232 kubenswrapper[2582]: W0917 19:06:23.155191    2582 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 17 19:06:23.162217 ip-10-0-131-232 kubenswrapper[2582]: W0917 19:06:23.155194    2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 17 19:06:23.162717 ip-10-0-131-232 kubenswrapper[2582]: W0917 19:06:23.155196    2582 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 17 19:06:23.162717 ip-10-0-131-232 kubenswrapper[2582]: W0917 19:06:23.155199    2582 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 17 19:06:23.162717 ip-10-0-131-232 kubenswrapper[2582]: W0917 19:06:23.155201    2582 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 17 19:06:23.162717 ip-10-0-131-232 kubenswrapper[2582]: W0917 19:06:23.155204    2582 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 17 19:06:23.162717 ip-10-0-131-232 kubenswrapper[2582]: W0917 19:06:23.155206    2582 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 17 19:06:23.162717 ip-10-0-131-232 kubenswrapper[2582]: W0917 19:06:23.155209    2582 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 17 19:06:23.162717 ip-10-0-131-232 kubenswrapper[2582]: W0917 19:06:23.155212    2582 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 17 19:06:23.162717 ip-10-0-131-232 kubenswrapper[2582]: W0917 19:06:23.155216    2582 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 17 19:06:23.162717 ip-10-0-131-232 kubenswrapper[2582]: W0917 19:06:23.155218    2582 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 17 19:06:23.162717 ip-10-0-131-232 kubenswrapper[2582]: W0917 19:06:23.155221    2582 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 17 19:06:23.162717 ip-10-0-131-232 kubenswrapper[2582]: W0917 19:06:23.155223    2582 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 17 19:06:23.162717 ip-10-0-131-232 kubenswrapper[2582]: W0917 19:06:23.155225    2582 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 17 19:06:23.162717 ip-10-0-131-232 kubenswrapper[2582]: W0917 19:06:23.155228    2582 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 17 19:06:23.162717 ip-10-0-131-232 kubenswrapper[2582]: W0917 19:06:23.155231    2582 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 17 19:06:23.162717 ip-10-0-131-232 kubenswrapper[2582]: W0917 19:06:23.155233    2582 feature_gate.go:328] unrecognized feature gate: Example
Sep 17 19:06:23.162717 ip-10-0-131-232 kubenswrapper[2582]: W0917 19:06:23.155237    2582 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 17 19:06:23.162717 ip-10-0-131-232 kubenswrapper[2582]: W0917 19:06:23.155240    2582 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 17 19:06:23.162717 ip-10-0-131-232 kubenswrapper[2582]: W0917 19:06:23.155243    2582 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 17 19:06:23.162717 ip-10-0-131-232 kubenswrapper[2582]: W0917 19:06:23.155245    2582 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 17 19:06:23.163213 ip-10-0-131-232 kubenswrapper[2582]: W0917 19:06:23.155249    2582 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 17 19:06:23.163213 ip-10-0-131-232 kubenswrapper[2582]: W0917 19:06:23.155252    2582 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 17 19:06:23.163213 ip-10-0-131-232 kubenswrapper[2582]: W0917 19:06:23.155255    2582 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 17 19:06:23.163213 ip-10-0-131-232 kubenswrapper[2582]: W0917 19:06:23.155257    2582 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 17 19:06:23.163213 ip-10-0-131-232 kubenswrapper[2582]: W0917 19:06:23.155261    2582 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 17 19:06:23.163213 ip-10-0-131-232 kubenswrapper[2582]: W0917 19:06:23.155263    2582 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 17 19:06:23.163213 ip-10-0-131-232 kubenswrapper[2582]: W0917 19:06:23.155266    2582 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 17 19:06:23.163213 ip-10-0-131-232 kubenswrapper[2582]: W0917 19:06:23.155269    2582 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 17 19:06:23.163213 ip-10-0-131-232 kubenswrapper[2582]: W0917 19:06:23.155271    2582 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 17 19:06:23.163213 ip-10-0-131-232 kubenswrapper[2582]: W0917 19:06:23.155274    2582 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 17 19:06:23.163213 ip-10-0-131-232 kubenswrapper[2582]: W0917 19:06:23.155276    2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 17 19:06:23.163213 ip-10-0-131-232 kubenswrapper[2582]: W0917 19:06:23.155279    2582 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 17 19:06:23.163213 ip-10-0-131-232 kubenswrapper[2582]: W0917 19:06:23.155281    2582 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 17 19:06:23.163213 ip-10-0-131-232 kubenswrapper[2582]: W0917 19:06:23.155285    2582 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 17 19:06:23.163213 ip-10-0-131-232 kubenswrapper[2582]: W0917 19:06:23.155288    2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 17 19:06:23.163213 ip-10-0-131-232 kubenswrapper[2582]: W0917 19:06:23.155291    2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 17 19:06:23.163213 ip-10-0-131-232 kubenswrapper[2582]: W0917 19:06:23.155294    2582 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 17 19:06:23.163213 ip-10-0-131-232 kubenswrapper[2582]: W0917 19:06:23.155296    2582 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 17 19:06:23.163646 ip-10-0-131-232 kubenswrapper[2582]: W0917 19:06:23.155299    2582 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 17 19:06:23.163646 ip-10-0-131-232 kubenswrapper[2582]: W0917 19:06:23.155301    2582 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 17 19:06:23.163646 ip-10-0-131-232 kubenswrapper[2582]: W0917 19:06:23.155305    2582 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 17 19:06:23.163646 ip-10-0-131-232 kubenswrapper[2582]: W0917 19:06:23.155308    2582 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 17 19:06:23.163646 ip-10-0-131-232 kubenswrapper[2582]: W0917 19:06:23.155310    2582 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 17 19:06:23.163646 ip-10-0-131-232 kubenswrapper[2582]: W0917 19:06:23.155313    2582 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 17 19:06:23.163646 ip-10-0-131-232 kubenswrapper[2582]: W0917 19:06:23.155315    2582 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 17 19:06:23.163646 ip-10-0-131-232 kubenswrapper[2582]: W0917 19:06:23.155317    2582 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 17 19:06:23.163646 ip-10-0-131-232 kubenswrapper[2582]: W0917 19:06:23.155320    2582 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 17 19:06:23.163646 ip-10-0-131-232 kubenswrapper[2582]: W0917 19:06:23.155323    2582 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 17 19:06:23.163646 ip-10-0-131-232 kubenswrapper[2582]: W0917 19:06:23.155325    2582 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 17 19:06:23.163646 ip-10-0-131-232 kubenswrapper[2582]: W0917 19:06:23.155328    2582 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 17 19:06:23.163646 ip-10-0-131-232 kubenswrapper[2582]: W0917 19:06:23.155330    2582 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 17 19:06:23.163646 ip-10-0-131-232 kubenswrapper[2582]: W0917 19:06:23.155332    2582 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 17 19:06:23.163646 ip-10-0-131-232 kubenswrapper[2582]: W0917 19:06:23.155335    2582 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 17 19:06:23.163646 ip-10-0-131-232 kubenswrapper[2582]: W0917 19:06:23.155337    2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 17 19:06:23.163646 ip-10-0-131-232 kubenswrapper[2582]: W0917 19:06:23.155340    2582 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 17 19:06:23.163646 ip-10-0-131-232 kubenswrapper[2582]: W0917 19:06:23.155342    2582 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 17 19:06:23.163646 ip-10-0-131-232 kubenswrapper[2582]: W0917 19:06:23.155347    2582 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 17 19:06:23.163646 ip-10-0-131-232 kubenswrapper[2582]: W0917 19:06:23.155350    2582 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 17 19:06:23.164200 ip-10-0-131-232 kubenswrapper[2582]: W0917 19:06:23.155352    2582 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 17 19:06:23.164200 ip-10-0-131-232 kubenswrapper[2582]: W0917 19:06:23.155356    2582 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 17 19:06:23.164200 ip-10-0-131-232 kubenswrapper[2582]: W0917 19:06:23.155360    2582 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 17 19:06:23.164200 ip-10-0-131-232 kubenswrapper[2582]: W0917 19:06:23.155364    2582 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 17 19:06:23.164200 ip-10-0-131-232 kubenswrapper[2582]: W0917 19:06:23.155368    2582 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 17 19:06:23.164200 ip-10-0-131-232 kubenswrapper[2582]: W0917 19:06:23.155372    2582 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 17 19:06:23.164200 ip-10-0-131-232 kubenswrapper[2582]: W0917 19:06:23.155375    2582 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 17 19:06:23.164200 ip-10-0-131-232 kubenswrapper[2582]: W0917 19:06:23.155378    2582 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 17 19:06:23.164200 ip-10-0-131-232 kubenswrapper[2582]: W0917 19:06:23.155381    2582 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 17 19:06:23.164200 ip-10-0-131-232 kubenswrapper[2582]: W0917 19:06:23.155384    2582 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 17 19:06:23.164200 ip-10-0-131-232 kubenswrapper[2582]: W0917 19:06:23.155387    2582 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 17 19:06:23.164200 ip-10-0-131-232 kubenswrapper[2582]: W0917 19:06:23.155389    2582 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 17 19:06:23.164200 ip-10-0-131-232 kubenswrapper[2582]: W0917 19:06:23.155392    2582 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 17 19:06:23.164200 ip-10-0-131-232 kubenswrapper[2582]: W0917 19:06:23.155394    2582 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 17 19:06:23.164200 ip-10-0-131-232 kubenswrapper[2582]: W0917 19:06:23.155399    2582 feature_gate.go:328] unrecognized feature gate: Example2
Sep 17 19:06:23.164200 ip-10-0-131-232 kubenswrapper[2582]: W0917 19:06:23.155401    2582 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 17 19:06:23.164200 ip-10-0-131-232 kubenswrapper[2582]: W0917 19:06:23.155405    2582 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 17 19:06:23.164200 ip-10-0-131-232 kubenswrapper[2582]: W0917 19:06:23.155407    2582 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 17 19:06:23.164200 ip-10-0-131-232 kubenswrapper[2582]: W0917 19:06:23.155410    2582 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 17 19:06:23.164200 ip-10-0-131-232 kubenswrapper[2582]: W0917 19:06:23.155412    2582 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 17 19:06:23.164679 ip-10-0-131-232 kubenswrapper[2582]: W0917 19:06:23.155414    2582 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 17 19:06:23.164679 ip-10-0-131-232 kubenswrapper[2582]: W0917 19:06:23.155417    2582 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 17 19:06:23.164679 ip-10-0-131-232 kubenswrapper[2582]: W0917 19:06:23.155420    2582 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 17 19:06:23.164679 ip-10-0-131-232 kubenswrapper[2582]: W0917 19:06:23.155422    2582 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 17 19:06:23.164679 ip-10-0-131-232 kubenswrapper[2582]: W0917 19:06:23.155425    2582 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 17 19:06:23.164679 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:06:23.156098    2582 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 17 19:06:23.164679 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:06:23.163915    2582 server.go:529] "Kubelet version" kubeletVersion="v1.34.9"
Sep 17 19:06:23.164679 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:06:23.163929    2582 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Sep 17 19:06:23.164679 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:06:23.163948    2582 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Sep 17 19:06:23.164679 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:06:23.163954    2582 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Sep 17 19:06:23.164679 ip-10-0-131-232 kubenswrapper[2582]: W0917 19:06:23.164009    2582 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 17 19:06:23.164679 ip-10-0-131-232 kubenswrapper[2582]: W0917 19:06:23.164014    2582 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 17 19:06:23.164679 ip-10-0-131-232 kubenswrapper[2582]: W0917 19:06:23.164017    2582 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 17 19:06:23.164679 ip-10-0-131-232 kubenswrapper[2582]: W0917 19:06:23.164019    2582 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 17 19:06:23.165039 ip-10-0-131-232 kubenswrapper[2582]: W0917 19:06:23.164022    2582 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 17 19:06:23.165039 ip-10-0-131-232 kubenswrapper[2582]: W0917 19:06:23.164025    2582 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 17 19:06:23.165039 ip-10-0-131-232 kubenswrapper[2582]: W0917 19:06:23.164028    2582 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 17 19:06:23.165039 ip-10-0-131-232 kubenswrapper[2582]: W0917 19:06:23.164030    2582 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 17 19:06:23.165039 ip-10-0-131-232 kubenswrapper[2582]: W0917 19:06:23.164033    2582 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 17 19:06:23.165039 ip-10-0-131-232 kubenswrapper[2582]: W0917 19:06:23.164036    2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 17 19:06:23.165039 ip-10-0-131-232 kubenswrapper[2582]: W0917 19:06:23.164038    2582 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 17 19:06:23.165039 ip-10-0-131-232 kubenswrapper[2582]: W0917 19:06:23.164041    2582 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 17 19:06:23.165039 ip-10-0-131-232 kubenswrapper[2582]: W0917 19:06:23.164044    2582 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 17 19:06:23.165039 ip-10-0-131-232 kubenswrapper[2582]: W0917 19:06:23.164046    2582 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 17 19:06:23.165039 ip-10-0-131-232 kubenswrapper[2582]: W0917 19:06:23.164049    2582 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 17 19:06:23.165039 ip-10-0-131-232 kubenswrapper[2582]: W0917 19:06:23.164051    2582 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 17 19:06:23.165039 ip-10-0-131-232 kubenswrapper[2582]: W0917 19:06:23.164054    2582 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 17 19:06:23.165039 ip-10-0-131-232 kubenswrapper[2582]: W0917 19:06:23.164056    2582 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 17 19:06:23.165039 ip-10-0-131-232 kubenswrapper[2582]: W0917 19:06:23.164059    2582 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 17 19:06:23.165039 ip-10-0-131-232 kubenswrapper[2582]: W0917 19:06:23.164061    2582 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 17 19:06:23.165039 ip-10-0-131-232 kubenswrapper[2582]: W0917 19:06:23.164064    2582 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 17 19:06:23.165039 ip-10-0-131-232 kubenswrapper[2582]: W0917 19:06:23.164066    2582 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 17 19:06:23.165039 ip-10-0-131-232 kubenswrapper[2582]: W0917 19:06:23.164069    2582 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 17 19:06:23.165507 ip-10-0-131-232 kubenswrapper[2582]: W0917 19:06:23.164071    2582 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 17 19:06:23.165507 ip-10-0-131-232 kubenswrapper[2582]: W0917 19:06:23.164074    2582 feature_gate.go:328] unrecognized feature gate: Example2
Sep 17 19:06:23.165507 ip-10-0-131-232 kubenswrapper[2582]: W0917 19:06:23.164077    2582 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 17 19:06:23.165507 ip-10-0-131-232 kubenswrapper[2582]: W0917 19:06:23.164080    2582 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 17 19:06:23.165507 ip-10-0-131-232 kubenswrapper[2582]: W0917 19:06:23.164082    2582 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 17 19:06:23.165507 ip-10-0-131-232 kubenswrapper[2582]: W0917 19:06:23.164084    2582 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 17 19:06:23.165507 ip-10-0-131-232 kubenswrapper[2582]: W0917 19:06:23.164087    2582 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 17 19:06:23.165507 ip-10-0-131-232 kubenswrapper[2582]: W0917 19:06:23.164090    2582 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 17 19:06:23.165507 ip-10-0-131-232 kubenswrapper[2582]: W0917 19:06:23.164092    2582 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 17 19:06:23.165507 ip-10-0-131-232 kubenswrapper[2582]: W0917 19:06:23.164095    2582 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 17 19:06:23.165507 ip-10-0-131-232 kubenswrapper[2582]: W0917 19:06:23.164097    2582 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 17 19:06:23.165507 ip-10-0-131-232 kubenswrapper[2582]: W0917 19:06:23.164099    2582 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 17 19:06:23.165507 ip-10-0-131-232 kubenswrapper[2582]: W0917 19:06:23.164102    2582 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 17 19:06:23.165507 ip-10-0-131-232 kubenswrapper[2582]: W0917 19:06:23.164105    2582 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 17 19:06:23.165507 ip-10-0-131-232 kubenswrapper[2582]: W0917 19:06:23.164107    2582 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 17 19:06:23.165507 ip-10-0-131-232 kubenswrapper[2582]: W0917 19:06:23.164110    2582 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 17 19:06:23.165507 ip-10-0-131-232 kubenswrapper[2582]: W0917 19:06:23.164112    2582 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 17 19:06:23.165507 ip-10-0-131-232 kubenswrapper[2582]: W0917 19:06:23.164114    2582 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 17 19:06:23.165507 ip-10-0-131-232 kubenswrapper[2582]: W0917 19:06:23.164117    2582 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 17 19:06:23.165507 ip-10-0-131-232 kubenswrapper[2582]: W0917 19:06:23.164120    2582 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 17 19:06:23.165999 ip-10-0-131-232 kubenswrapper[2582]: W0917 19:06:23.164122    2582 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 17 19:06:23.165999 ip-10-0-131-232 kubenswrapper[2582]: W0917 19:06:23.164125    2582 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 17 19:06:23.165999 ip-10-0-131-232 kubenswrapper[2582]: W0917 19:06:23.164127    2582 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 17 19:06:23.165999 ip-10-0-131-232 kubenswrapper[2582]: W0917 19:06:23.164129    2582 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 17 19:06:23.165999 ip-10-0-131-232 kubenswrapper[2582]: W0917 19:06:23.164132    2582 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 17 19:06:23.165999 ip-10-0-131-232 kubenswrapper[2582]: W0917 19:06:23.164135    2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 17 19:06:23.165999 ip-10-0-131-232 kubenswrapper[2582]: W0917 19:06:23.164137    2582 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 17 19:06:23.165999 ip-10-0-131-232 kubenswrapper[2582]: W0917 19:06:23.164139    2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 17 19:06:23.165999 ip-10-0-131-232 kubenswrapper[2582]: W0917 19:06:23.164142    2582 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 17 19:06:23.165999 ip-10-0-131-232 kubenswrapper[2582]: W0917 19:06:23.164144    2582 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 17 19:06:23.165999 ip-10-0-131-232 kubenswrapper[2582]: W0917 19:06:23.164147    2582 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 17 19:06:23.165999 ip-10-0-131-232 kubenswrapper[2582]: W0917 19:06:23.164149    2582 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 17 19:06:23.165999 ip-10-0-131-232 kubenswrapper[2582]: W0917 19:06:23.164152    2582 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 17 19:06:23.165999 ip-10-0-131-232 kubenswrapper[2582]: W0917 19:06:23.164154    2582 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 17 19:06:23.165999 ip-10-0-131-232 kubenswrapper[2582]: W0917 19:06:23.164164    2582 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 17 19:06:23.165999 ip-10-0-131-232 kubenswrapper[2582]: W0917 19:06:23.164167    2582 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 17 19:06:23.165999 ip-10-0-131-232 kubenswrapper[2582]: W0917 19:06:23.164170    2582 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 17 19:06:23.165999 ip-10-0-131-232 kubenswrapper[2582]: W0917 19:06:23.164172    2582 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 17 19:06:23.165999 ip-10-0-131-232 kubenswrapper[2582]: W0917 19:06:23.164175    2582 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 17 19:06:23.165999 ip-10-0-131-232 kubenswrapper[2582]: W0917 19:06:23.164178    2582 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 17 19:06:23.166487 ip-10-0-131-232 kubenswrapper[2582]: W0917 19:06:23.164180    2582 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 17 19:06:23.166487 ip-10-0-131-232 kubenswrapper[2582]: W0917 19:06:23.164183    2582 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 17 19:06:23.166487 ip-10-0-131-232 kubenswrapper[2582]: W0917 19:06:23.164185    2582 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 17 19:06:23.166487 ip-10-0-131-232 kubenswrapper[2582]: W0917 19:06:23.164188    2582 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 17 19:06:23.166487 ip-10-0-131-232 kubenswrapper[2582]: W0917 19:06:23.164191    2582 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 17 19:06:23.166487 ip-10-0-131-232 kubenswrapper[2582]: W0917 19:06:23.164193    2582 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 17 19:06:23.166487 ip-10-0-131-232 kubenswrapper[2582]: W0917 19:06:23.164195    2582 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 17 19:06:23.166487 ip-10-0-131-232 kubenswrapper[2582]: W0917 19:06:23.164198    2582 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 17 19:06:23.166487 ip-10-0-131-232 kubenswrapper[2582]: W0917 19:06:23.164200    2582 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 17 19:06:23.166487 ip-10-0-131-232 kubenswrapper[2582]: W0917 19:06:23.164203    2582 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 17 19:06:23.166487 ip-10-0-131-232 kubenswrapper[2582]: W0917 19:06:23.164206    2582 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 17 19:06:23.166487 ip-10-0-131-232 kubenswrapper[2582]: W0917 19:06:23.164208    2582 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 17 19:06:23.166487 ip-10-0-131-232 kubenswrapper[2582]: W0917 19:06:23.164211    2582 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 17 19:06:23.166487 ip-10-0-131-232 kubenswrapper[2582]: W0917 19:06:23.164214    2582 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 17 19:06:23.166487 ip-10-0-131-232 kubenswrapper[2582]: W0917 19:06:23.164216    2582 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 17 19:06:23.166487 ip-10-0-131-232 kubenswrapper[2582]: W0917 19:06:23.164220    2582 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 17 19:06:23.166487 ip-10-0-131-232 kubenswrapper[2582]: W0917 19:06:23.164224    2582 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 17 19:06:23.166487 ip-10-0-131-232 kubenswrapper[2582]: W0917 19:06:23.164227    2582 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 17 19:06:23.166487 ip-10-0-131-232 kubenswrapper[2582]: W0917 19:06:23.164229    2582 feature_gate.go:328] unrecognized feature gate: Example
Sep 17 19:06:23.166487 ip-10-0-131-232 kubenswrapper[2582]: W0917 19:06:23.164232    2582 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 17 19:06:23.166993 ip-10-0-131-232 kubenswrapper[2582]: W0917 19:06:23.164236    2582 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 17 19:06:23.166993 ip-10-0-131-232 kubenswrapper[2582]: W0917 19:06:23.164239    2582 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 17 19:06:23.166993 ip-10-0-131-232 kubenswrapper[2582]: W0917 19:06:23.164242    2582 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 17 19:06:23.166993 ip-10-0-131-232 kubenswrapper[2582]: W0917 19:06:23.164245    2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 17 19:06:23.166993 ip-10-0-131-232 kubenswrapper[2582]: W0917 19:06:23.164247    2582 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 17 19:06:23.166993 ip-10-0-131-232 kubenswrapper[2582]: W0917 19:06:23.164250    2582 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 17 19:06:23.166993 ip-10-0-131-232 kubenswrapper[2582]: W0917 19:06:23.164253    2582 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 17 19:06:23.166993 ip-10-0-131-232 kubenswrapper[2582]: W0917 19:06:23.164256    2582 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 17 19:06:23.166993 ip-10-0-131-232 kubenswrapper[2582]: W0917 19:06:23.164260    2582 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 17 19:06:23.166993 ip-10-0-131-232 kubenswrapper[2582]: W0917 19:06:23.164263    2582 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 17 19:06:23.166993 ip-10-0-131-232 kubenswrapper[2582]: W0917 19:06:23.164266    2582 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 17 19:06:23.166993 ip-10-0-131-232 kubenswrapper[2582]: W0917 19:06:23.164269    2582 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 17 19:06:23.166993 ip-10-0-131-232 kubenswrapper[2582]: W0917 19:06:23.164272    2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 17 19:06:23.166993 ip-10-0-131-232 kubenswrapper[2582]: W0917 19:06:23.164275    2582 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 17 19:06:23.166993 ip-10-0-131-232 kubenswrapper[2582]: W0917 19:06:23.164278    2582 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 17 19:06:23.166993 ip-10-0-131-232 kubenswrapper[2582]: W0917 19:06:23.164280    2582 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 17 19:06:23.167397 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:06:23.164285    2582 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 17 19:06:23.167397 ip-10-0-131-232 kubenswrapper[2582]: W0917 19:06:23.164395    2582 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 17 19:06:23.167397 ip-10-0-131-232 kubenswrapper[2582]: W0917 19:06:23.164400    2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 17 19:06:23.167397 ip-10-0-131-232 kubenswrapper[2582]: W0917 19:06:23.164403    2582 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 17 19:06:23.167397 ip-10-0-131-232 kubenswrapper[2582]: W0917 19:06:23.164405    2582 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 17 19:06:23.167397 ip-10-0-131-232 kubenswrapper[2582]: W0917 19:06:23.164408    2582 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 17 19:06:23.167397 ip-10-0-131-232 kubenswrapper[2582]: W0917 19:06:23.164411    2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 17 19:06:23.167397 ip-10-0-131-232 kubenswrapper[2582]: W0917 19:06:23.164414    2582 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 17 19:06:23.167397 ip-10-0-131-232 kubenswrapper[2582]: W0917 19:06:23.164418    2582 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 17 19:06:23.167397 ip-10-0-131-232 kubenswrapper[2582]: W0917 19:06:23.164420    2582 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 17 19:06:23.167397 ip-10-0-131-232 kubenswrapper[2582]: W0917 19:06:23.164422    2582 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 17 19:06:23.167397 ip-10-0-131-232 kubenswrapper[2582]: W0917 19:06:23.164425    2582 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 17 19:06:23.167397 ip-10-0-131-232 kubenswrapper[2582]: W0917 19:06:23.164427    2582 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 17 19:06:23.167397 ip-10-0-131-232 kubenswrapper[2582]: W0917 19:06:23.164430    2582 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 17 19:06:23.167744 ip-10-0-131-232 kubenswrapper[2582]: W0917 19:06:23.164432    2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 17 19:06:23.167744 ip-10-0-131-232 kubenswrapper[2582]: W0917 19:06:23.164435    2582 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 17 19:06:23.167744 ip-10-0-131-232 kubenswrapper[2582]: W0917 19:06:23.164437    2582 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 17 19:06:23.167744 ip-10-0-131-232 kubenswrapper[2582]: W0917 19:06:23.164440    2582 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 17 19:06:23.167744 ip-10-0-131-232 kubenswrapper[2582]: W0917 19:06:23.164442    2582 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 17 19:06:23.167744 ip-10-0-131-232 kubenswrapper[2582]: W0917 19:06:23.164445    2582 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 17 19:06:23.167744 ip-10-0-131-232 kubenswrapper[2582]: W0917 19:06:23.164447    2582 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 17 19:06:23.167744 ip-10-0-131-232 kubenswrapper[2582]: W0917 19:06:23.164450    2582 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 17 19:06:23.167744 ip-10-0-131-232 kubenswrapper[2582]: W0917 19:06:23.164452    2582 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 17 19:06:23.167744 ip-10-0-131-232 kubenswrapper[2582]: W0917 19:06:23.164454    2582 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 17 19:06:23.167744 ip-10-0-131-232 kubenswrapper[2582]: W0917 19:06:23.164457    2582 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 17 19:06:23.167744 ip-10-0-131-232 kubenswrapper[2582]: W0917 19:06:23.164460    2582 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 17 19:06:23.167744 ip-10-0-131-232 kubenswrapper[2582]: W0917 19:06:23.164462    2582 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 17 19:06:23.167744 ip-10-0-131-232 kubenswrapper[2582]: W0917 19:06:23.164465    2582 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 17 19:06:23.167744 ip-10-0-131-232 kubenswrapper[2582]: W0917 19:06:23.164467    2582 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 17 19:06:23.167744 ip-10-0-131-232 kubenswrapper[2582]: W0917 19:06:23.164470    2582 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 17 19:06:23.167744 ip-10-0-131-232 kubenswrapper[2582]: W0917 19:06:23.164472    2582 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 17 19:06:23.167744 ip-10-0-131-232 kubenswrapper[2582]: W0917 19:06:23.164474    2582 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 17 19:06:23.167744 ip-10-0-131-232 kubenswrapper[2582]: W0917 19:06:23.164477    2582 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 17 19:06:23.167744 ip-10-0-131-232 kubenswrapper[2582]: W0917 19:06:23.164479    2582 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 17 19:06:23.168333 ip-10-0-131-232 kubenswrapper[2582]: W0917 19:06:23.164481    2582 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 17 19:06:23.168333 ip-10-0-131-232 kubenswrapper[2582]: W0917 19:06:23.164485    2582 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 17 19:06:23.168333 ip-10-0-131-232 kubenswrapper[2582]: W0917 19:06:23.164488    2582 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 17 19:06:23.168333 ip-10-0-131-232 kubenswrapper[2582]: W0917 19:06:23.164491    2582 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 17 19:06:23.168333 ip-10-0-131-232 kubenswrapper[2582]: W0917 19:06:23.164495    2582 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 17 19:06:23.168333 ip-10-0-131-232 kubenswrapper[2582]: W0917 19:06:23.164498    2582 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 17 19:06:23.168333 ip-10-0-131-232 kubenswrapper[2582]: W0917 19:06:23.164501    2582 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 17 19:06:23.168333 ip-10-0-131-232 kubenswrapper[2582]: W0917 19:06:23.164504    2582 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 17 19:06:23.168333 ip-10-0-131-232 kubenswrapper[2582]: W0917 19:06:23.164506    2582 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 17 19:06:23.168333 ip-10-0-131-232 kubenswrapper[2582]: W0917 19:06:23.164509    2582 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 17 19:06:23.168333 ip-10-0-131-232 kubenswrapper[2582]: W0917 19:06:23.164512    2582 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 17 19:06:23.168333 ip-10-0-131-232 kubenswrapper[2582]: W0917 19:06:23.164515    2582 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 17 19:06:23.168333 ip-10-0-131-232 kubenswrapper[2582]: W0917 19:06:23.164518    2582 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 17 19:06:23.168333 ip-10-0-131-232 kubenswrapper[2582]: W0917 19:06:23.164521    2582 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 17 19:06:23.168333 ip-10-0-131-232 kubenswrapper[2582]: W0917 19:06:23.164523    2582 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 17 19:06:23.168333 ip-10-0-131-232 kubenswrapper[2582]: W0917 19:06:23.164526    2582 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 17 19:06:23.168333 ip-10-0-131-232 kubenswrapper[2582]: W0917 19:06:23.164528    2582 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 17 19:06:23.168333 ip-10-0-131-232 kubenswrapper[2582]: W0917 19:06:23.164530    2582 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 17 19:06:23.168333 ip-10-0-131-232 kubenswrapper[2582]: W0917 19:06:23.164533    2582 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 17 19:06:23.168796 ip-10-0-131-232 kubenswrapper[2582]: W0917 19:06:23.164535    2582 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 17 19:06:23.168796 ip-10-0-131-232 kubenswrapper[2582]: W0917 19:06:23.164537    2582 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 17 19:06:23.168796 ip-10-0-131-232 kubenswrapper[2582]: W0917 19:06:23.164540    2582 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 17 19:06:23.168796 ip-10-0-131-232 kubenswrapper[2582]: W0917 19:06:23.164542    2582 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 17 19:06:23.168796 ip-10-0-131-232 kubenswrapper[2582]: W0917 19:06:23.164545    2582 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 17 19:06:23.168796 ip-10-0-131-232 kubenswrapper[2582]: W0917 19:06:23.164547    2582 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 17 19:06:23.168796 ip-10-0-131-232 kubenswrapper[2582]: W0917 19:06:23.164550    2582 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 17 19:06:23.168796 ip-10-0-131-232 kubenswrapper[2582]: W0917 19:06:23.164552    2582 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 17 19:06:23.168796 ip-10-0-131-232 kubenswrapper[2582]: W0917 19:06:23.164555    2582 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 17 19:06:23.168796 ip-10-0-131-232 kubenswrapper[2582]: W0917 19:06:23.164557    2582 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 17 19:06:23.168796 ip-10-0-131-232 kubenswrapper[2582]: W0917 19:06:23.164560    2582 feature_gate.go:328] unrecognized feature gate: Example
Sep 17 19:06:23.168796 ip-10-0-131-232 kubenswrapper[2582]: W0917 19:06:23.164562    2582 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 17 19:06:23.168796 ip-10-0-131-232 kubenswrapper[2582]: W0917 19:06:23.164564    2582 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 17 19:06:23.168796 ip-10-0-131-232 kubenswrapper[2582]: W0917 19:06:23.164566    2582 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 17 19:06:23.168796 ip-10-0-131-232 kubenswrapper[2582]: W0917 19:06:23.164569    2582 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 17 19:06:23.168796 ip-10-0-131-232 kubenswrapper[2582]: W0917 19:06:23.164571    2582 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 17 19:06:23.168796 ip-10-0-131-232 kubenswrapper[2582]: W0917 19:06:23.164574    2582 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 17 19:06:23.168796 ip-10-0-131-232 kubenswrapper[2582]: W0917 19:06:23.164576    2582 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 17 19:06:23.168796 ip-10-0-131-232 kubenswrapper[2582]: W0917 19:06:23.164578    2582 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 17 19:06:23.168796 ip-10-0-131-232 kubenswrapper[2582]: W0917 19:06:23.164581    2582 feature_gate.go:328] unrecognized feature gate: Example2
Sep 17 19:06:23.169297 ip-10-0-131-232 kubenswrapper[2582]: W0917 19:06:23.164583    2582 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 17 19:06:23.169297 ip-10-0-131-232 kubenswrapper[2582]: W0917 19:06:23.164586    2582 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 17 19:06:23.169297 ip-10-0-131-232 kubenswrapper[2582]: W0917 19:06:23.164588    2582 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 17 19:06:23.169297 ip-10-0-131-232 kubenswrapper[2582]: W0917 19:06:23.164590    2582 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 17 19:06:23.169297 ip-10-0-131-232 kubenswrapper[2582]: W0917 19:06:23.164593    2582 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 17 19:06:23.169297 ip-10-0-131-232 kubenswrapper[2582]: W0917 19:06:23.164595    2582 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 17 19:06:23.169297 ip-10-0-131-232 kubenswrapper[2582]: W0917 19:06:23.164598    2582 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 17 19:06:23.169297 ip-10-0-131-232 kubenswrapper[2582]: W0917 19:06:23.164600    2582 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 17 19:06:23.169297 ip-10-0-131-232 kubenswrapper[2582]: W0917 19:06:23.164603    2582 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 17 19:06:23.169297 ip-10-0-131-232 kubenswrapper[2582]: W0917 19:06:23.164605    2582 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 17 19:06:23.169297 ip-10-0-131-232 kubenswrapper[2582]: W0917 19:06:23.164607    2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 17 19:06:23.169297 ip-10-0-131-232 kubenswrapper[2582]: W0917 19:06:23.164610    2582 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 17 19:06:23.169297 ip-10-0-131-232 kubenswrapper[2582]: W0917 19:06:23.164612    2582 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 17 19:06:23.169297 ip-10-0-131-232 kubenswrapper[2582]: W0917 19:06:23.164615    2582 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 17 19:06:23.169297 ip-10-0-131-232 kubenswrapper[2582]: W0917 19:06:23.164617    2582 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 17 19:06:23.169297 ip-10-0-131-232 kubenswrapper[2582]: W0917 19:06:23.164619    2582 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 17 19:06:23.169297 ip-10-0-131-232 kubenswrapper[2582]: W0917 19:06:23.164622    2582 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 17 19:06:23.169297 ip-10-0-131-232 kubenswrapper[2582]: W0917 19:06:23.164625    2582 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 17 19:06:23.169297 ip-10-0-131-232 kubenswrapper[2582]: W0917 19:06:23.164627    2582 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 17 19:06:23.169297 ip-10-0-131-232 kubenswrapper[2582]: W0917 19:06:23.164630    2582 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 17 19:06:23.169785 ip-10-0-131-232 kubenswrapper[2582]: W0917 19:06:23.164632    2582 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 17 19:06:23.169785 ip-10-0-131-232 kubenswrapper[2582]: W0917 19:06:23.164635    2582 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 17 19:06:23.169785 ip-10-0-131-232 kubenswrapper[2582]: W0917 19:06:23.164637    2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 17 19:06:23.169785 ip-10-0-131-232 kubenswrapper[2582]: W0917 19:06:23.164639    2582 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 17 19:06:23.169785 ip-10-0-131-232 kubenswrapper[2582]: W0917 19:06:23.164642    2582 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 17 19:06:23.169785 ip-10-0-131-232 kubenswrapper[2582]: W0917 19:06:23.164644    2582 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 17 19:06:23.169785 ip-10-0-131-232 kubenswrapper[2582]: W0917 19:06:23.164646    2582 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 17 19:06:23.169785 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:06:23.164652    2582 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 17 19:06:23.169785 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:06:23.164742    2582 server.go:962] "Client rotation is on, will bootstrap in background"
Sep 17 19:06:23.169785 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:06:23.167045    2582 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir"
Sep 17 19:06:23.169785 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:06:23.168090    2582 server.go:1019] "Starting client certificate rotation"
Sep 17 19:06:23.169785 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:06:23.168191    2582 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Sep 17 19:06:23.169785 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:06:23.168235    2582 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet"
Sep 17 19:06:23.170927 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:06:23.170913    2582 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Sep 17 19:06:23.172379 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:06:23.172362    2582 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Sep 17 19:06:23.183627 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:06:23.183610    2582 log.go:25] "Validated CRI v1 runtime API"
Sep 17 19:06:23.185371 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:06:23.185359    2582 log.go:25] "Validated CRI v1 image API"
Sep 17 19:06:23.186317 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:06:23.186296    2582 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Sep 17 19:06:23.192654 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:06:23.192637    2582 fs.go:135] Filesystem UUIDs: map[7B77-95E7:/dev/nvme0n1p2 ccfaf1be-23bf-4abd-bb05-2cd5d42aca6d:/dev/nvme0n1p3 de3c48a8-fd44-4e17-9cfd-c29cc77d21b7:/dev/nvme0n1p4]
Sep 17 19:06:23.192715 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:06:23.192654    2582 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}]
Sep 17 19:06:23.205045 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:06:23.204932    2582 manager.go:216] Machine: {Timestamp:2026-09-17 19:06:23.196532879 +0000 UTC m=+0.356130530 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3102071 MemoryCapacity:32812158976 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec273115f8a574daa087752e72548d7e SystemUUID:ec273115-f8a5-74da-a087-752e72548d7e BootID:7d854f96-33c2-4bfc-8b4a-2b5385c27af1 Filesystems:[{Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16406077440 Type:vfs Inodes:4005390 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6562435072 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6111232 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16406081536 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:ff:f3:83:c6:2b Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:ff:f3:83:c6:2b Speed:0 Mtu:9001} {Name:ovs-system MacAddress:76:b3:8b:a2:f9:bc Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:32812158976 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:34603008 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Sep 17 19:06:23.205045 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:06:23.205041    2582 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Sep 17 19:06:23.205162 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:06:23.205120    2582 manager.go:232] Version: {KernelVersion:5.14.0-570.141.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260914-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Sep 17 19:06:23.206019 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:06:23.206003    2582 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Sep 17 19:06:23.209129 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:06:23.209104    2582 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Sep 17 19:06:23.209256 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:06:23.209134    2582 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-131-232.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Sep 17 19:06:23.210852 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:06:23.210842    2582 topology_manager.go:138] "Creating topology manager with none policy"
Sep 17 19:06:23.210888 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:06:23.210854    2582 container_manager_linux.go:309] "Creating device plugin manager"
Sep 17 19:06:23.210888 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:06:23.210866    2582 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Sep 17 19:06:23.211731 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:06:23.211721    2582 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Sep 17 19:06:23.211802 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:06:23.211792    2582 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager"
Sep 17 19:06:23.216070 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:06:23.216059    2582 state_mem.go:36] "Initialized new in-memory state store"
Sep 17 19:06:23.216195 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:06:23.216185    2582 server.go:1240] "Using root directory" path="/var/lib/kubelet"
Sep 17 19:06:23.216230 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:06:23.216216    2582 kubelet.go:486] "Attempting to sync node with API server"
Sep 17 19:06:23.216230 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:06:23.216228    2582 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests"
Sep 17 19:06:23.216282 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:06:23.216250    2582 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Sep 17 19:06:23.216282 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:06:23.216261    2582 kubelet.go:398] "Adding apiserver pod source"
Sep 17 19:06:23.216282 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:06:23.216270    2582 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Sep 17 19:06:23.216629 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:06:23.216618    2582 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Sep 17 19:06:23.216664 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:06:23.216636    2582 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Sep 17 19:06:23.219139 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:06:23.219124    2582 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.13-6.rhaos4.21.git6253162.el9" apiVersion="v1"
Sep 17 19:06:23.221958 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:06:23.221945    2582 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Sep 17 19:06:23.221997 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:06:23.221975    2582 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Sep 17 19:06:23.223286 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:06:23.223267    2582 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Sep 17 19:06:23.223372 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:06:23.223291    2582 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Sep 17 19:06:23.223372 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:06:23.223302    2582 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Sep 17 19:06:23.223372 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:06:23.223311    2582 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Sep 17 19:06:23.223372 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:06:23.223334    2582 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Sep 17 19:06:23.223372 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:06:23.223342    2582 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Sep 17 19:06:23.223372 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:06:23.223360    2582 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Sep 17 19:06:23.223372 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:06:23.223369    2582 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Sep 17 19:06:23.223564 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:06:23.223378    2582 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Sep 17 19:06:23.223564 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:06:23.223387    2582 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Sep 17 19:06:23.223564 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:06:23.223406    2582 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Sep 17 19:06:23.223564 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:06:23.223414    2582 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Sep 17 19:06:23.224320 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:06:23.224307    2582 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Sep 17 19:06:23.224320 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:06:23.224320    2582 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Sep 17 19:06:23.225922 ip-10-0-131-232 kubenswrapper[2582]: E0917 19:06:23.225885    2582 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service"
Sep 17 19:06:23.225959 ip-10-0-131-232 kubenswrapper[2582]: E0917 19:06:23.225925    2582 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-131-232.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node"
Sep 17 19:06:23.228138 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:06:23.228126    2582 server.go:1268] "Started kubelet"
Sep 17 19:06:23.228241 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:06:23.228205    2582 server.go:180] "Starting to listen" address="0.0.0.0" port=10250
Sep 17 19:06:23.228707 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:06:23.228364    2582 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Sep 17 19:06:23.228835 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:06:23.228821    2582 server_v1.go:49] "podresources" method="list" useActivePods=true
Sep 17 19:06:23.228977 ip-10-0-131-232 systemd[1]: Started Kubernetes Kubelet.
Sep 17 19:06:23.233169 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:06:23.233148    2582 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Sep 17 19:06:23.234576 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:06:23.234550    2582 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-42lj6"
Sep 17 19:06:23.234868 ip-10-0-131-232 kubenswrapper[2582]: E0917 19:06:23.234843    2582 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache"
Sep 17 19:06:23.235402 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:06:23.235383    2582 server.go:310] "Adding debug handlers to kubelet server"
Sep 17 19:06:23.240077 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:06:23.240060    2582 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-131-232.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Sep 17 19:06:23.241052 ip-10-0-131-232 kubenswrapper[2582]: E0917 19:06:23.239975    2582 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-131-232.ec2.internal.18d6312fc8c78fb6  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-131-232.ec2.internal,UID:ip-10-0-131-232.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-131-232.ec2.internal,},FirstTimestamp:2026-09-17 19:06:23.228096438 +0000 UTC m=+0.387694083,LastTimestamp:2026-09-17 19:06:23.228096438 +0000 UTC m=+0.387694083,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-131-232.ec2.internal,}"
Sep 17 19:06:23.241191 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:06:23.241171    2582 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Sep 17 19:06:23.241885 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:06:23.241858    2582 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer"
Sep 17 19:06:23.242051 ip-10-0-131-232 kubenswrapper[2582]: E0917 19:06:23.242029    2582 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-131-232.ec2.internal\" not found"
Sep 17 19:06:23.242581 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:06:23.242565    2582 factory.go:55] Registering systemd factory
Sep 17 19:06:23.242581 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:06:23.242582    2582 factory.go:223] Registration of the systemd container factory successfully
Sep 17 19:06:23.242745 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:06:23.242689    2582 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Sep 17 19:06:23.242745 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:06:23.242734    2582 volume_manager.go:311] "The desired_state_of_world populator starts"
Sep 17 19:06:23.242745 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:06:23.242745    2582 volume_manager.go:313] "Starting Kubelet Volume Manager"
Sep 17 19:06:23.242848 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:06:23.242803    2582 factory.go:153] Registering CRI-O factory
Sep 17 19:06:23.242848 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:06:23.242817    2582 reconstruct.go:97] "Volume reconstruction finished"
Sep 17 19:06:23.242848 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:06:23.242823    2582 reconciler.go:29] "Reconciler: start to sync state"
Sep 17 19:06:23.242848 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:06:23.242818    2582 factory.go:223] Registration of the crio container factory successfully
Sep 17 19:06:23.243010 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:06:23.242867    2582 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Sep 17 19:06:23.243010 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:06:23.242880    2582 factory.go:103] Registering Raw factory
Sep 17 19:06:23.243010 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:06:23.242889    2582 manager.go:1196] Started watching for new ooms in manager
Sep 17 19:06:23.243361 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:06:23.243295    2582 manager.go:349] Starting recovery of all containers
Sep 17 19:06:23.245041 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:06:23.245021    2582 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-42lj6"
Sep 17 19:06:23.246532 ip-10-0-131-232 kubenswrapper[2582]: E0917 19:06:23.246503    2582 reflector.go:205] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver"
Sep 17 19:06:23.246610 ip-10-0-131-232 kubenswrapper[2582]: E0917 19:06:23.246576    2582 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-131-232.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms"
Sep 17 19:06:23.254190 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:06:23.253733    2582 manager.go:354] Recovery completed
Sep 17 19:06:23.255543 ip-10-0-131-232 kubenswrapper[2582]: E0917 19:06:23.255517    2582 watcher.go:152] Failed to watch directory "/sys/fs/cgroup/system.slice/systemd-update-utmp-runlevel.service": inotify_add_watch /sys/fs/cgroup/system.slice/systemd-update-utmp-runlevel.service: no such file or directory
Sep 17 19:06:23.258598 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:06:23.258586    2582 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 17 19:06:23.260412 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:06:23.260396    2582 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-232.ec2.internal" event="NodeHasSufficientMemory"
Sep 17 19:06:23.260485 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:06:23.260428    2582 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-232.ec2.internal" event="NodeHasNoDiskPressure"
Sep 17 19:06:23.260485 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:06:23.260442    2582 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-232.ec2.internal" event="NodeHasSufficientPID"
Sep 17 19:06:23.261036 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:06:23.261023    2582 cpu_manager.go:222] "Starting CPU manager" policy="none"
Sep 17 19:06:23.261036 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:06:23.261035    2582 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 17 19:06:23.261132 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:06:23.261035    2582 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s"
Sep 17 19:06:23.261132 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:06:23.261081    2582 state_mem.go:36] "Initialized new in-memory state store"
Sep 17 19:06:23.261697 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:06:23.261684    2582 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-232.ec2.internal" event="NodeHasSufficientMemory"
Sep 17 19:06:23.261748 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:06:23.261708    2582 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-232.ec2.internal" event="NodeHasNoDiskPressure"
Sep 17 19:06:23.261748 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:06:23.261721    2582 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-232.ec2.internal" event="NodeHasSufficientPID"
Sep 17 19:06:23.263026 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:06:23.263012    2582 policy_none.go:49] "None policy: Start"
Sep 17 19:06:23.263026 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:06:23.263028    2582 memory_manager.go:187] "Starting memorymanager" policy="None"
Sep 17 19:06:23.263133 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:06:23.263037    2582 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Sep 17 19:06:23.263220 ip-10-0-131-232 kubenswrapper[2582]: E0917 19:06:23.263062    2582 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-131-232.ec2.internal.18d6312fcab4a8a4  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-131-232.ec2.internal,UID:ip-10-0-131-232.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-131-232.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-131-232.ec2.internal,},FirstTimestamp:2026-09-17 19:06:23.260412068 +0000 UTC m=+0.420009718,LastTimestamp:2026-09-17 19:06:23.260412068 +0000 UTC m=+0.420009718,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-131-232.ec2.internal,}"
Sep 17 19:06:23.264938 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:06:23.264926    2582 policy_none.go:47] "Start"
Sep 17 19:06:23.304428 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:06:23.304412    2582 manager.go:337] "Starting Device Plugin manager"
Sep 17 19:06:23.321009 ip-10-0-131-232 kubenswrapper[2582]: E0917 19:06:23.304456    2582 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Sep 17 19:06:23.321009 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:06:23.304468    2582 server.go:87] "Starting device plugin registration server"
Sep 17 19:06:23.321009 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:06:23.304761    2582 eviction_manager.go:189] "Eviction manager: starting control loop"
Sep 17 19:06:23.321009 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:06:23.304773    2582 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Sep 17 19:06:23.321009 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:06:23.304866    2582 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Sep 17 19:06:23.321009 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:06:23.304971    2582 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts"
Sep 17 19:06:23.321009 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:06:23.304980    2582 plugin_manager.go:118] "Starting Kubelet Plugin Manager"
Sep 17 19:06:23.321009 ip-10-0-131-232 kubenswrapper[2582]: E0917 19:06:23.305428    2582 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Sep 17 19:06:23.321009 ip-10-0-131-232 kubenswrapper[2582]: E0917 19:06:23.305471    2582 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-131-232.ec2.internal\" not found"
Sep 17 19:06:23.373270 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:06:23.373238    2582 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Sep 17 19:06:23.374564 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:06:23.374544    2582 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Sep 17 19:06:23.374648 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:06:23.374584    2582 status_manager.go:244] "Starting to sync pod status with apiserver"
Sep 17 19:06:23.374648 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:06:23.374604    2582 kubelet.go:2448] "Starting kubelet main sync loop"
Sep 17 19:06:23.374648 ip-10-0-131-232 kubenswrapper[2582]: E0917 19:06:23.374636    2582 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful"
Sep 17 19:06:23.377531 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:06:23.377514    2582 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 17 19:06:23.405508 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:06:23.405493    2582 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 17 19:06:23.406367 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:06:23.406353    2582 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-232.ec2.internal" event="NodeHasSufficientMemory"
Sep 17 19:06:23.406446 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:06:23.406382    2582 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-232.ec2.internal" event="NodeHasNoDiskPressure"
Sep 17 19:06:23.406446 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:06:23.406397    2582 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-232.ec2.internal" event="NodeHasSufficientPID"
Sep 17 19:06:23.406446 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:06:23.406423    2582 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-131-232.ec2.internal"
Sep 17 19:06:23.419438 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:06:23.419420    2582 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-131-232.ec2.internal"
Sep 17 19:06:23.419495 ip-10-0-131-232 kubenswrapper[2582]: E0917 19:06:23.419443    2582 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-131-232.ec2.internal\": node \"ip-10-0-131-232.ec2.internal\" not found"
Sep 17 19:06:23.444441 ip-10-0-131-232 kubenswrapper[2582]: E0917 19:06:23.444416    2582 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-131-232.ec2.internal\" not found"
Sep 17 19:06:23.475209 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:06:23.475182    2582 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-232.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-131-232.ec2.internal"]
Sep 17 19:06:23.475315 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:06:23.475255    2582 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 17 19:06:23.476837 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:06:23.476822    2582 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-232.ec2.internal" event="NodeHasSufficientMemory"
Sep 17 19:06:23.476930 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:06:23.476851    2582 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-232.ec2.internal" event="NodeHasNoDiskPressure"
Sep 17 19:06:23.476930 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:06:23.476866    2582 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-232.ec2.internal" event="NodeHasSufficientPID"
Sep 17 19:06:23.478816 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:06:23.478804    2582 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 17 19:06:23.478963 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:06:23.478935    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-232.ec2.internal"
Sep 17 19:06:23.479004 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:06:23.478981    2582 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 17 19:06:23.479561 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:06:23.479539    2582 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-232.ec2.internal" event="NodeHasSufficientMemory"
Sep 17 19:06:23.479561 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:06:23.479565    2582 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-232.ec2.internal" event="NodeHasNoDiskPressure"
Sep 17 19:06:23.479694 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:06:23.479578    2582 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-232.ec2.internal" event="NodeHasSufficientPID"
Sep 17 19:06:23.479694 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:06:23.479590    2582 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-232.ec2.internal" event="NodeHasSufficientMemory"
Sep 17 19:06:23.479694 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:06:23.479615    2582 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-232.ec2.internal" event="NodeHasNoDiskPressure"
Sep 17 19:06:23.479694 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:06:23.479631    2582 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-232.ec2.internal" event="NodeHasSufficientPID"
Sep 17 19:06:23.481614 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:06:23.481596    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-131-232.ec2.internal"
Sep 17 19:06:23.481703 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:06:23.481626    2582 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 17 19:06:23.482307 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:06:23.482289    2582 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-232.ec2.internal" event="NodeHasSufficientMemory"
Sep 17 19:06:23.482397 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:06:23.482321    2582 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-232.ec2.internal" event="NodeHasNoDiskPressure"
Sep 17 19:06:23.482397 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:06:23.482334    2582 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-232.ec2.internal" event="NodeHasSufficientPID"
Sep 17 19:06:23.507644 ip-10-0-131-232 kubenswrapper[2582]: E0917 19:06:23.507617    2582 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-131-232.ec2.internal\" not found" node="ip-10-0-131-232.ec2.internal"
Sep 17 19:06:23.512227 ip-10-0-131-232 kubenswrapper[2582]: E0917 19:06:23.512208    2582 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-131-232.ec2.internal\" not found" node="ip-10-0-131-232.ec2.internal"
Sep 17 19:06:23.544352 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:06:23.544322    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/5a96bc0ce60b5e0ec092cd8768bc466e-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-131-232.ec2.internal\" (UID: \"5a96bc0ce60b5e0ec092cd8768bc466e\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-232.ec2.internal"
Sep 17 19:06:23.544352 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:06:23.544351    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/5a96bc0ce60b5e0ec092cd8768bc466e-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-131-232.ec2.internal\" (UID: \"5a96bc0ce60b5e0ec092cd8768bc466e\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-232.ec2.internal"
Sep 17 19:06:23.544552 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:06:23.544376    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/a1f9bd0034d8d01c3b2f70180b25c21f-config\") pod \"kube-apiserver-proxy-ip-10-0-131-232.ec2.internal\" (UID: \"a1f9bd0034d8d01c3b2f70180b25c21f\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-131-232.ec2.internal"
Sep 17 19:06:23.544552 ip-10-0-131-232 kubenswrapper[2582]: E0917 19:06:23.544456    2582 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-131-232.ec2.internal\" not found"
Sep 17 19:06:23.645208 ip-10-0-131-232 kubenswrapper[2582]: E0917 19:06:23.645129    2582 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-131-232.ec2.internal\" not found"
Sep 17 19:06:23.645208 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:06:23.645185    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/a1f9bd0034d8d01c3b2f70180b25c21f-config\") pod \"kube-apiserver-proxy-ip-10-0-131-232.ec2.internal\" (UID: \"a1f9bd0034d8d01c3b2f70180b25c21f\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-131-232.ec2.internal"
Sep 17 19:06:23.645208 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:06:23.645208    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/5a96bc0ce60b5e0ec092cd8768bc466e-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-131-232.ec2.internal\" (UID: \"5a96bc0ce60b5e0ec092cd8768bc466e\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-232.ec2.internal"
Sep 17 19:06:23.645427 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:06:23.645225    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/5a96bc0ce60b5e0ec092cd8768bc466e-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-131-232.ec2.internal\" (UID: \"5a96bc0ce60b5e0ec092cd8768bc466e\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-232.ec2.internal"
Sep 17 19:06:23.645427 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:06:23.645264    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/5a96bc0ce60b5e0ec092cd8768bc466e-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-131-232.ec2.internal\" (UID: \"5a96bc0ce60b5e0ec092cd8768bc466e\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-232.ec2.internal"
Sep 17 19:06:23.645427 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:06:23.645290    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/a1f9bd0034d8d01c3b2f70180b25c21f-config\") pod \"kube-apiserver-proxy-ip-10-0-131-232.ec2.internal\" (UID: \"a1f9bd0034d8d01c3b2f70180b25c21f\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-131-232.ec2.internal"
Sep 17 19:06:23.645427 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:06:23.645302    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/5a96bc0ce60b5e0ec092cd8768bc466e-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-131-232.ec2.internal\" (UID: \"5a96bc0ce60b5e0ec092cd8768bc466e\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-232.ec2.internal"
Sep 17 19:06:23.746244 ip-10-0-131-232 kubenswrapper[2582]: E0917 19:06:23.746207    2582 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-131-232.ec2.internal\" not found"
Sep 17 19:06:23.810437 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:06:23.810401    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-232.ec2.internal"
Sep 17 19:06:23.816311 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:06:23.816293    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-131-232.ec2.internal"
Sep 17 19:06:23.847276 ip-10-0-131-232 kubenswrapper[2582]: E0917 19:06:23.847244    2582 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-131-232.ec2.internal\" not found"
Sep 17 19:06:23.947832 ip-10-0-131-232 kubenswrapper[2582]: E0917 19:06:23.947747    2582 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-131-232.ec2.internal\" not found"
Sep 17 19:06:24.048136 ip-10-0-131-232 kubenswrapper[2582]: E0917 19:06:24.048104    2582 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-131-232.ec2.internal\" not found"
Sep 17 19:06:24.061331 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:06:24.061306    2582 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 17 19:06:24.142420 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:06:24.142390    2582 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-131-232.ec2.internal"
Sep 17 19:06:24.154734 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:06:24.154707    2582 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Sep 17 19:06:24.156281 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:06:24.156256    2582 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-232.ec2.internal"
Sep 17 19:06:24.161920 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:06:24.161884    2582 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 17 19:06:24.167545 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:06:24.167530    2582 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials"
Sep 17 19:06:24.167648 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:06:24.167632    2582 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Sep 17 19:06:24.167706 ip-10-0-131-232 kubenswrapper[2582]: E0917 19:06:24.167666    2582 request.go:1196] "Unexpected error when reading response body" err="read tcp 10.0.131.232:41998->23.21.151.204:6443: use of closed network connection"
Sep 17 19:06:24.167706 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:06:24.167671    2582 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Sep 17 19:06:24.167706 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:06:24.167688    2582 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Sep 17 19:06:24.167706 ip-10-0-131-232 kubenswrapper[2582]: E0917 19:06:24.167703    2582 kubelet.go:3250] "Failed creating a mirror pod" err="unexpected error when reading response body. Please retry. Original error: read tcp 10.0.131.232:41998->23.21.151.204:6443: use of closed network connection" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-232.ec2.internal"
Sep 17 19:06:24.216916 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:06:24.216854    2582 apiserver.go:52] "Watching apiserver"
Sep 17 19:06:24.234570 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:06:24.234553    2582 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Sep 17 19:06:24.236140 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:06:24.235675    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-network-diagnostics/network-check-target-jhjrs","openshift-ovn-kubernetes/ovnkube-node-kdm8s","kube-system/global-pull-secret-syncer-f6kjs","kube-system/konnectivity-agent-qwhmv","openshift-dns/node-resolver-cwtkf","openshift-multus/multus-additional-cni-plugins-wz7lk","openshift-multus/multus-fppr7","openshift-network-operator/iptables-alerter-kzblg","kube-system/kube-apiserver-proxy-ip-10-0-131-232.ec2.internal","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qxfrl","openshift-cluster-node-tuning-operator/tuned-g8qm4","openshift-image-registry/node-ca-64g8v","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-232.ec2.internal","openshift-multus/network-metrics-daemon-hj5r7"]
Sep 17 19:06:24.238281 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:06:24.238255    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-fppr7"
Sep 17 19:06:24.240449 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:06:24.240417    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-g8qm4"
Sep 17 19:06:24.241065 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:06:24.241046    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Sep 17 19:06:24.241176 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:06:24.241140    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-qnkzk\""
Sep 17 19:06:24.241176 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:06:24.241141    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Sep 17 19:06:24.241289 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:06:24.241209    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Sep 17 19:06:24.241289 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:06:24.241243    2582 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving"
Sep 17 19:06:24.241418 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:06:24.241399    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Sep 17 19:06:24.245001 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:06:24.244973    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-f6kjs"
Sep 17 19:06:24.245105 ip-10-0-131-232 kubenswrapper[2582]: E0917 19:06:24.245081    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-f6kjs" podUID="7cd887c0-2cc8-4363-b553-8d2b4e10f158"
Sep 17 19:06:24.245166 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:06:24.245140    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Sep 17 19:06:24.245324 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:06:24.245303    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-dcc46\""
Sep 17 19:06:24.245421 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:06:24.245325    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Sep 17 19:06:24.245421 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:06:24.245379    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-qwhmv"
Sep 17 19:06:24.247112 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:06:24.247089    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-cwtkf"
Sep 17 19:06:24.247597 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:06:24.247578    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Sep 17 19:06:24.247699 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:06:24.247634    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Sep 17 19:06:24.247864 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:06:24.247849    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-2klb7\""
Sep 17 19:06:24.248772 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:06:24.248753    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/aa2ade5e-8564-46a7-819c-75c57cb0961d-system-cni-dir\") pod \"multus-fppr7\" (UID: \"aa2ade5e-8564-46a7-819c-75c57cb0961d\") " pod="openshift-multus/multus-fppr7"
Sep 17 19:06:24.248849 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:06:24.248785    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/aa2ade5e-8564-46a7-819c-75c57cb0961d-multus-conf-dir\") pod \"multus-fppr7\" (UID: \"aa2ade5e-8564-46a7-819c-75c57cb0961d\") " pod="openshift-multus/multus-fppr7"
Sep 17 19:06:24.248849 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:06:24.248811    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/f661fc84-37f7-4cb9-a5b5-be6673fbd0dd-host\") pod \"tuned-g8qm4\" (UID: \"f661fc84-37f7-4cb9-a5b5-be6673fbd0dd\") " pod="openshift-cluster-node-tuning-operator/tuned-g8qm4"
Sep 17 19:06:24.248849 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:06:24.248835    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/46629fc1-07ce-45b8-9d73-453abfa383b9-hosts-file\") pod \"node-resolver-cwtkf\" (UID: \"46629fc1-07ce-45b8-9d73-453abfa383b9\") " pod="openshift-dns/node-resolver-cwtkf"
Sep 17 19:06:24.248990 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:06:24.248852    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/aa2ade5e-8564-46a7-819c-75c57cb0961d-host-run-netns\") pod \"multus-fppr7\" (UID: \"aa2ade5e-8564-46a7-819c-75c57cb0961d\") " pod="openshift-multus/multus-fppr7"
Sep 17 19:06:24.248990 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:06:24.248867    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/aa2ade5e-8564-46a7-819c-75c57cb0961d-multus-daemon-config\") pod \"multus-fppr7\" (UID: \"aa2ade5e-8564-46a7-819c-75c57cb0961d\") " pod="openshift-multus/multus-fppr7"
Sep 17 19:06:24.248990 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:06:24.248931    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/f661fc84-37f7-4cb9-a5b5-be6673fbd0dd-etc-sysconfig\") pod \"tuned-g8qm4\" (UID: \"f661fc84-37f7-4cb9-a5b5-be6673fbd0dd\") " pod="openshift-cluster-node-tuning-operator/tuned-g8qm4"
Sep 17 19:06:24.248990 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:06:24.248967    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/f661fc84-37f7-4cb9-a5b5-be6673fbd0dd-etc-sysctl-d\") pod \"tuned-g8qm4\" (UID: \"f661fc84-37f7-4cb9-a5b5-be6673fbd0dd\") " pod="openshift-cluster-node-tuning-operator/tuned-g8qm4"
Sep 17 19:06:24.249136 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:06:24.248995    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/f661fc84-37f7-4cb9-a5b5-be6673fbd0dd-etc-systemd\") pod \"tuned-g8qm4\" (UID: \"f661fc84-37f7-4cb9-a5b5-be6673fbd0dd\") " pod="openshift-cluster-node-tuning-operator/tuned-g8qm4"
Sep 17 19:06:24.249136 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:06:24.249037    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/f661fc84-37f7-4cb9-a5b5-be6673fbd0dd-var-lib-kubelet\") pod \"tuned-g8qm4\" (UID: \"f661fc84-37f7-4cb9-a5b5-be6673fbd0dd\") " pod="openshift-cluster-node-tuning-operator/tuned-g8qm4"
Sep 17 19:06:24.249136 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:06:24.249080    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/f661fc84-37f7-4cb9-a5b5-be6673fbd0dd-tmp\") pod \"tuned-g8qm4\" (UID: \"f661fc84-37f7-4cb9-a5b5-be6673fbd0dd\") " pod="openshift-cluster-node-tuning-operator/tuned-g8qm4"
Sep 17 19:06:24.249136 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:06:24.249096    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/aa2ade5e-8564-46a7-819c-75c57cb0961d-multus-socket-dir-parent\") pod \"multus-fppr7\" (UID: \"aa2ade5e-8564-46a7-819c-75c57cb0961d\") " pod="openshift-multus/multus-fppr7"
Sep 17 19:06:24.249136 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:06:24.249112    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/aa2ade5e-8564-46a7-819c-75c57cb0961d-host-run-multus-certs\") pod \"multus-fppr7\" (UID: \"aa2ade5e-8564-46a7-819c-75c57cb0961d\") " pod="openshift-multus/multus-fppr7"
Sep 17 19:06:24.249136 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:06:24.249128    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-22chq\" (UniqueName: \"kubernetes.io/projected/aa2ade5e-8564-46a7-819c-75c57cb0961d-kube-api-access-22chq\") pod \"multus-fppr7\" (UID: \"aa2ade5e-8564-46a7-819c-75c57cb0961d\") " pod="openshift-multus/multus-fppr7"
Sep 17 19:06:24.249402 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:06:24.249143    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/7cd887c0-2cc8-4363-b553-8d2b4e10f158-kubelet-config\") pod \"global-pull-secret-syncer-f6kjs\" (UID: \"7cd887c0-2cc8-4363-b553-8d2b4e10f158\") " pod="kube-system/global-pull-secret-syncer-f6kjs"
Sep 17 19:06:24.249402 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:06:24.249183    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/7cd887c0-2cc8-4363-b553-8d2b4e10f158-dbus\") pod \"global-pull-secret-syncer-f6kjs\" (UID: \"7cd887c0-2cc8-4363-b553-8d2b4e10f158\") " pod="kube-system/global-pull-secret-syncer-f6kjs"
Sep 17 19:06:24.249402 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:06:24.249226    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/fe7c72bc-812d-4272-8352-78f8fa2c19d1-agent-certs\") pod \"konnectivity-agent-qwhmv\" (UID: \"fe7c72bc-812d-4272-8352-78f8fa2c19d1\") " pod="kube-system/konnectivity-agent-qwhmv"
Sep 17 19:06:24.249402 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:06:24.249250    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/aa2ade5e-8564-46a7-819c-75c57cb0961d-cnibin\") pod \"multus-fppr7\" (UID: \"aa2ade5e-8564-46a7-819c-75c57cb0961d\") " pod="openshift-multus/multus-fppr7"
Sep 17 19:06:24.249402 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:06:24.249274    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/aa2ade5e-8564-46a7-819c-75c57cb0961d-host-run-k8s-cni-cncf-io\") pod \"multus-fppr7\" (UID: \"aa2ade5e-8564-46a7-819c-75c57cb0961d\") " pod="openshift-multus/multus-fppr7"
Sep 17 19:06:24.249402 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:06:24.249302    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/f661fc84-37f7-4cb9-a5b5-be6673fbd0dd-etc-modprobe-d\") pod \"tuned-g8qm4\" (UID: \"f661fc84-37f7-4cb9-a5b5-be6673fbd0dd\") " pod="openshift-cluster-node-tuning-operator/tuned-g8qm4"
Sep 17 19:06:24.249402 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:06:24.249339    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xdgj6\" (UniqueName: \"kubernetes.io/projected/f661fc84-37f7-4cb9-a5b5-be6673fbd0dd-kube-api-access-xdgj6\") pod \"tuned-g8qm4\" (UID: \"f661fc84-37f7-4cb9-a5b5-be6673fbd0dd\") " pod="openshift-cluster-node-tuning-operator/tuned-g8qm4"
Sep 17 19:06:24.249402 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:06:24.249389    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/7cd887c0-2cc8-4363-b553-8d2b4e10f158-original-pull-secret\") pod \"global-pull-secret-syncer-f6kjs\" (UID: \"7cd887c0-2cc8-4363-b553-8d2b4e10f158\") " pod="kube-system/global-pull-secret-syncer-f6kjs"
Sep 17 19:06:24.249757 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:06:24.249413    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/aa2ade5e-8564-46a7-819c-75c57cb0961d-hostroot\") pod \"multus-fppr7\" (UID: \"aa2ade5e-8564-46a7-819c-75c57cb0961d\") " pod="openshift-multus/multus-fppr7"
Sep 17 19:06:24.249757 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:06:24.249426    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/f661fc84-37f7-4cb9-a5b5-be6673fbd0dd-run\") pod \"tuned-g8qm4\" (UID: \"f661fc84-37f7-4cb9-a5b5-be6673fbd0dd\") " pod="openshift-cluster-node-tuning-operator/tuned-g8qm4"
Sep 17 19:06:24.249757 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:06:24.249441    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/fe7c72bc-812d-4272-8352-78f8fa2c19d1-konnectivity-ca\") pod \"konnectivity-agent-qwhmv\" (UID: \"fe7c72bc-812d-4272-8352-78f8fa2c19d1\") " pod="kube-system/konnectivity-agent-qwhmv"
Sep 17 19:06:24.249757 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:06:24.249457    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/aa2ade5e-8564-46a7-819c-75c57cb0961d-cni-binary-copy\") pod \"multus-fppr7\" (UID: \"aa2ade5e-8564-46a7-819c-75c57cb0961d\") " pod="openshift-multus/multus-fppr7"
Sep 17 19:06:24.249757 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:06:24.249481    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/aa2ade5e-8564-46a7-819c-75c57cb0961d-host-var-lib-cni-bin\") pod \"multus-fppr7\" (UID: \"aa2ade5e-8564-46a7-819c-75c57cb0961d\") " pod="openshift-multus/multus-fppr7"
Sep 17 19:06:24.249757 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:06:24.249509    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/aa2ade5e-8564-46a7-819c-75c57cb0961d-host-var-lib-cni-multus\") pod \"multus-fppr7\" (UID: \"aa2ade5e-8564-46a7-819c-75c57cb0961d\") " pod="openshift-multus/multus-fppr7"
Sep 17 19:06:24.249757 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:06:24.249526    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/aa2ade5e-8564-46a7-819c-75c57cb0961d-host-var-lib-kubelet\") pod \"multus-fppr7\" (UID: \"aa2ade5e-8564-46a7-819c-75c57cb0961d\") " pod="openshift-multus/multus-fppr7"
Sep 17 19:06:24.249757 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:06:24.249542    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/f661fc84-37f7-4cb9-a5b5-be6673fbd0dd-sys\") pod \"tuned-g8qm4\" (UID: \"f661fc84-37f7-4cb9-a5b5-be6673fbd0dd\") " pod="openshift-cluster-node-tuning-operator/tuned-g8qm4"
Sep 17 19:06:24.249757 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:06:24.249565    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/f661fc84-37f7-4cb9-a5b5-be6673fbd0dd-etc-tuned\") pod \"tuned-g8qm4\" (UID: \"f661fc84-37f7-4cb9-a5b5-be6673fbd0dd\") " pod="openshift-cluster-node-tuning-operator/tuned-g8qm4"
Sep 17 19:06:24.249757 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:06:24.249588    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5h6q4\" (UniqueName: \"kubernetes.io/projected/46629fc1-07ce-45b8-9d73-453abfa383b9-kube-api-access-5h6q4\") pod \"node-resolver-cwtkf\" (UID: \"46629fc1-07ce-45b8-9d73-453abfa383b9\") " pod="openshift-dns/node-resolver-cwtkf"
Sep 17 19:06:24.249757 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:06:24.249624    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/aa2ade5e-8564-46a7-819c-75c57cb0961d-multus-cni-dir\") pod \"multus-fppr7\" (UID: \"aa2ade5e-8564-46a7-819c-75c57cb0961d\") " pod="openshift-multus/multus-fppr7"
Sep 17 19:06:24.249757 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:06:24.249649    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-wz7lk"
Sep 17 19:06:24.249757 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:06:24.249654    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/46629fc1-07ce-45b8-9d73-453abfa383b9-tmp-dir\") pod \"node-resolver-cwtkf\" (UID: \"46629fc1-07ce-45b8-9d73-453abfa383b9\") " pod="openshift-dns/node-resolver-cwtkf"
Sep 17 19:06:24.249757 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:06:24.249676    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/aa2ade5e-8564-46a7-819c-75c57cb0961d-etc-kubernetes\") pod \"multus-fppr7\" (UID: \"aa2ade5e-8564-46a7-819c-75c57cb0961d\") " pod="openshift-multus/multus-fppr7"
Sep 17 19:06:24.249757 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:06:24.249699    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f661fc84-37f7-4cb9-a5b5-be6673fbd0dd-etc-kubernetes\") pod \"tuned-g8qm4\" (UID: \"f661fc84-37f7-4cb9-a5b5-be6673fbd0dd\") " pod="openshift-cluster-node-tuning-operator/tuned-g8qm4"
Sep 17 19:06:24.249757 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:06:24.249748    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/f661fc84-37f7-4cb9-a5b5-be6673fbd0dd-etc-sysctl-conf\") pod \"tuned-g8qm4\" (UID: \"f661fc84-37f7-4cb9-a5b5-be6673fbd0dd\") " pod="openshift-cluster-node-tuning-operator/tuned-g8qm4"
Sep 17 19:06:24.250407 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:06:24.249774    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/f661fc84-37f7-4cb9-a5b5-be6673fbd0dd-lib-modules\") pod \"tuned-g8qm4\" (UID: \"f661fc84-37f7-4cb9-a5b5-be6673fbd0dd\") " pod="openshift-cluster-node-tuning-operator/tuned-g8qm4"
Sep 17 19:06:24.250479 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:06:24.250462    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Sep 17 19:06:24.250819 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:06:24.250564    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Sep 17 19:06:24.250819 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:06:24.250808    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-h6r4v\""
Sep 17 19:06:24.251637 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:06:24.251594    2582 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-09-16 19:01:24 +0000 UTC" deadline="2028-05-16 17:22:52.332869298 +0000 UTC"
Sep 17 19:06:24.251637 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:06:24.251637    2582 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="14566h16m28.081234982s"
Sep 17 19:06:24.251910 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:06:24.251875    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Sep 17 19:06:24.252115 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:06:24.251997    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Sep 17 19:06:24.252377 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:06:24.252352    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-hj5r7"
Sep 17 19:06:24.252460 ip-10-0-131-232 kubenswrapper[2582]: E0917 19:06:24.252433    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-hj5r7" podUID="cc1b159b-69e7-404d-9a25-bcf0ff8148c5"
Sep 17 19:06:24.252530 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:06:24.252468    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-wzg6z\""
Sep 17 19:06:24.253497 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:06:24.253479    2582 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Sep 17 19:06:24.254250 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:06:24.254221    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-kzblg"
Sep 17 19:06:24.256514 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:06:24.256494    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Sep 17 19:06:24.256771 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:06:24.256750    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-lmqvs\""
Sep 17 19:06:24.256883 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:06:24.256866    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Sep 17 19:06:24.256988 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:06:24.256929    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Sep 17 19:06:24.257082 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:06:24.257068    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-kdm8s"
Sep 17 19:06:24.258735 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:06:24.258713    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qxfrl"
Sep 17 19:06:24.259661 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:06:24.259640    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Sep 17 19:06:24.259740 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:06:24.259662    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Sep 17 19:06:24.259740 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:06:24.259721    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Sep 17 19:06:24.260314 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:06:24.260285    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Sep 17 19:06:24.260410 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:06:24.260386    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Sep 17 19:06:24.260470 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:06:24.260428    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Sep 17 19:06:24.260470 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:06:24.260441    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-76vv6\""
Sep 17 19:06:24.261014 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:06:24.260994    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Sep 17 19:06:24.261293 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:06:24.261278    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Sep 17 19:06:24.261392 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:06:24.261363    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Sep 17 19:06:24.261392 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:06:24.261381    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-jhjrs"
Sep 17 19:06:24.261495 ip-10-0-131-232 kubenswrapper[2582]: E0917 19:06:24.261440    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-jhjrs" podUID="07daf4ff-9d55-4476-aefd-df39d86849a6"
Sep 17 19:06:24.261550 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:06:24.261495    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-csmsx\""
Sep 17 19:06:24.263742 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:06:24.263725    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-64g8v"
Sep 17 19:06:24.267194 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:06:24.266198    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Sep 17 19:06:24.267194 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:06:24.266931    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Sep 17 19:06:24.267194 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:06:24.267030    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Sep 17 19:06:24.267194 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:06:24.267051    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-mx6q8\""
Sep 17 19:06:24.282086 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:06:24.282066    2582 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-ttrgj"
Sep 17 19:06:24.291548 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:06:24.291531    2582 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-ttrgj"
Sep 17 19:06:24.330774 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:06:24.330756    2582 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 17 19:06:24.343634 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:06:24.343621    2582 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Sep 17 19:06:24.350228 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:06:24.350211    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/7cd887c0-2cc8-4363-b553-8d2b4e10f158-dbus\") pod \"global-pull-secret-syncer-f6kjs\" (UID: \"7cd887c0-2cc8-4363-b553-8d2b4e10f158\") " pod="kube-system/global-pull-secret-syncer-f6kjs"
Sep 17 19:06:24.350283 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:06:24.350238    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qdvst\" (UniqueName: \"kubernetes.io/projected/03964cb9-d415-48a8-9a6a-834979450459-kube-api-access-qdvst\") pod \"multus-additional-cni-plugins-wz7lk\" (UID: \"03964cb9-d415-48a8-9a6a-834979450459\") " pod="openshift-multus/multus-additional-cni-plugins-wz7lk"
Sep 17 19:06:24.350283 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:06:24.350257    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/c1b79c18-774f-4b78-a6a9-7ea8a229aaae-host-cni-netd\") pod \"ovnkube-node-kdm8s\" (UID: \"c1b79c18-774f-4b78-a6a9-7ea8a229aaae\") " pod="openshift-ovn-kubernetes/ovnkube-node-kdm8s"
Sep 17 19:06:24.350283 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:06:24.350272    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/aa2ade5e-8564-46a7-819c-75c57cb0961d-cnibin\") pod \"multus-fppr7\" (UID: \"aa2ade5e-8564-46a7-819c-75c57cb0961d\") " pod="openshift-multus/multus-fppr7"
Sep 17 19:06:24.350408 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:06:24.350291    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/f661fc84-37f7-4cb9-a5b5-be6673fbd0dd-etc-modprobe-d\") pod \"tuned-g8qm4\" (UID: \"f661fc84-37f7-4cb9-a5b5-be6673fbd0dd\") " pod="openshift-cluster-node-tuning-operator/tuned-g8qm4"
Sep 17 19:06:24.350408 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:06:24.350323    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xdgj6\" (UniqueName: \"kubernetes.io/projected/f661fc84-37f7-4cb9-a5b5-be6673fbd0dd-kube-api-access-xdgj6\") pod \"tuned-g8qm4\" (UID: \"f661fc84-37f7-4cb9-a5b5-be6673fbd0dd\") " pod="openshift-cluster-node-tuning-operator/tuned-g8qm4"
Sep 17 19:06:24.350408 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:06:24.350345    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/7cd887c0-2cc8-4363-b553-8d2b4e10f158-original-pull-secret\") pod \"global-pull-secret-syncer-f6kjs\" (UID: \"7cd887c0-2cc8-4363-b553-8d2b4e10f158\") " pod="kube-system/global-pull-secret-syncer-f6kjs"
Sep 17 19:06:24.350408 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:06:24.350365    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/03964cb9-d415-48a8-9a6a-834979450459-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-wz7lk\" (UID: \"03964cb9-d415-48a8-9a6a-834979450459\") " pod="openshift-multus/multus-additional-cni-plugins-wz7lk"
Sep 17 19:06:24.350408 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:06:24.350388    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/03964cb9-d415-48a8-9a6a-834979450459-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-wz7lk\" (UID: \"03964cb9-d415-48a8-9a6a-834979450459\") " pod="openshift-multus/multus-additional-cni-plugins-wz7lk"
Sep 17 19:06:24.350408 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:06:24.350385    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/aa2ade5e-8564-46a7-819c-75c57cb0961d-cnibin\") pod \"multus-fppr7\" (UID: \"aa2ade5e-8564-46a7-819c-75c57cb0961d\") " pod="openshift-multus/multus-fppr7"
Sep 17 19:06:24.350654 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:06:24.350411    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-df487\" (UniqueName: \"kubernetes.io/projected/e884baf0-e11b-4964-9913-07095d7d1f21-kube-api-access-df487\") pod \"aws-ebs-csi-driver-node-qxfrl\" (UID: \"e884baf0-e11b-4964-9913-07095d7d1f21\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qxfrl"
Sep 17 19:06:24.350654 ip-10-0-131-232 kubenswrapper[2582]: E0917 19:06:24.350448    2582 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 17 19:06:24.350654 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:06:24.350452    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/c1b79c18-774f-4b78-a6a9-7ea8a229aaae-etc-openvswitch\") pod \"ovnkube-node-kdm8s\" (UID: \"c1b79c18-774f-4b78-a6a9-7ea8a229aaae\") " pod="openshift-ovn-kubernetes/ovnkube-node-kdm8s"
Sep 17 19:06:24.350654 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:06:24.350481    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/c1b79c18-774f-4b78-a6a9-7ea8a229aaae-host-run-ovn-kubernetes\") pod \"ovnkube-node-kdm8s\" (UID: \"c1b79c18-774f-4b78-a6a9-7ea8a229aaae\") " pod="openshift-ovn-kubernetes/ovnkube-node-kdm8s"
Sep 17 19:06:24.350654 ip-10-0-131-232 kubenswrapper[2582]: E0917 19:06:24.350507    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7cd887c0-2cc8-4363-b553-8d2b4e10f158-original-pull-secret podName:7cd887c0-2cc8-4363-b553-8d2b4e10f158 nodeName:}" failed. No retries permitted until 2026-09-17 19:06:24.850492676 +0000 UTC m=+2.010090325 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/7cd887c0-2cc8-4363-b553-8d2b4e10f158-original-pull-secret") pod "global-pull-secret-syncer-f6kjs" (UID: "7cd887c0-2cc8-4363-b553-8d2b4e10f158") : object "kube-system"/"original-pull-secret" not registered
Sep 17 19:06:24.350654 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:06:24.350531    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/c1b79c18-774f-4b78-a6a9-7ea8a229aaae-host-cni-bin\") pod \"ovnkube-node-kdm8s\" (UID: \"c1b79c18-774f-4b78-a6a9-7ea8a229aaae\") " pod="openshift-ovn-kubernetes/ovnkube-node-kdm8s"
Sep 17 19:06:24.350654 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:06:24.350551    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/7cd887c0-2cc8-4363-b553-8d2b4e10f158-dbus\") pod \"global-pull-secret-syncer-f6kjs\" (UID: \"7cd887c0-2cc8-4363-b553-8d2b4e10f158\") " pod="kube-system/global-pull-secret-syncer-f6kjs"
Sep 17 19:06:24.350654 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:06:24.350558    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kftwn\" (UniqueName: \"kubernetes.io/projected/07daf4ff-9d55-4476-aefd-df39d86849a6-kube-api-access-kftwn\") pod \"network-check-target-jhjrs\" (UID: \"07daf4ff-9d55-4476-aefd-df39d86849a6\") " pod="openshift-network-diagnostics/network-check-target-jhjrs"
Sep 17 19:06:24.350654 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:06:24.350577    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/f661fc84-37f7-4cb9-a5b5-be6673fbd0dd-etc-modprobe-d\") pod \"tuned-g8qm4\" (UID: \"f661fc84-37f7-4cb9-a5b5-be6673fbd0dd\") " pod="openshift-cluster-node-tuning-operator/tuned-g8qm4"
Sep 17 19:06:24.350654 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:06:24.350585    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/aa2ade5e-8564-46a7-819c-75c57cb0961d-host-var-lib-cni-multus\") pod \"multus-fppr7\" (UID: \"aa2ade5e-8564-46a7-819c-75c57cb0961d\") " pod="openshift-multus/multus-fppr7"
Sep 17 19:06:24.350654 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:06:24.350601    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/aa2ade5e-8564-46a7-819c-75c57cb0961d-host-var-lib-kubelet\") pod \"multus-fppr7\" (UID: \"aa2ade5e-8564-46a7-819c-75c57cb0961d\") " pod="openshift-multus/multus-fppr7"
Sep 17 19:06:24.350654 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:06:24.350618    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/f661fc84-37f7-4cb9-a5b5-be6673fbd0dd-sys\") pod \"tuned-g8qm4\" (UID: \"f661fc84-37f7-4cb9-a5b5-be6673fbd0dd\") " pod="openshift-cluster-node-tuning-operator/tuned-g8qm4"
Sep 17 19:06:24.350654 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:06:24.350639    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/f902d60d-52fe-4458-94ce-21fc761c4bb5-host-slash\") pod \"iptables-alerter-kzblg\" (UID: \"f902d60d-52fe-4458-94ce-21fc761c4bb5\") " pod="openshift-network-operator/iptables-alerter-kzblg"
Sep 17 19:06:24.350654 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:06:24.350655    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/c1b79c18-774f-4b78-a6a9-7ea8a229aaae-systemd-units\") pod \"ovnkube-node-kdm8s\" (UID: \"c1b79c18-774f-4b78-a6a9-7ea8a229aaae\") " pod="openshift-ovn-kubernetes/ovnkube-node-kdm8s"
Sep 17 19:06:24.350654 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:06:24.350658    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/aa2ade5e-8564-46a7-819c-75c57cb0961d-host-var-lib-kubelet\") pod \"multus-fppr7\" (UID: \"aa2ade5e-8564-46a7-819c-75c57cb0961d\") " pod="openshift-multus/multus-fppr7"
Sep 17 19:06:24.351348 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:06:24.350669    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/c1b79c18-774f-4b78-a6a9-7ea8a229aaae-log-socket\") pod \"ovnkube-node-kdm8s\" (UID: \"c1b79c18-774f-4b78-a6a9-7ea8a229aaae\") " pod="openshift-ovn-kubernetes/ovnkube-node-kdm8s"
Sep 17 19:06:24.351348 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:06:24.350678    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/f661fc84-37f7-4cb9-a5b5-be6673fbd0dd-sys\") pod \"tuned-g8qm4\" (UID: \"f661fc84-37f7-4cb9-a5b5-be6673fbd0dd\") " pod="openshift-cluster-node-tuning-operator/tuned-g8qm4"
Sep 17 19:06:24.351348 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:06:24.350690    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/c1b79c18-774f-4b78-a6a9-7ea8a229aaae-var-lib-openvswitch\") pod \"ovnkube-node-kdm8s\" (UID: \"c1b79c18-774f-4b78-a6a9-7ea8a229aaae\") " pod="openshift-ovn-kubernetes/ovnkube-node-kdm8s"
Sep 17 19:06:24.351348 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:06:24.350710    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/aa2ade5e-8564-46a7-819c-75c57cb0961d-host-var-lib-cni-multus\") pod \"multus-fppr7\" (UID: \"aa2ade5e-8564-46a7-819c-75c57cb0961d\") " pod="openshift-multus/multus-fppr7"
Sep 17 19:06:24.351348 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:06:24.350748    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/c1b79c18-774f-4b78-a6a9-7ea8a229aaae-run-openvswitch\") pod \"ovnkube-node-kdm8s\" (UID: \"c1b79c18-774f-4b78-a6a9-7ea8a229aaae\") " pod="openshift-ovn-kubernetes/ovnkube-node-kdm8s"
Sep 17 19:06:24.351348 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:06:24.350776    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/f661fc84-37f7-4cb9-a5b5-be6673fbd0dd-etc-sysctl-conf\") pod \"tuned-g8qm4\" (UID: \"f661fc84-37f7-4cb9-a5b5-be6673fbd0dd\") " pod="openshift-cluster-node-tuning-operator/tuned-g8qm4"
Sep 17 19:06:24.351348 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:06:24.350821    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/03964cb9-d415-48a8-9a6a-834979450459-tuning-conf-dir\") pod \"multus-additional-cni-plugins-wz7lk\" (UID: \"03964cb9-d415-48a8-9a6a-834979450459\") " pod="openshift-multus/multus-additional-cni-plugins-wz7lk"
Sep 17 19:06:24.351348 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:06:24.350845    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/c1b79c18-774f-4b78-a6a9-7ea8a229aaae-host-slash\") pod \"ovnkube-node-kdm8s\" (UID: \"c1b79c18-774f-4b78-a6a9-7ea8a229aaae\") " pod="openshift-ovn-kubernetes/ovnkube-node-kdm8s"
Sep 17 19:06:24.351348 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:06:24.350870    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/c1b79c18-774f-4b78-a6a9-7ea8a229aaae-ovn-node-metrics-cert\") pod \"ovnkube-node-kdm8s\" (UID: \"c1b79c18-774f-4b78-a6a9-7ea8a229aaae\") " pod="openshift-ovn-kubernetes/ovnkube-node-kdm8s"
Sep 17 19:06:24.351348 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:06:24.350885    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/f661fc84-37f7-4cb9-a5b5-be6673fbd0dd-etc-sysctl-conf\") pod \"tuned-g8qm4\" (UID: \"f661fc84-37f7-4cb9-a5b5-be6673fbd0dd\") " pod="openshift-cluster-node-tuning-operator/tuned-g8qm4"
Sep 17 19:06:24.351348 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:06:24.350911    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/aa2ade5e-8564-46a7-819c-75c57cb0961d-system-cni-dir\") pod \"multus-fppr7\" (UID: \"aa2ade5e-8564-46a7-819c-75c57cb0961d\") " pod="openshift-multus/multus-fppr7"
Sep 17 19:06:24.351348 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:06:24.350939    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/aa2ade5e-8564-46a7-819c-75c57cb0961d-multus-conf-dir\") pod \"multus-fppr7\" (UID: \"aa2ade5e-8564-46a7-819c-75c57cb0961d\") " pod="openshift-multus/multus-fppr7"
Sep 17 19:06:24.351348 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:06:24.350957    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/f661fc84-37f7-4cb9-a5b5-be6673fbd0dd-host\") pod \"tuned-g8qm4\" (UID: \"f661fc84-37f7-4cb9-a5b5-be6673fbd0dd\") " pod="openshift-cluster-node-tuning-operator/tuned-g8qm4"
Sep 17 19:06:24.351348 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:06:24.350988    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/f661fc84-37f7-4cb9-a5b5-be6673fbd0dd-host\") pod \"tuned-g8qm4\" (UID: \"f661fc84-37f7-4cb9-a5b5-be6673fbd0dd\") " pod="openshift-cluster-node-tuning-operator/tuned-g8qm4"
Sep 17 19:06:24.351348 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:06:24.350993    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/aa2ade5e-8564-46a7-819c-75c57cb0961d-system-cni-dir\") pod \"multus-fppr7\" (UID: \"aa2ade5e-8564-46a7-819c-75c57cb0961d\") " pod="openshift-multus/multus-fppr7"
Sep 17 19:06:24.351348 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:06:24.350993    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/aa2ade5e-8564-46a7-819c-75c57cb0961d-multus-conf-dir\") pod \"multus-fppr7\" (UID: \"aa2ade5e-8564-46a7-819c-75c57cb0961d\") " pod="openshift-multus/multus-fppr7"
Sep 17 19:06:24.351348 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:06:24.351012    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/46629fc1-07ce-45b8-9d73-453abfa383b9-hosts-file\") pod \"node-resolver-cwtkf\" (UID: \"46629fc1-07ce-45b8-9d73-453abfa383b9\") " pod="openshift-dns/node-resolver-cwtkf"
Sep 17 19:06:24.352077 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:06:24.351041    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/e884baf0-e11b-4964-9913-07095d7d1f21-socket-dir\") pod \"aws-ebs-csi-driver-node-qxfrl\" (UID: \"e884baf0-e11b-4964-9913-07095d7d1f21\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qxfrl"
Sep 17 19:06:24.352077 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:06:24.351049    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/46629fc1-07ce-45b8-9d73-453abfa383b9-hosts-file\") pod \"node-resolver-cwtkf\" (UID: \"46629fc1-07ce-45b8-9d73-453abfa383b9\") " pod="openshift-dns/node-resolver-cwtkf"
Sep 17 19:06:24.352077 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:06:24.351064    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/e884baf0-e11b-4964-9913-07095d7d1f21-device-dir\") pod \"aws-ebs-csi-driver-node-qxfrl\" (UID: \"e884baf0-e11b-4964-9913-07095d7d1f21\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qxfrl"
Sep 17 19:06:24.352077 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:06:24.351091    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/aa2ade5e-8564-46a7-819c-75c57cb0961d-multus-daemon-config\") pod \"multus-fppr7\" (UID: \"aa2ade5e-8564-46a7-819c-75c57cb0961d\") " pod="openshift-multus/multus-fppr7"
Sep 17 19:06:24.352077 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:06:24.351114    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/f661fc84-37f7-4cb9-a5b5-be6673fbd0dd-var-lib-kubelet\") pod \"tuned-g8qm4\" (UID: \"f661fc84-37f7-4cb9-a5b5-be6673fbd0dd\") " pod="openshift-cluster-node-tuning-operator/tuned-g8qm4"
Sep 17 19:06:24.352077 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:06:24.351138    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/f661fc84-37f7-4cb9-a5b5-be6673fbd0dd-tmp\") pod \"tuned-g8qm4\" (UID: \"f661fc84-37f7-4cb9-a5b5-be6673fbd0dd\") " pod="openshift-cluster-node-tuning-operator/tuned-g8qm4"
Sep 17 19:06:24.352077 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:06:24.351163    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/e884baf0-e11b-4964-9913-07095d7d1f21-etc-selinux\") pod \"aws-ebs-csi-driver-node-qxfrl\" (UID: \"e884baf0-e11b-4964-9913-07095d7d1f21\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qxfrl"
Sep 17 19:06:24.352077 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:06:24.351180    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/f661fc84-37f7-4cb9-a5b5-be6673fbd0dd-var-lib-kubelet\") pod \"tuned-g8qm4\" (UID: \"f661fc84-37f7-4cb9-a5b5-be6673fbd0dd\") " pod="openshift-cluster-node-tuning-operator/tuned-g8qm4"
Sep 17 19:06:24.352077 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:06:24.351188    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/271e6574-9235-474f-b0bc-06ec57117390-serviceca\") pod \"node-ca-64g8v\" (UID: \"271e6574-9235-474f-b0bc-06ec57117390\") " pod="openshift-image-registry/node-ca-64g8v"
Sep 17 19:06:24.352077 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:06:24.351213    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/7cd887c0-2cc8-4363-b553-8d2b4e10f158-kubelet-config\") pod \"global-pull-secret-syncer-f6kjs\" (UID: \"7cd887c0-2cc8-4363-b553-8d2b4e10f158\") " pod="kube-system/global-pull-secret-syncer-f6kjs"
Sep 17 19:06:24.352077 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:06:24.351237    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/fe7c72bc-812d-4272-8352-78f8fa2c19d1-agent-certs\") pod \"konnectivity-agent-qwhmv\" (UID: \"fe7c72bc-812d-4272-8352-78f8fa2c19d1\") " pod="kube-system/konnectivity-agent-qwhmv"
Sep 17 19:06:24.352077 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:06:24.351261    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/03964cb9-d415-48a8-9a6a-834979450459-system-cni-dir\") pod \"multus-additional-cni-plugins-wz7lk\" (UID: \"03964cb9-d415-48a8-9a6a-834979450459\") " pod="openshift-multus/multus-additional-cni-plugins-wz7lk"
Sep 17 19:06:24.352077 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:06:24.351269    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/7cd887c0-2cc8-4363-b553-8d2b4e10f158-kubelet-config\") pod \"global-pull-secret-syncer-f6kjs\" (UID: \"7cd887c0-2cc8-4363-b553-8d2b4e10f158\") " pod="kube-system/global-pull-secret-syncer-f6kjs"
Sep 17 19:06:24.352077 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:06:24.351286    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/03964cb9-d415-48a8-9a6a-834979450459-cnibin\") pod \"multus-additional-cni-plugins-wz7lk\" (UID: \"03964cb9-d415-48a8-9a6a-834979450459\") " pod="openshift-multus/multus-additional-cni-plugins-wz7lk"
Sep 17 19:06:24.352077 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:06:24.351410    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/aa2ade5e-8564-46a7-819c-75c57cb0961d-host-run-k8s-cni-cncf-io\") pod \"multus-fppr7\" (UID: \"aa2ade5e-8564-46a7-819c-75c57cb0961d\") " pod="openshift-multus/multus-fppr7"
Sep 17 19:06:24.352077 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:06:24.351439    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/c1b79c18-774f-4b78-a6a9-7ea8a229aaae-node-log\") pod \"ovnkube-node-kdm8s\" (UID: \"c1b79c18-774f-4b78-a6a9-7ea8a229aaae\") " pod="openshift-ovn-kubernetes/ovnkube-node-kdm8s"
Sep 17 19:06:24.352077 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:06:24.351469    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/c1b79c18-774f-4b78-a6a9-7ea8a229aaae-env-overrides\") pod \"ovnkube-node-kdm8s\" (UID: \"c1b79c18-774f-4b78-a6a9-7ea8a229aaae\") " pod="openshift-ovn-kubernetes/ovnkube-node-kdm8s"
Sep 17 19:06:24.352785 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:06:24.351484    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/aa2ade5e-8564-46a7-819c-75c57cb0961d-host-run-k8s-cni-cncf-io\") pod \"multus-fppr7\" (UID: \"aa2ade5e-8564-46a7-819c-75c57cb0961d\") " pod="openshift-multus/multus-fppr7"
Sep 17 19:06:24.352785 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:06:24.351502    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/aa2ade5e-8564-46a7-819c-75c57cb0961d-hostroot\") pod \"multus-fppr7\" (UID: \"aa2ade5e-8564-46a7-819c-75c57cb0961d\") " pod="openshift-multus/multus-fppr7"
Sep 17 19:06:24.352785 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:06:24.351556    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/f661fc84-37f7-4cb9-a5b5-be6673fbd0dd-run\") pod \"tuned-g8qm4\" (UID: \"f661fc84-37f7-4cb9-a5b5-be6673fbd0dd\") " pod="openshift-cluster-node-tuning-operator/tuned-g8qm4"
Sep 17 19:06:24.352785 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:06:24.351599    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/fe7c72bc-812d-4272-8352-78f8fa2c19d1-konnectivity-ca\") pod \"konnectivity-agent-qwhmv\" (UID: \"fe7c72bc-812d-4272-8352-78f8fa2c19d1\") " pod="kube-system/konnectivity-agent-qwhmv"
Sep 17 19:06:24.352785 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:06:24.351604    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/aa2ade5e-8564-46a7-819c-75c57cb0961d-hostroot\") pod \"multus-fppr7\" (UID: \"aa2ade5e-8564-46a7-819c-75c57cb0961d\") " pod="openshift-multus/multus-fppr7"
Sep 17 19:06:24.352785 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:06:24.351619    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/aa2ade5e-8564-46a7-819c-75c57cb0961d-multus-daemon-config\") pod \"multus-fppr7\" (UID: \"aa2ade5e-8564-46a7-819c-75c57cb0961d\") " pod="openshift-multus/multus-fppr7"
Sep 17 19:06:24.352785 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:06:24.351626    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/c1b79c18-774f-4b78-a6a9-7ea8a229aaae-ovnkube-config\") pod \"ovnkube-node-kdm8s\" (UID: \"c1b79c18-774f-4b78-a6a9-7ea8a229aaae\") " pod="openshift-ovn-kubernetes/ovnkube-node-kdm8s"
Sep 17 19:06:24.352785 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:06:24.351652    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/c1b79c18-774f-4b78-a6a9-7ea8a229aaae-ovnkube-script-lib\") pod \"ovnkube-node-kdm8s\" (UID: \"c1b79c18-774f-4b78-a6a9-7ea8a229aaae\") " pod="openshift-ovn-kubernetes/ovnkube-node-kdm8s"
Sep 17 19:06:24.352785 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:06:24.351675    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8hv9p\" (UniqueName: \"kubernetes.io/projected/271e6574-9235-474f-b0bc-06ec57117390-kube-api-access-8hv9p\") pod \"node-ca-64g8v\" (UID: \"271e6574-9235-474f-b0bc-06ec57117390\") " pod="openshift-image-registry/node-ca-64g8v"
Sep 17 19:06:24.352785 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:06:24.351678    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/f661fc84-37f7-4cb9-a5b5-be6673fbd0dd-run\") pod \"tuned-g8qm4\" (UID: \"f661fc84-37f7-4cb9-a5b5-be6673fbd0dd\") " pod="openshift-cluster-node-tuning-operator/tuned-g8qm4"
Sep 17 19:06:24.352785 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:06:24.351698    2582 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Sep 17 19:06:24.352785 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:06:24.351713    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/aa2ade5e-8564-46a7-819c-75c57cb0961d-cni-binary-copy\") pod \"multus-fppr7\" (UID: \"aa2ade5e-8564-46a7-819c-75c57cb0961d\") " pod="openshift-multus/multus-fppr7"
Sep 17 19:06:24.352785 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:06:24.351792    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/aa2ade5e-8564-46a7-819c-75c57cb0961d-host-var-lib-cni-bin\") pod \"multus-fppr7\" (UID: \"aa2ade5e-8564-46a7-819c-75c57cb0961d\") " pod="openshift-multus/multus-fppr7"
Sep 17 19:06:24.352785 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:06:24.351816    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/f661fc84-37f7-4cb9-a5b5-be6673fbd0dd-etc-tuned\") pod \"tuned-g8qm4\" (UID: \"f661fc84-37f7-4cb9-a5b5-be6673fbd0dd\") " pod="openshift-cluster-node-tuning-operator/tuned-g8qm4"
Sep 17 19:06:24.352785 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:06:24.351830    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5h6q4\" (UniqueName: \"kubernetes.io/projected/46629fc1-07ce-45b8-9d73-453abfa383b9-kube-api-access-5h6q4\") pod \"node-resolver-cwtkf\" (UID: \"46629fc1-07ce-45b8-9d73-453abfa383b9\") " pod="openshift-dns/node-resolver-cwtkf"
Sep 17 19:06:24.352785 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:06:24.351847    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/03964cb9-d415-48a8-9a6a-834979450459-cni-binary-copy\") pod \"multus-additional-cni-plugins-wz7lk\" (UID: \"03964cb9-d415-48a8-9a6a-834979450459\") " pod="openshift-multus/multus-additional-cni-plugins-wz7lk"
Sep 17 19:06:24.352785 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:06:24.351871    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-psk99\" (UniqueName: \"kubernetes.io/projected/cc1b159b-69e7-404d-9a25-bcf0ff8148c5-kube-api-access-psk99\") pod \"network-metrics-daemon-hj5r7\" (UID: \"cc1b159b-69e7-404d-9a25-bcf0ff8148c5\") " pod="openshift-multus/network-metrics-daemon-hj5r7"
Sep 17 19:06:24.352785 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:06:24.351889    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/aa2ade5e-8564-46a7-819c-75c57cb0961d-host-var-lib-cni-bin\") pod \"multus-fppr7\" (UID: \"aa2ade5e-8564-46a7-819c-75c57cb0961d\") " pod="openshift-multus/multus-fppr7"
Sep 17 19:06:24.353371 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:06:24.351911    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/f902d60d-52fe-4458-94ce-21fc761c4bb5-iptables-alerter-script\") pod \"iptables-alerter-kzblg\" (UID: \"f902d60d-52fe-4458-94ce-21fc761c4bb5\") " pod="openshift-network-operator/iptables-alerter-kzblg"
Sep 17 19:06:24.353371 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:06:24.351935    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/c1b79c18-774f-4b78-a6a9-7ea8a229aaae-host-run-netns\") pod \"ovnkube-node-kdm8s\" (UID: \"c1b79c18-774f-4b78-a6a9-7ea8a229aaae\") " pod="openshift-ovn-kubernetes/ovnkube-node-kdm8s"
Sep 17 19:06:24.353371 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:06:24.351958    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/aa2ade5e-8564-46a7-819c-75c57cb0961d-multus-cni-dir\") pod \"multus-fppr7\" (UID: \"aa2ade5e-8564-46a7-819c-75c57cb0961d\") " pod="openshift-multus/multus-fppr7"
Sep 17 19:06:24.353371 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:06:24.351981    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/46629fc1-07ce-45b8-9d73-453abfa383b9-tmp-dir\") pod \"node-resolver-cwtkf\" (UID: \"46629fc1-07ce-45b8-9d73-453abfa383b9\") " pod="openshift-dns/node-resolver-cwtkf"
Sep 17 19:06:24.353371 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:06:24.352031    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/c1b79c18-774f-4b78-a6a9-7ea8a229aaae-host-kubelet\") pod \"ovnkube-node-kdm8s\" (UID: \"c1b79c18-774f-4b78-a6a9-7ea8a229aaae\") " pod="openshift-ovn-kubernetes/ovnkube-node-kdm8s"
Sep 17 19:06:24.353371 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:06:24.352057    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/c1b79c18-774f-4b78-a6a9-7ea8a229aaae-run-systemd\") pod \"ovnkube-node-kdm8s\" (UID: \"c1b79c18-774f-4b78-a6a9-7ea8a229aaae\") " pod="openshift-ovn-kubernetes/ovnkube-node-kdm8s"
Sep 17 19:06:24.353371 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:06:24.352132    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/c1b79c18-774f-4b78-a6a9-7ea8a229aaae-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-kdm8s\" (UID: \"c1b79c18-774f-4b78-a6a9-7ea8a229aaae\") " pod="openshift-ovn-kubernetes/ovnkube-node-kdm8s"
Sep 17 19:06:24.353371 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:06:24.352154    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/fe7c72bc-812d-4272-8352-78f8fa2c19d1-konnectivity-ca\") pod \"konnectivity-agent-qwhmv\" (UID: \"fe7c72bc-812d-4272-8352-78f8fa2c19d1\") " pod="kube-system/konnectivity-agent-qwhmv"
Sep 17 19:06:24.353371 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:06:24.352171    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/aa2ade5e-8564-46a7-819c-75c57cb0961d-cni-binary-copy\") pod \"multus-fppr7\" (UID: \"aa2ade5e-8564-46a7-819c-75c57cb0961d\") " pod="openshift-multus/multus-fppr7"
Sep 17 19:06:24.353371 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:06:24.352195    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/271e6574-9235-474f-b0bc-06ec57117390-host\") pod \"node-ca-64g8v\" (UID: \"271e6574-9235-474f-b0bc-06ec57117390\") " pod="openshift-image-registry/node-ca-64g8v"
Sep 17 19:06:24.353371 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:06:24.352219    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/aa2ade5e-8564-46a7-819c-75c57cb0961d-etc-kubernetes\") pod \"multus-fppr7\" (UID: \"aa2ade5e-8564-46a7-819c-75c57cb0961d\") " pod="openshift-multus/multus-fppr7"
Sep 17 19:06:24.353371 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:06:24.352235    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/aa2ade5e-8564-46a7-819c-75c57cb0961d-multus-cni-dir\") pod \"multus-fppr7\" (UID: \"aa2ade5e-8564-46a7-819c-75c57cb0961d\") " pod="openshift-multus/multus-fppr7"
Sep 17 19:06:24.353371 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:06:24.352251    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f661fc84-37f7-4cb9-a5b5-be6673fbd0dd-etc-kubernetes\") pod \"tuned-g8qm4\" (UID: \"f661fc84-37f7-4cb9-a5b5-be6673fbd0dd\") " pod="openshift-cluster-node-tuning-operator/tuned-g8qm4"
Sep 17 19:06:24.353371 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:06:24.352278    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/f661fc84-37f7-4cb9-a5b5-be6673fbd0dd-lib-modules\") pod \"tuned-g8qm4\" (UID: \"f661fc84-37f7-4cb9-a5b5-be6673fbd0dd\") " pod="openshift-cluster-node-tuning-operator/tuned-g8qm4"
Sep 17 19:06:24.353371 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:06:24.352285    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/aa2ade5e-8564-46a7-819c-75c57cb0961d-etc-kubernetes\") pod \"multus-fppr7\" (UID: \"aa2ade5e-8564-46a7-819c-75c57cb0961d\") " pod="openshift-multus/multus-fppr7"
Sep 17 19:06:24.353371 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:06:24.352294    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f661fc84-37f7-4cb9-a5b5-be6673fbd0dd-etc-kubernetes\") pod \"tuned-g8qm4\" (UID: \"f661fc84-37f7-4cb9-a5b5-be6673fbd0dd\") " pod="openshift-cluster-node-tuning-operator/tuned-g8qm4"
Sep 17 19:06:24.353371 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:06:24.352272    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/46629fc1-07ce-45b8-9d73-453abfa383b9-tmp-dir\") pod \"node-resolver-cwtkf\" (UID: \"46629fc1-07ce-45b8-9d73-453abfa383b9\") " pod="openshift-dns/node-resolver-cwtkf"
Sep 17 19:06:24.353829 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:06:24.352298    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/e884baf0-e11b-4964-9913-07095d7d1f21-registration-dir\") pod \"aws-ebs-csi-driver-node-qxfrl\" (UID: \"e884baf0-e11b-4964-9913-07095d7d1f21\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qxfrl"
Sep 17 19:06:24.353829 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:06:24.352373    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gv65p\" (UniqueName: \"kubernetes.io/projected/f902d60d-52fe-4458-94ce-21fc761c4bb5-kube-api-access-gv65p\") pod \"iptables-alerter-kzblg\" (UID: \"f902d60d-52fe-4458-94ce-21fc761c4bb5\") " pod="openshift-network-operator/iptables-alerter-kzblg"
Sep 17 19:06:24.353829 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:06:24.352398    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/c1b79c18-774f-4b78-a6a9-7ea8a229aaae-run-ovn\") pod \"ovnkube-node-kdm8s\" (UID: \"c1b79c18-774f-4b78-a6a9-7ea8a229aaae\") " pod="openshift-ovn-kubernetes/ovnkube-node-kdm8s"
Sep 17 19:06:24.353829 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:06:24.352419    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/e884baf0-e11b-4964-9913-07095d7d1f21-kubelet-dir\") pod \"aws-ebs-csi-driver-node-qxfrl\" (UID: \"e884baf0-e11b-4964-9913-07095d7d1f21\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qxfrl"
Sep 17 19:06:24.353829 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:06:24.352443    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/e884baf0-e11b-4964-9913-07095d7d1f21-sys-fs\") pod \"aws-ebs-csi-driver-node-qxfrl\" (UID: \"e884baf0-e11b-4964-9913-07095d7d1f21\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qxfrl"
Sep 17 19:06:24.353829 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:06:24.352465    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/f661fc84-37f7-4cb9-a5b5-be6673fbd0dd-lib-modules\") pod \"tuned-g8qm4\" (UID: \"f661fc84-37f7-4cb9-a5b5-be6673fbd0dd\") " pod="openshift-cluster-node-tuning-operator/tuned-g8qm4"
Sep 17 19:06:24.353829 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:06:24.352472    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g2vgd\" (UniqueName: \"kubernetes.io/projected/c1b79c18-774f-4b78-a6a9-7ea8a229aaae-kube-api-access-g2vgd\") pod \"ovnkube-node-kdm8s\" (UID: \"c1b79c18-774f-4b78-a6a9-7ea8a229aaae\") " pod="openshift-ovn-kubernetes/ovnkube-node-kdm8s"
Sep 17 19:06:24.353829 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:06:24.352513    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/aa2ade5e-8564-46a7-819c-75c57cb0961d-host-run-netns\") pod \"multus-fppr7\" (UID: \"aa2ade5e-8564-46a7-819c-75c57cb0961d\") " pod="openshift-multus/multus-fppr7"
Sep 17 19:06:24.353829 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:06:24.352555    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/f661fc84-37f7-4cb9-a5b5-be6673fbd0dd-etc-sysconfig\") pod \"tuned-g8qm4\" (UID: \"f661fc84-37f7-4cb9-a5b5-be6673fbd0dd\") " pod="openshift-cluster-node-tuning-operator/tuned-g8qm4"
Sep 17 19:06:24.353829 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:06:24.352575    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/f661fc84-37f7-4cb9-a5b5-be6673fbd0dd-etc-sysctl-d\") pod \"tuned-g8qm4\" (UID: \"f661fc84-37f7-4cb9-a5b5-be6673fbd0dd\") " pod="openshift-cluster-node-tuning-operator/tuned-g8qm4"
Sep 17 19:06:24.353829 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:06:24.352593    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/f661fc84-37f7-4cb9-a5b5-be6673fbd0dd-etc-systemd\") pod \"tuned-g8qm4\" (UID: \"f661fc84-37f7-4cb9-a5b5-be6673fbd0dd\") " pod="openshift-cluster-node-tuning-operator/tuned-g8qm4"
Sep 17 19:06:24.353829 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:06:24.352595    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/aa2ade5e-8564-46a7-819c-75c57cb0961d-host-run-netns\") pod \"multus-fppr7\" (UID: \"aa2ade5e-8564-46a7-819c-75c57cb0961d\") " pod="openshift-multus/multus-fppr7"
Sep 17 19:06:24.353829 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:06:24.352614    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cc1b159b-69e7-404d-9a25-bcf0ff8148c5-metrics-certs\") pod \"network-metrics-daemon-hj5r7\" (UID: \"cc1b159b-69e7-404d-9a25-bcf0ff8148c5\") " pod="openshift-multus/network-metrics-daemon-hj5r7"
Sep 17 19:06:24.353829 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:06:24.352634    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/aa2ade5e-8564-46a7-819c-75c57cb0961d-multus-socket-dir-parent\") pod \"multus-fppr7\" (UID: \"aa2ade5e-8564-46a7-819c-75c57cb0961d\") " pod="openshift-multus/multus-fppr7"
Sep 17 19:06:24.353829 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:06:24.352655    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/aa2ade5e-8564-46a7-819c-75c57cb0961d-host-run-multus-certs\") pod \"multus-fppr7\" (UID: \"aa2ade5e-8564-46a7-819c-75c57cb0961d\") " pod="openshift-multus/multus-fppr7"
Sep 17 19:06:24.353829 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:06:24.352676    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-22chq\" (UniqueName: \"kubernetes.io/projected/aa2ade5e-8564-46a7-819c-75c57cb0961d-kube-api-access-22chq\") pod \"multus-fppr7\" (UID: \"aa2ade5e-8564-46a7-819c-75c57cb0961d\") " pod="openshift-multus/multus-fppr7"
Sep 17 19:06:24.353829 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:06:24.352695    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/f661fc84-37f7-4cb9-a5b5-be6673fbd0dd-etc-systemd\") pod \"tuned-g8qm4\" (UID: \"f661fc84-37f7-4cb9-a5b5-be6673fbd0dd\") " pod="openshift-cluster-node-tuning-operator/tuned-g8qm4"
Sep 17 19:06:24.354442 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:06:24.352707    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/f661fc84-37f7-4cb9-a5b5-be6673fbd0dd-etc-sysctl-d\") pod \"tuned-g8qm4\" (UID: \"f661fc84-37f7-4cb9-a5b5-be6673fbd0dd\") " pod="openshift-cluster-node-tuning-operator/tuned-g8qm4"
Sep 17 19:06:24.354442 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:06:24.352758    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/f661fc84-37f7-4cb9-a5b5-be6673fbd0dd-etc-sysconfig\") pod \"tuned-g8qm4\" (UID: \"f661fc84-37f7-4cb9-a5b5-be6673fbd0dd\") " pod="openshift-cluster-node-tuning-operator/tuned-g8qm4"
Sep 17 19:06:24.354442 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:06:24.352789    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/aa2ade5e-8564-46a7-819c-75c57cb0961d-host-run-multus-certs\") pod \"multus-fppr7\" (UID: \"aa2ade5e-8564-46a7-819c-75c57cb0961d\") " pod="openshift-multus/multus-fppr7"
Sep 17 19:06:24.354442 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:06:24.352876    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/aa2ade5e-8564-46a7-819c-75c57cb0961d-multus-socket-dir-parent\") pod \"multus-fppr7\" (UID: \"aa2ade5e-8564-46a7-819c-75c57cb0961d\") " pod="openshift-multus/multus-fppr7"
Sep 17 19:06:24.355207 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:06:24.355189    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/f661fc84-37f7-4cb9-a5b5-be6673fbd0dd-etc-tuned\") pod \"tuned-g8qm4\" (UID: \"f661fc84-37f7-4cb9-a5b5-be6673fbd0dd\") " pod="openshift-cluster-node-tuning-operator/tuned-g8qm4"
Sep 17 19:06:24.355497 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:06:24.355478    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/f661fc84-37f7-4cb9-a5b5-be6673fbd0dd-tmp\") pod \"tuned-g8qm4\" (UID: \"f661fc84-37f7-4cb9-a5b5-be6673fbd0dd\") " pod="openshift-cluster-node-tuning-operator/tuned-g8qm4"
Sep 17 19:06:24.355706 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:06:24.355688    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/fe7c72bc-812d-4272-8352-78f8fa2c19d1-agent-certs\") pod \"konnectivity-agent-qwhmv\" (UID: \"fe7c72bc-812d-4272-8352-78f8fa2c19d1\") " pod="kube-system/konnectivity-agent-qwhmv"
Sep 17 19:06:24.359547 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:06:24.359524    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xdgj6\" (UniqueName: \"kubernetes.io/projected/f661fc84-37f7-4cb9-a5b5-be6673fbd0dd-kube-api-access-xdgj6\") pod \"tuned-g8qm4\" (UID: \"f661fc84-37f7-4cb9-a5b5-be6673fbd0dd\") " pod="openshift-cluster-node-tuning-operator/tuned-g8qm4"
Sep 17 19:06:24.360351 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:06:24.360332    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5h6q4\" (UniqueName: \"kubernetes.io/projected/46629fc1-07ce-45b8-9d73-453abfa383b9-kube-api-access-5h6q4\") pod \"node-resolver-cwtkf\" (UID: \"46629fc1-07ce-45b8-9d73-453abfa383b9\") " pod="openshift-dns/node-resolver-cwtkf"
Sep 17 19:06:24.360949 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:06:24.360936    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-22chq\" (UniqueName: \"kubernetes.io/projected/aa2ade5e-8564-46a7-819c-75c57cb0961d-kube-api-access-22chq\") pod \"multus-fppr7\" (UID: \"aa2ade5e-8564-46a7-819c-75c57cb0961d\") " pod="openshift-multus/multus-fppr7"
Sep 17 19:06:24.386098 ip-10-0-131-232 kubenswrapper[2582]: W0917 19:06:24.385874    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5a96bc0ce60b5e0ec092cd8768bc466e.slice/crio-ea8c5c4aa39874f45e5681fd31d2e42bd0b620b5fc5e1913c59ea6350869fe88 WatchSource:0}: Error finding container ea8c5c4aa39874f45e5681fd31d2e42bd0b620b5fc5e1913c59ea6350869fe88: Status 404 returned error can't find the container with id ea8c5c4aa39874f45e5681fd31d2e42bd0b620b5fc5e1913c59ea6350869fe88
Sep 17 19:06:24.387655 ip-10-0-131-232 kubenswrapper[2582]: W0917 19:06:24.387636    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda1f9bd0034d8d01c3b2f70180b25c21f.slice/crio-d8f1c6329ccd2522505f459e6d92c9f9a2ebf6192b0f0f4fd313a9a8c308b9b0 WatchSource:0}: Error finding container d8f1c6329ccd2522505f459e6d92c9f9a2ebf6192b0f0f4fd313a9a8c308b9b0: Status 404 returned error can't find the container with id d8f1c6329ccd2522505f459e6d92c9f9a2ebf6192b0f0f4fd313a9a8c308b9b0
Sep 17 19:06:24.392325 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:06:24.392304    2582 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 17 19:06:24.453255 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:06:24.453230    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/271e6574-9235-474f-b0bc-06ec57117390-host\") pod \"node-ca-64g8v\" (UID: \"271e6574-9235-474f-b0bc-06ec57117390\") " pod="openshift-image-registry/node-ca-64g8v"
Sep 17 19:06:24.453368 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:06:24.453262    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/e884baf0-e11b-4964-9913-07095d7d1f21-registration-dir\") pod \"aws-ebs-csi-driver-node-qxfrl\" (UID: \"e884baf0-e11b-4964-9913-07095d7d1f21\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qxfrl"
Sep 17 19:06:24.453368 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:06:24.453299    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gv65p\" (UniqueName: \"kubernetes.io/projected/f902d60d-52fe-4458-94ce-21fc761c4bb5-kube-api-access-gv65p\") pod \"iptables-alerter-kzblg\" (UID: \"f902d60d-52fe-4458-94ce-21fc761c4bb5\") " pod="openshift-network-operator/iptables-alerter-kzblg"
Sep 17 19:06:24.453368 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:06:24.453327    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/c1b79c18-774f-4b78-a6a9-7ea8a229aaae-run-ovn\") pod \"ovnkube-node-kdm8s\" (UID: \"c1b79c18-774f-4b78-a6a9-7ea8a229aaae\") " pod="openshift-ovn-kubernetes/ovnkube-node-kdm8s"
Sep 17 19:06:24.453368 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:06:24.453351    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/e884baf0-e11b-4964-9913-07095d7d1f21-kubelet-dir\") pod \"aws-ebs-csi-driver-node-qxfrl\" (UID: \"e884baf0-e11b-4964-9913-07095d7d1f21\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qxfrl"
Sep 17 19:06:24.453368 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:06:24.453356    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/271e6574-9235-474f-b0bc-06ec57117390-host\") pod \"node-ca-64g8v\" (UID: \"271e6574-9235-474f-b0bc-06ec57117390\") " pod="openshift-image-registry/node-ca-64g8v"
Sep 17 19:06:24.453618 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:06:24.453328    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/e884baf0-e11b-4964-9913-07095d7d1f21-registration-dir\") pod \"aws-ebs-csi-driver-node-qxfrl\" (UID: \"e884baf0-e11b-4964-9913-07095d7d1f21\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qxfrl"
Sep 17 19:06:24.453618 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:06:24.453373    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/e884baf0-e11b-4964-9913-07095d7d1f21-sys-fs\") pod \"aws-ebs-csi-driver-node-qxfrl\" (UID: \"e884baf0-e11b-4964-9913-07095d7d1f21\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qxfrl"
Sep 17 19:06:24.453618 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:06:24.453405    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/c1b79c18-774f-4b78-a6a9-7ea8a229aaae-run-ovn\") pod \"ovnkube-node-kdm8s\" (UID: \"c1b79c18-774f-4b78-a6a9-7ea8a229aaae\") " pod="openshift-ovn-kubernetes/ovnkube-node-kdm8s"
Sep 17 19:06:24.453618 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:06:24.453413    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/e884baf0-e11b-4964-9913-07095d7d1f21-kubelet-dir\") pod \"aws-ebs-csi-driver-node-qxfrl\" (UID: \"e884baf0-e11b-4964-9913-07095d7d1f21\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qxfrl"
Sep 17 19:06:24.453618 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:06:24.453413    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/e884baf0-e11b-4964-9913-07095d7d1f21-sys-fs\") pod \"aws-ebs-csi-driver-node-qxfrl\" (UID: \"e884baf0-e11b-4964-9913-07095d7d1f21\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qxfrl"
Sep 17 19:06:24.453618 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:06:24.453447    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g2vgd\" (UniqueName: \"kubernetes.io/projected/c1b79c18-774f-4b78-a6a9-7ea8a229aaae-kube-api-access-g2vgd\") pod \"ovnkube-node-kdm8s\" (UID: \"c1b79c18-774f-4b78-a6a9-7ea8a229aaae\") " pod="openshift-ovn-kubernetes/ovnkube-node-kdm8s"
Sep 17 19:06:24.453618 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:06:24.453478    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cc1b159b-69e7-404d-9a25-bcf0ff8148c5-metrics-certs\") pod \"network-metrics-daemon-hj5r7\" (UID: \"cc1b159b-69e7-404d-9a25-bcf0ff8148c5\") " pod="openshift-multus/network-metrics-daemon-hj5r7"
Sep 17 19:06:24.453618 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:06:24.453496    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qdvst\" (UniqueName: \"kubernetes.io/projected/03964cb9-d415-48a8-9a6a-834979450459-kube-api-access-qdvst\") pod \"multus-additional-cni-plugins-wz7lk\" (UID: \"03964cb9-d415-48a8-9a6a-834979450459\") " pod="openshift-multus/multus-additional-cni-plugins-wz7lk"
Sep 17 19:06:24.453618 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:06:24.453521    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/c1b79c18-774f-4b78-a6a9-7ea8a229aaae-host-cni-netd\") pod \"ovnkube-node-kdm8s\" (UID: \"c1b79c18-774f-4b78-a6a9-7ea8a229aaae\") " pod="openshift-ovn-kubernetes/ovnkube-node-kdm8s"
Sep 17 19:06:24.453618 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:06:24.453549    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/03964cb9-d415-48a8-9a6a-834979450459-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-wz7lk\" (UID: \"03964cb9-d415-48a8-9a6a-834979450459\") " pod="openshift-multus/multus-additional-cni-plugins-wz7lk"
Sep 17 19:06:24.453618 ip-10-0-131-232 kubenswrapper[2582]: E0917 19:06:24.453551    2582 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 19:06:24.453618 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:06:24.453581    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/c1b79c18-774f-4b78-a6a9-7ea8a229aaae-host-cni-netd\") pod \"ovnkube-node-kdm8s\" (UID: \"c1b79c18-774f-4b78-a6a9-7ea8a229aaae\") " pod="openshift-ovn-kubernetes/ovnkube-node-kdm8s"
Sep 17 19:06:24.453618 ip-10-0-131-232 kubenswrapper[2582]: E0917 19:06:24.453611    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cc1b159b-69e7-404d-9a25-bcf0ff8148c5-metrics-certs podName:cc1b159b-69e7-404d-9a25-bcf0ff8148c5 nodeName:}" failed. No retries permitted until 2026-09-17 19:06:24.953592608 +0000 UTC m=+2.113190256 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/cc1b159b-69e7-404d-9a25-bcf0ff8148c5-metrics-certs") pod "network-metrics-daemon-hj5r7" (UID: "cc1b159b-69e7-404d-9a25-bcf0ff8148c5") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 19:06:24.454249 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:06:24.453630    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/03964cb9-d415-48a8-9a6a-834979450459-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-wz7lk\" (UID: \"03964cb9-d415-48a8-9a6a-834979450459\") " pod="openshift-multus/multus-additional-cni-plugins-wz7lk"
Sep 17 19:06:24.454249 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:06:24.453656    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-df487\" (UniqueName: \"kubernetes.io/projected/e884baf0-e11b-4964-9913-07095d7d1f21-kube-api-access-df487\") pod \"aws-ebs-csi-driver-node-qxfrl\" (UID: \"e884baf0-e11b-4964-9913-07095d7d1f21\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qxfrl"
Sep 17 19:06:24.454249 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:06:24.453681    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/c1b79c18-774f-4b78-a6a9-7ea8a229aaae-etc-openvswitch\") pod \"ovnkube-node-kdm8s\" (UID: \"c1b79c18-774f-4b78-a6a9-7ea8a229aaae\") " pod="openshift-ovn-kubernetes/ovnkube-node-kdm8s"
Sep 17 19:06:24.454249 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:06:24.453704    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/c1b79c18-774f-4b78-a6a9-7ea8a229aaae-host-run-ovn-kubernetes\") pod \"ovnkube-node-kdm8s\" (UID: \"c1b79c18-774f-4b78-a6a9-7ea8a229aaae\") " pod="openshift-ovn-kubernetes/ovnkube-node-kdm8s"
Sep 17 19:06:24.454249 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:06:24.453724    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/c1b79c18-774f-4b78-a6a9-7ea8a229aaae-host-cni-bin\") pod \"ovnkube-node-kdm8s\" (UID: \"c1b79c18-774f-4b78-a6a9-7ea8a229aaae\") " pod="openshift-ovn-kubernetes/ovnkube-node-kdm8s"
Sep 17 19:06:24.454249 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:06:24.453781    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/c1b79c18-774f-4b78-a6a9-7ea8a229aaae-host-cni-bin\") pod \"ovnkube-node-kdm8s\" (UID: \"c1b79c18-774f-4b78-a6a9-7ea8a229aaae\") " pod="openshift-ovn-kubernetes/ovnkube-node-kdm8s"
Sep 17 19:06:24.454249 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:06:24.453780    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/c1b79c18-774f-4b78-a6a9-7ea8a229aaae-host-run-ovn-kubernetes\") pod \"ovnkube-node-kdm8s\" (UID: \"c1b79c18-774f-4b78-a6a9-7ea8a229aaae\") " pod="openshift-ovn-kubernetes/ovnkube-node-kdm8s"
Sep 17 19:06:24.454249 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:06:24.453777    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/c1b79c18-774f-4b78-a6a9-7ea8a229aaae-etc-openvswitch\") pod \"ovnkube-node-kdm8s\" (UID: \"c1b79c18-774f-4b78-a6a9-7ea8a229aaae\") " pod="openshift-ovn-kubernetes/ovnkube-node-kdm8s"
Sep 17 19:06:24.454249 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:06:24.453805    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kftwn\" (UniqueName: \"kubernetes.io/projected/07daf4ff-9d55-4476-aefd-df39d86849a6-kube-api-access-kftwn\") pod \"network-check-target-jhjrs\" (UID: \"07daf4ff-9d55-4476-aefd-df39d86849a6\") " pod="openshift-network-diagnostics/network-check-target-jhjrs"
Sep 17 19:06:24.454249 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:06:24.453841    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/f902d60d-52fe-4458-94ce-21fc761c4bb5-host-slash\") pod \"iptables-alerter-kzblg\" (UID: \"f902d60d-52fe-4458-94ce-21fc761c4bb5\") " pod="openshift-network-operator/iptables-alerter-kzblg"
Sep 17 19:06:24.454249 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:06:24.453919    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/c1b79c18-774f-4b78-a6a9-7ea8a229aaae-systemd-units\") pod \"ovnkube-node-kdm8s\" (UID: \"c1b79c18-774f-4b78-a6a9-7ea8a229aaae\") " pod="openshift-ovn-kubernetes/ovnkube-node-kdm8s"
Sep 17 19:06:24.454249 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:06:24.453969    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/f902d60d-52fe-4458-94ce-21fc761c4bb5-host-slash\") pod \"iptables-alerter-kzblg\" (UID: \"f902d60d-52fe-4458-94ce-21fc761c4bb5\") " pod="openshift-network-operator/iptables-alerter-kzblg"
Sep 17 19:06:24.454249 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:06:24.453971    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/c1b79c18-774f-4b78-a6a9-7ea8a229aaae-systemd-units\") pod \"ovnkube-node-kdm8s\" (UID: \"c1b79c18-774f-4b78-a6a9-7ea8a229aaae\") " pod="openshift-ovn-kubernetes/ovnkube-node-kdm8s"
Sep 17 19:06:24.454249 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:06:24.454012    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/c1b79c18-774f-4b78-a6a9-7ea8a229aaae-log-socket\") pod \"ovnkube-node-kdm8s\" (UID: \"c1b79c18-774f-4b78-a6a9-7ea8a229aaae\") " pod="openshift-ovn-kubernetes/ovnkube-node-kdm8s"
Sep 17 19:06:24.454249 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:06:24.454038    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/c1b79c18-774f-4b78-a6a9-7ea8a229aaae-var-lib-openvswitch\") pod \"ovnkube-node-kdm8s\" (UID: \"c1b79c18-774f-4b78-a6a9-7ea8a229aaae\") " pod="openshift-ovn-kubernetes/ovnkube-node-kdm8s"
Sep 17 19:06:24.454249 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:06:24.454062    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/c1b79c18-774f-4b78-a6a9-7ea8a229aaae-run-openvswitch\") pod \"ovnkube-node-kdm8s\" (UID: \"c1b79c18-774f-4b78-a6a9-7ea8a229aaae\") " pod="openshift-ovn-kubernetes/ovnkube-node-kdm8s"
Sep 17 19:06:24.454249 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:06:24.454083    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/03964cb9-d415-48a8-9a6a-834979450459-tuning-conf-dir\") pod \"multus-additional-cni-plugins-wz7lk\" (UID: \"03964cb9-d415-48a8-9a6a-834979450459\") " pod="openshift-multus/multus-additional-cni-plugins-wz7lk"
Sep 17 19:06:24.455058 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:06:24.454104    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/c1b79c18-774f-4b78-a6a9-7ea8a229aaae-host-slash\") pod \"ovnkube-node-kdm8s\" (UID: \"c1b79c18-774f-4b78-a6a9-7ea8a229aaae\") " pod="openshift-ovn-kubernetes/ovnkube-node-kdm8s"
Sep 17 19:06:24.455058 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:06:24.454111    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/c1b79c18-774f-4b78-a6a9-7ea8a229aaae-var-lib-openvswitch\") pod \"ovnkube-node-kdm8s\" (UID: \"c1b79c18-774f-4b78-a6a9-7ea8a229aaae\") " pod="openshift-ovn-kubernetes/ovnkube-node-kdm8s"
Sep 17 19:06:24.455058 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:06:24.454129    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/c1b79c18-774f-4b78-a6a9-7ea8a229aaae-host-slash\") pod \"ovnkube-node-kdm8s\" (UID: \"c1b79c18-774f-4b78-a6a9-7ea8a229aaae\") " pod="openshift-ovn-kubernetes/ovnkube-node-kdm8s"
Sep 17 19:06:24.455058 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:06:24.454105    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/c1b79c18-774f-4b78-a6a9-7ea8a229aaae-run-openvswitch\") pod \"ovnkube-node-kdm8s\" (UID: \"c1b79c18-774f-4b78-a6a9-7ea8a229aaae\") " pod="openshift-ovn-kubernetes/ovnkube-node-kdm8s"
Sep 17 19:06:24.455058 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:06:24.454127    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/c1b79c18-774f-4b78-a6a9-7ea8a229aaae-ovn-node-metrics-cert\") pod \"ovnkube-node-kdm8s\" (UID: \"c1b79c18-774f-4b78-a6a9-7ea8a229aaae\") " pod="openshift-ovn-kubernetes/ovnkube-node-kdm8s"
Sep 17 19:06:24.455058 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:06:24.454083    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/c1b79c18-774f-4b78-a6a9-7ea8a229aaae-log-socket\") pod \"ovnkube-node-kdm8s\" (UID: \"c1b79c18-774f-4b78-a6a9-7ea8a229aaae\") " pod="openshift-ovn-kubernetes/ovnkube-node-kdm8s"
Sep 17 19:06:24.455058 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:06:24.454232    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/e884baf0-e11b-4964-9913-07095d7d1f21-socket-dir\") pod \"aws-ebs-csi-driver-node-qxfrl\" (UID: \"e884baf0-e11b-4964-9913-07095d7d1f21\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qxfrl"
Sep 17 19:06:24.455058 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:06:24.454238    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/03964cb9-d415-48a8-9a6a-834979450459-tuning-conf-dir\") pod \"multus-additional-cni-plugins-wz7lk\" (UID: \"03964cb9-d415-48a8-9a6a-834979450459\") " pod="openshift-multus/multus-additional-cni-plugins-wz7lk"
Sep 17 19:06:24.455058 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:06:24.454260    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/e884baf0-e11b-4964-9913-07095d7d1f21-device-dir\") pod \"aws-ebs-csi-driver-node-qxfrl\" (UID: \"e884baf0-e11b-4964-9913-07095d7d1f21\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qxfrl"
Sep 17 19:06:24.455058 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:06:24.454286    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/e884baf0-e11b-4964-9913-07095d7d1f21-etc-selinux\") pod \"aws-ebs-csi-driver-node-qxfrl\" (UID: \"e884baf0-e11b-4964-9913-07095d7d1f21\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qxfrl"
Sep 17 19:06:24.455058 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:06:24.454309    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/271e6574-9235-474f-b0bc-06ec57117390-serviceca\") pod \"node-ca-64g8v\" (UID: \"271e6574-9235-474f-b0bc-06ec57117390\") " pod="openshift-image-registry/node-ca-64g8v"
Sep 17 19:06:24.455058 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:06:24.454322    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/e884baf0-e11b-4964-9913-07095d7d1f21-socket-dir\") pod \"aws-ebs-csi-driver-node-qxfrl\" (UID: \"e884baf0-e11b-4964-9913-07095d7d1f21\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qxfrl"
Sep 17 19:06:24.455058 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:06:24.454335    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/03964cb9-d415-48a8-9a6a-834979450459-system-cni-dir\") pod \"multus-additional-cni-plugins-wz7lk\" (UID: \"03964cb9-d415-48a8-9a6a-834979450459\") " pod="openshift-multus/multus-additional-cni-plugins-wz7lk"
Sep 17 19:06:24.455058 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:06:24.454341    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/e884baf0-e11b-4964-9913-07095d7d1f21-device-dir\") pod \"aws-ebs-csi-driver-node-qxfrl\" (UID: \"e884baf0-e11b-4964-9913-07095d7d1f21\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qxfrl"
Sep 17 19:06:24.455058 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:06:24.454359    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/03964cb9-d415-48a8-9a6a-834979450459-cnibin\") pod \"multus-additional-cni-plugins-wz7lk\" (UID: \"03964cb9-d415-48a8-9a6a-834979450459\") " pod="openshift-multus/multus-additional-cni-plugins-wz7lk"
Sep 17 19:06:24.455058 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:06:24.454374    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/03964cb9-d415-48a8-9a6a-834979450459-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-wz7lk\" (UID: \"03964cb9-d415-48a8-9a6a-834979450459\") " pod="openshift-multus/multus-additional-cni-plugins-wz7lk"
Sep 17 19:06:24.455058 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:06:24.454383    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/c1b79c18-774f-4b78-a6a9-7ea8a229aaae-node-log\") pod \"ovnkube-node-kdm8s\" (UID: \"c1b79c18-774f-4b78-a6a9-7ea8a229aaae\") " pod="openshift-ovn-kubernetes/ovnkube-node-kdm8s"
Sep 17 19:06:24.455575 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:06:24.454390    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/e884baf0-e11b-4964-9913-07095d7d1f21-etc-selinux\") pod \"aws-ebs-csi-driver-node-qxfrl\" (UID: \"e884baf0-e11b-4964-9913-07095d7d1f21\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qxfrl"
Sep 17 19:06:24.455575 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:06:24.454408    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/c1b79c18-774f-4b78-a6a9-7ea8a229aaae-env-overrides\") pod \"ovnkube-node-kdm8s\" (UID: \"c1b79c18-774f-4b78-a6a9-7ea8a229aaae\") " pod="openshift-ovn-kubernetes/ovnkube-node-kdm8s"
Sep 17 19:06:24.455575 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:06:24.454406    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/03964cb9-d415-48a8-9a6a-834979450459-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-wz7lk\" (UID: \"03964cb9-d415-48a8-9a6a-834979450459\") " pod="openshift-multus/multus-additional-cni-plugins-wz7lk"
Sep 17 19:06:24.455575 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:06:24.454432    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/03964cb9-d415-48a8-9a6a-834979450459-system-cni-dir\") pod \"multus-additional-cni-plugins-wz7lk\" (UID: \"03964cb9-d415-48a8-9a6a-834979450459\") " pod="openshift-multus/multus-additional-cni-plugins-wz7lk"
Sep 17 19:06:24.455575 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:06:24.454425    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/03964cb9-d415-48a8-9a6a-834979450459-cnibin\") pod \"multus-additional-cni-plugins-wz7lk\" (UID: \"03964cb9-d415-48a8-9a6a-834979450459\") " pod="openshift-multus/multus-additional-cni-plugins-wz7lk"
Sep 17 19:06:24.455575 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:06:24.454454    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/c1b79c18-774f-4b78-a6a9-7ea8a229aaae-node-log\") pod \"ovnkube-node-kdm8s\" (UID: \"c1b79c18-774f-4b78-a6a9-7ea8a229aaae\") " pod="openshift-ovn-kubernetes/ovnkube-node-kdm8s"
Sep 17 19:06:24.455575 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:06:24.454480    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/c1b79c18-774f-4b78-a6a9-7ea8a229aaae-ovnkube-config\") pod \"ovnkube-node-kdm8s\" (UID: \"c1b79c18-774f-4b78-a6a9-7ea8a229aaae\") " pod="openshift-ovn-kubernetes/ovnkube-node-kdm8s"
Sep 17 19:06:24.455575 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:06:24.454498    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/c1b79c18-774f-4b78-a6a9-7ea8a229aaae-ovnkube-script-lib\") pod \"ovnkube-node-kdm8s\" (UID: \"c1b79c18-774f-4b78-a6a9-7ea8a229aaae\") " pod="openshift-ovn-kubernetes/ovnkube-node-kdm8s"
Sep 17 19:06:24.455575 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:06:24.454514    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8hv9p\" (UniqueName: \"kubernetes.io/projected/271e6574-9235-474f-b0bc-06ec57117390-kube-api-access-8hv9p\") pod \"node-ca-64g8v\" (UID: \"271e6574-9235-474f-b0bc-06ec57117390\") " pod="openshift-image-registry/node-ca-64g8v"
Sep 17 19:06:24.455575 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:06:24.454538    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/03964cb9-d415-48a8-9a6a-834979450459-cni-binary-copy\") pod \"multus-additional-cni-plugins-wz7lk\" (UID: \"03964cb9-d415-48a8-9a6a-834979450459\") " pod="openshift-multus/multus-additional-cni-plugins-wz7lk"
Sep 17 19:06:24.455575 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:06:24.454554    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-psk99\" (UniqueName: \"kubernetes.io/projected/cc1b159b-69e7-404d-9a25-bcf0ff8148c5-kube-api-access-psk99\") pod \"network-metrics-daemon-hj5r7\" (UID: \"cc1b159b-69e7-404d-9a25-bcf0ff8148c5\") " pod="openshift-multus/network-metrics-daemon-hj5r7"
Sep 17 19:06:24.455575 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:06:24.454568    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/f902d60d-52fe-4458-94ce-21fc761c4bb5-iptables-alerter-script\") pod \"iptables-alerter-kzblg\" (UID: \"f902d60d-52fe-4458-94ce-21fc761c4bb5\") " pod="openshift-network-operator/iptables-alerter-kzblg"
Sep 17 19:06:24.455575 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:06:24.454583    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/c1b79c18-774f-4b78-a6a9-7ea8a229aaae-host-run-netns\") pod \"ovnkube-node-kdm8s\" (UID: \"c1b79c18-774f-4b78-a6a9-7ea8a229aaae\") " pod="openshift-ovn-kubernetes/ovnkube-node-kdm8s"
Sep 17 19:06:24.455575 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:06:24.454611    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/c1b79c18-774f-4b78-a6a9-7ea8a229aaae-host-kubelet\") pod \"ovnkube-node-kdm8s\" (UID: \"c1b79c18-774f-4b78-a6a9-7ea8a229aaae\") " pod="openshift-ovn-kubernetes/ovnkube-node-kdm8s"
Sep 17 19:06:24.455575 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:06:24.454626    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/c1b79c18-774f-4b78-a6a9-7ea8a229aaae-run-systemd\") pod \"ovnkube-node-kdm8s\" (UID: \"c1b79c18-774f-4b78-a6a9-7ea8a229aaae\") " pod="openshift-ovn-kubernetes/ovnkube-node-kdm8s"
Sep 17 19:06:24.455575 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:06:24.454641    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/c1b79c18-774f-4b78-a6a9-7ea8a229aaae-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-kdm8s\" (UID: \"c1b79c18-774f-4b78-a6a9-7ea8a229aaae\") " pod="openshift-ovn-kubernetes/ovnkube-node-kdm8s"
Sep 17 19:06:24.455575 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:06:24.454686    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/c1b79c18-774f-4b78-a6a9-7ea8a229aaae-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-kdm8s\" (UID: \"c1b79c18-774f-4b78-a6a9-7ea8a229aaae\") " pod="openshift-ovn-kubernetes/ovnkube-node-kdm8s"
Sep 17 19:06:24.456054 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:06:24.454824    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/271e6574-9235-474f-b0bc-06ec57117390-serviceca\") pod \"node-ca-64g8v\" (UID: \"271e6574-9235-474f-b0bc-06ec57117390\") " pod="openshift-image-registry/node-ca-64g8v"
Sep 17 19:06:24.456054 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:06:24.454919    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/c1b79c18-774f-4b78-a6a9-7ea8a229aaae-host-kubelet\") pod \"ovnkube-node-kdm8s\" (UID: \"c1b79c18-774f-4b78-a6a9-7ea8a229aaae\") " pod="openshift-ovn-kubernetes/ovnkube-node-kdm8s"
Sep 17 19:06:24.456054 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:06:24.454962    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/c1b79c18-774f-4b78-a6a9-7ea8a229aaae-run-systemd\") pod \"ovnkube-node-kdm8s\" (UID: \"c1b79c18-774f-4b78-a6a9-7ea8a229aaae\") " pod="openshift-ovn-kubernetes/ovnkube-node-kdm8s"
Sep 17 19:06:24.456054 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:06:24.454972    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/c1b79c18-774f-4b78-a6a9-7ea8a229aaae-host-run-netns\") pod \"ovnkube-node-kdm8s\" (UID: \"c1b79c18-774f-4b78-a6a9-7ea8a229aaae\") " pod="openshift-ovn-kubernetes/ovnkube-node-kdm8s"
Sep 17 19:06:24.456054 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:06:24.455049    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/c1b79c18-774f-4b78-a6a9-7ea8a229aaae-ovnkube-config\") pod \"ovnkube-node-kdm8s\" (UID: \"c1b79c18-774f-4b78-a6a9-7ea8a229aaae\") " pod="openshift-ovn-kubernetes/ovnkube-node-kdm8s"
Sep 17 19:06:24.456054 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:06:24.455049    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/c1b79c18-774f-4b78-a6a9-7ea8a229aaae-ovnkube-script-lib\") pod \"ovnkube-node-kdm8s\" (UID: \"c1b79c18-774f-4b78-a6a9-7ea8a229aaae\") " pod="openshift-ovn-kubernetes/ovnkube-node-kdm8s"
Sep 17 19:06:24.456054 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:06:24.455071    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/c1b79c18-774f-4b78-a6a9-7ea8a229aaae-env-overrides\") pod \"ovnkube-node-kdm8s\" (UID: \"c1b79c18-774f-4b78-a6a9-7ea8a229aaae\") " pod="openshift-ovn-kubernetes/ovnkube-node-kdm8s"
Sep 17 19:06:24.456054 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:06:24.455343    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/f902d60d-52fe-4458-94ce-21fc761c4bb5-iptables-alerter-script\") pod \"iptables-alerter-kzblg\" (UID: \"f902d60d-52fe-4458-94ce-21fc761c4bb5\") " pod="openshift-network-operator/iptables-alerter-kzblg"
Sep 17 19:06:24.456054 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:06:24.455363    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/03964cb9-d415-48a8-9a6a-834979450459-cni-binary-copy\") pod \"multus-additional-cni-plugins-wz7lk\" (UID: \"03964cb9-d415-48a8-9a6a-834979450459\") " pod="openshift-multus/multus-additional-cni-plugins-wz7lk"
Sep 17 19:06:24.456458 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:06:24.456443    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/c1b79c18-774f-4b78-a6a9-7ea8a229aaae-ovn-node-metrics-cert\") pod \"ovnkube-node-kdm8s\" (UID: \"c1b79c18-774f-4b78-a6a9-7ea8a229aaae\") " pod="openshift-ovn-kubernetes/ovnkube-node-kdm8s"
Sep 17 19:06:24.462527 ip-10-0-131-232 kubenswrapper[2582]: E0917 19:06:24.462513    2582 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 17 19:06:24.462569 ip-10-0-131-232 kubenswrapper[2582]: E0917 19:06:24.462529    2582 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 17 19:06:24.462569 ip-10-0-131-232 kubenswrapper[2582]: E0917 19:06:24.462538    2582 projected.go:196] Error preparing data for projected volume kube-api-access-kftwn for pod openshift-network-diagnostics/network-check-target-jhjrs: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 19:06:24.462641 ip-10-0-131-232 kubenswrapper[2582]: E0917 19:06:24.462579    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/07daf4ff-9d55-4476-aefd-df39d86849a6-kube-api-access-kftwn podName:07daf4ff-9d55-4476-aefd-df39d86849a6 nodeName:}" failed. No retries permitted until 2026-09-17 19:06:24.962567239 +0000 UTC m=+2.122164871 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-kftwn" (UniqueName: "kubernetes.io/projected/07daf4ff-9d55-4476-aefd-df39d86849a6-kube-api-access-kftwn") pod "network-check-target-jhjrs" (UID: "07daf4ff-9d55-4476-aefd-df39d86849a6") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 19:06:24.465192 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:06:24.465170    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-g2vgd\" (UniqueName: \"kubernetes.io/projected/c1b79c18-774f-4b78-a6a9-7ea8a229aaae-kube-api-access-g2vgd\") pod \"ovnkube-node-kdm8s\" (UID: \"c1b79c18-774f-4b78-a6a9-7ea8a229aaae\") " pod="openshift-ovn-kubernetes/ovnkube-node-kdm8s"
Sep 17 19:06:24.466360 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:06:24.466331    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8hv9p\" (UniqueName: \"kubernetes.io/projected/271e6574-9235-474f-b0bc-06ec57117390-kube-api-access-8hv9p\") pod \"node-ca-64g8v\" (UID: \"271e6574-9235-474f-b0bc-06ec57117390\") " pod="openshift-image-registry/node-ca-64g8v"
Sep 17 19:06:24.466360 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:06:24.466345    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gv65p\" (UniqueName: \"kubernetes.io/projected/f902d60d-52fe-4458-94ce-21fc761c4bb5-kube-api-access-gv65p\") pod \"iptables-alerter-kzblg\" (UID: \"f902d60d-52fe-4458-94ce-21fc761c4bb5\") " pod="openshift-network-operator/iptables-alerter-kzblg"
Sep 17 19:06:24.466495 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:06:24.466417    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-df487\" (UniqueName: \"kubernetes.io/projected/e884baf0-e11b-4964-9913-07095d7d1f21-kube-api-access-df487\") pod \"aws-ebs-csi-driver-node-qxfrl\" (UID: \"e884baf0-e11b-4964-9913-07095d7d1f21\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qxfrl"
Sep 17 19:06:24.466630 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:06:24.466611    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qdvst\" (UniqueName: \"kubernetes.io/projected/03964cb9-d415-48a8-9a6a-834979450459-kube-api-access-qdvst\") pod \"multus-additional-cni-plugins-wz7lk\" (UID: \"03964cb9-d415-48a8-9a6a-834979450459\") " pod="openshift-multus/multus-additional-cni-plugins-wz7lk"
Sep 17 19:06:24.466755 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:06:24.466742    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-psk99\" (UniqueName: \"kubernetes.io/projected/cc1b159b-69e7-404d-9a25-bcf0ff8148c5-kube-api-access-psk99\") pod \"network-metrics-daemon-hj5r7\" (UID: \"cc1b159b-69e7-404d-9a25-bcf0ff8148c5\") " pod="openshift-multus/network-metrics-daemon-hj5r7"
Sep 17 19:06:25.954801 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:06:25.954780    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-fppr7"
Sep 17 19:06:25.965077 ip-10-0-131-232 kubenswrapper[2582]: W0917 19:06:25.965047    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaa2ade5e_8564_46a7_819c_75c57cb0961d.slice/crio-12b55b5f845225ecd5d63b1852279b2b6fc78895c8fb179edd37804bf84684df WatchSource:0}: Error finding container 12b55b5f845225ecd5d63b1852279b2b6fc78895c8fb179edd37804bf84684df: Status 404 returned error can't find the container with id 12b55b5f845225ecd5d63b1852279b2b6fc78895c8fb179edd37804bf84684df
Sep 17 19:06:25.971205 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:06:25.971186    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-g8qm4"
Sep 17 19:06:25.977264 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:06:25.977238    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-qwhmv"
Sep 17 19:06:25.978987 ip-10-0-131-232 kubenswrapper[2582]: W0917 19:06:25.978967    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf661fc84_37f7_4cb9_a5b5_be6673fbd0dd.slice/crio-8f43ddb4bba7bfe044195d17176267d5454d40747bd7aaeffd0c537055a537bc WatchSource:0}: Error finding container 8f43ddb4bba7bfe044195d17176267d5454d40747bd7aaeffd0c537055a537bc: Status 404 returned error can't find the container with id 8f43ddb4bba7bfe044195d17176267d5454d40747bd7aaeffd0c537055a537bc
Sep 17 19:06:25.985533 ip-10-0-131-232 kubenswrapper[2582]: W0917 19:06:25.985515    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfe7c72bc_812d_4272_8352_78f8fa2c19d1.slice/crio-db834e82ff28b648498ce5168291c0a0d640371ee65b2c8a077952170541abe1 WatchSource:0}: Error finding container db834e82ff28b648498ce5168291c0a0d640371ee65b2c8a077952170541abe1: Status 404 returned error can't find the container with id db834e82ff28b648498ce5168291c0a0d640371ee65b2c8a077952170541abe1
Sep 17 19:06:26.000901 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:06:26.000876    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-cwtkf"
Sep 17 19:06:26.006726 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:06:26.006700    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-wz7lk"
Sep 17 19:06:26.008724 ip-10-0-131-232 kubenswrapper[2582]: W0917 19:06:26.008702    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod46629fc1_07ce_45b8_9d73_453abfa383b9.slice/crio-9c47c9e198810c6901a39f31264cc680ff94522ead433ea59555d223da4565a1 WatchSource:0}: Error finding container 9c47c9e198810c6901a39f31264cc680ff94522ead433ea59555d223da4565a1: Status 404 returned error can't find the container with id 9c47c9e198810c6901a39f31264cc680ff94522ead433ea59555d223da4565a1
Sep 17 19:06:26.015207 ip-10-0-131-232 kubenswrapper[2582]: W0917 19:06:26.015188    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod03964cb9_d415_48a8_9a6a_834979450459.slice/crio-1e8d7a0afe4555026dd855c4b6cc66e334715906ea022031746b0263b9fc3a54 WatchSource:0}: Error finding container 1e8d7a0afe4555026dd855c4b6cc66e334715906ea022031746b0263b9fc3a54: Status 404 returned error can't find the container with id 1e8d7a0afe4555026dd855c4b6cc66e334715906ea022031746b0263b9fc3a54
Sep 17 19:06:26.023643 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:06:26.023624    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-kzblg"
Sep 17 19:06:26.032436 ip-10-0-131-232 kubenswrapper[2582]: W0917 19:06:26.032417    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf902d60d_52fe_4458_94ce_21fc761c4bb5.slice/crio-0b849f1c0811e161ca6914334dfd844bba0f38f9f867d1a17ebdc9379d6506b3 WatchSource:0}: Error finding container 0b849f1c0811e161ca6914334dfd844bba0f38f9f867d1a17ebdc9379d6506b3: Status 404 returned error can't find the container with id 0b849f1c0811e161ca6914334dfd844bba0f38f9f867d1a17ebdc9379d6506b3
Sep 17 19:06:26.043693 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:06:26.043670    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-kdm8s"
Sep 17 19:06:26.051741 ip-10-0-131-232 kubenswrapper[2582]: W0917 19:06:26.051713    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc1b79c18_774f_4b78_a6a9_7ea8a229aaae.slice/crio-d939cd777e8148f186d45ee3aa445b77b7bc8a066b5c5bdbb9ffd9b3dbd82751 WatchSource:0}: Error finding container d939cd777e8148f186d45ee3aa445b77b7bc8a066b5c5bdbb9ffd9b3dbd82751: Status 404 returned error can't find the container with id d939cd777e8148f186d45ee3aa445b77b7bc8a066b5c5bdbb9ffd9b3dbd82751
Sep 17 19:06:26.062545 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:06:26.062524    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qxfrl"
Sep 17 19:06:26.066366 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:06:26.066341    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-64g8v"
Sep 17 19:06:26.071553 ip-10-0-131-232 kubenswrapper[2582]: W0917 19:06:26.071534    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode884baf0_e11b_4964_9913_07095d7d1f21.slice/crio-8889e8dc9cd4b9fa73921ed835ad8eff5c593f29a95f23a228f167bb6083f295 WatchSource:0}: Error finding container 8889e8dc9cd4b9fa73921ed835ad8eff5c593f29a95f23a228f167bb6083f295: Status 404 returned error can't find the container with id 8889e8dc9cd4b9fa73921ed835ad8eff5c593f29a95f23a228f167bb6083f295
Sep 17 19:06:26.075845 ip-10-0-131-232 kubenswrapper[2582]: W0917 19:06:26.075804    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod271e6574_9235_474f_b0bc_06ec57117390.slice/crio-bc8cc52eebb87fc13ab9bf4d96033d890379aeaf7d6da8bbf9cebaad2864c474 WatchSource:0}: Error finding container bc8cc52eebb87fc13ab9bf4d96033d890379aeaf7d6da8bbf9cebaad2864c474: Status 404 returned error can't find the container with id bc8cc52eebb87fc13ab9bf4d96033d890379aeaf7d6da8bbf9cebaad2864c474
Sep 17 19:06:26.245751 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:06:26.245662    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/7cd887c0-2cc8-4363-b553-8d2b4e10f158-original-pull-secret\") pod \"global-pull-secret-syncer-f6kjs\" (UID: \"7cd887c0-2cc8-4363-b553-8d2b4e10f158\") " pod="kube-system/global-pull-secret-syncer-f6kjs"
Sep 17 19:06:26.245921 ip-10-0-131-232 kubenswrapper[2582]: E0917 19:06:26.245803    2582 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 17 19:06:26.245921 ip-10-0-131-232 kubenswrapper[2582]: E0917 19:06:26.245887    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7cd887c0-2cc8-4363-b553-8d2b4e10f158-original-pull-secret podName:7cd887c0-2cc8-4363-b553-8d2b4e10f158 nodeName:}" failed. No retries permitted until 2026-09-17 19:06:27.245869566 +0000 UTC m=+3.017540611 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/7cd887c0-2cc8-4363-b553-8d2b4e10f158-original-pull-secret") pod "global-pull-secret-syncer-f6kjs" (UID: "7cd887c0-2cc8-4363-b553-8d2b4e10f158") : object "kube-system"/"original-pull-secret" not registered
Sep 17 19:06:26.346376 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:06:26.346340    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cc1b159b-69e7-404d-9a25-bcf0ff8148c5-metrics-certs\") pod \"network-metrics-daemon-hj5r7\" (UID: \"cc1b159b-69e7-404d-9a25-bcf0ff8148c5\") " pod="openshift-multus/network-metrics-daemon-hj5r7"
Sep 17 19:06:26.346621 ip-10-0-131-232 kubenswrapper[2582]: E0917 19:06:26.346596    2582 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 19:06:26.346743 ip-10-0-131-232 kubenswrapper[2582]: E0917 19:06:26.346690    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cc1b159b-69e7-404d-9a25-bcf0ff8148c5-metrics-certs podName:cc1b159b-69e7-404d-9a25-bcf0ff8148c5 nodeName:}" failed. No retries permitted until 2026-09-17 19:06:27.346670012 +0000 UTC m=+3.118341059 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/cc1b159b-69e7-404d-9a25-bcf0ff8148c5-metrics-certs") pod "network-metrics-daemon-hj5r7" (UID: "cc1b159b-69e7-404d-9a25-bcf0ff8148c5") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 19:06:26.449841 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:06:26.447016    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kftwn\" (UniqueName: \"kubernetes.io/projected/07daf4ff-9d55-4476-aefd-df39d86849a6-kube-api-access-kftwn\") pod \"network-check-target-jhjrs\" (UID: \"07daf4ff-9d55-4476-aefd-df39d86849a6\") " pod="openshift-network-diagnostics/network-check-target-jhjrs"
Sep 17 19:06:26.449841 ip-10-0-131-232 kubenswrapper[2582]: E0917 19:06:26.447221    2582 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 17 19:06:26.449841 ip-10-0-131-232 kubenswrapper[2582]: E0917 19:06:26.447244    2582 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 17 19:06:26.449841 ip-10-0-131-232 kubenswrapper[2582]: E0917 19:06:26.447256    2582 projected.go:196] Error preparing data for projected volume kube-api-access-kftwn for pod openshift-network-diagnostics/network-check-target-jhjrs: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 19:06:26.449841 ip-10-0-131-232 kubenswrapper[2582]: E0917 19:06:26.447313    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/07daf4ff-9d55-4476-aefd-df39d86849a6-kube-api-access-kftwn podName:07daf4ff-9d55-4476-aefd-df39d86849a6 nodeName:}" failed. No retries permitted until 2026-09-17 19:06:27.447295769 +0000 UTC m=+3.218966814 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-kftwn" (UniqueName: "kubernetes.io/projected/07daf4ff-9d55-4476-aefd-df39d86849a6-kube-api-access-kftwn") pod "network-check-target-jhjrs" (UID: "07daf4ff-9d55-4476-aefd-df39d86849a6") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 19:06:26.590117 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:06:26.590035    2582 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 17 19:06:26.682750 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:06:26.682698    2582 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-09-16 19:01:25 +0000 UTC" deadline="2028-04-14 03:57:20.07053517 +0000 UTC"
Sep 17 19:06:26.682750 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:06:26.682756    2582 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13784h50m53.387792084s"
Sep 17 19:06:26.782490 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:06:26.782295    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-g8qm4" event={"ID":"f661fc84-37f7-4cb9-a5b5-be6673fbd0dd","Type":"ContainerStarted","Data":"8f43ddb4bba7bfe044195d17176267d5454d40747bd7aaeffd0c537055a537bc"}
Sep 17 19:06:26.798369 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:06:26.798230    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-fppr7" event={"ID":"aa2ade5e-8564-46a7-819c-75c57cb0961d","Type":"ContainerStarted","Data":"12b55b5f845225ecd5d63b1852279b2b6fc78895c8fb179edd37804bf84684df"}
Sep 17 19:06:26.806450 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:06:26.806415    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-kzblg" event={"ID":"f902d60d-52fe-4458-94ce-21fc761c4bb5","Type":"ContainerStarted","Data":"0b849f1c0811e161ca6914334dfd844bba0f38f9f867d1a17ebdc9379d6506b3"}
Sep 17 19:06:26.810757 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:06:26.810669    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-wz7lk" event={"ID":"03964cb9-d415-48a8-9a6a-834979450459","Type":"ContainerStarted","Data":"1e8d7a0afe4555026dd855c4b6cc66e334715906ea022031746b0263b9fc3a54"}
Sep 17 19:06:26.824300 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:06:26.824154    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-232.ec2.internal" event={"ID":"5a96bc0ce60b5e0ec092cd8768bc466e","Type":"ContainerStarted","Data":"ea8c5c4aa39874f45e5681fd31d2e42bd0b620b5fc5e1913c59ea6350869fe88"}
Sep 17 19:06:26.845361 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:06:26.845278    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-131-232.ec2.internal" event={"ID":"a1f9bd0034d8d01c3b2f70180b25c21f","Type":"ContainerStarted","Data":"d8f1c6329ccd2522505f459e6d92c9f9a2ebf6192b0f0f4fd313a9a8c308b9b0"}
Sep 17 19:06:26.854352 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:06:26.854317    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-64g8v" event={"ID":"271e6574-9235-474f-b0bc-06ec57117390","Type":"ContainerStarted","Data":"bc8cc52eebb87fc13ab9bf4d96033d890379aeaf7d6da8bbf9cebaad2864c474"}
Sep 17 19:06:26.867863 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:06:26.866666    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qxfrl" event={"ID":"e884baf0-e11b-4964-9913-07095d7d1f21","Type":"ContainerStarted","Data":"8889e8dc9cd4b9fa73921ed835ad8eff5c593f29a95f23a228f167bb6083f295"}
Sep 17 19:06:26.881577 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:06:26.881450    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-kdm8s" event={"ID":"c1b79c18-774f-4b78-a6a9-7ea8a229aaae","Type":"ContainerStarted","Data":"d939cd777e8148f186d45ee3aa445b77b7bc8a066b5c5bdbb9ffd9b3dbd82751"}
Sep 17 19:06:26.886618 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:06:26.886574    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-cwtkf" event={"ID":"46629fc1-07ce-45b8-9d73-453abfa383b9","Type":"ContainerStarted","Data":"9c47c9e198810c6901a39f31264cc680ff94522ead433ea59555d223da4565a1"}
Sep 17 19:06:26.889810 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:06:26.889782    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-qwhmv" event={"ID":"fe7c72bc-812d-4272-8352-78f8fa2c19d1","Type":"ContainerStarted","Data":"db834e82ff28b648498ce5168291c0a0d640371ee65b2c8a077952170541abe1"}
Sep 17 19:06:27.254053 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:06:27.253955    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/7cd887c0-2cc8-4363-b553-8d2b4e10f158-original-pull-secret\") pod \"global-pull-secret-syncer-f6kjs\" (UID: \"7cd887c0-2cc8-4363-b553-8d2b4e10f158\") " pod="kube-system/global-pull-secret-syncer-f6kjs"
Sep 17 19:06:27.254221 ip-10-0-131-232 kubenswrapper[2582]: E0917 19:06:27.254149    2582 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 17 19:06:27.254221 ip-10-0-131-232 kubenswrapper[2582]: E0917 19:06:27.254213    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7cd887c0-2cc8-4363-b553-8d2b4e10f158-original-pull-secret podName:7cd887c0-2cc8-4363-b553-8d2b4e10f158 nodeName:}" failed. No retries permitted until 2026-09-17 19:06:29.254194712 +0000 UTC m=+5.025865805 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/7cd887c0-2cc8-4363-b553-8d2b4e10f158-original-pull-secret") pod "global-pull-secret-syncer-f6kjs" (UID: "7cd887c0-2cc8-4363-b553-8d2b4e10f158") : object "kube-system"/"original-pull-secret" not registered
Sep 17 19:06:27.354912 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:06:27.354795    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cc1b159b-69e7-404d-9a25-bcf0ff8148c5-metrics-certs\") pod \"network-metrics-daemon-hj5r7\" (UID: \"cc1b159b-69e7-404d-9a25-bcf0ff8148c5\") " pod="openshift-multus/network-metrics-daemon-hj5r7"
Sep 17 19:06:27.355110 ip-10-0-131-232 kubenswrapper[2582]: E0917 19:06:27.355013    2582 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 19:06:27.355110 ip-10-0-131-232 kubenswrapper[2582]: E0917 19:06:27.355093    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cc1b159b-69e7-404d-9a25-bcf0ff8148c5-metrics-certs podName:cc1b159b-69e7-404d-9a25-bcf0ff8148c5 nodeName:}" failed. No retries permitted until 2026-09-17 19:06:29.355070809 +0000 UTC m=+5.126741872 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/cc1b159b-69e7-404d-9a25-bcf0ff8148c5-metrics-certs") pod "network-metrics-daemon-hj5r7" (UID: "cc1b159b-69e7-404d-9a25-bcf0ff8148c5") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 19:06:27.455943 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:06:27.455835    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kftwn\" (UniqueName: \"kubernetes.io/projected/07daf4ff-9d55-4476-aefd-df39d86849a6-kube-api-access-kftwn\") pod \"network-check-target-jhjrs\" (UID: \"07daf4ff-9d55-4476-aefd-df39d86849a6\") " pod="openshift-network-diagnostics/network-check-target-jhjrs"
Sep 17 19:06:27.456130 ip-10-0-131-232 kubenswrapper[2582]: E0917 19:06:27.455981    2582 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 17 19:06:27.456130 ip-10-0-131-232 kubenswrapper[2582]: E0917 19:06:27.456009    2582 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 17 19:06:27.456130 ip-10-0-131-232 kubenswrapper[2582]: E0917 19:06:27.456021    2582 projected.go:196] Error preparing data for projected volume kube-api-access-kftwn for pod openshift-network-diagnostics/network-check-target-jhjrs: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 19:06:27.456130 ip-10-0-131-232 kubenswrapper[2582]: E0917 19:06:27.456081    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/07daf4ff-9d55-4476-aefd-df39d86849a6-kube-api-access-kftwn podName:07daf4ff-9d55-4476-aefd-df39d86849a6 nodeName:}" failed. No retries permitted until 2026-09-17 19:06:29.456063372 +0000 UTC m=+5.227734434 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-kftwn" (UniqueName: "kubernetes.io/projected/07daf4ff-9d55-4476-aefd-df39d86849a6-kube-api-access-kftwn") pod "network-check-target-jhjrs" (UID: "07daf4ff-9d55-4476-aefd-df39d86849a6") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 19:06:27.614651 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:06:27.614576    2582 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 17 19:06:27.683859 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:06:27.683781    2582 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-09-16 19:01:25 +0000 UTC" deadline="2028-05-07 10:19:22.66878779 +0000 UTC"
Sep 17 19:06:27.683859 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:06:27.683856    2582 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14343h12m54.984934934s"
Sep 17 19:06:27.763644 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:06:27.763598    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-f6kjs"
Sep 17 19:06:27.763786 ip-10-0-131-232 kubenswrapper[2582]: E0917 19:06:27.763722    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-f6kjs" podUID="7cd887c0-2cc8-4363-b553-8d2b4e10f158"
Sep 17 19:06:27.764262 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:06:27.764242    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-jhjrs"
Sep 17 19:06:27.764356 ip-10-0-131-232 kubenswrapper[2582]: E0917 19:06:27.764339    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-jhjrs" podUID="07daf4ff-9d55-4476-aefd-df39d86849a6"
Sep 17 19:06:27.764415 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:06:27.764402    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-hj5r7"
Sep 17 19:06:27.764551 ip-10-0-131-232 kubenswrapper[2582]: E0917 19:06:27.764532    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-hj5r7" podUID="cc1b159b-69e7-404d-9a25-bcf0ff8148c5"
Sep 17 19:06:27.970089 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:06:27.969998    2582 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 17 19:06:29.269636 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:06:29.269598    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/7cd887c0-2cc8-4363-b553-8d2b4e10f158-original-pull-secret\") pod \"global-pull-secret-syncer-f6kjs\" (UID: \"7cd887c0-2cc8-4363-b553-8d2b4e10f158\") " pod="kube-system/global-pull-secret-syncer-f6kjs"
Sep 17 19:06:29.270298 ip-10-0-131-232 kubenswrapper[2582]: E0917 19:06:29.269777    2582 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 17 19:06:29.270298 ip-10-0-131-232 kubenswrapper[2582]: E0917 19:06:29.269861    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7cd887c0-2cc8-4363-b553-8d2b4e10f158-original-pull-secret podName:7cd887c0-2cc8-4363-b553-8d2b4e10f158 nodeName:}" failed. No retries permitted until 2026-09-17 19:06:33.269842795 +0000 UTC m=+9.041513849 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/7cd887c0-2cc8-4363-b553-8d2b4e10f158-original-pull-secret") pod "global-pull-secret-syncer-f6kjs" (UID: "7cd887c0-2cc8-4363-b553-8d2b4e10f158") : object "kube-system"/"original-pull-secret" not registered
Sep 17 19:06:29.370930 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:06:29.370896    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cc1b159b-69e7-404d-9a25-bcf0ff8148c5-metrics-certs\") pod \"network-metrics-daemon-hj5r7\" (UID: \"cc1b159b-69e7-404d-9a25-bcf0ff8148c5\") " pod="openshift-multus/network-metrics-daemon-hj5r7"
Sep 17 19:06:29.371091 ip-10-0-131-232 kubenswrapper[2582]: E0917 19:06:29.371074    2582 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 19:06:29.371153 ip-10-0-131-232 kubenswrapper[2582]: E0917 19:06:29.371142    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cc1b159b-69e7-404d-9a25-bcf0ff8148c5-metrics-certs podName:cc1b159b-69e7-404d-9a25-bcf0ff8148c5 nodeName:}" failed. No retries permitted until 2026-09-17 19:06:33.37112382 +0000 UTC m=+9.142794862 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/cc1b159b-69e7-404d-9a25-bcf0ff8148c5-metrics-certs") pod "network-metrics-daemon-hj5r7" (UID: "cc1b159b-69e7-404d-9a25-bcf0ff8148c5") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 19:06:29.471842 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:06:29.471790    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kftwn\" (UniqueName: \"kubernetes.io/projected/07daf4ff-9d55-4476-aefd-df39d86849a6-kube-api-access-kftwn\") pod \"network-check-target-jhjrs\" (UID: \"07daf4ff-9d55-4476-aefd-df39d86849a6\") " pod="openshift-network-diagnostics/network-check-target-jhjrs"
Sep 17 19:06:29.472054 ip-10-0-131-232 kubenswrapper[2582]: E0917 19:06:29.472031    2582 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 17 19:06:29.472145 ip-10-0-131-232 kubenswrapper[2582]: E0917 19:06:29.472062    2582 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 17 19:06:29.472145 ip-10-0-131-232 kubenswrapper[2582]: E0917 19:06:29.472075    2582 projected.go:196] Error preparing data for projected volume kube-api-access-kftwn for pod openshift-network-diagnostics/network-check-target-jhjrs: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 19:06:29.472145 ip-10-0-131-232 kubenswrapper[2582]: E0917 19:06:29.472137    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/07daf4ff-9d55-4476-aefd-df39d86849a6-kube-api-access-kftwn podName:07daf4ff-9d55-4476-aefd-df39d86849a6 nodeName:}" failed. No retries permitted until 2026-09-17 19:06:33.472119628 +0000 UTC m=+9.243790685 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-kftwn" (UniqueName: "kubernetes.io/projected/07daf4ff-9d55-4476-aefd-df39d86849a6-kube-api-access-kftwn") pod "network-check-target-jhjrs" (UID: "07daf4ff-9d55-4476-aefd-df39d86849a6") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 19:06:29.765096 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:06:29.764571    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-f6kjs"
Sep 17 19:06:29.765096 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:06:29.764607    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-jhjrs"
Sep 17 19:06:29.765096 ip-10-0-131-232 kubenswrapper[2582]: E0917 19:06:29.764703    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-f6kjs" podUID="7cd887c0-2cc8-4363-b553-8d2b4e10f158"
Sep 17 19:06:29.765096 ip-10-0-131-232 kubenswrapper[2582]: E0917 19:06:29.764763    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-jhjrs" podUID="07daf4ff-9d55-4476-aefd-df39d86849a6"
Sep 17 19:06:29.765096 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:06:29.764840    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-hj5r7"
Sep 17 19:06:29.765096 ip-10-0-131-232 kubenswrapper[2582]: E0917 19:06:29.764930    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-hj5r7" podUID="cc1b159b-69e7-404d-9a25-bcf0ff8148c5"
Sep 17 19:06:31.762798 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:06:31.762758    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-jhjrs"
Sep 17 19:06:31.763255 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:06:31.762808    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-hj5r7"
Sep 17 19:06:31.763255 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:06:31.762885    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-f6kjs"
Sep 17 19:06:31.763255 ip-10-0-131-232 kubenswrapper[2582]: E0917 19:06:31.762902    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-jhjrs" podUID="07daf4ff-9d55-4476-aefd-df39d86849a6"
Sep 17 19:06:31.763255 ip-10-0-131-232 kubenswrapper[2582]: E0917 19:06:31.762988    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-f6kjs" podUID="7cd887c0-2cc8-4363-b553-8d2b4e10f158"
Sep 17 19:06:31.763255 ip-10-0-131-232 kubenswrapper[2582]: E0917 19:06:31.763083    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-hj5r7" podUID="cc1b159b-69e7-404d-9a25-bcf0ff8148c5"
Sep 17 19:06:33.304681 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:06:33.304634    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/7cd887c0-2cc8-4363-b553-8d2b4e10f158-original-pull-secret\") pod \"global-pull-secret-syncer-f6kjs\" (UID: \"7cd887c0-2cc8-4363-b553-8d2b4e10f158\") " pod="kube-system/global-pull-secret-syncer-f6kjs"
Sep 17 19:06:33.305206 ip-10-0-131-232 kubenswrapper[2582]: E0917 19:06:33.304849    2582 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 17 19:06:33.305206 ip-10-0-131-232 kubenswrapper[2582]: E0917 19:06:33.304918    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7cd887c0-2cc8-4363-b553-8d2b4e10f158-original-pull-secret podName:7cd887c0-2cc8-4363-b553-8d2b4e10f158 nodeName:}" failed. No retries permitted until 2026-09-17 19:06:41.304901634 +0000 UTC m=+17.076572676 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/7cd887c0-2cc8-4363-b553-8d2b4e10f158-original-pull-secret") pod "global-pull-secret-syncer-f6kjs" (UID: "7cd887c0-2cc8-4363-b553-8d2b4e10f158") : object "kube-system"/"original-pull-secret" not registered
Sep 17 19:06:33.405545 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:06:33.405452    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cc1b159b-69e7-404d-9a25-bcf0ff8148c5-metrics-certs\") pod \"network-metrics-daemon-hj5r7\" (UID: \"cc1b159b-69e7-404d-9a25-bcf0ff8148c5\") " pod="openshift-multus/network-metrics-daemon-hj5r7"
Sep 17 19:06:33.405709 ip-10-0-131-232 kubenswrapper[2582]: E0917 19:06:33.405630    2582 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 19:06:33.405709 ip-10-0-131-232 kubenswrapper[2582]: E0917 19:06:33.405694    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cc1b159b-69e7-404d-9a25-bcf0ff8148c5-metrics-certs podName:cc1b159b-69e7-404d-9a25-bcf0ff8148c5 nodeName:}" failed. No retries permitted until 2026-09-17 19:06:41.405675831 +0000 UTC m=+17.177346876 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/cc1b159b-69e7-404d-9a25-bcf0ff8148c5-metrics-certs") pod "network-metrics-daemon-hj5r7" (UID: "cc1b159b-69e7-404d-9a25-bcf0ff8148c5") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 19:06:33.506413 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:06:33.506372    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kftwn\" (UniqueName: \"kubernetes.io/projected/07daf4ff-9d55-4476-aefd-df39d86849a6-kube-api-access-kftwn\") pod \"network-check-target-jhjrs\" (UID: \"07daf4ff-9d55-4476-aefd-df39d86849a6\") " pod="openshift-network-diagnostics/network-check-target-jhjrs"
Sep 17 19:06:33.506571 ip-10-0-131-232 kubenswrapper[2582]: E0917 19:06:33.506529    2582 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 17 19:06:33.506571 ip-10-0-131-232 kubenswrapper[2582]: E0917 19:06:33.506550    2582 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 17 19:06:33.506571 ip-10-0-131-232 kubenswrapper[2582]: E0917 19:06:33.506561    2582 projected.go:196] Error preparing data for projected volume kube-api-access-kftwn for pod openshift-network-diagnostics/network-check-target-jhjrs: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 19:06:33.506718 ip-10-0-131-232 kubenswrapper[2582]: E0917 19:06:33.506618    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/07daf4ff-9d55-4476-aefd-df39d86849a6-kube-api-access-kftwn podName:07daf4ff-9d55-4476-aefd-df39d86849a6 nodeName:}" failed. No retries permitted until 2026-09-17 19:06:41.506598569 +0000 UTC m=+17.278269629 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-kftwn" (UniqueName: "kubernetes.io/projected/07daf4ff-9d55-4476-aefd-df39d86849a6-kube-api-access-kftwn") pod "network-check-target-jhjrs" (UID: "07daf4ff-9d55-4476-aefd-df39d86849a6") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 19:06:33.763273 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:06:33.763195    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-f6kjs"
Sep 17 19:06:33.763273 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:06:33.763239    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-hj5r7"
Sep 17 19:06:33.763489 ip-10-0-131-232 kubenswrapper[2582]: E0917 19:06:33.763309    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-f6kjs" podUID="7cd887c0-2cc8-4363-b553-8d2b4e10f158"
Sep 17 19:06:33.763489 ip-10-0-131-232 kubenswrapper[2582]: E0917 19:06:33.763368    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-hj5r7" podUID="cc1b159b-69e7-404d-9a25-bcf0ff8148c5"
Sep 17 19:06:33.763489 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:06:33.763238    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-jhjrs"
Sep 17 19:06:33.763489 ip-10-0-131-232 kubenswrapper[2582]: E0917 19:06:33.763460    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-jhjrs" podUID="07daf4ff-9d55-4476-aefd-df39d86849a6"
Sep 17 19:06:35.763489 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:06:35.763457    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-f6kjs"
Sep 17 19:06:35.763980 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:06:35.763457    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-hj5r7"
Sep 17 19:06:35.763980 ip-10-0-131-232 kubenswrapper[2582]: E0917 19:06:35.763587    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-f6kjs" podUID="7cd887c0-2cc8-4363-b553-8d2b4e10f158"
Sep 17 19:06:35.763980 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:06:35.763637    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-jhjrs"
Sep 17 19:06:35.763980 ip-10-0-131-232 kubenswrapper[2582]: E0917 19:06:35.763725    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-jhjrs" podUID="07daf4ff-9d55-4476-aefd-df39d86849a6"
Sep 17 19:06:35.763980 ip-10-0-131-232 kubenswrapper[2582]: E0917 19:06:35.763803    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-hj5r7" podUID="cc1b159b-69e7-404d-9a25-bcf0ff8148c5"
Sep 17 19:06:37.763707 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:06:37.763629    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-f6kjs"
Sep 17 19:06:37.763707 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:06:37.763666    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-jhjrs"
Sep 17 19:06:37.764192 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:06:37.763728    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-hj5r7"
Sep 17 19:06:37.764192 ip-10-0-131-232 kubenswrapper[2582]: E0917 19:06:37.763744    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-f6kjs" podUID="7cd887c0-2cc8-4363-b553-8d2b4e10f158"
Sep 17 19:06:37.764192 ip-10-0-131-232 kubenswrapper[2582]: E0917 19:06:37.763839    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-hj5r7" podUID="cc1b159b-69e7-404d-9a25-bcf0ff8148c5"
Sep 17 19:06:37.764192 ip-10-0-131-232 kubenswrapper[2582]: E0917 19:06:37.763895    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-jhjrs" podUID="07daf4ff-9d55-4476-aefd-df39d86849a6"
Sep 17 19:06:39.763079 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:06:39.763046    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-f6kjs"
Sep 17 19:06:39.763079 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:06:39.763080    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-jhjrs"
Sep 17 19:06:39.763079 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:06:39.763091    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-hj5r7"
Sep 17 19:06:39.763595 ip-10-0-131-232 kubenswrapper[2582]: E0917 19:06:39.763186    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-f6kjs" podUID="7cd887c0-2cc8-4363-b553-8d2b4e10f158"
Sep 17 19:06:39.763595 ip-10-0-131-232 kubenswrapper[2582]: E0917 19:06:39.763249    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-jhjrs" podUID="07daf4ff-9d55-4476-aefd-df39d86849a6"
Sep 17 19:06:39.763595 ip-10-0-131-232 kubenswrapper[2582]: E0917 19:06:39.763301    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-hj5r7" podUID="cc1b159b-69e7-404d-9a25-bcf0ff8148c5"
Sep 17 19:06:41.365921 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:06:41.365877    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/7cd887c0-2cc8-4363-b553-8d2b4e10f158-original-pull-secret\") pod \"global-pull-secret-syncer-f6kjs\" (UID: \"7cd887c0-2cc8-4363-b553-8d2b4e10f158\") " pod="kube-system/global-pull-secret-syncer-f6kjs"
Sep 17 19:06:41.366564 ip-10-0-131-232 kubenswrapper[2582]: E0917 19:06:41.366035    2582 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 17 19:06:41.366564 ip-10-0-131-232 kubenswrapper[2582]: E0917 19:06:41.366150    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7cd887c0-2cc8-4363-b553-8d2b4e10f158-original-pull-secret podName:7cd887c0-2cc8-4363-b553-8d2b4e10f158 nodeName:}" failed. No retries permitted until 2026-09-17 19:06:57.366125954 +0000 UTC m=+33.137796999 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/7cd887c0-2cc8-4363-b553-8d2b4e10f158-original-pull-secret") pod "global-pull-secret-syncer-f6kjs" (UID: "7cd887c0-2cc8-4363-b553-8d2b4e10f158") : object "kube-system"/"original-pull-secret" not registered
Sep 17 19:06:41.467219 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:06:41.467175    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cc1b159b-69e7-404d-9a25-bcf0ff8148c5-metrics-certs\") pod \"network-metrics-daemon-hj5r7\" (UID: \"cc1b159b-69e7-404d-9a25-bcf0ff8148c5\") " pod="openshift-multus/network-metrics-daemon-hj5r7"
Sep 17 19:06:41.467428 ip-10-0-131-232 kubenswrapper[2582]: E0917 19:06:41.467403    2582 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 19:06:41.467571 ip-10-0-131-232 kubenswrapper[2582]: E0917 19:06:41.467475    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cc1b159b-69e7-404d-9a25-bcf0ff8148c5-metrics-certs podName:cc1b159b-69e7-404d-9a25-bcf0ff8148c5 nodeName:}" failed. No retries permitted until 2026-09-17 19:06:57.467457541 +0000 UTC m=+33.239128580 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/cc1b159b-69e7-404d-9a25-bcf0ff8148c5-metrics-certs") pod "network-metrics-daemon-hj5r7" (UID: "cc1b159b-69e7-404d-9a25-bcf0ff8148c5") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 19:06:41.568242 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:06:41.568165    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kftwn\" (UniqueName: \"kubernetes.io/projected/07daf4ff-9d55-4476-aefd-df39d86849a6-kube-api-access-kftwn\") pod \"network-check-target-jhjrs\" (UID: \"07daf4ff-9d55-4476-aefd-df39d86849a6\") " pod="openshift-network-diagnostics/network-check-target-jhjrs"
Sep 17 19:06:41.568389 ip-10-0-131-232 kubenswrapper[2582]: E0917 19:06:41.568309    2582 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 17 19:06:41.568389 ip-10-0-131-232 kubenswrapper[2582]: E0917 19:06:41.568327    2582 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 17 19:06:41.568389 ip-10-0-131-232 kubenswrapper[2582]: E0917 19:06:41.568338    2582 projected.go:196] Error preparing data for projected volume kube-api-access-kftwn for pod openshift-network-diagnostics/network-check-target-jhjrs: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 19:06:41.568534 ip-10-0-131-232 kubenswrapper[2582]: E0917 19:06:41.568396    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/07daf4ff-9d55-4476-aefd-df39d86849a6-kube-api-access-kftwn podName:07daf4ff-9d55-4476-aefd-df39d86849a6 nodeName:}" failed. No retries permitted until 2026-09-17 19:06:57.568378345 +0000 UTC m=+33.340049398 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-kftwn" (UniqueName: "kubernetes.io/projected/07daf4ff-9d55-4476-aefd-df39d86849a6-kube-api-access-kftwn") pod "network-check-target-jhjrs" (UID: "07daf4ff-9d55-4476-aefd-df39d86849a6") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 19:06:41.763747 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:06:41.763705    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-jhjrs"
Sep 17 19:06:41.763923 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:06:41.763807    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-f6kjs"
Sep 17 19:06:41.763923 ip-10-0-131-232 kubenswrapper[2582]: E0917 19:06:41.763834    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-jhjrs" podUID="07daf4ff-9d55-4476-aefd-df39d86849a6"
Sep 17 19:06:41.763923 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:06:41.763848    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-hj5r7"
Sep 17 19:06:41.763923 ip-10-0-131-232 kubenswrapper[2582]: E0917 19:06:41.763916    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-f6kjs" podUID="7cd887c0-2cc8-4363-b553-8d2b4e10f158"
Sep 17 19:06:41.764086 ip-10-0-131-232 kubenswrapper[2582]: E0917 19:06:41.763987    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-hj5r7" podUID="cc1b159b-69e7-404d-9a25-bcf0ff8148c5"
Sep 17 19:06:43.763239 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:06:43.763199    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-jhjrs"
Sep 17 19:06:43.763631 ip-10-0-131-232 kubenswrapper[2582]: E0917 19:06:43.763309    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-jhjrs" podUID="07daf4ff-9d55-4476-aefd-df39d86849a6"
Sep 17 19:06:43.763631 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:06:43.763199    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-f6kjs"
Sep 17 19:06:43.763631 ip-10-0-131-232 kubenswrapper[2582]: E0917 19:06:43.763386    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-f6kjs" podUID="7cd887c0-2cc8-4363-b553-8d2b4e10f158"
Sep 17 19:06:43.763631 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:06:43.763199    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-hj5r7"
Sep 17 19:06:43.763631 ip-10-0-131-232 kubenswrapper[2582]: E0917 19:06:43.763464    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-hj5r7" podUID="cc1b159b-69e7-404d-9a25-bcf0ff8148c5"
Sep 17 19:06:44.949706 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:06:44.949456    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-g8qm4" event={"ID":"f661fc84-37f7-4cb9-a5b5-be6673fbd0dd","Type":"ContainerStarted","Data":"b2e231572c2264bfc98207b01c363351d66e89df4cd095df054a77e819f404d9"}
Sep 17 19:06:44.968092 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:06:44.968026    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-g8qm4" podStartSLOduration=2.233933426 podStartE2EDuration="20.968006892s" totalImagesPullingTime="18.734073466s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 19:06:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 19:06:44.967807461 +0000 UTC m=+20.739478523" watchObservedRunningTime="2026-09-17 19:06:44.968006892 +0000 UTC m=+20.739677956"
Sep 17 19:06:44.968681 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:06:44.968582    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-131-232.ec2.internal" event={"ID":"a1f9bd0034d8d01c3b2f70180b25c21f","Type":"ContainerStarted","Data":"394fbfb42bb2dbad3c8195d17a490e193fb45c0d21270b0b746e01dc5d3f04df"}
Sep 17 19:06:44.985580 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:06:44.985529    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-131-232.ec2.internal" podStartSLOduration=19.985508646 podStartE2EDuration="19.985508646s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 19:06:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 19:06:44.984457211 +0000 UTC m=+20.756128273" watchObservedRunningTime="2026-09-17 19:06:44.985508646 +0000 UTC m=+20.757179708"
Sep 17 19:06:45.763900 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:06:45.763684    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-hj5r7"
Sep 17 19:06:45.764069 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:06:45.763717    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-f6kjs"
Sep 17 19:06:45.764069 ip-10-0-131-232 kubenswrapper[2582]: E0917 19:06:45.764013    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-hj5r7" podUID="cc1b159b-69e7-404d-9a25-bcf0ff8148c5"
Sep 17 19:06:45.764069 ip-10-0-131-232 kubenswrapper[2582]: E0917 19:06:45.764055    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-f6kjs" podUID="7cd887c0-2cc8-4363-b553-8d2b4e10f158"
Sep 17 19:06:45.764069 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:06:45.763723    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-jhjrs"
Sep 17 19:06:45.764247 ip-10-0-131-232 kubenswrapper[2582]: E0917 19:06:45.764141    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-jhjrs" podUID="07daf4ff-9d55-4476-aefd-df39d86849a6"
Sep 17 19:06:45.972406 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:06:45.972372    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qxfrl" event={"ID":"e884baf0-e11b-4964-9913-07095d7d1f21","Type":"ContainerStarted","Data":"7f1a9643be06a4fcf7c3220b220447e391cfa66c0dd1ca3e9e39ea52be53b41e"}
Sep 17 19:06:45.974910 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:06:45.974890    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-kdm8s_c1b79c18-774f-4b78-a6a9-7ea8a229aaae/ovn-acl-logging/0.log"
Sep 17 19:06:45.975206 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:06:45.975186    2582 generic.go:356] "Generic (PLEG): container finished" podID="c1b79c18-774f-4b78-a6a9-7ea8a229aaae" containerID="53046ddc8c2e7aa52ba0c47848f0adef6439c224448b40443f823b1d066be500" exitCode=1
Sep 17 19:06:45.975284 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:06:45.975259    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-kdm8s" event={"ID":"c1b79c18-774f-4b78-a6a9-7ea8a229aaae","Type":"ContainerStarted","Data":"004bbe9641be5f8cf90abad2ffbde7577a5a2ed2b28ca64b187bbb4bdbb50de7"}
Sep 17 19:06:45.975348 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:06:45.975333    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-kdm8s" event={"ID":"c1b79c18-774f-4b78-a6a9-7ea8a229aaae","Type":"ContainerStarted","Data":"b4fb51fc62b4d9a6ec9664b4602ebf1796281e3bef85ef0e13ebf887f8aa630f"}
Sep 17 19:06:45.975381 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:06:45.975369    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-kdm8s" event={"ID":"c1b79c18-774f-4b78-a6a9-7ea8a229aaae","Type":"ContainerStarted","Data":"31d6028122c72be14fff45a1561bae078e51927723a0b35d823b3788e19c88f9"}
Sep 17 19:06:45.975429 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:06:45.975397    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-kdm8s" event={"ID":"c1b79c18-774f-4b78-a6a9-7ea8a229aaae","Type":"ContainerStarted","Data":"83cdfed4e1bd010fc886512a7f56247ae93564eb7520e1383bbb2803d59f6568"}
Sep 17 19:06:45.975473 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:06:45.975428    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-kdm8s" event={"ID":"c1b79c18-774f-4b78-a6a9-7ea8a229aaae","Type":"ContainerDied","Data":"53046ddc8c2e7aa52ba0c47848f0adef6439c224448b40443f823b1d066be500"}
Sep 17 19:06:45.975519 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:06:45.975475    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-kdm8s" event={"ID":"c1b79c18-774f-4b78-a6a9-7ea8a229aaae","Type":"ContainerStarted","Data":"cfdeef6aeddae93a5c18a5d1d21935e559d8cfa20798324bc781a94b407ed628"}
Sep 17 19:06:45.976532 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:06:45.976509    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-cwtkf" event={"ID":"46629fc1-07ce-45b8-9d73-453abfa383b9","Type":"ContainerStarted","Data":"5994abe051c25b28cea0c1d7ac78bf48c40308595800fd531259c1b457ea2011"}
Sep 17 19:06:45.977896 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:06:45.977858    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-qwhmv" event={"ID":"fe7c72bc-812d-4272-8352-78f8fa2c19d1","Type":"ContainerStarted","Data":"7677e7eeec1199dfc6ecbcf215e7dde2c31686b55d3b073a35bf688c37615e9c"}
Sep 17 19:06:45.979308 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:06:45.979258    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-fppr7" event={"ID":"aa2ade5e-8564-46a7-819c-75c57cb0961d","Type":"ContainerStarted","Data":"f88794ebca379e2c34d16ef377aa130b9948454e83c3759561cea213bd8d01d4"}
Sep 17 19:06:45.980784 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:06:45.980759    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-kzblg" event={"ID":"f902d60d-52fe-4458-94ce-21fc761c4bb5","Type":"ContainerStarted","Data":"f97d1e48ad088bbd9320a53afdce01363627db50f5f3e53a4bc88be125168675"}
Sep 17 19:06:45.982297 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:06:45.982277    2582 generic.go:356] "Generic (PLEG): container finished" podID="03964cb9-d415-48a8-9a6a-834979450459" containerID="279a137adcb9ef73040e479d2fab85497f5da133c094b1832f5daed80b5fa270" exitCode=0
Sep 17 19:06:45.982389 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:06:45.982309    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-wz7lk" event={"ID":"03964cb9-d415-48a8-9a6a-834979450459","Type":"ContainerDied","Data":"279a137adcb9ef73040e479d2fab85497f5da133c094b1832f5daed80b5fa270"}
Sep 17 19:06:45.984313 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:06:45.984287    2582 generic.go:356] "Generic (PLEG): container finished" podID="5a96bc0ce60b5e0ec092cd8768bc466e" containerID="935f7c261201b68f01a0cf7af671e48779152643cc9f04c1e44367b5de89e401" exitCode=0
Sep 17 19:06:45.984427 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:06:45.984364    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-232.ec2.internal" event={"ID":"5a96bc0ce60b5e0ec092cd8768bc466e","Type":"ContainerDied","Data":"935f7c261201b68f01a0cf7af671e48779152643cc9f04c1e44367b5de89e401"}
Sep 17 19:06:45.986335 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:06:45.986312    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-64g8v" event={"ID":"271e6574-9235-474f-b0bc-06ec57117390","Type":"ContainerStarted","Data":"31344a39fc264ec9736b3bcc46d9cd46a2f979b9db553075fec1d59aadd7992c"}
Sep 17 19:06:45.992564 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:06:45.992521    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-cwtkf" podStartSLOduration=3.288225045 podStartE2EDuration="21.992509018s" totalImagesPullingTime="18.704283973s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 19:06:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 19:06:45.991957907 +0000 UTC m=+21.763628968" watchObservedRunningTime="2026-09-17 19:06:45.992509018 +0000 UTC m=+21.764180080"
Sep 17 19:06:46.009506 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:06:46.009456    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-64g8v" podStartSLOduration=3.517297568 podStartE2EDuration="22.009438363s" totalImagesPullingTime="18.492140795s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 19:06:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 19:06:46.009176413 +0000 UTC m=+21.780847476" watchObservedRunningTime="2026-09-17 19:06:46.009438363 +0000 UTC m=+21.781109425"
Sep 17 19:06:46.027449 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:06:46.027400    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-qwhmv" podStartSLOduration=3.433998783 podStartE2EDuration="22.027379987s" totalImagesPullingTime="18.593381204s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 19:06:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 19:06:46.027324689 +0000 UTC m=+21.798995762" watchObservedRunningTime="2026-09-17 19:06:46.027379987 +0000 UTC m=+21.799051047"
Sep 17 19:06:46.061637 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:06:46.061588    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-kzblg" podStartSLOduration=3.515246557 podStartE2EDuration="22.061568389s" totalImagesPullingTime="18.546321832s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 19:06:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 19:06:46.045497549 +0000 UTC m=+21.817168612" watchObservedRunningTime="2026-09-17 19:06:46.061568389 +0000 UTC m=+21.833239452"
Sep 17 19:06:46.083059 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:06:46.083012    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-fppr7" podStartSLOduration=3.081742813 podStartE2EDuration="22.082994761s" totalImagesPullingTime="19.001251948s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 19:06:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 19:06:46.08268361 +0000 UTC m=+21.854354672" watchObservedRunningTime="2026-09-17 19:06:46.082994761 +0000 UTC m=+21.854665822"
Sep 17 19:06:46.635303 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:06:46.635280    2582 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Sep 17 19:06:46.710862 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:06:46.710579    2582 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-09-17T19:06:46.635298168Z","UUID":"1796ec5a-f91f-46d0-a49d-dd525e71fcec","Handler":null,"Name":"","Endpoint":""}
Sep 17 19:06:46.712281 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:06:46.712257    2582 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Sep 17 19:06:46.712411 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:06:46.712293    2582 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Sep 17 19:06:46.993238 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:06:46.993183    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qxfrl" event={"ID":"e884baf0-e11b-4964-9913-07095d7d1f21","Type":"ContainerStarted","Data":"53390ba04a25c6b7fd85f17b073d442564adcd487b9c607818f06622bb604a37"}
Sep 17 19:06:46.995512 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:06:46.995461    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-232.ec2.internal" event={"ID":"5a96bc0ce60b5e0ec092cd8768bc466e","Type":"ContainerStarted","Data":"a32c21c1bcd48e5539f36242d9312713fff5c3cf2ef89dbc6e89bfcd5a47eb3c"}
Sep 17 19:06:47.012132 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:06:47.012070    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-232.ec2.internal" podStartSLOduration=22.012052696 podStartE2EDuration="22.012052696s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 19:06:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 19:06:47.011744143 +0000 UTC m=+22.783415205" watchObservedRunningTime="2026-09-17 19:06:47.012052696 +0000 UTC m=+22.783723758"
Sep 17 19:06:47.763031 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:06:47.762997    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-hj5r7"
Sep 17 19:06:47.763031 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:06:47.763027    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-jhjrs"
Sep 17 19:06:47.763265 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:06:47.763027    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-f6kjs"
Sep 17 19:06:47.763265 ip-10-0-131-232 kubenswrapper[2582]: E0917 19:06:47.763108    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-hj5r7" podUID="cc1b159b-69e7-404d-9a25-bcf0ff8148c5"
Sep 17 19:06:47.763265 ip-10-0-131-232 kubenswrapper[2582]: E0917 19:06:47.763222    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-jhjrs" podUID="07daf4ff-9d55-4476-aefd-df39d86849a6"
Sep 17 19:06:47.763391 ip-10-0-131-232 kubenswrapper[2582]: E0917 19:06:47.763319    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-f6kjs" podUID="7cd887c0-2cc8-4363-b553-8d2b4e10f158"
Sep 17 19:06:47.999625 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:06:47.999590    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qxfrl" event={"ID":"e884baf0-e11b-4964-9913-07095d7d1f21","Type":"ContainerStarted","Data":"427d7b3def7d730678d6a9108c31ee814ecfe344eee8c88d5f54abce8acf6e68"}
Sep 17 19:06:48.003232 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:06:48.003166    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-kdm8s_c1b79c18-774f-4b78-a6a9-7ea8a229aaae/ovn-acl-logging/0.log"
Sep 17 19:06:48.003642 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:06:48.003620    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-kdm8s" event={"ID":"c1b79c18-774f-4b78-a6a9-7ea8a229aaae","Type":"ContainerStarted","Data":"3569a0700353b7ad67fa96b05896cf249e4f392d290ff56602183f19e3b2199e"}
Sep 17 19:06:48.021500 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:06:48.021452    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qxfrl" podStartSLOduration=3.092332897 podStartE2EDuration="24.021437352s" totalImagesPullingTime="20.929104455s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 19:06:24 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 19:06:48.021027658 +0000 UTC m=+23.792698719" watchObservedRunningTime="2026-09-17 19:06:48.021437352 +0000 UTC m=+23.793108417"
Sep 17 19:06:49.763758 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:06:49.763723    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-hj5r7"
Sep 17 19:06:49.763758 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:06:49.763737    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-jhjrs"
Sep 17 19:06:49.764319 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:06:49.763862    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-f6kjs"
Sep 17 19:06:49.764319 ip-10-0-131-232 kubenswrapper[2582]: E0917 19:06:49.763883    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-hj5r7" podUID="cc1b159b-69e7-404d-9a25-bcf0ff8148c5"
Sep 17 19:06:49.764319 ip-10-0-131-232 kubenswrapper[2582]: E0917 19:06:49.763965    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-f6kjs" podUID="7cd887c0-2cc8-4363-b553-8d2b4e10f158"
Sep 17 19:06:49.764319 ip-10-0-131-232 kubenswrapper[2582]: E0917 19:06:49.764022    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-jhjrs" podUID="07daf4ff-9d55-4476-aefd-df39d86849a6"
Sep 17 19:06:50.519623 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:06:50.519440    2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-qwhmv"
Sep 17 19:06:50.520131 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:06:50.520102    2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-qwhmv"
Sep 17 19:06:51.013604 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:06:51.013578    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-kdm8s_c1b79c18-774f-4b78-a6a9-7ea8a229aaae/ovn-acl-logging/0.log"
Sep 17 19:06:51.014063 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:06:51.013975    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-kdm8s" event={"ID":"c1b79c18-774f-4b78-a6a9-7ea8a229aaae","Type":"ContainerStarted","Data":"56c253b9f65d11317086b75e485861c8d56c0afc6df8b96ebb076c192653addc"}
Sep 17 19:06:51.014234 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:06:51.014207    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-kdm8s"
Sep 17 19:06:51.014394 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:06:51.014379    2582 scope.go:117] "RemoveContainer" containerID="53046ddc8c2e7aa52ba0c47848f0adef6439c224448b40443f823b1d066be500"
Sep 17 19:06:51.015909 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:06:51.015887    2582 generic.go:356] "Generic (PLEG): container finished" podID="03964cb9-d415-48a8-9a6a-834979450459" containerID="d541eb53983127a1fe4dd1d5b7e916f25a96234461cd15642ecb8a392b196031" exitCode=0
Sep 17 19:06:51.016008 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:06:51.015971    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-wz7lk" event={"ID":"03964cb9-d415-48a8-9a6a-834979450459","Type":"ContainerDied","Data":"d541eb53983127a1fe4dd1d5b7e916f25a96234461cd15642ecb8a392b196031"}
Sep 17 19:06:51.016266 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:06:51.016244    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-qwhmv"
Sep 17 19:06:51.016793 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:06:51.016775    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-qwhmv"
Sep 17 19:06:51.031578 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:06:51.031560    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-kdm8s"
Sep 17 19:06:51.763801 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:06:51.763774    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-hj5r7"
Sep 17 19:06:51.763971 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:06:51.763776    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-jhjrs"
Sep 17 19:06:51.763971 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:06:51.763779    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-f6kjs"
Sep 17 19:06:51.763971 ip-10-0-131-232 kubenswrapper[2582]: E0917 19:06:51.763902    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-hj5r7" podUID="cc1b159b-69e7-404d-9a25-bcf0ff8148c5"
Sep 17 19:06:51.764096 ip-10-0-131-232 kubenswrapper[2582]: E0917 19:06:51.763994    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-jhjrs" podUID="07daf4ff-9d55-4476-aefd-df39d86849a6"
Sep 17 19:06:51.764197 ip-10-0-131-232 kubenswrapper[2582]: E0917 19:06:51.764164    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-f6kjs" podUID="7cd887c0-2cc8-4363-b553-8d2b4e10f158"
Sep 17 19:06:52.021658 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:06:52.021600    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-kdm8s_c1b79c18-774f-4b78-a6a9-7ea8a229aaae/ovn-acl-logging/0.log"
Sep 17 19:06:52.022285 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:06:52.022006    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-kdm8s" event={"ID":"c1b79c18-774f-4b78-a6a9-7ea8a229aaae","Type":"ContainerStarted","Data":"c6b69bec9c030dbd787f53553c50deb143c5f44b40cba21ad482c469bc8a28fb"}
Sep 17 19:06:52.022285 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:06:52.022053    2582 prober_manager.go:312] "Failed to trigger a manual run" probe="Readiness"
Sep 17 19:06:52.022285 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:06:52.022232    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-kdm8s"
Sep 17 19:06:52.024087 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:06:52.024064    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-wz7lk" event={"ID":"03964cb9-d415-48a8-9a6a-834979450459","Type":"ContainerStarted","Data":"deef24fecf7d4bf4d293898f390a2454ca67c9b68c535603901ac9028cf43481"}
Sep 17 19:06:52.039440 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:06:52.039422    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-kdm8s"
Sep 17 19:06:52.058003 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:06:52.057963    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-kdm8s" podStartSLOduration=9.361748758 podStartE2EDuration="28.057939244s" totalImagesPullingTime="18.696190486s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 19:06:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 19:06:52.057414712 +0000 UTC m=+27.829085772" watchObservedRunningTime="2026-09-17 19:06:52.057939244 +0000 UTC m=+27.829610304"
Sep 17 19:06:52.363660 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:06:52.363577    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-f6kjs"]
Sep 17 19:06:52.363811 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:06:52.363709    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-f6kjs"
Sep 17 19:06:52.363885 ip-10-0-131-232 kubenswrapper[2582]: E0917 19:06:52.363814    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-f6kjs" podUID="7cd887c0-2cc8-4363-b553-8d2b4e10f158"
Sep 17 19:06:52.367426 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:06:52.367277    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-jhjrs"]
Sep 17 19:06:52.367558 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:06:52.367526    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-jhjrs"
Sep 17 19:06:52.367630 ip-10-0-131-232 kubenswrapper[2582]: E0917 19:06:52.367600    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-jhjrs" podUID="07daf4ff-9d55-4476-aefd-df39d86849a6"
Sep 17 19:06:52.367894 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:06:52.367876    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-hj5r7"]
Sep 17 19:06:52.368003 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:06:52.367986    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-hj5r7"
Sep 17 19:06:52.368108 ip-10-0-131-232 kubenswrapper[2582]: E0917 19:06:52.368070    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-hj5r7" podUID="cc1b159b-69e7-404d-9a25-bcf0ff8148c5"
Sep 17 19:06:52.505428 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:06:52.505402    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-kdm8s"
Sep 17 19:06:53.028422 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:06:53.028397    2582 generic.go:356] "Generic (PLEG): container finished" podID="03964cb9-d415-48a8-9a6a-834979450459" containerID="deef24fecf7d4bf4d293898f390a2454ca67c9b68c535603901ac9028cf43481" exitCode=0
Sep 17 19:06:53.028795 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:06:53.028478    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-wz7lk" event={"ID":"03964cb9-d415-48a8-9a6a-834979450459","Type":"ContainerDied","Data":"deef24fecf7d4bf4d293898f390a2454ca67c9b68c535603901ac9028cf43481"}
Sep 17 19:06:53.763654 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:06:53.763482    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-jhjrs"
Sep 17 19:06:53.763767 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:06:53.763544    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-f6kjs"
Sep 17 19:06:53.763767 ip-10-0-131-232 kubenswrapper[2582]: E0917 19:06:53.763731    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-jhjrs" podUID="07daf4ff-9d55-4476-aefd-df39d86849a6"
Sep 17 19:06:53.763846 ip-10-0-131-232 kubenswrapper[2582]: E0917 19:06:53.763804    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-f6kjs" podUID="7cd887c0-2cc8-4363-b553-8d2b4e10f158"
Sep 17 19:06:54.033311 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:06:54.033279    2582 generic.go:356] "Generic (PLEG): container finished" podID="03964cb9-d415-48a8-9a6a-834979450459" containerID="e33394a030b30f3e732da9651e89bf231092bab7ef33cc58276847280adeaeb5" exitCode=0
Sep 17 19:06:54.033727 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:06:54.033358    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-wz7lk" event={"ID":"03964cb9-d415-48a8-9a6a-834979450459","Type":"ContainerDied","Data":"e33394a030b30f3e732da9651e89bf231092bab7ef33cc58276847280adeaeb5"}
Sep 17 19:06:54.764497 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:06:54.764471    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-hj5r7"
Sep 17 19:06:54.764706 ip-10-0-131-232 kubenswrapper[2582]: E0917 19:06:54.764563    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-hj5r7" podUID="cc1b159b-69e7-404d-9a25-bcf0ff8148c5"
Sep 17 19:06:55.763145 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:06:55.763110    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-jhjrs"
Sep 17 19:06:55.763145 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:06:55.763151    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-f6kjs"
Sep 17 19:06:55.763635 ip-10-0-131-232 kubenswrapper[2582]: E0917 19:06:55.763243    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-jhjrs" podUID="07daf4ff-9d55-4476-aefd-df39d86849a6"
Sep 17 19:06:55.763635 ip-10-0-131-232 kubenswrapper[2582]: E0917 19:06:55.763377    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-f6kjs" podUID="7cd887c0-2cc8-4363-b553-8d2b4e10f158"
Sep 17 19:06:56.763200 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:06:56.763164    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-hj5r7"
Sep 17 19:06:56.763728 ip-10-0-131-232 kubenswrapper[2582]: E0917 19:06:56.763294    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-hj5r7" podUID="cc1b159b-69e7-404d-9a25-bcf0ff8148c5"
Sep 17 19:06:57.383563 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:06:57.383525    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/7cd887c0-2cc8-4363-b553-8d2b4e10f158-original-pull-secret\") pod \"global-pull-secret-syncer-f6kjs\" (UID: \"7cd887c0-2cc8-4363-b553-8d2b4e10f158\") " pod="kube-system/global-pull-secret-syncer-f6kjs"
Sep 17 19:06:57.383747 ip-10-0-131-232 kubenswrapper[2582]: E0917 19:06:57.383672    2582 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 17 19:06:57.383747 ip-10-0-131-232 kubenswrapper[2582]: E0917 19:06:57.383736    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7cd887c0-2cc8-4363-b553-8d2b4e10f158-original-pull-secret podName:7cd887c0-2cc8-4363-b553-8d2b4e10f158 nodeName:}" failed. No retries permitted until 2026-09-17 19:07:29.383721259 +0000 UTC m=+65.155392312 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/7cd887c0-2cc8-4363-b553-8d2b4e10f158-original-pull-secret") pod "global-pull-secret-syncer-f6kjs" (UID: "7cd887c0-2cc8-4363-b553-8d2b4e10f158") : object "kube-system"/"original-pull-secret" not registered
Sep 17 19:06:57.483853 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:06:57.483810    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cc1b159b-69e7-404d-9a25-bcf0ff8148c5-metrics-certs\") pod \"network-metrics-daemon-hj5r7\" (UID: \"cc1b159b-69e7-404d-9a25-bcf0ff8148c5\") " pod="openshift-multus/network-metrics-daemon-hj5r7"
Sep 17 19:06:57.484035 ip-10-0-131-232 kubenswrapper[2582]: E0917 19:06:57.483989    2582 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 19:06:57.484097 ip-10-0-131-232 kubenswrapper[2582]: E0917 19:06:57.484070    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cc1b159b-69e7-404d-9a25-bcf0ff8148c5-metrics-certs podName:cc1b159b-69e7-404d-9a25-bcf0ff8148c5 nodeName:}" failed. No retries permitted until 2026-09-17 19:07:29.484047844 +0000 UTC m=+65.255718908 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/cc1b159b-69e7-404d-9a25-bcf0ff8148c5-metrics-certs") pod "network-metrics-daemon-hj5r7" (UID: "cc1b159b-69e7-404d-9a25-bcf0ff8148c5") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 19:06:57.584677 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:06:57.584637    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kftwn\" (UniqueName: \"kubernetes.io/projected/07daf4ff-9d55-4476-aefd-df39d86849a6-kube-api-access-kftwn\") pod \"network-check-target-jhjrs\" (UID: \"07daf4ff-9d55-4476-aefd-df39d86849a6\") " pod="openshift-network-diagnostics/network-check-target-jhjrs"
Sep 17 19:06:57.584868 ip-10-0-131-232 kubenswrapper[2582]: E0917 19:06:57.584841    2582 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 17 19:06:57.584868 ip-10-0-131-232 kubenswrapper[2582]: E0917 19:06:57.584867    2582 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 17 19:06:57.584984 ip-10-0-131-232 kubenswrapper[2582]: E0917 19:06:57.584878    2582 projected.go:196] Error preparing data for projected volume kube-api-access-kftwn for pod openshift-network-diagnostics/network-check-target-jhjrs: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 19:06:57.584984 ip-10-0-131-232 kubenswrapper[2582]: E0917 19:06:57.584943    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/07daf4ff-9d55-4476-aefd-df39d86849a6-kube-api-access-kftwn podName:07daf4ff-9d55-4476-aefd-df39d86849a6 nodeName:}" failed. No retries permitted until 2026-09-17 19:07:29.584923243 +0000 UTC m=+65.356594298 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-kftwn" (UniqueName: "kubernetes.io/projected/07daf4ff-9d55-4476-aefd-df39d86849a6-kube-api-access-kftwn") pod "network-check-target-jhjrs" (UID: "07daf4ff-9d55-4476-aefd-df39d86849a6") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 19:06:57.763206 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:06:57.763127    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-f6kjs"
Sep 17 19:06:57.763206 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:06:57.763159    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-jhjrs"
Sep 17 19:06:57.763745 ip-10-0-131-232 kubenswrapper[2582]: E0917 19:06:57.763222    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-f6kjs" podUID="7cd887c0-2cc8-4363-b553-8d2b4e10f158"
Sep 17 19:06:57.763745 ip-10-0-131-232 kubenswrapper[2582]: E0917 19:06:57.763344    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-jhjrs" podUID="07daf4ff-9d55-4476-aefd-df39d86849a6"
Sep 17 19:06:58.523363 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:06:58.523340    2582 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-232.ec2.internal" event="NodeReady"
Sep 17 19:06:58.523483 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:06:58.523473    2582 kubelet_node_status.go:489] "Fast updating node status as it just became ready"
Sep 17 19:06:58.596118 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:06:58.596060    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-4b2jj"]
Sep 17 19:06:58.605376 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:06:58.605360    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-4b2jj"
Sep 17 19:06:58.606711 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:06:58.606693    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-67qx4"]
Sep 17 19:06:58.609718 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:06:58.609702    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-cmft5\""
Sep 17 19:06:58.619126 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:06:58.619107    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Sep 17 19:06:58.619338 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:06:58.619326    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-67qx4"
Sep 17 19:06:58.619404 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:06:58.619392    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Sep 17 19:06:58.620016 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:06:58.619998    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Sep 17 19:06:58.623840 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:06:58.623769    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Sep 17 19:06:58.623840 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:06:58.623774    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Sep 17 19:06:58.624187 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:06:58.624171    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-rj77f\""
Sep 17 19:06:58.648303 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:06:58.648269    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-4b2jj"]
Sep 17 19:06:58.649046 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:06:58.649029    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-67qx4"]
Sep 17 19:06:58.688865 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:06:58.688804    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-kff2k"]
Sep 17 19:06:58.692850 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:06:58.692804    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ccsgs\" (UniqueName: \"kubernetes.io/projected/413bc45c-0bcb-421e-aa86-c1b75aea7d11-kube-api-access-ccsgs\") pod \"dns-default-67qx4\" (UID: \"413bc45c-0bcb-421e-aa86-c1b75aea7d11\") " pod="openshift-dns/dns-default-67qx4"
Sep 17 19:06:58.692993 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:06:58.692864    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/e6357f17-007b-4c5c-83c8-7a3d1c2c0e85-cert\") pod \"ingress-canary-4b2jj\" (UID: \"e6357f17-007b-4c5c-83c8-7a3d1c2c0e85\") " pod="openshift-ingress-canary/ingress-canary-4b2jj"
Sep 17 19:06:58.692993 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:06:58.692952    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6fl4g\" (UniqueName: \"kubernetes.io/projected/e6357f17-007b-4c5c-83c8-7a3d1c2c0e85-kube-api-access-6fl4g\") pod \"ingress-canary-4b2jj\" (UID: \"e6357f17-007b-4c5c-83c8-7a3d1c2c0e85\") " pod="openshift-ingress-canary/ingress-canary-4b2jj"
Sep 17 19:06:58.692993 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:06:58.692985    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/413bc45c-0bcb-421e-aa86-c1b75aea7d11-config-volume\") pod \"dns-default-67qx4\" (UID: \"413bc45c-0bcb-421e-aa86-c1b75aea7d11\") " pod="openshift-dns/dns-default-67qx4"
Sep 17 19:06:58.693136 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:06:58.693001    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/413bc45c-0bcb-421e-aa86-c1b75aea7d11-metrics-tls\") pod \"dns-default-67qx4\" (UID: \"413bc45c-0bcb-421e-aa86-c1b75aea7d11\") " pod="openshift-dns/dns-default-67qx4"
Sep 17 19:06:58.693136 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:06:58.693028    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/413bc45c-0bcb-421e-aa86-c1b75aea7d11-tmp-dir\") pod \"dns-default-67qx4\" (UID: \"413bc45c-0bcb-421e-aa86-c1b75aea7d11\") " pod="openshift-dns/dns-default-67qx4"
Sep 17 19:06:58.709463 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:06:58.709438    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-kff2k"
Sep 17 19:06:58.716716 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:06:58.716694    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Sep 17 19:06:58.716813 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:06:58.716731    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Sep 17 19:06:58.716902 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:06:58.716884    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-vsnnc\""
Sep 17 19:06:58.716947 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:06:58.716893    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Sep 17 19:06:58.717013 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:06:58.716991    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Sep 17 19:06:58.719219 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:06:58.719200    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-kff2k"]
Sep 17 19:06:58.763019 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:06:58.762995    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-hj5r7"
Sep 17 19:06:58.766440 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:06:58.766398    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Sep 17 19:06:58.766778 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:06:58.766458    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-lsqsv\""
Sep 17 19:06:58.794085 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:06:58.794060    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/984299cc-0712-45a1-89b5-883a6508992c-crio-socket\") pod \"insights-runtime-extractor-kff2k\" (UID: \"984299cc-0712-45a1-89b5-883a6508992c\") " pod="openshift-insights/insights-runtime-extractor-kff2k"
Sep 17 19:06:58.794181 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:06:58.794088    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/984299cc-0712-45a1-89b5-883a6508992c-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-kff2k\" (UID: \"984299cc-0712-45a1-89b5-883a6508992c\") " pod="openshift-insights/insights-runtime-extractor-kff2k"
Sep 17 19:06:58.794181 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:06:58.794124    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6fl4g\" (UniqueName: \"kubernetes.io/projected/e6357f17-007b-4c5c-83c8-7a3d1c2c0e85-kube-api-access-6fl4g\") pod \"ingress-canary-4b2jj\" (UID: \"e6357f17-007b-4c5c-83c8-7a3d1c2c0e85\") " pod="openshift-ingress-canary/ingress-canary-4b2jj"
Sep 17 19:06:58.794270 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:06:58.794189    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/413bc45c-0bcb-421e-aa86-c1b75aea7d11-config-volume\") pod \"dns-default-67qx4\" (UID: \"413bc45c-0bcb-421e-aa86-c1b75aea7d11\") " pod="openshift-dns/dns-default-67qx4"
Sep 17 19:06:58.794270 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:06:58.794212    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/413bc45c-0bcb-421e-aa86-c1b75aea7d11-metrics-tls\") pod \"dns-default-67qx4\" (UID: \"413bc45c-0bcb-421e-aa86-c1b75aea7d11\") " pod="openshift-dns/dns-default-67qx4"
Sep 17 19:06:58.794270 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:06:58.794232    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/413bc45c-0bcb-421e-aa86-c1b75aea7d11-tmp-dir\") pod \"dns-default-67qx4\" (UID: \"413bc45c-0bcb-421e-aa86-c1b75aea7d11\") " pod="openshift-dns/dns-default-67qx4"
Sep 17 19:06:58.794270 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:06:58.794252    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/984299cc-0712-45a1-89b5-883a6508992c-data-volume\") pod \"insights-runtime-extractor-kff2k\" (UID: \"984299cc-0712-45a1-89b5-883a6508992c\") " pod="openshift-insights/insights-runtime-extractor-kff2k"
Sep 17 19:06:58.794461 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:06:58.794295    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ccsgs\" (UniqueName: \"kubernetes.io/projected/413bc45c-0bcb-421e-aa86-c1b75aea7d11-kube-api-access-ccsgs\") pod \"dns-default-67qx4\" (UID: \"413bc45c-0bcb-421e-aa86-c1b75aea7d11\") " pod="openshift-dns/dns-default-67qx4"
Sep 17 19:06:58.794461 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:06:58.794320    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/984299cc-0712-45a1-89b5-883a6508992c-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-kff2k\" (UID: \"984299cc-0712-45a1-89b5-883a6508992c\") " pod="openshift-insights/insights-runtime-extractor-kff2k"
Sep 17 19:06:58.794461 ip-10-0-131-232 kubenswrapper[2582]: E0917 19:06:58.794344    2582 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 17 19:06:58.794461 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:06:58.794389    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/e6357f17-007b-4c5c-83c8-7a3d1c2c0e85-cert\") pod \"ingress-canary-4b2jj\" (UID: \"e6357f17-007b-4c5c-83c8-7a3d1c2c0e85\") " pod="openshift-ingress-canary/ingress-canary-4b2jj"
Sep 17 19:06:58.794461 ip-10-0-131-232 kubenswrapper[2582]: E0917 19:06:58.794415    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/413bc45c-0bcb-421e-aa86-c1b75aea7d11-metrics-tls podName:413bc45c-0bcb-421e-aa86-c1b75aea7d11 nodeName:}" failed. No retries permitted until 2026-09-17 19:06:59.29439351 +0000 UTC m=+35.066064569 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/413bc45c-0bcb-421e-aa86-c1b75aea7d11-metrics-tls") pod "dns-default-67qx4" (UID: "413bc45c-0bcb-421e-aa86-c1b75aea7d11") : secret "dns-default-metrics-tls" not found
Sep 17 19:06:58.794461 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:06:58.794447    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qkpvq\" (UniqueName: \"kubernetes.io/projected/984299cc-0712-45a1-89b5-883a6508992c-kube-api-access-qkpvq\") pod \"insights-runtime-extractor-kff2k\" (UID: \"984299cc-0712-45a1-89b5-883a6508992c\") " pod="openshift-insights/insights-runtime-extractor-kff2k"
Sep 17 19:06:58.794706 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:06:58.794689    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/413bc45c-0bcb-421e-aa86-c1b75aea7d11-tmp-dir\") pod \"dns-default-67qx4\" (UID: \"413bc45c-0bcb-421e-aa86-c1b75aea7d11\") " pod="openshift-dns/dns-default-67qx4"
Sep 17 19:06:58.794837 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:06:58.794800    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/413bc45c-0bcb-421e-aa86-c1b75aea7d11-config-volume\") pod \"dns-default-67qx4\" (UID: \"413bc45c-0bcb-421e-aa86-c1b75aea7d11\") " pod="openshift-dns/dns-default-67qx4"
Sep 17 19:06:58.798816 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:06:58.798788    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/e6357f17-007b-4c5c-83c8-7a3d1c2c0e85-cert\") pod \"ingress-canary-4b2jj\" (UID: \"e6357f17-007b-4c5c-83c8-7a3d1c2c0e85\") " pod="openshift-ingress-canary/ingress-canary-4b2jj"
Sep 17 19:06:58.809120 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:06:58.809100    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ccsgs\" (UniqueName: \"kubernetes.io/projected/413bc45c-0bcb-421e-aa86-c1b75aea7d11-kube-api-access-ccsgs\") pod \"dns-default-67qx4\" (UID: \"413bc45c-0bcb-421e-aa86-c1b75aea7d11\") " pod="openshift-dns/dns-default-67qx4"
Sep 17 19:06:58.809416 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:06:58.809399    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6fl4g\" (UniqueName: \"kubernetes.io/projected/e6357f17-007b-4c5c-83c8-7a3d1c2c0e85-kube-api-access-6fl4g\") pod \"ingress-canary-4b2jj\" (UID: \"e6357f17-007b-4c5c-83c8-7a3d1c2c0e85\") " pod="openshift-ingress-canary/ingress-canary-4b2jj"
Sep 17 19:06:58.895031 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:06:58.895008    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/984299cc-0712-45a1-89b5-883a6508992c-data-volume\") pod \"insights-runtime-extractor-kff2k\" (UID: \"984299cc-0712-45a1-89b5-883a6508992c\") " pod="openshift-insights/insights-runtime-extractor-kff2k"
Sep 17 19:06:58.895164 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:06:58.895080    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/984299cc-0712-45a1-89b5-883a6508992c-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-kff2k\" (UID: \"984299cc-0712-45a1-89b5-883a6508992c\") " pod="openshift-insights/insights-runtime-extractor-kff2k"
Sep 17 19:06:58.895164 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:06:58.895127    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qkpvq\" (UniqueName: \"kubernetes.io/projected/984299cc-0712-45a1-89b5-883a6508992c-kube-api-access-qkpvq\") pod \"insights-runtime-extractor-kff2k\" (UID: \"984299cc-0712-45a1-89b5-883a6508992c\") " pod="openshift-insights/insights-runtime-extractor-kff2k"
Sep 17 19:06:58.895270 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:06:58.895169    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/984299cc-0712-45a1-89b5-883a6508992c-crio-socket\") pod \"insights-runtime-extractor-kff2k\" (UID: \"984299cc-0712-45a1-89b5-883a6508992c\") " pod="openshift-insights/insights-runtime-extractor-kff2k"
Sep 17 19:06:58.895270 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:06:58.895192    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/984299cc-0712-45a1-89b5-883a6508992c-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-kff2k\" (UID: \"984299cc-0712-45a1-89b5-883a6508992c\") " pod="openshift-insights/insights-runtime-extractor-kff2k"
Sep 17 19:06:58.895270 ip-10-0-131-232 kubenswrapper[2582]: E0917 19:06:58.895209    2582 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 17 19:06:58.895270 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:06:58.895255    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/984299cc-0712-45a1-89b5-883a6508992c-data-volume\") pod \"insights-runtime-extractor-kff2k\" (UID: \"984299cc-0712-45a1-89b5-883a6508992c\") " pod="openshift-insights/insights-runtime-extractor-kff2k"
Sep 17 19:06:58.895433 ip-10-0-131-232 kubenswrapper[2582]: E0917 19:06:58.895280    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/984299cc-0712-45a1-89b5-883a6508992c-insights-runtime-extractor-tls podName:984299cc-0712-45a1-89b5-883a6508992c nodeName:}" failed. No retries permitted until 2026-09-17 19:06:59.395264038 +0000 UTC m=+35.166935079 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/984299cc-0712-45a1-89b5-883a6508992c-insights-runtime-extractor-tls") pod "insights-runtime-extractor-kff2k" (UID: "984299cc-0712-45a1-89b5-883a6508992c") : secret "insights-runtime-extractor-tls" not found
Sep 17 19:06:58.895433 ip-10-0-131-232 kubenswrapper[2582]: E0917 19:06:58.895281    2582 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 17 19:06:58.895433 ip-10-0-131-232 kubenswrapper[2582]: E0917 19:06:58.895352    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/984299cc-0712-45a1-89b5-883a6508992c-kube-rbac-proxy-cm podName:984299cc-0712-45a1-89b5-883a6508992c nodeName:}" failed. No retries permitted until 2026-09-17 19:06:59.395335071 +0000 UTC m=+35.167006131 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/984299cc-0712-45a1-89b5-883a6508992c-kube-rbac-proxy-cm") pod "insights-runtime-extractor-kff2k" (UID: "984299cc-0712-45a1-89b5-883a6508992c") : configmap "kube-rbac-proxy" not found
Sep 17 19:06:58.895433 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:06:58.895401    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/984299cc-0712-45a1-89b5-883a6508992c-crio-socket\") pod \"insights-runtime-extractor-kff2k\" (UID: \"984299cc-0712-45a1-89b5-883a6508992c\") " pod="openshift-insights/insights-runtime-extractor-kff2k"
Sep 17 19:06:58.906062 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:06:58.906038    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qkpvq\" (UniqueName: \"kubernetes.io/projected/984299cc-0712-45a1-89b5-883a6508992c-kube-api-access-qkpvq\") pod \"insights-runtime-extractor-kff2k\" (UID: \"984299cc-0712-45a1-89b5-883a6508992c\") " pod="openshift-insights/insights-runtime-extractor-kff2k"
Sep 17 19:06:58.914094 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:06:58.914081    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-4b2jj"
Sep 17 19:06:59.049839 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:06:59.049647    2582 generic.go:356] "Generic (PLEG): container finished" podID="03964cb9-d415-48a8-9a6a-834979450459" containerID="eda4136d9f427259bfb5c32adecde3b3fac79ec15d36704d633c057d6a249f0d" exitCode=0
Sep 17 19:06:59.049978 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:06:59.049708    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-wz7lk" event={"ID":"03964cb9-d415-48a8-9a6a-834979450459","Type":"ContainerDied","Data":"eda4136d9f427259bfb5c32adecde3b3fac79ec15d36704d633c057d6a249f0d"}
Sep 17 19:06:59.256691 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:06:59.256660    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-4b2jj"]
Sep 17 19:06:59.290576 ip-10-0-131-232 kubenswrapper[2582]: W0917 19:06:59.290546    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode6357f17_007b_4c5c_83c8_7a3d1c2c0e85.slice/crio-ae86ca6699213a3f81cc65b580d1304df9938340ee1dfb9571e5a7fdfac84c91 WatchSource:0}: Error finding container ae86ca6699213a3f81cc65b580d1304df9938340ee1dfb9571e5a7fdfac84c91: Status 404 returned error can't find the container with id ae86ca6699213a3f81cc65b580d1304df9938340ee1dfb9571e5a7fdfac84c91
Sep 17 19:06:59.299401 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:06:59.299383    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/413bc45c-0bcb-421e-aa86-c1b75aea7d11-metrics-tls\") pod \"dns-default-67qx4\" (UID: \"413bc45c-0bcb-421e-aa86-c1b75aea7d11\") " pod="openshift-dns/dns-default-67qx4"
Sep 17 19:06:59.299547 ip-10-0-131-232 kubenswrapper[2582]: E0917 19:06:59.299525    2582 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 17 19:06:59.299597 ip-10-0-131-232 kubenswrapper[2582]: E0917 19:06:59.299585    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/413bc45c-0bcb-421e-aa86-c1b75aea7d11-metrics-tls podName:413bc45c-0bcb-421e-aa86-c1b75aea7d11 nodeName:}" failed. No retries permitted until 2026-09-17 19:07:00.299570808 +0000 UTC m=+36.071241873 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/413bc45c-0bcb-421e-aa86-c1b75aea7d11-metrics-tls") pod "dns-default-67qx4" (UID: "413bc45c-0bcb-421e-aa86-c1b75aea7d11") : secret "dns-default-metrics-tls" not found
Sep 17 19:06:59.400563 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:06:59.400506    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/984299cc-0712-45a1-89b5-883a6508992c-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-kff2k\" (UID: \"984299cc-0712-45a1-89b5-883a6508992c\") " pod="openshift-insights/insights-runtime-extractor-kff2k"
Sep 17 19:06:59.400563 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:06:59.400558    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/984299cc-0712-45a1-89b5-883a6508992c-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-kff2k\" (UID: \"984299cc-0712-45a1-89b5-883a6508992c\") " pod="openshift-insights/insights-runtime-extractor-kff2k"
Sep 17 19:06:59.400695 ip-10-0-131-232 kubenswrapper[2582]: E0917 19:06:59.400616    2582 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 17 19:06:59.400695 ip-10-0-131-232 kubenswrapper[2582]: E0917 19:06:59.400648    2582 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 17 19:06:59.400695 ip-10-0-131-232 kubenswrapper[2582]: E0917 19:06:59.400665    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/984299cc-0712-45a1-89b5-883a6508992c-kube-rbac-proxy-cm podName:984299cc-0712-45a1-89b5-883a6508992c nodeName:}" failed. No retries permitted until 2026-09-17 19:07:00.400651598 +0000 UTC m=+36.172322638 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/984299cc-0712-45a1-89b5-883a6508992c-kube-rbac-proxy-cm") pod "insights-runtime-extractor-kff2k" (UID: "984299cc-0712-45a1-89b5-883a6508992c") : configmap "kube-rbac-proxy" not found
Sep 17 19:06:59.400695 ip-10-0-131-232 kubenswrapper[2582]: E0917 19:06:59.400694    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/984299cc-0712-45a1-89b5-883a6508992c-insights-runtime-extractor-tls podName:984299cc-0712-45a1-89b5-883a6508992c nodeName:}" failed. No retries permitted until 2026-09-17 19:07:00.400682127 +0000 UTC m=+36.172353184 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/984299cc-0712-45a1-89b5-883a6508992c-insights-runtime-extractor-tls") pod "insights-runtime-extractor-kff2k" (UID: "984299cc-0712-45a1-89b5-883a6508992c") : secret "insights-runtime-extractor-tls" not found
Sep 17 19:06:59.735612 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:06:59.735539    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-cwtkf_46629fc1-07ce-45b8-9d73-453abfa383b9/dns-node-resolver/0.log"
Sep 17 19:06:59.762921 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:06:59.762893    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-jhjrs"
Sep 17 19:06:59.763090 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:06:59.763068    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-f6kjs"
Sep 17 19:06:59.769339 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:06:59.769050    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Sep 17 19:06:59.769339 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:06:59.769167    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-72wlh\""
Sep 17 19:06:59.769870 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:06:59.769361    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Sep 17 19:06:59.770200 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:06:59.770186    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Sep 17 19:07:00.054281 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:07:00.054198    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-4b2jj" event={"ID":"e6357f17-007b-4c5c-83c8-7a3d1c2c0e85","Type":"ContainerStarted","Data":"ae86ca6699213a3f81cc65b580d1304df9938340ee1dfb9571e5a7fdfac84c91"}
Sep 17 19:07:00.057931 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:07:00.057903    2582 generic.go:356] "Generic (PLEG): container finished" podID="03964cb9-d415-48a8-9a6a-834979450459" containerID="0f2a1ed1bf216ff5c2e9e6509c06a8b2e32164fb2a4425a3c5037c5f7d716ccf" exitCode=0
Sep 17 19:07:00.058081 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:07:00.057948    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-wz7lk" event={"ID":"03964cb9-d415-48a8-9a6a-834979450459","Type":"ContainerDied","Data":"0f2a1ed1bf216ff5c2e9e6509c06a8b2e32164fb2a4425a3c5037c5f7d716ccf"}
Sep 17 19:07:00.281607 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:07:00.281578    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-64g8v_271e6574-9235-474f-b0bc-06ec57117390/node-ca/0.log"
Sep 17 19:07:00.308994 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:07:00.308915    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/413bc45c-0bcb-421e-aa86-c1b75aea7d11-metrics-tls\") pod \"dns-default-67qx4\" (UID: \"413bc45c-0bcb-421e-aa86-c1b75aea7d11\") " pod="openshift-dns/dns-default-67qx4"
Sep 17 19:07:00.309126 ip-10-0-131-232 kubenswrapper[2582]: E0917 19:07:00.309059    2582 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 17 19:07:00.309126 ip-10-0-131-232 kubenswrapper[2582]: E0917 19:07:00.309121    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/413bc45c-0bcb-421e-aa86-c1b75aea7d11-metrics-tls podName:413bc45c-0bcb-421e-aa86-c1b75aea7d11 nodeName:}" failed. No retries permitted until 2026-09-17 19:07:02.309105003 +0000 UTC m=+38.080776043 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/413bc45c-0bcb-421e-aa86-c1b75aea7d11-metrics-tls") pod "dns-default-67qx4" (UID: "413bc45c-0bcb-421e-aa86-c1b75aea7d11") : secret "dns-default-metrics-tls" not found
Sep 17 19:07:00.410204 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:07:00.410163    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/984299cc-0712-45a1-89b5-883a6508992c-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-kff2k\" (UID: \"984299cc-0712-45a1-89b5-883a6508992c\") " pod="openshift-insights/insights-runtime-extractor-kff2k"
Sep 17 19:07:00.410368 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:07:00.410234    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/984299cc-0712-45a1-89b5-883a6508992c-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-kff2k\" (UID: \"984299cc-0712-45a1-89b5-883a6508992c\") " pod="openshift-insights/insights-runtime-extractor-kff2k"
Sep 17 19:07:00.410368 ip-10-0-131-232 kubenswrapper[2582]: E0917 19:07:00.410337    2582 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 17 19:07:00.410368 ip-10-0-131-232 kubenswrapper[2582]: E0917 19:07:00.410354    2582 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 17 19:07:00.410512 ip-10-0-131-232 kubenswrapper[2582]: E0917 19:07:00.410413    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/984299cc-0712-45a1-89b5-883a6508992c-insights-runtime-extractor-tls podName:984299cc-0712-45a1-89b5-883a6508992c nodeName:}" failed. No retries permitted until 2026-09-17 19:07:02.410392631 +0000 UTC m=+38.182063685 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/984299cc-0712-45a1-89b5-883a6508992c-insights-runtime-extractor-tls") pod "insights-runtime-extractor-kff2k" (UID: "984299cc-0712-45a1-89b5-883a6508992c") : secret "insights-runtime-extractor-tls" not found
Sep 17 19:07:00.410512 ip-10-0-131-232 kubenswrapper[2582]: E0917 19:07:00.410429    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/984299cc-0712-45a1-89b5-883a6508992c-kube-rbac-proxy-cm podName:984299cc-0712-45a1-89b5-883a6508992c nodeName:}" failed. No retries permitted until 2026-09-17 19:07:02.410422325 +0000 UTC m=+38.182093365 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/984299cc-0712-45a1-89b5-883a6508992c-kube-rbac-proxy-cm") pod "insights-runtime-extractor-kff2k" (UID: "984299cc-0712-45a1-89b5-883a6508992c") : configmap "kube-rbac-proxy" not found
Sep 17 19:07:01.064843 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:07:01.064789    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-wz7lk" event={"ID":"03964cb9-d415-48a8-9a6a-834979450459","Type":"ContainerStarted","Data":"66b7b3e8e14099e73b2ec1f00dcc62adb0580255e5341cc09359c98f4178d5b6"}
Sep 17 19:07:01.092705 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:07:01.092647    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-wz7lk" podStartSLOduration=11.930645078 podStartE2EDuration="37.092629918s" totalImagesPullingTime="25.16198484s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 19:06:24 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 19:07:01.091223783 +0000 UTC m=+36.862894847" watchObservedRunningTime="2026-09-17 19:07:01.092629918 +0000 UTC m=+36.864300996"
Sep 17 19:07:02.068679 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:07:02.068395    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-4b2jj" event={"ID":"e6357f17-007b-4c5c-83c8-7a3d1c2c0e85","Type":"ContainerStarted","Data":"3c766bf77239207381cc57565362251c5f9cc769f3a35c10e2f45950ad0a354e"}
Sep 17 19:07:02.085008 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:07:02.084975    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-55889f7b47-2w6p2"]
Sep 17 19:07:02.105964 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:07:02.105939    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-55889f7b47-2w6p2"
Sep 17 19:07:02.108940 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:07:02.108919    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"serving-cert\""
Sep 17 19:07:02.109057 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:07:02.108919    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-storage-version-migrator-operator-dockercfg-6tssb\""
Sep 17 19:07:02.109259 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:07:02.109244    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"openshift-service-ca.crt\""
Sep 17 19:07:02.109294 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:07:02.109283    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-root-ca.crt\""
Sep 17 19:07:02.109404 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:07:02.109380    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"config\""
Sep 17 19:07:02.112026 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:07:02.111976    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-4b2jj" podStartSLOduration=1.811196134 podStartE2EDuration="4.11195566s" totalImagesPullingTime="2.300759526s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 19:06:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 19:07:02.103191679 +0000 UTC m=+37.874862741" watchObservedRunningTime="2026-09-17 19:07:02.11195566 +0000 UTC m=+37.883626720"
Sep 17 19:07:02.113766 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:07:02.113013    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-55889f7b47-2w6p2"]
Sep 17 19:07:02.224804 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:07:02.224769    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k57g6\" (UniqueName: \"kubernetes.io/projected/7074d19e-e52e-460a-a702-44130277bea5-kube-api-access-k57g6\") pod \"kube-storage-version-migrator-operator-55889f7b47-2w6p2\" (UID: \"7074d19e-e52e-460a-a702-44130277bea5\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-55889f7b47-2w6p2"
Sep 17 19:07:02.225005 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:07:02.224913    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/7074d19e-e52e-460a-a702-44130277bea5-serving-cert\") pod \"kube-storage-version-migrator-operator-55889f7b47-2w6p2\" (UID: \"7074d19e-e52e-460a-a702-44130277bea5\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-55889f7b47-2w6p2"
Sep 17 19:07:02.225005 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:07:02.224945    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/7074d19e-e52e-460a-a702-44130277bea5-config\") pod \"kube-storage-version-migrator-operator-55889f7b47-2w6p2\" (UID: \"7074d19e-e52e-460a-a702-44130277bea5\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-55889f7b47-2w6p2"
Sep 17 19:07:02.325363 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:07:02.325297    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/7074d19e-e52e-460a-a702-44130277bea5-serving-cert\") pod \"kube-storage-version-migrator-operator-55889f7b47-2w6p2\" (UID: \"7074d19e-e52e-460a-a702-44130277bea5\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-55889f7b47-2w6p2"
Sep 17 19:07:02.325363 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:07:02.325331    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/7074d19e-e52e-460a-a702-44130277bea5-config\") pod \"kube-storage-version-migrator-operator-55889f7b47-2w6p2\" (UID: \"7074d19e-e52e-460a-a702-44130277bea5\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-55889f7b47-2w6p2"
Sep 17 19:07:02.325478 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:07:02.325368    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/413bc45c-0bcb-421e-aa86-c1b75aea7d11-metrics-tls\") pod \"dns-default-67qx4\" (UID: \"413bc45c-0bcb-421e-aa86-c1b75aea7d11\") " pod="openshift-dns/dns-default-67qx4"
Sep 17 19:07:02.325478 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:07:02.325397    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k57g6\" (UniqueName: \"kubernetes.io/projected/7074d19e-e52e-460a-a702-44130277bea5-kube-api-access-k57g6\") pod \"kube-storage-version-migrator-operator-55889f7b47-2w6p2\" (UID: \"7074d19e-e52e-460a-a702-44130277bea5\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-55889f7b47-2w6p2"
Sep 17 19:07:02.325545 ip-10-0-131-232 kubenswrapper[2582]: E0917 19:07:02.325520    2582 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 17 19:07:02.325635 ip-10-0-131-232 kubenswrapper[2582]: E0917 19:07:02.325623    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/413bc45c-0bcb-421e-aa86-c1b75aea7d11-metrics-tls podName:413bc45c-0bcb-421e-aa86-c1b75aea7d11 nodeName:}" failed. No retries permitted until 2026-09-17 19:07:06.325606292 +0000 UTC m=+42.097277333 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/413bc45c-0bcb-421e-aa86-c1b75aea7d11-metrics-tls") pod "dns-default-67qx4" (UID: "413bc45c-0bcb-421e-aa86-c1b75aea7d11") : secret "dns-default-metrics-tls" not found
Sep 17 19:07:02.325942 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:07:02.325926    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/7074d19e-e52e-460a-a702-44130277bea5-config\") pod \"kube-storage-version-migrator-operator-55889f7b47-2w6p2\" (UID: \"7074d19e-e52e-460a-a702-44130277bea5\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-55889f7b47-2w6p2"
Sep 17 19:07:02.328654 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:07:02.328630    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/7074d19e-e52e-460a-a702-44130277bea5-serving-cert\") pod \"kube-storage-version-migrator-operator-55889f7b47-2w6p2\" (UID: \"7074d19e-e52e-460a-a702-44130277bea5\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-55889f7b47-2w6p2"
Sep 17 19:07:02.336779 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:07:02.336752    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-k57g6\" (UniqueName: \"kubernetes.io/projected/7074d19e-e52e-460a-a702-44130277bea5-kube-api-access-k57g6\") pod \"kube-storage-version-migrator-operator-55889f7b47-2w6p2\" (UID: \"7074d19e-e52e-460a-a702-44130277bea5\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-55889f7b47-2w6p2"
Sep 17 19:07:02.415972 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:07:02.415931    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-55889f7b47-2w6p2"
Sep 17 19:07:02.426303 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:07:02.426263    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/984299cc-0712-45a1-89b5-883a6508992c-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-kff2k\" (UID: \"984299cc-0712-45a1-89b5-883a6508992c\") " pod="openshift-insights/insights-runtime-extractor-kff2k"
Sep 17 19:07:02.426425 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:07:02.426341    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/984299cc-0712-45a1-89b5-883a6508992c-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-kff2k\" (UID: \"984299cc-0712-45a1-89b5-883a6508992c\") " pod="openshift-insights/insights-runtime-extractor-kff2k"
Sep 17 19:07:02.426501 ip-10-0-131-232 kubenswrapper[2582]: E0917 19:07:02.426420    2582 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 17 19:07:02.426501 ip-10-0-131-232 kubenswrapper[2582]: E0917 19:07:02.426428    2582 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 17 19:07:02.426501 ip-10-0-131-232 kubenswrapper[2582]: E0917 19:07:02.426484    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/984299cc-0712-45a1-89b5-883a6508992c-insights-runtime-extractor-tls podName:984299cc-0712-45a1-89b5-883a6508992c nodeName:}" failed. No retries permitted until 2026-09-17 19:07:06.426467882 +0000 UTC m=+42.198138922 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/984299cc-0712-45a1-89b5-883a6508992c-insights-runtime-extractor-tls") pod "insights-runtime-extractor-kff2k" (UID: "984299cc-0712-45a1-89b5-883a6508992c") : secret "insights-runtime-extractor-tls" not found
Sep 17 19:07:02.426501 ip-10-0-131-232 kubenswrapper[2582]: E0917 19:07:02.426498    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/984299cc-0712-45a1-89b5-883a6508992c-kube-rbac-proxy-cm podName:984299cc-0712-45a1-89b5-883a6508992c nodeName:}" failed. No retries permitted until 2026-09-17 19:07:06.426492092 +0000 UTC m=+42.198163131 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/984299cc-0712-45a1-89b5-883a6508992c-kube-rbac-proxy-cm") pod "insights-runtime-extractor-kff2k" (UID: "984299cc-0712-45a1-89b5-883a6508992c") : configmap "kube-rbac-proxy" not found
Sep 17 19:07:02.699657 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:07:02.699627    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-55889f7b47-2w6p2"]
Sep 17 19:07:02.702856 ip-10-0-131-232 kubenswrapper[2582]: W0917 19:07:02.702814    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7074d19e_e52e_460a_a702_44130277bea5.slice/crio-d64b3cb882fa4887285caf6129066193c7a2c8f757c2fa3e3c5073485d6da1ec WatchSource:0}: Error finding container d64b3cb882fa4887285caf6129066193c7a2c8f757c2fa3e3c5073485d6da1ec: Status 404 returned error can't find the container with id d64b3cb882fa4887285caf6129066193c7a2c8f757c2fa3e3c5073485d6da1ec
Sep 17 19:07:02.833157 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:07:02.833119    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-network-diagnostics/network-check-source-7cbc45fddc-h5bj4"]
Sep 17 19:07:02.856218 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:07:02.856193    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-7cbc45fddc-h5bj4"]
Sep 17 19:07:02.856345 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:07:02.856284    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-7cbc45fddc-h5bj4"
Sep 17 19:07:02.858888 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:07:02.858863    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-diagnostics-dockercfg-tg764\""
Sep 17 19:07:02.930865 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:07:02.930816    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gkv74\" (UniqueName: \"kubernetes.io/projected/8d44cde2-a7be-41dd-b5aa-f2eda8e86447-kube-api-access-gkv74\") pod \"network-check-source-7cbc45fddc-h5bj4\" (UID: \"8d44cde2-a7be-41dd-b5aa-f2eda8e86447\") " pod="openshift-network-diagnostics/network-check-source-7cbc45fddc-h5bj4"
Sep 17 19:07:03.031897 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:07:03.031786    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gkv74\" (UniqueName: \"kubernetes.io/projected/8d44cde2-a7be-41dd-b5aa-f2eda8e86447-kube-api-access-gkv74\") pod \"network-check-source-7cbc45fddc-h5bj4\" (UID: \"8d44cde2-a7be-41dd-b5aa-f2eda8e86447\") " pod="openshift-network-diagnostics/network-check-source-7cbc45fddc-h5bj4"
Sep 17 19:07:03.041548 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:07:03.041514    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gkv74\" (UniqueName: \"kubernetes.io/projected/8d44cde2-a7be-41dd-b5aa-f2eda8e86447-kube-api-access-gkv74\") pod \"network-check-source-7cbc45fddc-h5bj4\" (UID: \"8d44cde2-a7be-41dd-b5aa-f2eda8e86447\") " pod="openshift-network-diagnostics/network-check-source-7cbc45fddc-h5bj4"
Sep 17 19:07:03.072462 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:07:03.072428    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-55889f7b47-2w6p2" event={"ID":"7074d19e-e52e-460a-a702-44130277bea5","Type":"ContainerStarted","Data":"d64b3cb882fa4887285caf6129066193c7a2c8f757c2fa3e3c5073485d6da1ec"}
Sep 17 19:07:03.083975 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:07:03.083941    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-service-ca-operator/service-ca-operator-57cdc4b768-r2wgc"]
Sep 17 19:07:03.105903 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:07:03.105872    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca-operator/service-ca-operator-57cdc4b768-r2wgc"]
Sep 17 19:07:03.106048 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:07:03.105989    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca-operator/service-ca-operator-57cdc4b768-r2wgc"
Sep 17 19:07:03.108919 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:07:03.108892    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-dockercfg-762r8\""
Sep 17 19:07:03.109268 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:07:03.109254    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"kube-root-ca.crt\""
Sep 17 19:07:03.109382 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:07:03.109360    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-config\""
Sep 17 19:07:03.109606 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:07:03.109593    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"serving-cert\""
Sep 17 19:07:03.109699 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:07:03.109675    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"openshift-service-ca.crt\""
Sep 17 19:07:03.165588 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:07:03.165552    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-7cbc45fddc-h5bj4"
Sep 17 19:07:03.234050 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:07:03.233987    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/839a4dbf-a176-4b22-95ec-00dd4973161a-config\") pod \"service-ca-operator-57cdc4b768-r2wgc\" (UID: \"839a4dbf-a176-4b22-95ec-00dd4973161a\") " pod="openshift-service-ca-operator/service-ca-operator-57cdc4b768-r2wgc"
Sep 17 19:07:03.234194 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:07:03.234049    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qw457\" (UniqueName: \"kubernetes.io/projected/839a4dbf-a176-4b22-95ec-00dd4973161a-kube-api-access-qw457\") pod \"service-ca-operator-57cdc4b768-r2wgc\" (UID: \"839a4dbf-a176-4b22-95ec-00dd4973161a\") " pod="openshift-service-ca-operator/service-ca-operator-57cdc4b768-r2wgc"
Sep 17 19:07:03.234194 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:07:03.234140    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/839a4dbf-a176-4b22-95ec-00dd4973161a-serving-cert\") pod \"service-ca-operator-57cdc4b768-r2wgc\" (UID: \"839a4dbf-a176-4b22-95ec-00dd4973161a\") " pod="openshift-service-ca-operator/service-ca-operator-57cdc4b768-r2wgc"
Sep 17 19:07:03.267020 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:07:03.266986    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-5cb5bc568b-xhl4k"]
Sep 17 19:07:03.301359 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:07:03.301289    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-5cb5bc568b-xhl4k"]
Sep 17 19:07:03.301490 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:07:03.301412    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-5cb5bc568b-xhl4k"
Sep 17 19:07:03.305302 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:07:03.305281    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\""
Sep 17 19:07:03.306200 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:07:03.306178    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\""
Sep 17 19:07:03.306348 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:07:03.306260    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\""
Sep 17 19:07:03.306488 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:07:03.306473    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-4tjxz\""
Sep 17 19:07:03.311705 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:07:03.311688    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\""
Sep 17 19:07:03.335072 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:07:03.335046    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qw457\" (UniqueName: \"kubernetes.io/projected/839a4dbf-a176-4b22-95ec-00dd4973161a-kube-api-access-qw457\") pod \"service-ca-operator-57cdc4b768-r2wgc\" (UID: \"839a4dbf-a176-4b22-95ec-00dd4973161a\") " pod="openshift-service-ca-operator/service-ca-operator-57cdc4b768-r2wgc"
Sep 17 19:07:03.335208 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:07:03.335094    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/839a4dbf-a176-4b22-95ec-00dd4973161a-serving-cert\") pod \"service-ca-operator-57cdc4b768-r2wgc\" (UID: \"839a4dbf-a176-4b22-95ec-00dd4973161a\") " pod="openshift-service-ca-operator/service-ca-operator-57cdc4b768-r2wgc"
Sep 17 19:07:03.335208 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:07:03.335156    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/839a4dbf-a176-4b22-95ec-00dd4973161a-config\") pod \"service-ca-operator-57cdc4b768-r2wgc\" (UID: \"839a4dbf-a176-4b22-95ec-00dd4973161a\") " pod="openshift-service-ca-operator/service-ca-operator-57cdc4b768-r2wgc"
Sep 17 19:07:03.335633 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:07:03.335615    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/839a4dbf-a176-4b22-95ec-00dd4973161a-config\") pod \"service-ca-operator-57cdc4b768-r2wgc\" (UID: \"839a4dbf-a176-4b22-95ec-00dd4973161a\") " pod="openshift-service-ca-operator/service-ca-operator-57cdc4b768-r2wgc"
Sep 17 19:07:03.337814 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:07:03.337797    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/839a4dbf-a176-4b22-95ec-00dd4973161a-serving-cert\") pod \"service-ca-operator-57cdc4b768-r2wgc\" (UID: \"839a4dbf-a176-4b22-95ec-00dd4973161a\") " pod="openshift-service-ca-operator/service-ca-operator-57cdc4b768-r2wgc"
Sep 17 19:07:03.344236 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:07:03.344218    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qw457\" (UniqueName: \"kubernetes.io/projected/839a4dbf-a176-4b22-95ec-00dd4973161a-kube-api-access-qw457\") pod \"service-ca-operator-57cdc4b768-r2wgc\" (UID: \"839a4dbf-a176-4b22-95ec-00dd4973161a\") " pod="openshift-service-ca-operator/service-ca-operator-57cdc4b768-r2wgc"
Sep 17 19:07:03.414523 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:07:03.414490    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca-operator/service-ca-operator-57cdc4b768-r2wgc"
Sep 17 19:07:03.435966 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:07:03.435942    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/d7ce149d-9931-4d71-be5b-b668f59766e1-registry-tls\") pod \"image-registry-5cb5bc568b-xhl4k\" (UID: \"d7ce149d-9931-4d71-be5b-b668f59766e1\") " pod="openshift-image-registry/image-registry-5cb5bc568b-xhl4k"
Sep 17 19:07:03.436096 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:07:03.435970    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/d7ce149d-9931-4d71-be5b-b668f59766e1-bound-sa-token\") pod \"image-registry-5cb5bc568b-xhl4k\" (UID: \"d7ce149d-9931-4d71-be5b-b668f59766e1\") " pod="openshift-image-registry/image-registry-5cb5bc568b-xhl4k"
Sep 17 19:07:03.436096 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:07:03.436032    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fvw9m\" (UniqueName: \"kubernetes.io/projected/d7ce149d-9931-4d71-be5b-b668f59766e1-kube-api-access-fvw9m\") pod \"image-registry-5cb5bc568b-xhl4k\" (UID: \"d7ce149d-9931-4d71-be5b-b668f59766e1\") " pod="openshift-image-registry/image-registry-5cb5bc568b-xhl4k"
Sep 17 19:07:03.436178 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:07:03.436109    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/d7ce149d-9931-4d71-be5b-b668f59766e1-ca-trust-extracted\") pod \"image-registry-5cb5bc568b-xhl4k\" (UID: \"d7ce149d-9931-4d71-be5b-b668f59766e1\") " pod="openshift-image-registry/image-registry-5cb5bc568b-xhl4k"
Sep 17 19:07:03.436178 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:07:03.436140    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/d7ce149d-9931-4d71-be5b-b668f59766e1-trusted-ca\") pod \"image-registry-5cb5bc568b-xhl4k\" (UID: \"d7ce149d-9931-4d71-be5b-b668f59766e1\") " pod="openshift-image-registry/image-registry-5cb5bc568b-xhl4k"
Sep 17 19:07:03.436178 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:07:03.436164    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/d7ce149d-9931-4d71-be5b-b668f59766e1-installation-pull-secrets\") pod \"image-registry-5cb5bc568b-xhl4k\" (UID: \"d7ce149d-9931-4d71-be5b-b668f59766e1\") " pod="openshift-image-registry/image-registry-5cb5bc568b-xhl4k"
Sep 17 19:07:03.436287 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:07:03.436200    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/d7ce149d-9931-4d71-be5b-b668f59766e1-registry-certificates\") pod \"image-registry-5cb5bc568b-xhl4k\" (UID: \"d7ce149d-9931-4d71-be5b-b668f59766e1\") " pod="openshift-image-registry/image-registry-5cb5bc568b-xhl4k"
Sep 17 19:07:03.436287 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:07:03.436244    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/d7ce149d-9931-4d71-be5b-b668f59766e1-image-registry-private-configuration\") pod \"image-registry-5cb5bc568b-xhl4k\" (UID: \"d7ce149d-9931-4d71-be5b-b668f59766e1\") " pod="openshift-image-registry/image-registry-5cb5bc568b-xhl4k"
Sep 17 19:07:03.459708 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:07:03.459673    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-7cbc45fddc-h5bj4"]
Sep 17 19:07:03.466337 ip-10-0-131-232 kubenswrapper[2582]: W0917 19:07:03.466304    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8d44cde2_a7be_41dd_b5aa_f2eda8e86447.slice/crio-a431ffb0f595d0237c16da2f47562b97a9ff97e44ef04b7240ab81f417a931d1 WatchSource:0}: Error finding container a431ffb0f595d0237c16da2f47562b97a9ff97e44ef04b7240ab81f417a931d1: Status 404 returned error can't find the container with id a431ffb0f595d0237c16da2f47562b97a9ff97e44ef04b7240ab81f417a931d1
Sep 17 19:07:03.537518 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:07:03.537481    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/d7ce149d-9931-4d71-be5b-b668f59766e1-ca-trust-extracted\") pod \"image-registry-5cb5bc568b-xhl4k\" (UID: \"d7ce149d-9931-4d71-be5b-b668f59766e1\") " pod="openshift-image-registry/image-registry-5cb5bc568b-xhl4k"
Sep 17 19:07:03.537518 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:07:03.537520    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/d7ce149d-9931-4d71-be5b-b668f59766e1-trusted-ca\") pod \"image-registry-5cb5bc568b-xhl4k\" (UID: \"d7ce149d-9931-4d71-be5b-b668f59766e1\") " pod="openshift-image-registry/image-registry-5cb5bc568b-xhl4k"
Sep 17 19:07:03.537746 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:07:03.537540    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/d7ce149d-9931-4d71-be5b-b668f59766e1-installation-pull-secrets\") pod \"image-registry-5cb5bc568b-xhl4k\" (UID: \"d7ce149d-9931-4d71-be5b-b668f59766e1\") " pod="openshift-image-registry/image-registry-5cb5bc568b-xhl4k"
Sep 17 19:07:03.537746 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:07:03.537562    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/d7ce149d-9931-4d71-be5b-b668f59766e1-registry-certificates\") pod \"image-registry-5cb5bc568b-xhl4k\" (UID: \"d7ce149d-9931-4d71-be5b-b668f59766e1\") " pod="openshift-image-registry/image-registry-5cb5bc568b-xhl4k"
Sep 17 19:07:03.537746 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:07:03.537581    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/d7ce149d-9931-4d71-be5b-b668f59766e1-image-registry-private-configuration\") pod \"image-registry-5cb5bc568b-xhl4k\" (UID: \"d7ce149d-9931-4d71-be5b-b668f59766e1\") " pod="openshift-image-registry/image-registry-5cb5bc568b-xhl4k"
Sep 17 19:07:03.537746 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:07:03.537613    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/d7ce149d-9931-4d71-be5b-b668f59766e1-registry-tls\") pod \"image-registry-5cb5bc568b-xhl4k\" (UID: \"d7ce149d-9931-4d71-be5b-b668f59766e1\") " pod="openshift-image-registry/image-registry-5cb5bc568b-xhl4k"
Sep 17 19:07:03.537746 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:07:03.537628    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/d7ce149d-9931-4d71-be5b-b668f59766e1-bound-sa-token\") pod \"image-registry-5cb5bc568b-xhl4k\" (UID: \"d7ce149d-9931-4d71-be5b-b668f59766e1\") " pod="openshift-image-registry/image-registry-5cb5bc568b-xhl4k"
Sep 17 19:07:03.537746 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:07:03.537666    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fvw9m\" (UniqueName: \"kubernetes.io/projected/d7ce149d-9931-4d71-be5b-b668f59766e1-kube-api-access-fvw9m\") pod \"image-registry-5cb5bc568b-xhl4k\" (UID: \"d7ce149d-9931-4d71-be5b-b668f59766e1\") " pod="openshift-image-registry/image-registry-5cb5bc568b-xhl4k"
Sep 17 19:07:03.538016 ip-10-0-131-232 kubenswrapper[2582]: E0917 19:07:03.537774    2582 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 17 19:07:03.538061 ip-10-0-131-232 kubenswrapper[2582]: E0917 19:07:03.538023    2582 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-5cb5bc568b-xhl4k: secret "image-registry-tls" not found
Sep 17 19:07:03.538113 ip-10-0-131-232 kubenswrapper[2582]: E0917 19:07:03.538106    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/d7ce149d-9931-4d71-be5b-b668f59766e1-registry-tls podName:d7ce149d-9931-4d71-be5b-b668f59766e1 nodeName:}" failed. No retries permitted until 2026-09-17 19:07:04.038082627 +0000 UTC m=+39.809753690 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/d7ce149d-9931-4d71-be5b-b668f59766e1-registry-tls") pod "image-registry-5cb5bc568b-xhl4k" (UID: "d7ce149d-9931-4d71-be5b-b668f59766e1") : secret "image-registry-tls" not found
Sep 17 19:07:03.538318 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:07:03.538290    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/d7ce149d-9931-4d71-be5b-b668f59766e1-ca-trust-extracted\") pod \"image-registry-5cb5bc568b-xhl4k\" (UID: \"d7ce149d-9931-4d71-be5b-b668f59766e1\") " pod="openshift-image-registry/image-registry-5cb5bc568b-xhl4k"
Sep 17 19:07:03.538710 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:07:03.538692    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/d7ce149d-9931-4d71-be5b-b668f59766e1-trusted-ca\") pod \"image-registry-5cb5bc568b-xhl4k\" (UID: \"d7ce149d-9931-4d71-be5b-b668f59766e1\") " pod="openshift-image-registry/image-registry-5cb5bc568b-xhl4k"
Sep 17 19:07:03.538759 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:07:03.538645    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/d7ce149d-9931-4d71-be5b-b668f59766e1-registry-certificates\") pod \"image-registry-5cb5bc568b-xhl4k\" (UID: \"d7ce149d-9931-4d71-be5b-b668f59766e1\") " pod="openshift-image-registry/image-registry-5cb5bc568b-xhl4k"
Sep 17 19:07:03.540469 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:07:03.540449    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/d7ce149d-9931-4d71-be5b-b668f59766e1-image-registry-private-configuration\") pod \"image-registry-5cb5bc568b-xhl4k\" (UID: \"d7ce149d-9931-4d71-be5b-b668f59766e1\") " pod="openshift-image-registry/image-registry-5cb5bc568b-xhl4k"
Sep 17 19:07:03.540517 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:07:03.540474    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/d7ce149d-9931-4d71-be5b-b668f59766e1-installation-pull-secrets\") pod \"image-registry-5cb5bc568b-xhl4k\" (UID: \"d7ce149d-9931-4d71-be5b-b668f59766e1\") " pod="openshift-image-registry/image-registry-5cb5bc568b-xhl4k"
Sep 17 19:07:03.547050 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:07:03.547026    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/d7ce149d-9931-4d71-be5b-b668f59766e1-bound-sa-token\") pod \"image-registry-5cb5bc568b-xhl4k\" (UID: \"d7ce149d-9931-4d71-be5b-b668f59766e1\") " pod="openshift-image-registry/image-registry-5cb5bc568b-xhl4k"
Sep 17 19:07:03.547778 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:07:03.547761    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fvw9m\" (UniqueName: \"kubernetes.io/projected/d7ce149d-9931-4d71-be5b-b668f59766e1-kube-api-access-fvw9m\") pod \"image-registry-5cb5bc568b-xhl4k\" (UID: \"d7ce149d-9931-4d71-be5b-b668f59766e1\") " pod="openshift-image-registry/image-registry-5cb5bc568b-xhl4k"
Sep 17 19:07:03.719426 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:07:03.719394    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca-operator/service-ca-operator-57cdc4b768-r2wgc"]
Sep 17 19:07:03.723046 ip-10-0-131-232 kubenswrapper[2582]: W0917 19:07:03.723016    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod839a4dbf_a176_4b22_95ec_00dd4973161a.slice/crio-0ec20fbae4bd51cacf7adc228234193b056b8b3ecbd9161d436767868384daf4 WatchSource:0}: Error finding container 0ec20fbae4bd51cacf7adc228234193b056b8b3ecbd9161d436767868384daf4: Status 404 returned error can't find the container with id 0ec20fbae4bd51cacf7adc228234193b056b8b3ecbd9161d436767868384daf4
Sep 17 19:07:04.042071 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:07:04.041984    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/d7ce149d-9931-4d71-be5b-b668f59766e1-registry-tls\") pod \"image-registry-5cb5bc568b-xhl4k\" (UID: \"d7ce149d-9931-4d71-be5b-b668f59766e1\") " pod="openshift-image-registry/image-registry-5cb5bc568b-xhl4k"
Sep 17 19:07:04.042247 ip-10-0-131-232 kubenswrapper[2582]: E0917 19:07:04.042174    2582 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 17 19:07:04.042247 ip-10-0-131-232 kubenswrapper[2582]: E0917 19:07:04.042198    2582 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-5cb5bc568b-xhl4k: secret "image-registry-tls" not found
Sep 17 19:07:04.042355 ip-10-0-131-232 kubenswrapper[2582]: E0917 19:07:04.042269    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/d7ce149d-9931-4d71-be5b-b668f59766e1-registry-tls podName:d7ce149d-9931-4d71-be5b-b668f59766e1 nodeName:}" failed. No retries permitted until 2026-09-17 19:07:05.042246176 +0000 UTC m=+40.813917217 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/d7ce149d-9931-4d71-be5b-b668f59766e1-registry-tls") pod "image-registry-5cb5bc568b-xhl4k" (UID: "d7ce149d-9931-4d71-be5b-b668f59766e1") : secret "image-registry-tls" not found
Sep 17 19:07:04.077609 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:07:04.077565    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-57cdc4b768-r2wgc" event={"ID":"839a4dbf-a176-4b22-95ec-00dd4973161a","Type":"ContainerStarted","Data":"0ec20fbae4bd51cacf7adc228234193b056b8b3ecbd9161d436767868384daf4"}
Sep 17 19:07:04.079104 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:07:04.079069    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-7cbc45fddc-h5bj4" event={"ID":"8d44cde2-a7be-41dd-b5aa-f2eda8e86447","Type":"ContainerStarted","Data":"a431ffb0f595d0237c16da2f47562b97a9ff97e44ef04b7240ab81f417a931d1"}
Sep 17 19:07:05.050412 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:07:05.050373    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/d7ce149d-9931-4d71-be5b-b668f59766e1-registry-tls\") pod \"image-registry-5cb5bc568b-xhl4k\" (UID: \"d7ce149d-9931-4d71-be5b-b668f59766e1\") " pod="openshift-image-registry/image-registry-5cb5bc568b-xhl4k"
Sep 17 19:07:05.050611 ip-10-0-131-232 kubenswrapper[2582]: E0917 19:07:05.050583    2582 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 17 19:07:05.050611 ip-10-0-131-232 kubenswrapper[2582]: E0917 19:07:05.050608    2582 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-5cb5bc568b-xhl4k: secret "image-registry-tls" not found
Sep 17 19:07:05.050721 ip-10-0-131-232 kubenswrapper[2582]: E0917 19:07:05.050678    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/d7ce149d-9931-4d71-be5b-b668f59766e1-registry-tls podName:d7ce149d-9931-4d71-be5b-b668f59766e1 nodeName:}" failed. No retries permitted until 2026-09-17 19:07:07.050655407 +0000 UTC m=+42.822326453 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/d7ce149d-9931-4d71-be5b-b668f59766e1-registry-tls") pod "image-registry-5cb5bc568b-xhl4k" (UID: "d7ce149d-9931-4d71-be5b-b668f59766e1") : secret "image-registry-tls" not found
Sep 17 19:07:06.360701 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:07:06.360665    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/413bc45c-0bcb-421e-aa86-c1b75aea7d11-metrics-tls\") pod \"dns-default-67qx4\" (UID: \"413bc45c-0bcb-421e-aa86-c1b75aea7d11\") " pod="openshift-dns/dns-default-67qx4"
Sep 17 19:07:06.361165 ip-10-0-131-232 kubenswrapper[2582]: E0917 19:07:06.360855    2582 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 17 19:07:06.361165 ip-10-0-131-232 kubenswrapper[2582]: E0917 19:07:06.360932    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/413bc45c-0bcb-421e-aa86-c1b75aea7d11-metrics-tls podName:413bc45c-0bcb-421e-aa86-c1b75aea7d11 nodeName:}" failed. No retries permitted until 2026-09-17 19:07:14.360917058 +0000 UTC m=+50.132588098 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/413bc45c-0bcb-421e-aa86-c1b75aea7d11-metrics-tls") pod "dns-default-67qx4" (UID: "413bc45c-0bcb-421e-aa86-c1b75aea7d11") : secret "dns-default-metrics-tls" not found
Sep 17 19:07:06.461815 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:07:06.461712    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/984299cc-0712-45a1-89b5-883a6508992c-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-kff2k\" (UID: \"984299cc-0712-45a1-89b5-883a6508992c\") " pod="openshift-insights/insights-runtime-extractor-kff2k"
Sep 17 19:07:06.461989 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:07:06.461837    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/984299cc-0712-45a1-89b5-883a6508992c-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-kff2k\" (UID: \"984299cc-0712-45a1-89b5-883a6508992c\") " pod="openshift-insights/insights-runtime-extractor-kff2k"
Sep 17 19:07:06.461989 ip-10-0-131-232 kubenswrapper[2582]: E0917 19:07:06.461890    2582 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 17 19:07:06.461989 ip-10-0-131-232 kubenswrapper[2582]: E0917 19:07:06.461961    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/984299cc-0712-45a1-89b5-883a6508992c-insights-runtime-extractor-tls podName:984299cc-0712-45a1-89b5-883a6508992c nodeName:}" failed. No retries permitted until 2026-09-17 19:07:14.461941952 +0000 UTC m=+50.233613015 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/984299cc-0712-45a1-89b5-883a6508992c-insights-runtime-extractor-tls") pod "insights-runtime-extractor-kff2k" (UID: "984299cc-0712-45a1-89b5-883a6508992c") : secret "insights-runtime-extractor-tls" not found
Sep 17 19:07:06.462447 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:07:06.462427    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/984299cc-0712-45a1-89b5-883a6508992c-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-kff2k\" (UID: \"984299cc-0712-45a1-89b5-883a6508992c\") " pod="openshift-insights/insights-runtime-extractor-kff2k"
Sep 17 19:07:07.066238 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:07:07.066188    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/d7ce149d-9931-4d71-be5b-b668f59766e1-registry-tls\") pod \"image-registry-5cb5bc568b-xhl4k\" (UID: \"d7ce149d-9931-4d71-be5b-b668f59766e1\") " pod="openshift-image-registry/image-registry-5cb5bc568b-xhl4k"
Sep 17 19:07:07.066437 ip-10-0-131-232 kubenswrapper[2582]: E0917 19:07:07.066371    2582 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 17 19:07:07.066437 ip-10-0-131-232 kubenswrapper[2582]: E0917 19:07:07.066394    2582 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-5cb5bc568b-xhl4k: secret "image-registry-tls" not found
Sep 17 19:07:07.066543 ip-10-0-131-232 kubenswrapper[2582]: E0917 19:07:07.066458    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/d7ce149d-9931-4d71-be5b-b668f59766e1-registry-tls podName:d7ce149d-9931-4d71-be5b-b668f59766e1 nodeName:}" failed. No retries permitted until 2026-09-17 19:07:11.066435976 +0000 UTC m=+46.838107016 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/d7ce149d-9931-4d71-be5b-b668f59766e1-registry-tls") pod "image-registry-5cb5bc568b-xhl4k" (UID: "d7ce149d-9931-4d71-be5b-b668f59766e1") : secret "image-registry-tls" not found
Sep 17 19:07:08.094787 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:07:08.094751    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-55889f7b47-2w6p2" event={"ID":"7074d19e-e52e-460a-a702-44130277bea5","Type":"ContainerStarted","Data":"d2071077a8d8f354fe20decec283293c66c60336a2f6dd1bb958f48a36ccda98"}
Sep 17 19:07:08.096439 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:07:08.096414    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-7cbc45fddc-h5bj4" event={"ID":"8d44cde2-a7be-41dd-b5aa-f2eda8e86447","Type":"ContainerStarted","Data":"a735fa301e101c4c3f66801c2b418891514d89aa7cd494f3cfe8656958a10824"}
Sep 17 19:07:08.098069 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:07:08.098047    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-57cdc4b768-r2wgc" event={"ID":"839a4dbf-a176-4b22-95ec-00dd4973161a","Type":"ContainerStarted","Data":"3c1395cb3398cb0a49d50b8a064edcc1ea55096fb68877bbc55d25c2fd989595"}
Sep 17 19:07:08.113664 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:07:08.113611    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-55889f7b47-2w6p2" podStartSLOduration=1.455606621 podStartE2EDuration="6.113593087s" totalImagesPullingTime="4.657986466s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 19:07:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 19:07:08.112934816 +0000 UTC m=+43.884605879" watchObservedRunningTime="2026-09-17 19:07:08.113593087 +0000 UTC m=+43.885264151"
Sep 17 19:07:08.131050 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:07:08.130994    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-source-7cbc45fddc-h5bj4" podStartSLOduration=2.206676399 podStartE2EDuration="6.13097437s" totalImagesPullingTime="3.924297971s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 19:07:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 19:07:08.13049255 +0000 UTC m=+43.902163611" watchObservedRunningTime="2026-09-17 19:07:08.13097437 +0000 UTC m=+43.902645432"
Sep 17 19:07:08.150095 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:07:08.150035    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-service-ca-operator/service-ca-operator-57cdc4b768-r2wgc" podStartSLOduration=1.494281658 podStartE2EDuration="5.150019868s" totalImagesPullingTime="3.65573821s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 19:07:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 19:07:08.148974333 +0000 UTC m=+43.920645396" watchObservedRunningTime="2026-09-17 19:07:08.150019868 +0000 UTC m=+43.921690930"
Sep 17 19:07:11.096164 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:07:11.096125    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/d7ce149d-9931-4d71-be5b-b668f59766e1-registry-tls\") pod \"image-registry-5cb5bc568b-xhl4k\" (UID: \"d7ce149d-9931-4d71-be5b-b668f59766e1\") " pod="openshift-image-registry/image-registry-5cb5bc568b-xhl4k"
Sep 17 19:07:11.096584 ip-10-0-131-232 kubenswrapper[2582]: E0917 19:07:11.096278    2582 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 17 19:07:11.096584 ip-10-0-131-232 kubenswrapper[2582]: E0917 19:07:11.096299    2582 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-5cb5bc568b-xhl4k: secret "image-registry-tls" not found
Sep 17 19:07:11.096584 ip-10-0-131-232 kubenswrapper[2582]: E0917 19:07:11.096358    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/d7ce149d-9931-4d71-be5b-b668f59766e1-registry-tls podName:d7ce149d-9931-4d71-be5b-b668f59766e1 nodeName:}" failed. No retries permitted until 2026-09-17 19:07:19.096341621 +0000 UTC m=+54.868012663 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/d7ce149d-9931-4d71-be5b-b668f59766e1-registry-tls") pod "image-registry-5cb5bc568b-xhl4k" (UID: "d7ce149d-9931-4d71-be5b-b668f59766e1") : secret "image-registry-tls" not found
Sep 17 19:07:12.106705 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:07:12.106669    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-service-ca/service-ca-55f6fc7b86-86t4c"]
Sep 17 19:07:12.138879 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:07:12.138814    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-55f6fc7b86-86t4c"]
Sep 17 19:07:12.139055 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:07:12.138947    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-55f6fc7b86-86t4c"
Sep 17 19:07:12.141728 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:07:12.141699    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"openshift-service-ca.crt\""
Sep 17 19:07:12.141728 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:07:12.141699    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"service-ca-dockercfg-bmpp5\""
Sep 17 19:07:12.142749 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:07:12.142728    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"signing-cabundle\""
Sep 17 19:07:12.142891 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:07:12.142796    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"signing-key\""
Sep 17 19:07:12.142891 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:07:12.142811    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"kube-root-ca.crt\""
Sep 17 19:07:12.203536 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:07:12.203502    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/f9874c04-e34f-4422-876e-af7ef38c6682-signing-cabundle\") pod \"service-ca-55f6fc7b86-86t4c\" (UID: \"f9874c04-e34f-4422-876e-af7ef38c6682\") " pod="openshift-service-ca/service-ca-55f6fc7b86-86t4c"
Sep 17 19:07:12.203704 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:07:12.203593    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k87gn\" (UniqueName: \"kubernetes.io/projected/f9874c04-e34f-4422-876e-af7ef38c6682-kube-api-access-k87gn\") pod \"service-ca-55f6fc7b86-86t4c\" (UID: \"f9874c04-e34f-4422-876e-af7ef38c6682\") " pod="openshift-service-ca/service-ca-55f6fc7b86-86t4c"
Sep 17 19:07:12.203704 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:07:12.203646    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/f9874c04-e34f-4422-876e-af7ef38c6682-signing-key\") pod \"service-ca-55f6fc7b86-86t4c\" (UID: \"f9874c04-e34f-4422-876e-af7ef38c6682\") " pod="openshift-service-ca/service-ca-55f6fc7b86-86t4c"
Sep 17 19:07:12.304191 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:07:12.304151    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k87gn\" (UniqueName: \"kubernetes.io/projected/f9874c04-e34f-4422-876e-af7ef38c6682-kube-api-access-k87gn\") pod \"service-ca-55f6fc7b86-86t4c\" (UID: \"f9874c04-e34f-4422-876e-af7ef38c6682\") " pod="openshift-service-ca/service-ca-55f6fc7b86-86t4c"
Sep 17 19:07:12.304354 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:07:12.304226    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/f9874c04-e34f-4422-876e-af7ef38c6682-signing-key\") pod \"service-ca-55f6fc7b86-86t4c\" (UID: \"f9874c04-e34f-4422-876e-af7ef38c6682\") " pod="openshift-service-ca/service-ca-55f6fc7b86-86t4c"
Sep 17 19:07:12.304354 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:07:12.304255    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/f9874c04-e34f-4422-876e-af7ef38c6682-signing-cabundle\") pod \"service-ca-55f6fc7b86-86t4c\" (UID: \"f9874c04-e34f-4422-876e-af7ef38c6682\") " pod="openshift-service-ca/service-ca-55f6fc7b86-86t4c"
Sep 17 19:07:12.305112 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:07:12.305096    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/f9874c04-e34f-4422-876e-af7ef38c6682-signing-cabundle\") pod \"service-ca-55f6fc7b86-86t4c\" (UID: \"f9874c04-e34f-4422-876e-af7ef38c6682\") " pod="openshift-service-ca/service-ca-55f6fc7b86-86t4c"
Sep 17 19:07:12.307030 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:07:12.307011    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/f9874c04-e34f-4422-876e-af7ef38c6682-signing-key\") pod \"service-ca-55f6fc7b86-86t4c\" (UID: \"f9874c04-e34f-4422-876e-af7ef38c6682\") " pod="openshift-service-ca/service-ca-55f6fc7b86-86t4c"
Sep 17 19:07:12.317516 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:07:12.317486    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-k87gn\" (UniqueName: \"kubernetes.io/projected/f9874c04-e34f-4422-876e-af7ef38c6682-kube-api-access-k87gn\") pod \"service-ca-55f6fc7b86-86t4c\" (UID: \"f9874c04-e34f-4422-876e-af7ef38c6682\") " pod="openshift-service-ca/service-ca-55f6fc7b86-86t4c"
Sep 17 19:07:12.447759 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:07:12.447726    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-55f6fc7b86-86t4c"
Sep 17 19:07:12.738694 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:07:12.738632    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-55f6fc7b86-86t4c"]
Sep 17 19:07:12.742121 ip-10-0-131-232 kubenswrapper[2582]: W0917 19:07:12.742089    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf9874c04_e34f_4422_876e_af7ef38c6682.slice/crio-6c70b8760364ed87dceb7fb966cec7443f9a9c06d6460902383b8d491ff2d145 WatchSource:0}: Error finding container 6c70b8760364ed87dceb7fb966cec7443f9a9c06d6460902383b8d491ff2d145: Status 404 returned error can't find the container with id 6c70b8760364ed87dceb7fb966cec7443f9a9c06d6460902383b8d491ff2d145
Sep 17 19:07:13.113795 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:07:13.113705    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-55f6fc7b86-86t4c" event={"ID":"f9874c04-e34f-4422-876e-af7ef38c6682","Type":"ContainerStarted","Data":"5ddb435a6b7cd3bb4af185922ca4f49adf2878c8b19fbd3a9294eb8fc1d1d4a9"}
Sep 17 19:07:13.114188 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:07:13.113803    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-55f6fc7b86-86t4c" event={"ID":"f9874c04-e34f-4422-876e-af7ef38c6682","Type":"ContainerStarted","Data":"6c70b8760364ed87dceb7fb966cec7443f9a9c06d6460902383b8d491ff2d145"}
Sep 17 19:07:13.136325 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:07:13.136263    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-service-ca/service-ca-55f6fc7b86-86t4c" podStartSLOduration=1.13624123 podStartE2EDuration="1.13624123s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 19:07:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 19:07:13.135958168 +0000 UTC m=+48.907629231" watchObservedRunningTime="2026-09-17 19:07:13.13624123 +0000 UTC m=+48.907912293"
Sep 17 19:07:14.421994 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:07:14.421956    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/413bc45c-0bcb-421e-aa86-c1b75aea7d11-metrics-tls\") pod \"dns-default-67qx4\" (UID: \"413bc45c-0bcb-421e-aa86-c1b75aea7d11\") " pod="openshift-dns/dns-default-67qx4"
Sep 17 19:07:14.422440 ip-10-0-131-232 kubenswrapper[2582]: E0917 19:07:14.422111    2582 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 17 19:07:14.422440 ip-10-0-131-232 kubenswrapper[2582]: E0917 19:07:14.422189    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/413bc45c-0bcb-421e-aa86-c1b75aea7d11-metrics-tls podName:413bc45c-0bcb-421e-aa86-c1b75aea7d11 nodeName:}" failed. No retries permitted until 2026-09-17 19:07:30.422167988 +0000 UTC m=+66.193839033 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/413bc45c-0bcb-421e-aa86-c1b75aea7d11-metrics-tls") pod "dns-default-67qx4" (UID: "413bc45c-0bcb-421e-aa86-c1b75aea7d11") : secret "dns-default-metrics-tls" not found
Sep 17 19:07:14.523139 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:07:14.523092    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/984299cc-0712-45a1-89b5-883a6508992c-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-kff2k\" (UID: \"984299cc-0712-45a1-89b5-883a6508992c\") " pod="openshift-insights/insights-runtime-extractor-kff2k"
Sep 17 19:07:14.523308 ip-10-0-131-232 kubenswrapper[2582]: E0917 19:07:14.523292    2582 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 17 19:07:14.523368 ip-10-0-131-232 kubenswrapper[2582]: E0917 19:07:14.523358    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/984299cc-0712-45a1-89b5-883a6508992c-insights-runtime-extractor-tls podName:984299cc-0712-45a1-89b5-883a6508992c nodeName:}" failed. No retries permitted until 2026-09-17 19:07:30.523339343 +0000 UTC m=+66.295010390 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/984299cc-0712-45a1-89b5-883a6508992c-insights-runtime-extractor-tls") pod "insights-runtime-extractor-kff2k" (UID: "984299cc-0712-45a1-89b5-883a6508992c") : secret "insights-runtime-extractor-tls" not found
Sep 17 19:07:19.158016 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:07:19.157978    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/d7ce149d-9931-4d71-be5b-b668f59766e1-registry-tls\") pod \"image-registry-5cb5bc568b-xhl4k\" (UID: \"d7ce149d-9931-4d71-be5b-b668f59766e1\") " pod="openshift-image-registry/image-registry-5cb5bc568b-xhl4k"
Sep 17 19:07:19.160435 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:07:19.160405    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/d7ce149d-9931-4d71-be5b-b668f59766e1-registry-tls\") pod \"image-registry-5cb5bc568b-xhl4k\" (UID: \"d7ce149d-9931-4d71-be5b-b668f59766e1\") " pod="openshift-image-registry/image-registry-5cb5bc568b-xhl4k"
Sep 17 19:07:19.211624 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:07:19.211589    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-5cb5bc568b-xhl4k"
Sep 17 19:07:19.493547 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:07:19.493467    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-5cb5bc568b-xhl4k"]
Sep 17 19:07:19.496734 ip-10-0-131-232 kubenswrapper[2582]: W0917 19:07:19.496709    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd7ce149d_9931_4d71_be5b_b668f59766e1.slice/crio-dcefff316cbb10151c6dde524e15db69f4072e801705b021c4eaeaefc73c3a51 WatchSource:0}: Error finding container dcefff316cbb10151c6dde524e15db69f4072e801705b021c4eaeaefc73c3a51: Status 404 returned error can't find the container with id dcefff316cbb10151c6dde524e15db69f4072e801705b021c4eaeaefc73c3a51
Sep 17 19:07:20.136108 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:07:20.136070    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-5cb5bc568b-xhl4k" event={"ID":"d7ce149d-9931-4d71-be5b-b668f59766e1","Type":"ContainerStarted","Data":"663da08b46f6833f52fc6a946ce6be8201f6185140b66783aec78490b8fa4ad1"}
Sep 17 19:07:20.136287 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:07:20.136126    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-5cb5bc568b-xhl4k" event={"ID":"d7ce149d-9931-4d71-be5b-b668f59766e1","Type":"ContainerStarted","Data":"dcefff316cbb10151c6dde524e15db69f4072e801705b021c4eaeaefc73c3a51"}
Sep 17 19:07:20.136287 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:07:20.136211    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-5cb5bc568b-xhl4k"
Sep 17 19:07:20.163508 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:07:20.162972    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/image-registry-5cb5bc568b-xhl4k" podStartSLOduration=17.162950951 podStartE2EDuration="17.162950951s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 19:07:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 19:07:20.162324182 +0000 UTC m=+55.933995245" watchObservedRunningTime="2026-09-17 19:07:20.162950951 +0000 UTC m=+55.934622014"
Sep 17 19:07:24.044608 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:07:24.044580    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-kdm8s"
Sep 17 19:07:29.437871 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:07:29.437809    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/7cd887c0-2cc8-4363-b553-8d2b4e10f158-original-pull-secret\") pod \"global-pull-secret-syncer-f6kjs\" (UID: \"7cd887c0-2cc8-4363-b553-8d2b4e10f158\") " pod="kube-system/global-pull-secret-syncer-f6kjs"
Sep 17 19:07:29.440744 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:07:29.440726    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Sep 17 19:07:29.451175 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:07:29.451160    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/7cd887c0-2cc8-4363-b553-8d2b4e10f158-original-pull-secret\") pod \"global-pull-secret-syncer-f6kjs\" (UID: \"7cd887c0-2cc8-4363-b553-8d2b4e10f158\") " pod="kube-system/global-pull-secret-syncer-f6kjs"
Sep 17 19:07:29.481442 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:07:29.481420    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-f6kjs"
Sep 17 19:07:29.538755 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:07:29.538732    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cc1b159b-69e7-404d-9a25-bcf0ff8148c5-metrics-certs\") pod \"network-metrics-daemon-hj5r7\" (UID: \"cc1b159b-69e7-404d-9a25-bcf0ff8148c5\") " pod="openshift-multus/network-metrics-daemon-hj5r7"
Sep 17 19:07:29.541364 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:07:29.541339    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Sep 17 19:07:29.551666 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:07:29.551647    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cc1b159b-69e7-404d-9a25-bcf0ff8148c5-metrics-certs\") pod \"network-metrics-daemon-hj5r7\" (UID: \"cc1b159b-69e7-404d-9a25-bcf0ff8148c5\") " pod="openshift-multus/network-metrics-daemon-hj5r7"
Sep 17 19:07:29.639076 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:07:29.639046    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kftwn\" (UniqueName: \"kubernetes.io/projected/07daf4ff-9d55-4476-aefd-df39d86849a6-kube-api-access-kftwn\") pod \"network-check-target-jhjrs\" (UID: \"07daf4ff-9d55-4476-aefd-df39d86849a6\") " pod="openshift-network-diagnostics/network-check-target-jhjrs"
Sep 17 19:07:29.641483 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:07:29.641464    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kftwn\" (UniqueName: \"kubernetes.io/projected/07daf4ff-9d55-4476-aefd-df39d86849a6-kube-api-access-kftwn\") pod \"network-check-target-jhjrs\" (UID: \"07daf4ff-9d55-4476-aefd-df39d86849a6\") " pod="openshift-network-diagnostics/network-check-target-jhjrs"
Sep 17 19:07:29.673994 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:07:29.673972    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-lsqsv\""
Sep 17 19:07:29.682094 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:07:29.682073    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-hj5r7"
Sep 17 19:07:29.779059 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:07:29.779038    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-72wlh\""
Sep 17 19:07:29.786512 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:07:29.786495    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-jhjrs"
Sep 17 19:07:29.799743 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:07:29.799725    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-f6kjs"]
Sep 17 19:07:29.804815 ip-10-0-131-232 kubenswrapper[2582]: W0917 19:07:29.804792    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7cd887c0_2cc8_4363_b553_8d2b4e10f158.slice/crio-bf9b7fb01792022a6860b148b5cbf8f688bedc7f81d922722338e598b206359b WatchSource:0}: Error finding container bf9b7fb01792022a6860b148b5cbf8f688bedc7f81d922722338e598b206359b: Status 404 returned error can't find the container with id bf9b7fb01792022a6860b148b5cbf8f688bedc7f81d922722338e598b206359b
Sep 17 19:07:29.951967 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:07:29.951902    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-hj5r7"]
Sep 17 19:07:29.954649 ip-10-0-131-232 kubenswrapper[2582]: W0917 19:07:29.954627    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcc1b159b_69e7_404d_9a25_bcf0ff8148c5.slice/crio-04870815cc6696fde32d259ad441cc0dd55e024a03c9d38d9766112dad86fc66 WatchSource:0}: Error finding container 04870815cc6696fde32d259ad441cc0dd55e024a03c9d38d9766112dad86fc66: Status 404 returned error can't find the container with id 04870815cc6696fde32d259ad441cc0dd55e024a03c9d38d9766112dad86fc66
Sep 17 19:07:30.059535 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:07:30.059505    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-jhjrs"]
Sep 17 19:07:30.062769 ip-10-0-131-232 kubenswrapper[2582]: W0917 19:07:30.062738    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod07daf4ff_9d55_4476_aefd_df39d86849a6.slice/crio-c8c75c0175497df92d38f1dababb6c9358de568c0244704885cffe7a58497e15 WatchSource:0}: Error finding container c8c75c0175497df92d38f1dababb6c9358de568c0244704885cffe7a58497e15: Status 404 returned error can't find the container with id c8c75c0175497df92d38f1dababb6c9358de568c0244704885cffe7a58497e15
Sep 17 19:07:30.167003 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:07:30.166968    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-f6kjs" event={"ID":"7cd887c0-2cc8-4363-b553-8d2b4e10f158","Type":"ContainerStarted","Data":"bf9b7fb01792022a6860b148b5cbf8f688bedc7f81d922722338e598b206359b"}
Sep 17 19:07:30.168408 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:07:30.168381    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-jhjrs" event={"ID":"07daf4ff-9d55-4476-aefd-df39d86849a6","Type":"ContainerStarted","Data":"8da42774a9768b224c7394ed877be2363d68c86cf0476848d6b5836f0e877416"}
Sep 17 19:07:30.168532 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:07:30.168427    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-jhjrs" event={"ID":"07daf4ff-9d55-4476-aefd-df39d86849a6","Type":"ContainerStarted","Data":"c8c75c0175497df92d38f1dababb6c9358de568c0244704885cffe7a58497e15"}
Sep 17 19:07:30.168532 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:07:30.168478    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-jhjrs"
Sep 17 19:07:30.169571 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:07:30.169551    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-hj5r7" event={"ID":"cc1b159b-69e7-404d-9a25-bcf0ff8148c5","Type":"ContainerStarted","Data":"04870815cc6696fde32d259ad441cc0dd55e024a03c9d38d9766112dad86fc66"}
Sep 17 19:07:30.444695 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:07:30.444664    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/413bc45c-0bcb-421e-aa86-c1b75aea7d11-metrics-tls\") pod \"dns-default-67qx4\" (UID: \"413bc45c-0bcb-421e-aa86-c1b75aea7d11\") " pod="openshift-dns/dns-default-67qx4"
Sep 17 19:07:30.448285 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:07:30.448246    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/413bc45c-0bcb-421e-aa86-c1b75aea7d11-metrics-tls\") pod \"dns-default-67qx4\" (UID: \"413bc45c-0bcb-421e-aa86-c1b75aea7d11\") " pod="openshift-dns/dns-default-67qx4"
Sep 17 19:07:30.545908 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:07:30.545859    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/984299cc-0712-45a1-89b5-883a6508992c-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-kff2k\" (UID: \"984299cc-0712-45a1-89b5-883a6508992c\") " pod="openshift-insights/insights-runtime-extractor-kff2k"
Sep 17 19:07:30.548953 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:07:30.548920    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/984299cc-0712-45a1-89b5-883a6508992c-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-kff2k\" (UID: \"984299cc-0712-45a1-89b5-883a6508992c\") " pod="openshift-insights/insights-runtime-extractor-kff2k"
Sep 17 19:07:30.730545 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:07:30.730464    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-rj77f\""
Sep 17 19:07:30.737229 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:07:30.737203    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-67qx4"
Sep 17 19:07:30.821555 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:07:30.821528    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-vsnnc\""
Sep 17 19:07:30.828580 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:07:30.828383    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-kff2k"
Sep 17 19:07:31.179032 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:07:31.178967    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-hj5r7" event={"ID":"cc1b159b-69e7-404d-9a25-bcf0ff8148c5","Type":"ContainerStarted","Data":"142518680d12f6be0cb996bc5ee1a6f5e0dc264933351f74fc54a85b9f9d9c99"}
Sep 17 19:07:31.179231 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:07:31.179092    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-hj5r7" event={"ID":"cc1b159b-69e7-404d-9a25-bcf0ff8148c5","Type":"ContainerStarted","Data":"46129787ebcfe89fde5803a04af1d4834835cb431dd53b647dc420d2a70520df"}
Sep 17 19:07:31.199429 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:07:31.199373    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-jhjrs" podStartSLOduration=67.199352624 podStartE2EDuration="1m7.199352624s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 19:06:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 19:07:30.204876832 +0000 UTC m=+65.976547894" watchObservedRunningTime="2026-09-17 19:07:31.199352624 +0000 UTC m=+66.971023689"
Sep 17 19:07:31.200250 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:07:31.200208    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-hj5r7" podStartSLOduration=66.218368166 podStartE2EDuration="1m7.200195704s" totalImagesPullingTime="981.827538ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 19:06:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 19:07:31.197926614 +0000 UTC m=+66.969597677" watchObservedRunningTime="2026-09-17 19:07:31.200195704 +0000 UTC m=+66.971866765"
Sep 17 19:07:31.207160 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:07:31.207129    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-67qx4"]
Sep 17 19:07:31.209058 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:07:31.209035    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-kff2k"]
Sep 17 19:07:31.210196 ip-10-0-131-232 kubenswrapper[2582]: W0917 19:07:31.210175    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod413bc45c_0bcb_421e_aa86_c1b75aea7d11.slice/crio-462c5af8c3622cb61a1aa19dff39061909232e4be6ab17f986db7f0564627540 WatchSource:0}: Error finding container 462c5af8c3622cb61a1aa19dff39061909232e4be6ab17f986db7f0564627540: Status 404 returned error can't find the container with id 462c5af8c3622cb61a1aa19dff39061909232e4be6ab17f986db7f0564627540
Sep 17 19:07:31.211107 ip-10-0-131-232 kubenswrapper[2582]: W0917 19:07:31.211073    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod984299cc_0712_45a1_89b5_883a6508992c.slice/crio-ecbb6ca9bed978165336f8513aad76563224584047bc719e34f03447e46bbec1 WatchSource:0}: Error finding container ecbb6ca9bed978165336f8513aad76563224584047bc719e34f03447e46bbec1: Status 404 returned error can't find the container with id ecbb6ca9bed978165336f8513aad76563224584047bc719e34f03447e46bbec1
Sep 17 19:07:32.186055 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:07:32.185966    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-kff2k" event={"ID":"984299cc-0712-45a1-89b5-883a6508992c","Type":"ContainerStarted","Data":"f4740551e12d5b5e4565b7ef4fb5997c19bc3d2b7aaf9d34be5de0c350921807"}
Sep 17 19:07:32.186055 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:07:32.186038    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-kff2k" event={"ID":"984299cc-0712-45a1-89b5-883a6508992c","Type":"ContainerStarted","Data":"b24906d4aa749df9231f81a09177f95587f75bd81fe3dcfa01e1d1637caccc8d"}
Sep 17 19:07:32.186572 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:07:32.186062    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-kff2k" event={"ID":"984299cc-0712-45a1-89b5-883a6508992c","Type":"ContainerStarted","Data":"ecbb6ca9bed978165336f8513aad76563224584047bc719e34f03447e46bbec1"}
Sep 17 19:07:32.187280 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:07:32.187252    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-67qx4" event={"ID":"413bc45c-0bcb-421e-aa86-c1b75aea7d11","Type":"ContainerStarted","Data":"462c5af8c3622cb61a1aa19dff39061909232e4be6ab17f986db7f0564627540"}
Sep 17 19:07:33.415679 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:07:33.414449    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5b6fb5464b-h4vdq"]
Sep 17 19:07:33.417950 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:07:33.417928    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5b6fb5464b-h4vdq"
Sep 17 19:07:33.422244 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:07:33.422220    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"openshift-service-ca.crt\""
Sep 17 19:07:33.423301 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:07:33.423277    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"kube-root-ca.crt\""
Sep 17 19:07:33.423411 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:07:33.423300    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-hub-kubeconfig\""
Sep 17 19:07:33.423411 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:07:33.423332    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-open-cluster-management.io-proxy-agent-signer-client-cert\""
Sep 17 19:07:33.423916 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:07:33.423899    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"open-cluster-management-image-pull-credentials\""
Sep 17 19:07:33.425407 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:07:33.425387    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-ca\""
Sep 17 19:07:33.427691 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:07:33.427669    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-service-proxy-server-certificates\""
Sep 17 19:07:33.446141 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:07:33.446108    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5b6fb5464b-h4vdq"]
Sep 17 19:07:33.472968 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:07:33.472599    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-5cb5bc568b-xhl4k"]
Sep 17 19:07:33.571374 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:07:33.571209    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c6svt\" (UniqueName: \"kubernetes.io/projected/3b569255-4548-40bf-992e-2a663457d33b-kube-api-access-c6svt\") pod \"cluster-proxy-proxy-agent-5b6fb5464b-h4vdq\" (UID: \"3b569255-4548-40bf-992e-2a663457d33b\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5b6fb5464b-h4vdq"
Sep 17 19:07:33.571374 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:07:33.571261    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/3b569255-4548-40bf-992e-2a663457d33b-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-5b6fb5464b-h4vdq\" (UID: \"3b569255-4548-40bf-992e-2a663457d33b\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5b6fb5464b-h4vdq"
Sep 17 19:07:33.571374 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:07:33.571294    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca\" (UniqueName: \"kubernetes.io/secret/3b569255-4548-40bf-992e-2a663457d33b-ca\") pod \"cluster-proxy-proxy-agent-5b6fb5464b-h4vdq\" (UID: \"3b569255-4548-40bf-992e-2a663457d33b\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5b6fb5464b-h4vdq"
Sep 17 19:07:33.571374 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:07:33.571364    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/3b569255-4548-40bf-992e-2a663457d33b-additional-service-ca\") pod \"cluster-proxy-proxy-agent-5b6fb5464b-h4vdq\" (UID: \"3b569255-4548-40bf-992e-2a663457d33b\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5b6fb5464b-h4vdq"
Sep 17 19:07:33.571731 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:07:33.571394    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/3b569255-4548-40bf-992e-2a663457d33b-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-5b6fb5464b-h4vdq\" (UID: \"3b569255-4548-40bf-992e-2a663457d33b\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5b6fb5464b-h4vdq"
Sep 17 19:07:33.571731 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:07:33.571469    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub\" (UniqueName: \"kubernetes.io/secret/3b569255-4548-40bf-992e-2a663457d33b-hub\") pod \"cluster-proxy-proxy-agent-5b6fb5464b-h4vdq\" (UID: \"3b569255-4548-40bf-992e-2a663457d33b\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5b6fb5464b-h4vdq"
Sep 17 19:07:33.672400 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:07:33.672324    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/3b569255-4548-40bf-992e-2a663457d33b-additional-service-ca\") pod \"cluster-proxy-proxy-agent-5b6fb5464b-h4vdq\" (UID: \"3b569255-4548-40bf-992e-2a663457d33b\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5b6fb5464b-h4vdq"
Sep 17 19:07:33.672400 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:07:33.672378    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/3b569255-4548-40bf-992e-2a663457d33b-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-5b6fb5464b-h4vdq\" (UID: \"3b569255-4548-40bf-992e-2a663457d33b\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5b6fb5464b-h4vdq"
Sep 17 19:07:33.672592 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:07:33.672410    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub\" (UniqueName: \"kubernetes.io/secret/3b569255-4548-40bf-992e-2a663457d33b-hub\") pod \"cluster-proxy-proxy-agent-5b6fb5464b-h4vdq\" (UID: \"3b569255-4548-40bf-992e-2a663457d33b\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5b6fb5464b-h4vdq"
Sep 17 19:07:33.672592 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:07:33.672450    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c6svt\" (UniqueName: \"kubernetes.io/projected/3b569255-4548-40bf-992e-2a663457d33b-kube-api-access-c6svt\") pod \"cluster-proxy-proxy-agent-5b6fb5464b-h4vdq\" (UID: \"3b569255-4548-40bf-992e-2a663457d33b\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5b6fb5464b-h4vdq"
Sep 17 19:07:33.672592 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:07:33.672483    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/3b569255-4548-40bf-992e-2a663457d33b-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-5b6fb5464b-h4vdq\" (UID: \"3b569255-4548-40bf-992e-2a663457d33b\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5b6fb5464b-h4vdq"
Sep 17 19:07:33.672592 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:07:33.672508    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca\" (UniqueName: \"kubernetes.io/secret/3b569255-4548-40bf-992e-2a663457d33b-ca\") pod \"cluster-proxy-proxy-agent-5b6fb5464b-h4vdq\" (UID: \"3b569255-4548-40bf-992e-2a663457d33b\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5b6fb5464b-h4vdq"
Sep 17 19:07:33.673247 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:07:33.673222    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/3b569255-4548-40bf-992e-2a663457d33b-additional-service-ca\") pod \"cluster-proxy-proxy-agent-5b6fb5464b-h4vdq\" (UID: \"3b569255-4548-40bf-992e-2a663457d33b\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5b6fb5464b-h4vdq"
Sep 17 19:07:33.675458 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:07:33.675436    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/3b569255-4548-40bf-992e-2a663457d33b-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-5b6fb5464b-h4vdq\" (UID: \"3b569255-4548-40bf-992e-2a663457d33b\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5b6fb5464b-h4vdq"
Sep 17 19:07:33.675653 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:07:33.675633    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub\" (UniqueName: \"kubernetes.io/secret/3b569255-4548-40bf-992e-2a663457d33b-hub\") pod \"cluster-proxy-proxy-agent-5b6fb5464b-h4vdq\" (UID: \"3b569255-4548-40bf-992e-2a663457d33b\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5b6fb5464b-h4vdq"
Sep 17 19:07:33.675727 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:07:33.675673    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca\" (UniqueName: \"kubernetes.io/secret/3b569255-4548-40bf-992e-2a663457d33b-ca\") pod \"cluster-proxy-proxy-agent-5b6fb5464b-h4vdq\" (UID: \"3b569255-4548-40bf-992e-2a663457d33b\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5b6fb5464b-h4vdq"
Sep 17 19:07:33.675802 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:07:33.675785    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/3b569255-4548-40bf-992e-2a663457d33b-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-5b6fb5464b-h4vdq\" (UID: \"3b569255-4548-40bf-992e-2a663457d33b\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5b6fb5464b-h4vdq"
Sep 17 19:07:33.688350 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:07:33.688329    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-c6svt\" (UniqueName: \"kubernetes.io/projected/3b569255-4548-40bf-992e-2a663457d33b-kube-api-access-c6svt\") pod \"cluster-proxy-proxy-agent-5b6fb5464b-h4vdq\" (UID: \"3b569255-4548-40bf-992e-2a663457d33b\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5b6fb5464b-h4vdq"
Sep 17 19:07:33.737834 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:07:33.737788    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5b6fb5464b-h4vdq"
Sep 17 19:07:34.485259 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:07:34.485172    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5b6fb5464b-h4vdq"]
Sep 17 19:07:34.489328 ip-10-0-131-232 kubenswrapper[2582]: W0917 19:07:34.489288    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod3b569255_4548_40bf_992e_2a663457d33b.slice/crio-2f27a6e76838ca52094d4802991e4619b81c39065354f5ced5d2cf9fac926b8a WatchSource:0}: Error finding container 2f27a6e76838ca52094d4802991e4619b81c39065354f5ced5d2cf9fac926b8a: Status 404 returned error can't find the container with id 2f27a6e76838ca52094d4802991e4619b81c39065354f5ced5d2cf9fac926b8a
Sep 17 19:07:35.198767 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:07:35.198730    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-f6kjs" event={"ID":"7cd887c0-2cc8-4363-b553-8d2b4e10f158","Type":"ContainerStarted","Data":"dee0de7958c566a04c655f305f9eadcdc284077ce0e7ad2ea5e0761a43546a51"}
Sep 17 19:07:35.200605 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:07:35.200572    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-67qx4" event={"ID":"413bc45c-0bcb-421e-aa86-c1b75aea7d11","Type":"ContainerStarted","Data":"1cc7604325ef6657e6de2b1be021009ce52e303f4a6ac3a0a9b3dadf0b2cd036"}
Sep 17 19:07:35.200739 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:07:35.200628    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-67qx4"
Sep 17 19:07:35.200739 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:07:35.200653    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-67qx4" event={"ID":"413bc45c-0bcb-421e-aa86-c1b75aea7d11","Type":"ContainerStarted","Data":"eca15fa767060fd2f8449d9e087524ada418ebda3cfe288fb339071d1edd0f41"}
Sep 17 19:07:35.201726 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:07:35.201700    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5b6fb5464b-h4vdq" event={"ID":"3b569255-4548-40bf-992e-2a663457d33b","Type":"ContainerStarted","Data":"2f27a6e76838ca52094d4802991e4619b81c39065354f5ced5d2cf9fac926b8a"}
Sep 17 19:07:35.217716 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:07:35.217676    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-f6kjs" podStartSLOduration=66.826293336 podStartE2EDuration="1m11.217664745s" totalImagesPullingTime="4.391371409s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 19:06:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 19:07:35.215903657 +0000 UTC m=+70.987574720" watchObservedRunningTime="2026-09-17 19:07:35.217664745 +0000 UTC m=+70.989335807"
Sep 17 19:07:35.235070 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:07:35.234860    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-67qx4" podStartSLOduration=34.262737282 podStartE2EDuration="37.234848864s" totalImagesPullingTime="2.972111582s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 19:06:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 19:07:35.233893274 +0000 UTC m=+71.005564336" watchObservedRunningTime="2026-09-17 19:07:35.234848864 +0000 UTC m=+71.006519907"
Sep 17 19:07:36.209532 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:07:36.209491    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-kff2k" event={"ID":"984299cc-0712-45a1-89b5-883a6508992c","Type":"ContainerStarted","Data":"110cef0a17af9e47472a431847328b9ab8d482b797e026d5a984480a974ae8f2"}
Sep 17 19:07:36.230326 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:07:36.230259    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-kff2k" podStartSLOduration=34.248239126 podStartE2EDuration="38.230237389s" totalImagesPullingTime="3.981998263s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 19:06:58 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 19:07:36.228329401 +0000 UTC m=+72.000000466" watchObservedRunningTime="2026-09-17 19:07:36.230237389 +0000 UTC m=+72.001908454"
Sep 17 19:07:40.465556 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:07:40.465512    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-rrzmt"]
Sep 17 19:07:40.467983 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:07:40.467960    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-rrzmt"
Sep 17 19:07:40.470494 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:07:40.470473    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\""
Sep 17 19:07:40.472282 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:07:40.472261    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
Sep 17 19:07:40.474006 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:07:40.472573    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\""
Sep 17 19:07:40.474006 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:07:40.472923    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
Sep 17 19:07:40.474006 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:07:40.473094    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\""
Sep 17 19:07:40.474006 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:07:40.473277    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-x7nzw\""
Sep 17 19:07:40.474006 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:07:40.473519    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\""
Sep 17 19:07:40.524736 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:07:40.524702    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/53864523-604f-40d9-be99-cab48397a2f5-node-exporter-tls\") pod \"node-exporter-rrzmt\" (UID: \"53864523-604f-40d9-be99-cab48397a2f5\") " pod="openshift-monitoring/node-exporter-rrzmt"
Sep 17 19:07:40.524945 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:07:40.524790    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/53864523-604f-40d9-be99-cab48397a2f5-metrics-client-ca\") pod \"node-exporter-rrzmt\" (UID: \"53864523-604f-40d9-be99-cab48397a2f5\") " pod="openshift-monitoring/node-exporter-rrzmt"
Sep 17 19:07:40.524945 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:07:40.524841    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5wt62\" (UniqueName: \"kubernetes.io/projected/53864523-604f-40d9-be99-cab48397a2f5-kube-api-access-5wt62\") pod \"node-exporter-rrzmt\" (UID: \"53864523-604f-40d9-be99-cab48397a2f5\") " pod="openshift-monitoring/node-exporter-rrzmt"
Sep 17 19:07:40.524945 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:07:40.524880    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/53864523-604f-40d9-be99-cab48397a2f5-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-rrzmt\" (UID: \"53864523-604f-40d9-be99-cab48397a2f5\") " pod="openshift-monitoring/node-exporter-rrzmt"
Sep 17 19:07:40.525117 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:07:40.524964    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/53864523-604f-40d9-be99-cab48397a2f5-node-exporter-wtmp\") pod \"node-exporter-rrzmt\" (UID: \"53864523-604f-40d9-be99-cab48397a2f5\") " pod="openshift-monitoring/node-exporter-rrzmt"
Sep 17 19:07:40.525117 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:07:40.524991    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/53864523-604f-40d9-be99-cab48397a2f5-node-exporter-textfile\") pod \"node-exporter-rrzmt\" (UID: \"53864523-604f-40d9-be99-cab48397a2f5\") " pod="openshift-monitoring/node-exporter-rrzmt"
Sep 17 19:07:40.525117 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:07:40.525022    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/53864523-604f-40d9-be99-cab48397a2f5-root\") pod \"node-exporter-rrzmt\" (UID: \"53864523-604f-40d9-be99-cab48397a2f5\") " pod="openshift-monitoring/node-exporter-rrzmt"
Sep 17 19:07:40.525117 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:07:40.525047    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/53864523-604f-40d9-be99-cab48397a2f5-node-exporter-accelerators-collector-config\") pod \"node-exporter-rrzmt\" (UID: \"53864523-604f-40d9-be99-cab48397a2f5\") " pod="openshift-monitoring/node-exporter-rrzmt"
Sep 17 19:07:40.525117 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:07:40.525092    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/53864523-604f-40d9-be99-cab48397a2f5-sys\") pod \"node-exporter-rrzmt\" (UID: \"53864523-604f-40d9-be99-cab48397a2f5\") " pod="openshift-monitoring/node-exporter-rrzmt"
Sep 17 19:07:40.626240 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:07:40.626195    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/53864523-604f-40d9-be99-cab48397a2f5-node-exporter-wtmp\") pod \"node-exporter-rrzmt\" (UID: \"53864523-604f-40d9-be99-cab48397a2f5\") " pod="openshift-monitoring/node-exporter-rrzmt"
Sep 17 19:07:40.626412 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:07:40.626253    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/53864523-604f-40d9-be99-cab48397a2f5-node-exporter-textfile\") pod \"node-exporter-rrzmt\" (UID: \"53864523-604f-40d9-be99-cab48397a2f5\") " pod="openshift-monitoring/node-exporter-rrzmt"
Sep 17 19:07:40.626412 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:07:40.626284    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/53864523-604f-40d9-be99-cab48397a2f5-root\") pod \"node-exporter-rrzmt\" (UID: \"53864523-604f-40d9-be99-cab48397a2f5\") " pod="openshift-monitoring/node-exporter-rrzmt"
Sep 17 19:07:40.626412 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:07:40.626308    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/53864523-604f-40d9-be99-cab48397a2f5-node-exporter-accelerators-collector-config\") pod \"node-exporter-rrzmt\" (UID: \"53864523-604f-40d9-be99-cab48397a2f5\") " pod="openshift-monitoring/node-exporter-rrzmt"
Sep 17 19:07:40.626412 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:07:40.626341    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/53864523-604f-40d9-be99-cab48397a2f5-sys\") pod \"node-exporter-rrzmt\" (UID: \"53864523-604f-40d9-be99-cab48397a2f5\") " pod="openshift-monitoring/node-exporter-rrzmt"
Sep 17 19:07:40.626412 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:07:40.626363    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/53864523-604f-40d9-be99-cab48397a2f5-node-exporter-tls\") pod \"node-exporter-rrzmt\" (UID: \"53864523-604f-40d9-be99-cab48397a2f5\") " pod="openshift-monitoring/node-exporter-rrzmt"
Sep 17 19:07:40.626412 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:07:40.626404    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/53864523-604f-40d9-be99-cab48397a2f5-metrics-client-ca\") pod \"node-exporter-rrzmt\" (UID: \"53864523-604f-40d9-be99-cab48397a2f5\") " pod="openshift-monitoring/node-exporter-rrzmt"
Sep 17 19:07:40.626743 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:07:40.626431    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5wt62\" (UniqueName: \"kubernetes.io/projected/53864523-604f-40d9-be99-cab48397a2f5-kube-api-access-5wt62\") pod \"node-exporter-rrzmt\" (UID: \"53864523-604f-40d9-be99-cab48397a2f5\") " pod="openshift-monitoring/node-exporter-rrzmt"
Sep 17 19:07:40.626743 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:07:40.626465    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/53864523-604f-40d9-be99-cab48397a2f5-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-rrzmt\" (UID: \"53864523-604f-40d9-be99-cab48397a2f5\") " pod="openshift-monitoring/node-exporter-rrzmt"
Sep 17 19:07:40.627283 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:07:40.626940    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/53864523-604f-40d9-be99-cab48397a2f5-root\") pod \"node-exporter-rrzmt\" (UID: \"53864523-604f-40d9-be99-cab48397a2f5\") " pod="openshift-monitoring/node-exporter-rrzmt"
Sep 17 19:07:40.627283 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:07:40.627158    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/53864523-604f-40d9-be99-cab48397a2f5-node-exporter-textfile\") pod \"node-exporter-rrzmt\" (UID: \"53864523-604f-40d9-be99-cab48397a2f5\") " pod="openshift-monitoring/node-exporter-rrzmt"
Sep 17 19:07:40.627516 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:07:40.627296    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/53864523-604f-40d9-be99-cab48397a2f5-node-exporter-wtmp\") pod \"node-exporter-rrzmt\" (UID: \"53864523-604f-40d9-be99-cab48397a2f5\") " pod="openshift-monitoring/node-exporter-rrzmt"
Sep 17 19:07:40.627516 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:07:40.627439    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/53864523-604f-40d9-be99-cab48397a2f5-node-exporter-accelerators-collector-config\") pod \"node-exporter-rrzmt\" (UID: \"53864523-604f-40d9-be99-cab48397a2f5\") " pod="openshift-monitoring/node-exporter-rrzmt"
Sep 17 19:07:40.627748 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:07:40.627727    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/53864523-604f-40d9-be99-cab48397a2f5-metrics-client-ca\") pod \"node-exporter-rrzmt\" (UID: \"53864523-604f-40d9-be99-cab48397a2f5\") " pod="openshift-monitoring/node-exporter-rrzmt"
Sep 17 19:07:40.627856 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:07:40.627795    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/53864523-604f-40d9-be99-cab48397a2f5-sys\") pod \"node-exporter-rrzmt\" (UID: \"53864523-604f-40d9-be99-cab48397a2f5\") " pod="openshift-monitoring/node-exporter-rrzmt"
Sep 17 19:07:40.629675 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:07:40.629654    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/53864523-604f-40d9-be99-cab48397a2f5-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-rrzmt\" (UID: \"53864523-604f-40d9-be99-cab48397a2f5\") " pod="openshift-monitoring/node-exporter-rrzmt"
Sep 17 19:07:40.639059 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:07:40.639032    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/53864523-604f-40d9-be99-cab48397a2f5-node-exporter-tls\") pod \"node-exporter-rrzmt\" (UID: \"53864523-604f-40d9-be99-cab48397a2f5\") " pod="openshift-monitoring/node-exporter-rrzmt"
Sep 17 19:07:40.654878 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:07:40.654852    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5wt62\" (UniqueName: \"kubernetes.io/projected/53864523-604f-40d9-be99-cab48397a2f5-kube-api-access-5wt62\") pod \"node-exporter-rrzmt\" (UID: \"53864523-604f-40d9-be99-cab48397a2f5\") " pod="openshift-monitoring/node-exporter-rrzmt"
Sep 17 19:07:40.781873 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:07:40.781781    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-rrzmt"
Sep 17 19:07:41.131449 ip-10-0-131-232 kubenswrapper[2582]: W0917 19:07:41.131418    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod53864523_604f_40d9_be99_cab48397a2f5.slice/crio-f030f02da0e3d525aa9ea8fa01248262f00a57b438158cd5131b486979bcc11f WatchSource:0}: Error finding container f030f02da0e3d525aa9ea8fa01248262f00a57b438158cd5131b486979bcc11f: Status 404 returned error can't find the container with id f030f02da0e3d525aa9ea8fa01248262f00a57b438158cd5131b486979bcc11f
Sep 17 19:07:41.230504 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:07:41.230445    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5b6fb5464b-h4vdq" event={"ID":"3b569255-4548-40bf-992e-2a663457d33b","Type":"ContainerStarted","Data":"4380c6fd35b19d0f9fed3434f0e5064bb5eafe597f72a8c7f3c50d34a39def2f"}
Sep 17 19:07:41.231805 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:07:41.231778    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-rrzmt" event={"ID":"53864523-604f-40d9-be99-cab48397a2f5","Type":"ContainerStarted","Data":"f030f02da0e3d525aa9ea8fa01248262f00a57b438158cd5131b486979bcc11f"}
Sep 17 19:07:42.237333 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:07:42.237294    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5b6fb5464b-h4vdq" event={"ID":"3b569255-4548-40bf-992e-2a663457d33b","Type":"ContainerStarted","Data":"9314f9155f46718f34b75801d29aa0e267578a6659c328d57fb319db979f1522"}
Sep 17 19:07:42.237855 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:07:42.237348    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5b6fb5464b-h4vdq" event={"ID":"3b569255-4548-40bf-992e-2a663457d33b","Type":"ContainerStarted","Data":"3e8d1ca1ffa07397a1dda9b3130a3f86c40c31d8e8ebf83eb23f28c551e6ebed"}
Sep 17 19:07:42.238965 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:07:42.238937    2582 generic.go:356] "Generic (PLEG): container finished" podID="53864523-604f-40d9-be99-cab48397a2f5" containerID="c16ced55fecee110b060a8f1c2a4054c459b8ead9656c74b2bf5992f4e5b0e80" exitCode=0
Sep 17 19:07:42.239059 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:07:42.238994    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-rrzmt" event={"ID":"53864523-604f-40d9-be99-cab48397a2f5","Type":"ContainerDied","Data":"c16ced55fecee110b060a8f1c2a4054c459b8ead9656c74b2bf5992f4e5b0e80"}
Sep 17 19:07:42.260034 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:07:42.259955    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5b6fb5464b-h4vdq" podStartSLOduration=2.586990239 podStartE2EDuration="9.259938797s" totalImagesPullingTime="6.672948558s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 19:07:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 19:07:42.257877414 +0000 UTC m=+78.029548476" watchObservedRunningTime="2026-09-17 19:07:42.259938797 +0000 UTC m=+78.031609859"
Sep 17 19:07:43.244073 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:07:43.244039    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-rrzmt" event={"ID":"53864523-604f-40d9-be99-cab48397a2f5","Type":"ContainerStarted","Data":"0d279eb9dc3cccb124f40273ccc68f7804b7be5e63221898f7f9367a6caf074a"}
Sep 17 19:07:43.244451 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:07:43.244093    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-rrzmt" event={"ID":"53864523-604f-40d9-be99-cab48397a2f5","Type":"ContainerStarted","Data":"d0faeb42442e514b6c1247a24d604480085bbe0bb24e569645fa9a179603a200"}
Sep 17 19:07:43.266877 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:07:43.266808    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-rrzmt" podStartSLOduration=2.334845808 podStartE2EDuration="3.266794345s" totalImagesPullingTime="858.964028ms" totalInitContainerRuntime="72.984509ms" isStatefulPod=true podCreationTimestamp="2026-09-17 19:07:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 19:07:43.26524582 +0000 UTC m=+79.036916883" watchObservedRunningTime="2026-09-17 19:07:43.266794345 +0000 UTC m=+79.038465442"
Sep 17 19:07:43.478525 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:07:43.478498    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-5cb5bc568b-xhl4k"
Sep 17 19:07:45.212695 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:07:45.212664    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-67qx4"
Sep 17 19:07:58.496565 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:07:58.496417    2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-image-registry/image-registry-5cb5bc568b-xhl4k" podUID="d7ce149d-9931-4d71-be5b-b668f59766e1" containerName="registry" containerID="cri-o://663da08b46f6833f52fc6a946ce6be8201f6185140b66783aec78490b8fa4ad1" gracePeriod=30
Sep 17 19:07:58.737940 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:07:58.737918    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-5cb5bc568b-xhl4k"
Sep 17 19:07:58.756662 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:07:58.756601    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/d7ce149d-9931-4d71-be5b-b668f59766e1-registry-tls\") pod \"d7ce149d-9931-4d71-be5b-b668f59766e1\" (UID: \"d7ce149d-9931-4d71-be5b-b668f59766e1\") "
Sep 17 19:07:58.756662 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:07:58.756631    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/d7ce149d-9931-4d71-be5b-b668f59766e1-bound-sa-token\") pod \"d7ce149d-9931-4d71-be5b-b668f59766e1\" (UID: \"d7ce149d-9931-4d71-be5b-b668f59766e1\") "
Sep 17 19:07:58.756810 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:07:58.756689    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/d7ce149d-9931-4d71-be5b-b668f59766e1-image-registry-private-configuration\") pod \"d7ce149d-9931-4d71-be5b-b668f59766e1\" (UID: \"d7ce149d-9931-4d71-be5b-b668f59766e1\") "
Sep 17 19:07:58.756888 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:07:58.756866    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/d7ce149d-9931-4d71-be5b-b668f59766e1-installation-pull-secrets\") pod \"d7ce149d-9931-4d71-be5b-b668f59766e1\" (UID: \"d7ce149d-9931-4d71-be5b-b668f59766e1\") "
Sep 17 19:07:58.756940 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:07:58.756911    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-fvw9m\" (UniqueName: \"kubernetes.io/projected/d7ce149d-9931-4d71-be5b-b668f59766e1-kube-api-access-fvw9m\") pod \"d7ce149d-9931-4d71-be5b-b668f59766e1\" (UID: \"d7ce149d-9931-4d71-be5b-b668f59766e1\") "
Sep 17 19:07:58.756995 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:07:58.756963    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/d7ce149d-9931-4d71-be5b-b668f59766e1-trusted-ca\") pod \"d7ce149d-9931-4d71-be5b-b668f59766e1\" (UID: \"d7ce149d-9931-4d71-be5b-b668f59766e1\") "
Sep 17 19:07:58.756995 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:07:58.756991    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/d7ce149d-9931-4d71-be5b-b668f59766e1-registry-certificates\") pod \"d7ce149d-9931-4d71-be5b-b668f59766e1\" (UID: \"d7ce149d-9931-4d71-be5b-b668f59766e1\") "
Sep 17 19:07:58.757097 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:07:58.757017    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/d7ce149d-9931-4d71-be5b-b668f59766e1-ca-trust-extracted\") pod \"d7ce149d-9931-4d71-be5b-b668f59766e1\" (UID: \"d7ce149d-9931-4d71-be5b-b668f59766e1\") "
Sep 17 19:07:58.757582 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:07:58.757431    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d7ce149d-9931-4d71-be5b-b668f59766e1-registry-certificates" (OuterVolumeSpecName: "registry-certificates") pod "d7ce149d-9931-4d71-be5b-b668f59766e1" (UID: "d7ce149d-9931-4d71-be5b-b668f59766e1"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 19:07:58.757939 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:07:58.757914    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d7ce149d-9931-4d71-be5b-b668f59766e1-trusted-ca" (OuterVolumeSpecName: "trusted-ca") pod "d7ce149d-9931-4d71-be5b-b668f59766e1" (UID: "d7ce149d-9931-4d71-be5b-b668f59766e1"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 19:07:58.761798 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:07:58.761597    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d7ce149d-9931-4d71-be5b-b668f59766e1-installation-pull-secrets" (OuterVolumeSpecName: "installation-pull-secrets") pod "d7ce149d-9931-4d71-be5b-b668f59766e1" (UID: "d7ce149d-9931-4d71-be5b-b668f59766e1"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 19:07:58.761798 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:07:58.761754    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d7ce149d-9931-4d71-be5b-b668f59766e1-image-registry-private-configuration" (OuterVolumeSpecName: "image-registry-private-configuration") pod "d7ce149d-9931-4d71-be5b-b668f59766e1" (UID: "d7ce149d-9931-4d71-be5b-b668f59766e1"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 19:07:58.761991 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:07:58.761874    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d7ce149d-9931-4d71-be5b-b668f59766e1-kube-api-access-fvw9m" (OuterVolumeSpecName: "kube-api-access-fvw9m") pod "d7ce149d-9931-4d71-be5b-b668f59766e1" (UID: "d7ce149d-9931-4d71-be5b-b668f59766e1"). InnerVolumeSpecName "kube-api-access-fvw9m". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 19:07:58.761991 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:07:58.761940    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d7ce149d-9931-4d71-be5b-b668f59766e1-bound-sa-token" (OuterVolumeSpecName: "bound-sa-token") pod "d7ce149d-9931-4d71-be5b-b668f59766e1" (UID: "d7ce149d-9931-4d71-be5b-b668f59766e1"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 19:07:58.762061 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:07:58.761989    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d7ce149d-9931-4d71-be5b-b668f59766e1-registry-tls" (OuterVolumeSpecName: "registry-tls") pod "d7ce149d-9931-4d71-be5b-b668f59766e1" (UID: "d7ce149d-9931-4d71-be5b-b668f59766e1"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 19:07:58.771653 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:07:58.771619    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d7ce149d-9931-4d71-be5b-b668f59766e1-ca-trust-extracted" (OuterVolumeSpecName: "ca-trust-extracted") pod "d7ce149d-9931-4d71-be5b-b668f59766e1" (UID: "d7ce149d-9931-4d71-be5b-b668f59766e1"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 19:07:58.858108 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:07:58.858059    2582 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/d7ce149d-9931-4d71-be5b-b668f59766e1-registry-tls\") on node \"ip-10-0-131-232.ec2.internal\" DevicePath \"\""
Sep 17 19:07:58.858108 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:07:58.858102    2582 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/d7ce149d-9931-4d71-be5b-b668f59766e1-bound-sa-token\") on node \"ip-10-0-131-232.ec2.internal\" DevicePath \"\""
Sep 17 19:07:58.858108 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:07:58.858113    2582 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/d7ce149d-9931-4d71-be5b-b668f59766e1-image-registry-private-configuration\") on node \"ip-10-0-131-232.ec2.internal\" DevicePath \"\""
Sep 17 19:07:58.858108 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:07:58.858122    2582 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/d7ce149d-9931-4d71-be5b-b668f59766e1-installation-pull-secrets\") on node \"ip-10-0-131-232.ec2.internal\" DevicePath \"\""
Sep 17 19:07:58.858357 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:07:58.858131    2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fvw9m\" (UniqueName: \"kubernetes.io/projected/d7ce149d-9931-4d71-be5b-b668f59766e1-kube-api-access-fvw9m\") on node \"ip-10-0-131-232.ec2.internal\" DevicePath \"\""
Sep 17 19:07:58.858357 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:07:58.858139    2582 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/d7ce149d-9931-4d71-be5b-b668f59766e1-trusted-ca\") on node \"ip-10-0-131-232.ec2.internal\" DevicePath \"\""
Sep 17 19:07:58.858357 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:07:58.858147    2582 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/d7ce149d-9931-4d71-be5b-b668f59766e1-registry-certificates\") on node \"ip-10-0-131-232.ec2.internal\" DevicePath \"\""
Sep 17 19:07:58.858357 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:07:58.858155    2582 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/d7ce149d-9931-4d71-be5b-b668f59766e1-ca-trust-extracted\") on node \"ip-10-0-131-232.ec2.internal\" DevicePath \"\""
Sep 17 19:07:59.298886 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:07:59.298848    2582 generic.go:356] "Generic (PLEG): container finished" podID="d7ce149d-9931-4d71-be5b-b668f59766e1" containerID="663da08b46f6833f52fc6a946ce6be8201f6185140b66783aec78490b8fa4ad1" exitCode=0
Sep 17 19:07:59.299050 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:07:59.298892    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-5cb5bc568b-xhl4k" event={"ID":"d7ce149d-9931-4d71-be5b-b668f59766e1","Type":"ContainerDied","Data":"663da08b46f6833f52fc6a946ce6be8201f6185140b66783aec78490b8fa4ad1"}
Sep 17 19:07:59.299050 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:07:59.298911    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-5cb5bc568b-xhl4k"
Sep 17 19:07:59.299050 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:07:59.298933    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-5cb5bc568b-xhl4k" event={"ID":"d7ce149d-9931-4d71-be5b-b668f59766e1","Type":"ContainerDied","Data":"dcefff316cbb10151c6dde524e15db69f4072e801705b021c4eaeaefc73c3a51"}
Sep 17 19:07:59.299050 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:07:59.298951    2582 scope.go:117] "RemoveContainer" containerID="663da08b46f6833f52fc6a946ce6be8201f6185140b66783aec78490b8fa4ad1"
Sep 17 19:07:59.307391 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:07:59.307373    2582 scope.go:117] "RemoveContainer" containerID="663da08b46f6833f52fc6a946ce6be8201f6185140b66783aec78490b8fa4ad1"
Sep 17 19:07:59.307685 ip-10-0-131-232 kubenswrapper[2582]: E0917 19:07:59.307665    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"663da08b46f6833f52fc6a946ce6be8201f6185140b66783aec78490b8fa4ad1\": container with ID starting with 663da08b46f6833f52fc6a946ce6be8201f6185140b66783aec78490b8fa4ad1 not found: ID does not exist" containerID="663da08b46f6833f52fc6a946ce6be8201f6185140b66783aec78490b8fa4ad1"
Sep 17 19:07:59.307748 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:07:59.307699    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"663da08b46f6833f52fc6a946ce6be8201f6185140b66783aec78490b8fa4ad1"} err="failed to get container status \"663da08b46f6833f52fc6a946ce6be8201f6185140b66783aec78490b8fa4ad1\": rpc error: code = NotFound desc = could not find container \"663da08b46f6833f52fc6a946ce6be8201f6185140b66783aec78490b8fa4ad1\": container with ID starting with 663da08b46f6833f52fc6a946ce6be8201f6185140b66783aec78490b8fa4ad1 not found: ID does not exist"
Sep 17 19:07:59.321521 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:07:59.321484    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-5cb5bc568b-xhl4k"]
Sep 17 19:07:59.325880 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:07:59.325851    2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-5cb5bc568b-xhl4k"]
Sep 17 19:08:00.767228 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:08:00.767193    2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d7ce149d-9931-4d71-be5b-b668f59766e1" path="/var/lib/kubelet/pods/d7ce149d-9931-4d71-be5b-b668f59766e1/volumes"
Sep 17 19:08:01.182368 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:08:01.182343    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-jhjrs"
Sep 17 19:08:08.309122 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:08:08.309080    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-78bfc78456-p7x4j"]
Sep 17 19:08:08.312756 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:08:08.309638    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d7ce149d-9931-4d71-be5b-b668f59766e1" containerName="registry"
Sep 17 19:08:08.312756 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:08:08.309654    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="d7ce149d-9931-4d71-be5b-b668f59766e1" containerName="registry"
Sep 17 19:08:08.312756 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:08:08.309705    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="d7ce149d-9931-4d71-be5b-b668f59766e1" containerName="registry"
Sep 17 19:08:08.362876 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:08:08.362809    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-78bfc78456-p7x4j"]
Sep 17 19:08:08.363208 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:08:08.363191    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-78bfc78456-p7x4j"
Sep 17 19:08:08.389800 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:08:08.389775    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\""
Sep 17 19:08:08.389968 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:08:08.389950    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-rlcgb\""
Sep 17 19:08:08.390425 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:08:08.390410    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\""
Sep 17 19:08:08.390504 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:08:08.390414    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\""
Sep 17 19:08:08.390919 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:08:08.390900    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\""
Sep 17 19:08:08.390919 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:08:08.390914    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\""
Sep 17 19:08:08.391050 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:08:08.390909    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\""
Sep 17 19:08:08.391508 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:08:08.391493    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\""
Sep 17 19:08:08.397181 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:08:08.397165    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\""
Sep 17 19:08:08.423271 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:08:08.423249    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/e78532b5-39ab-42d3-bf91-99703b1bb3b9-oauth-serving-cert\") pod \"console-78bfc78456-p7x4j\" (UID: \"e78532b5-39ab-42d3-bf91-99703b1bb3b9\") " pod="openshift-console/console-78bfc78456-p7x4j"
Sep 17 19:08:08.423350 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:08:08.423278    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e78532b5-39ab-42d3-bf91-99703b1bb3b9-trusted-ca-bundle\") pod \"console-78bfc78456-p7x4j\" (UID: \"e78532b5-39ab-42d3-bf91-99703b1bb3b9\") " pod="openshift-console/console-78bfc78456-p7x4j"
Sep 17 19:08:08.423350 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:08:08.423304    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/e78532b5-39ab-42d3-bf91-99703b1bb3b9-console-oauth-config\") pod \"console-78bfc78456-p7x4j\" (UID: \"e78532b5-39ab-42d3-bf91-99703b1bb3b9\") " pod="openshift-console/console-78bfc78456-p7x4j"
Sep 17 19:08:08.423350 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:08:08.423326    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/e78532b5-39ab-42d3-bf91-99703b1bb3b9-service-ca\") pod \"console-78bfc78456-p7x4j\" (UID: \"e78532b5-39ab-42d3-bf91-99703b1bb3b9\") " pod="openshift-console/console-78bfc78456-p7x4j"
Sep 17 19:08:08.423350 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:08:08.423348    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jc5gq\" (UniqueName: \"kubernetes.io/projected/e78532b5-39ab-42d3-bf91-99703b1bb3b9-kube-api-access-jc5gq\") pod \"console-78bfc78456-p7x4j\" (UID: \"e78532b5-39ab-42d3-bf91-99703b1bb3b9\") " pod="openshift-console/console-78bfc78456-p7x4j"
Sep 17 19:08:08.423524 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:08:08.423373    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/e78532b5-39ab-42d3-bf91-99703b1bb3b9-console-config\") pod \"console-78bfc78456-p7x4j\" (UID: \"e78532b5-39ab-42d3-bf91-99703b1bb3b9\") " pod="openshift-console/console-78bfc78456-p7x4j"
Sep 17 19:08:08.423524 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:08:08.423467    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/e78532b5-39ab-42d3-bf91-99703b1bb3b9-console-serving-cert\") pod \"console-78bfc78456-p7x4j\" (UID: \"e78532b5-39ab-42d3-bf91-99703b1bb3b9\") " pod="openshift-console/console-78bfc78456-p7x4j"
Sep 17 19:08:08.523795 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:08:08.523772    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/e78532b5-39ab-42d3-bf91-99703b1bb3b9-console-config\") pod \"console-78bfc78456-p7x4j\" (UID: \"e78532b5-39ab-42d3-bf91-99703b1bb3b9\") " pod="openshift-console/console-78bfc78456-p7x4j"
Sep 17 19:08:08.523903 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:08:08.523807    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/e78532b5-39ab-42d3-bf91-99703b1bb3b9-console-serving-cert\") pod \"console-78bfc78456-p7x4j\" (UID: \"e78532b5-39ab-42d3-bf91-99703b1bb3b9\") " pod="openshift-console/console-78bfc78456-p7x4j"
Sep 17 19:08:08.523903 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:08:08.523844    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/e78532b5-39ab-42d3-bf91-99703b1bb3b9-oauth-serving-cert\") pod \"console-78bfc78456-p7x4j\" (UID: \"e78532b5-39ab-42d3-bf91-99703b1bb3b9\") " pod="openshift-console/console-78bfc78456-p7x4j"
Sep 17 19:08:08.524025 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:08:08.523900    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e78532b5-39ab-42d3-bf91-99703b1bb3b9-trusted-ca-bundle\") pod \"console-78bfc78456-p7x4j\" (UID: \"e78532b5-39ab-42d3-bf91-99703b1bb3b9\") " pod="openshift-console/console-78bfc78456-p7x4j"
Sep 17 19:08:08.524025 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:08:08.523955    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/e78532b5-39ab-42d3-bf91-99703b1bb3b9-console-oauth-config\") pod \"console-78bfc78456-p7x4j\" (UID: \"e78532b5-39ab-42d3-bf91-99703b1bb3b9\") " pod="openshift-console/console-78bfc78456-p7x4j"
Sep 17 19:08:08.524025 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:08:08.523998    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/e78532b5-39ab-42d3-bf91-99703b1bb3b9-service-ca\") pod \"console-78bfc78456-p7x4j\" (UID: \"e78532b5-39ab-42d3-bf91-99703b1bb3b9\") " pod="openshift-console/console-78bfc78456-p7x4j"
Sep 17 19:08:08.524181 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:08:08.524033    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jc5gq\" (UniqueName: \"kubernetes.io/projected/e78532b5-39ab-42d3-bf91-99703b1bb3b9-kube-api-access-jc5gq\") pod \"console-78bfc78456-p7x4j\" (UID: \"e78532b5-39ab-42d3-bf91-99703b1bb3b9\") " pod="openshift-console/console-78bfc78456-p7x4j"
Sep 17 19:08:08.524563 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:08:08.524539    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/e78532b5-39ab-42d3-bf91-99703b1bb3b9-oauth-serving-cert\") pod \"console-78bfc78456-p7x4j\" (UID: \"e78532b5-39ab-42d3-bf91-99703b1bb3b9\") " pod="openshift-console/console-78bfc78456-p7x4j"
Sep 17 19:08:08.524647 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:08:08.524541    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/e78532b5-39ab-42d3-bf91-99703b1bb3b9-console-config\") pod \"console-78bfc78456-p7x4j\" (UID: \"e78532b5-39ab-42d3-bf91-99703b1bb3b9\") " pod="openshift-console/console-78bfc78456-p7x4j"
Sep 17 19:08:08.524899 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:08:08.524741    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/e78532b5-39ab-42d3-bf91-99703b1bb3b9-service-ca\") pod \"console-78bfc78456-p7x4j\" (UID: \"e78532b5-39ab-42d3-bf91-99703b1bb3b9\") " pod="openshift-console/console-78bfc78456-p7x4j"
Sep 17 19:08:08.524899 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:08:08.524741    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e78532b5-39ab-42d3-bf91-99703b1bb3b9-trusted-ca-bundle\") pod \"console-78bfc78456-p7x4j\" (UID: \"e78532b5-39ab-42d3-bf91-99703b1bb3b9\") " pod="openshift-console/console-78bfc78456-p7x4j"
Sep 17 19:08:08.526510 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:08:08.526482    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/e78532b5-39ab-42d3-bf91-99703b1bb3b9-console-oauth-config\") pod \"console-78bfc78456-p7x4j\" (UID: \"e78532b5-39ab-42d3-bf91-99703b1bb3b9\") " pod="openshift-console/console-78bfc78456-p7x4j"
Sep 17 19:08:08.526669 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:08:08.526652    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/e78532b5-39ab-42d3-bf91-99703b1bb3b9-console-serving-cert\") pod \"console-78bfc78456-p7x4j\" (UID: \"e78532b5-39ab-42d3-bf91-99703b1bb3b9\") " pod="openshift-console/console-78bfc78456-p7x4j"
Sep 17 19:08:08.533060 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:08:08.533040    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jc5gq\" (UniqueName: \"kubernetes.io/projected/e78532b5-39ab-42d3-bf91-99703b1bb3b9-kube-api-access-jc5gq\") pod \"console-78bfc78456-p7x4j\" (UID: \"e78532b5-39ab-42d3-bf91-99703b1bb3b9\") " pod="openshift-console/console-78bfc78456-p7x4j"
Sep 17 19:08:08.673115 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:08:08.673090    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-78bfc78456-p7x4j"
Sep 17 19:08:08.975973 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:08:08.975755    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-78bfc78456-p7x4j"]
Sep 17 19:08:08.978484 ip-10-0-131-232 kubenswrapper[2582]: W0917 19:08:08.978450    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode78532b5_39ab_42d3_bf91_99703b1bb3b9.slice/crio-6715366a30412738e65fcf838f95fc1be4e0acfedd47eb1d9aff39be3595fe4a WatchSource:0}: Error finding container 6715366a30412738e65fcf838f95fc1be4e0acfedd47eb1d9aff39be3595fe4a: Status 404 returned error can't find the container with id 6715366a30412738e65fcf838f95fc1be4e0acfedd47eb1d9aff39be3595fe4a
Sep 17 19:08:09.334937 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:08:09.334859    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-78bfc78456-p7x4j" event={"ID":"e78532b5-39ab-42d3-bf91-99703b1bb3b9","Type":"ContainerStarted","Data":"6715366a30412738e65fcf838f95fc1be4e0acfedd47eb1d9aff39be3595fe4a"}
Sep 17 19:08:13.352603 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:08:13.352505    2582 generic.go:356] "Generic (PLEG): container finished" podID="7074d19e-e52e-460a-a702-44130277bea5" containerID="d2071077a8d8f354fe20decec283293c66c60336a2f6dd1bb958f48a36ccda98" exitCode=0
Sep 17 19:08:13.352603 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:08:13.352576    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-55889f7b47-2w6p2" event={"ID":"7074d19e-e52e-460a-a702-44130277bea5","Type":"ContainerDied","Data":"d2071077a8d8f354fe20decec283293c66c60336a2f6dd1bb958f48a36ccda98"}
Sep 17 19:08:13.353177 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:08:13.352956    2582 scope.go:117] "RemoveContainer" containerID="d2071077a8d8f354fe20decec283293c66c60336a2f6dd1bb958f48a36ccda98"
Sep 17 19:08:13.354614 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:08:13.354568    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-78bfc78456-p7x4j" event={"ID":"e78532b5-39ab-42d3-bf91-99703b1bb3b9","Type":"ContainerStarted","Data":"3ea572d49ef7f413d317a9eea05e6c537c6df540b740886f48ea4b0e30990c55"}
Sep 17 19:08:13.404809 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:08:13.404752    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-78bfc78456-p7x4j" podStartSLOduration=1.390771754 podStartE2EDuration="5.404730223s" totalImagesPullingTime="4.013958469s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 19:08:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 19:08:13.403100114 +0000 UTC m=+109.174771178" watchObservedRunningTime="2026-09-17 19:08:13.404730223 +0000 UTC m=+109.176401286"
Sep 17 19:08:14.360478 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:08:14.360440    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-55889f7b47-2w6p2" event={"ID":"7074d19e-e52e-460a-a702-44130277bea5","Type":"ContainerStarted","Data":"bb17e3f6f24b4cc26adbaebe40c59fd5064a897cf0d8b728820639882cde9baf"}
Sep 17 19:08:18.674081 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:08:18.674042    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-78bfc78456-p7x4j"
Sep 17 19:08:18.674464 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:08:18.674131    2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-78bfc78456-p7x4j"
Sep 17 19:08:18.678855 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:08:18.678834    2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-78bfc78456-p7x4j"
Sep 17 19:08:19.381402 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:08:19.381369    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-78bfc78456-p7x4j"
Sep 17 19:08:21.377375 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:08:21.377348    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-rrzmt_53864523-604f-40d9-be99-cab48397a2f5/init-textfile/0.log"
Sep 17 19:08:21.578317 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:08:21.578290    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-rrzmt_53864523-604f-40d9-be99-cab48397a2f5/node-exporter/0.log"
Sep 17 19:08:21.777161 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:08:21.777133    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-rrzmt_53864523-604f-40d9-be99-cab48397a2f5/kube-rbac-proxy/0.log"
Sep 17 19:08:27.777264 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:08:27.777231    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_console-78bfc78456-p7x4j_e78532b5-39ab-42d3-bf91-99703b1bb3b9/console/0.log"
Sep 17 19:08:29.414961 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:08:29.414926    2582 generic.go:356] "Generic (PLEG): container finished" podID="839a4dbf-a176-4b22-95ec-00dd4973161a" containerID="3c1395cb3398cb0a49d50b8a064edcc1ea55096fb68877bbc55d25c2fd989595" exitCode=0
Sep 17 19:08:29.415319 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:08:29.414994    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-57cdc4b768-r2wgc" event={"ID":"839a4dbf-a176-4b22-95ec-00dd4973161a","Type":"ContainerDied","Data":"3c1395cb3398cb0a49d50b8a064edcc1ea55096fb68877bbc55d25c2fd989595"}
Sep 17 19:08:29.415366 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:08:29.415329    2582 scope.go:117] "RemoveContainer" containerID="3c1395cb3398cb0a49d50b8a064edcc1ea55096fb68877bbc55d25c2fd989595"
Sep 17 19:08:30.420691 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:08:30.420649    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-57cdc4b768-r2wgc" event={"ID":"839a4dbf-a176-4b22-95ec-00dd4973161a","Type":"ContainerStarted","Data":"138cb8cad9dd944f47a0bda5e9e9376e04251bdac471452ccc3562c87ffebbf1"}
Sep 17 19:09:41.149199 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:09:41.149161    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-78bfc78456-p7x4j"]
Sep 17 19:10:06.168438 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:10:06.168391    2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-78bfc78456-p7x4j" podUID="e78532b5-39ab-42d3-bf91-99703b1bb3b9" containerName="console" containerID="cri-o://3ea572d49ef7f413d317a9eea05e6c537c6df540b740886f48ea4b0e30990c55" gracePeriod=15
Sep 17 19:10:06.400140 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:10:06.400116    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-78bfc78456-p7x4j"
Sep 17 19:10:06.455100 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:10:06.455007    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/e78532b5-39ab-42d3-bf91-99703b1bb3b9-console-serving-cert\") pod \"e78532b5-39ab-42d3-bf91-99703b1bb3b9\" (UID: \"e78532b5-39ab-42d3-bf91-99703b1bb3b9\") "
Sep 17 19:10:06.455100 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:10:06.455069    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/e78532b5-39ab-42d3-bf91-99703b1bb3b9-oauth-serving-cert\") pod \"e78532b5-39ab-42d3-bf91-99703b1bb3b9\" (UID: \"e78532b5-39ab-42d3-bf91-99703b1bb3b9\") "
Sep 17 19:10:06.455328 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:10:06.455113    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/e78532b5-39ab-42d3-bf91-99703b1bb3b9-console-oauth-config\") pod \"e78532b5-39ab-42d3-bf91-99703b1bb3b9\" (UID: \"e78532b5-39ab-42d3-bf91-99703b1bb3b9\") "
Sep 17 19:10:06.455328 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:10:06.455138    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jc5gq\" (UniqueName: \"kubernetes.io/projected/e78532b5-39ab-42d3-bf91-99703b1bb3b9-kube-api-access-jc5gq\") pod \"e78532b5-39ab-42d3-bf91-99703b1bb3b9\" (UID: \"e78532b5-39ab-42d3-bf91-99703b1bb3b9\") "
Sep 17 19:10:06.455328 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:10:06.455166    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/e78532b5-39ab-42d3-bf91-99703b1bb3b9-service-ca\") pod \"e78532b5-39ab-42d3-bf91-99703b1bb3b9\" (UID: \"e78532b5-39ab-42d3-bf91-99703b1bb3b9\") "
Sep 17 19:10:06.455328 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:10:06.455194    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e78532b5-39ab-42d3-bf91-99703b1bb3b9-trusted-ca-bundle\") pod \"e78532b5-39ab-42d3-bf91-99703b1bb3b9\" (UID: \"e78532b5-39ab-42d3-bf91-99703b1bb3b9\") "
Sep 17 19:10:06.455328 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:10:06.455245    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/e78532b5-39ab-42d3-bf91-99703b1bb3b9-console-config\") pod \"e78532b5-39ab-42d3-bf91-99703b1bb3b9\" (UID: \"e78532b5-39ab-42d3-bf91-99703b1bb3b9\") "
Sep 17 19:10:06.455620 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:10:06.455583    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e78532b5-39ab-42d3-bf91-99703b1bb3b9-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "e78532b5-39ab-42d3-bf91-99703b1bb3b9" (UID: "e78532b5-39ab-42d3-bf91-99703b1bb3b9"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 19:10:06.455704 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:10:06.455621    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e78532b5-39ab-42d3-bf91-99703b1bb3b9-service-ca" (OuterVolumeSpecName: "service-ca") pod "e78532b5-39ab-42d3-bf91-99703b1bb3b9" (UID: "e78532b5-39ab-42d3-bf91-99703b1bb3b9"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 19:10:06.455704 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:10:06.455644    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e78532b5-39ab-42d3-bf91-99703b1bb3b9-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "e78532b5-39ab-42d3-bf91-99703b1bb3b9" (UID: "e78532b5-39ab-42d3-bf91-99703b1bb3b9"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 19:10:06.455794 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:10:06.455724    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e78532b5-39ab-42d3-bf91-99703b1bb3b9-console-config" (OuterVolumeSpecName: "console-config") pod "e78532b5-39ab-42d3-bf91-99703b1bb3b9" (UID: "e78532b5-39ab-42d3-bf91-99703b1bb3b9"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 19:10:06.457670 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:10:06.457645    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e78532b5-39ab-42d3-bf91-99703b1bb3b9-kube-api-access-jc5gq" (OuterVolumeSpecName: "kube-api-access-jc5gq") pod "e78532b5-39ab-42d3-bf91-99703b1bb3b9" (UID: "e78532b5-39ab-42d3-bf91-99703b1bb3b9"). InnerVolumeSpecName "kube-api-access-jc5gq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 19:10:06.457837 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:10:06.457796    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e78532b5-39ab-42d3-bf91-99703b1bb3b9-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "e78532b5-39ab-42d3-bf91-99703b1bb3b9" (UID: "e78532b5-39ab-42d3-bf91-99703b1bb3b9"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 19:10:06.457837 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:10:06.457807    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e78532b5-39ab-42d3-bf91-99703b1bb3b9-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "e78532b5-39ab-42d3-bf91-99703b1bb3b9" (UID: "e78532b5-39ab-42d3-bf91-99703b1bb3b9"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 19:10:06.556275 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:10:06.556243    2582 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/e78532b5-39ab-42d3-bf91-99703b1bb3b9-console-serving-cert\") on node \"ip-10-0-131-232.ec2.internal\" DevicePath \"\""
Sep 17 19:10:06.556275 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:10:06.556272    2582 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/e78532b5-39ab-42d3-bf91-99703b1bb3b9-oauth-serving-cert\") on node \"ip-10-0-131-232.ec2.internal\" DevicePath \"\""
Sep 17 19:10:06.556275 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:10:06.556281    2582 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/e78532b5-39ab-42d3-bf91-99703b1bb3b9-console-oauth-config\") on node \"ip-10-0-131-232.ec2.internal\" DevicePath \"\""
Sep 17 19:10:06.556500 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:10:06.556290    2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jc5gq\" (UniqueName: \"kubernetes.io/projected/e78532b5-39ab-42d3-bf91-99703b1bb3b9-kube-api-access-jc5gq\") on node \"ip-10-0-131-232.ec2.internal\" DevicePath \"\""
Sep 17 19:10:06.556500 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:10:06.556298    2582 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/e78532b5-39ab-42d3-bf91-99703b1bb3b9-service-ca\") on node \"ip-10-0-131-232.ec2.internal\" DevicePath \"\""
Sep 17 19:10:06.556500 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:10:06.556306    2582 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e78532b5-39ab-42d3-bf91-99703b1bb3b9-trusted-ca-bundle\") on node \"ip-10-0-131-232.ec2.internal\" DevicePath \"\""
Sep 17 19:10:06.556500 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:10:06.556314    2582 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/e78532b5-39ab-42d3-bf91-99703b1bb3b9-console-config\") on node \"ip-10-0-131-232.ec2.internal\" DevicePath \"\""
Sep 17 19:10:06.743913 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:10:06.743806    2582 generic.go:356] "Generic (PLEG): container finished" podID="e78532b5-39ab-42d3-bf91-99703b1bb3b9" containerID="3ea572d49ef7f413d317a9eea05e6c537c6df540b740886f48ea4b0e30990c55" exitCode=0
Sep 17 19:10:06.743913 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:10:06.743869    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-78bfc78456-p7x4j"
Sep 17 19:10:06.743913 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:10:06.743889    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-78bfc78456-p7x4j" event={"ID":"e78532b5-39ab-42d3-bf91-99703b1bb3b9","Type":"ContainerDied","Data":"3ea572d49ef7f413d317a9eea05e6c537c6df540b740886f48ea4b0e30990c55"}
Sep 17 19:10:06.744122 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:10:06.743948    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-78bfc78456-p7x4j" event={"ID":"e78532b5-39ab-42d3-bf91-99703b1bb3b9","Type":"ContainerDied","Data":"6715366a30412738e65fcf838f95fc1be4e0acfedd47eb1d9aff39be3595fe4a"}
Sep 17 19:10:06.744122 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:10:06.743969    2582 scope.go:117] "RemoveContainer" containerID="3ea572d49ef7f413d317a9eea05e6c537c6df540b740886f48ea4b0e30990c55"
Sep 17 19:10:06.752271 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:10:06.752247    2582 scope.go:117] "RemoveContainer" containerID="3ea572d49ef7f413d317a9eea05e6c537c6df540b740886f48ea4b0e30990c55"
Sep 17 19:10:06.754183 ip-10-0-131-232 kubenswrapper[2582]: E0917 19:10:06.754151    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3ea572d49ef7f413d317a9eea05e6c537c6df540b740886f48ea4b0e30990c55\": container with ID starting with 3ea572d49ef7f413d317a9eea05e6c537c6df540b740886f48ea4b0e30990c55 not found: ID does not exist" containerID="3ea572d49ef7f413d317a9eea05e6c537c6df540b740886f48ea4b0e30990c55"
Sep 17 19:10:06.754291 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:10:06.754204    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3ea572d49ef7f413d317a9eea05e6c537c6df540b740886f48ea4b0e30990c55"} err="failed to get container status \"3ea572d49ef7f413d317a9eea05e6c537c6df540b740886f48ea4b0e30990c55\": rpc error: code = NotFound desc = could not find container \"3ea572d49ef7f413d317a9eea05e6c537c6df540b740886f48ea4b0e30990c55\": container with ID starting with 3ea572d49ef7f413d317a9eea05e6c537c6df540b740886f48ea4b0e30990c55 not found: ID does not exist"
Sep 17 19:10:06.767515 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:10:06.767491    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-78bfc78456-p7x4j"]
Sep 17 19:10:06.771502 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:10:06.771479    2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-78bfc78456-p7x4j"]
Sep 17 19:10:08.767253 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:10:08.767217    2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e78532b5-39ab-42d3-bf91-99703b1bb3b9" path="/var/lib/kubelet/pods/e78532b5-39ab-42d3-bf91-99703b1bb3b9/volumes"
Sep 17 19:10:09.073343 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:10:09.073267    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1aqdjsk"]
Sep 17 19:10:09.073508 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:10:09.073496    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e78532b5-39ab-42d3-bf91-99703b1bb3b9" containerName="console"
Sep 17 19:10:09.073548 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:10:09.073509    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="e78532b5-39ab-42d3-bf91-99703b1bb3b9" containerName="console"
Sep 17 19:10:09.073581 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:10:09.073555    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="e78532b5-39ab-42d3-bf91-99703b1bb3b9" containerName="console"
Sep 17 19:10:09.075875 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:10:09.075858    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1aqdjsk"
Sep 17 19:10:09.078633 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:10:09.078613    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\""
Sep 17 19:10:09.079062 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:10:09.079047    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-zdtss\""
Sep 17 19:10:09.079631 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:10:09.079612    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\""
Sep 17 19:10:09.089471 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:10:09.089448    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1aqdjsk"]
Sep 17 19:10:09.180640 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:10:09.180601    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/a117dcb0-50c7-4523-a8aa-944497fa7cbc-bundle\") pod \"1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1aqdjsk\" (UID: \"a117dcb0-50c7-4523-a8aa-944497fa7cbc\") " pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1aqdjsk"
Sep 17 19:10:09.180856 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:10:09.180681    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2lfzk\" (UniqueName: \"kubernetes.io/projected/a117dcb0-50c7-4523-a8aa-944497fa7cbc-kube-api-access-2lfzk\") pod \"1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1aqdjsk\" (UID: \"a117dcb0-50c7-4523-a8aa-944497fa7cbc\") " pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1aqdjsk"
Sep 17 19:10:09.180856 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:10:09.180705    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/a117dcb0-50c7-4523-a8aa-944497fa7cbc-util\") pod \"1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1aqdjsk\" (UID: \"a117dcb0-50c7-4523-a8aa-944497fa7cbc\") " pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1aqdjsk"
Sep 17 19:10:09.281894 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:10:09.281848    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2lfzk\" (UniqueName: \"kubernetes.io/projected/a117dcb0-50c7-4523-a8aa-944497fa7cbc-kube-api-access-2lfzk\") pod \"1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1aqdjsk\" (UID: \"a117dcb0-50c7-4523-a8aa-944497fa7cbc\") " pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1aqdjsk"
Sep 17 19:10:09.282081 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:10:09.281902    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/a117dcb0-50c7-4523-a8aa-944497fa7cbc-util\") pod \"1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1aqdjsk\" (UID: \"a117dcb0-50c7-4523-a8aa-944497fa7cbc\") " pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1aqdjsk"
Sep 17 19:10:09.282081 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:10:09.281926    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/a117dcb0-50c7-4523-a8aa-944497fa7cbc-bundle\") pod \"1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1aqdjsk\" (UID: \"a117dcb0-50c7-4523-a8aa-944497fa7cbc\") " pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1aqdjsk"
Sep 17 19:10:09.282270 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:10:09.282250    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/a117dcb0-50c7-4523-a8aa-944497fa7cbc-util\") pod \"1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1aqdjsk\" (UID: \"a117dcb0-50c7-4523-a8aa-944497fa7cbc\") " pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1aqdjsk"
Sep 17 19:10:09.282308 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:10:09.282276    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/a117dcb0-50c7-4523-a8aa-944497fa7cbc-bundle\") pod \"1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1aqdjsk\" (UID: \"a117dcb0-50c7-4523-a8aa-944497fa7cbc\") " pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1aqdjsk"
Sep 17 19:10:09.291615 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:10:09.291587    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2lfzk\" (UniqueName: \"kubernetes.io/projected/a117dcb0-50c7-4523-a8aa-944497fa7cbc-kube-api-access-2lfzk\") pod \"1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1aqdjsk\" (UID: \"a117dcb0-50c7-4523-a8aa-944497fa7cbc\") " pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1aqdjsk"
Sep 17 19:10:09.384739 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:10:09.384699    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1aqdjsk"
Sep 17 19:10:09.655180 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:10:09.655100    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1aqdjsk"]
Sep 17 19:10:09.658412 ip-10-0-131-232 kubenswrapper[2582]: W0917 19:10:09.658387    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda117dcb0_50c7_4523_a8aa_944497fa7cbc.slice/crio-92e27629239c8905db0c5e3619db255ea31eb3eb6851ead033637ee5eb4139fd WatchSource:0}: Error finding container 92e27629239c8905db0c5e3619db255ea31eb3eb6851ead033637ee5eb4139fd: Status 404 returned error can't find the container with id 92e27629239c8905db0c5e3619db255ea31eb3eb6851ead033637ee5eb4139fd
Sep 17 19:10:09.757112 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:10:09.757072    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1aqdjsk" event={"ID":"a117dcb0-50c7-4523-a8aa-944497fa7cbc","Type":"ContainerStarted","Data":"92e27629239c8905db0c5e3619db255ea31eb3eb6851ead033637ee5eb4139fd"}
Sep 17 19:10:17.787632 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:10:17.787595    2582 generic.go:356] "Generic (PLEG): container finished" podID="a117dcb0-50c7-4523-a8aa-944497fa7cbc" containerID="f202a72076073d38f271ce7a471c09d121668b890b178b72354a10b3579481da" exitCode=0
Sep 17 19:10:17.788065 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:10:17.787683    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1aqdjsk" event={"ID":"a117dcb0-50c7-4523-a8aa-944497fa7cbc","Type":"ContainerDied","Data":"f202a72076073d38f271ce7a471c09d121668b890b178b72354a10b3579481da"}
Sep 17 19:10:19.797132 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:10:19.797043    2582 generic.go:356] "Generic (PLEG): container finished" podID="a117dcb0-50c7-4523-a8aa-944497fa7cbc" containerID="60ee21dd6b737b28422daabbafad55cbcdfeb365a93a0de562048f461a0a872d" exitCode=0
Sep 17 19:10:19.797548 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:10:19.797130    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1aqdjsk" event={"ID":"a117dcb0-50c7-4523-a8aa-944497fa7cbc","Type":"ContainerDied","Data":"60ee21dd6b737b28422daabbafad55cbcdfeb365a93a0de562048f461a0a872d"}
Sep 17 19:10:26.825417 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:10:26.825369    2582 generic.go:356] "Generic (PLEG): container finished" podID="a117dcb0-50c7-4523-a8aa-944497fa7cbc" containerID="f1523c42e2b661fec5885232a4ba34bbc92ceff3b74ff8e06a1493cde3f8b351" exitCode=0
Sep 17 19:10:26.825884 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:10:26.825463    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1aqdjsk" event={"ID":"a117dcb0-50c7-4523-a8aa-944497fa7cbc","Type":"ContainerDied","Data":"f1523c42e2b661fec5885232a4ba34bbc92ceff3b74ff8e06a1493cde3f8b351"}
Sep 17 19:10:27.945120 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:10:27.945094    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1aqdjsk"
Sep 17 19:10:28.130091 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:10:28.130011    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/a117dcb0-50c7-4523-a8aa-944497fa7cbc-bundle\") pod \"a117dcb0-50c7-4523-a8aa-944497fa7cbc\" (UID: \"a117dcb0-50c7-4523-a8aa-944497fa7cbc\") "
Sep 17 19:10:28.130091 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:10:28.130046    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2lfzk\" (UniqueName: \"kubernetes.io/projected/a117dcb0-50c7-4523-a8aa-944497fa7cbc-kube-api-access-2lfzk\") pod \"a117dcb0-50c7-4523-a8aa-944497fa7cbc\" (UID: \"a117dcb0-50c7-4523-a8aa-944497fa7cbc\") "
Sep 17 19:10:28.130091 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:10:28.130084    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/a117dcb0-50c7-4523-a8aa-944497fa7cbc-util\") pod \"a117dcb0-50c7-4523-a8aa-944497fa7cbc\" (UID: \"a117dcb0-50c7-4523-a8aa-944497fa7cbc\") "
Sep 17 19:10:28.130769 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:10:28.130703    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a117dcb0-50c7-4523-a8aa-944497fa7cbc-bundle" (OuterVolumeSpecName: "bundle") pod "a117dcb0-50c7-4523-a8aa-944497fa7cbc" (UID: "a117dcb0-50c7-4523-a8aa-944497fa7cbc"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 19:10:28.132399 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:10:28.132374    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a117dcb0-50c7-4523-a8aa-944497fa7cbc-kube-api-access-2lfzk" (OuterVolumeSpecName: "kube-api-access-2lfzk") pod "a117dcb0-50c7-4523-a8aa-944497fa7cbc" (UID: "a117dcb0-50c7-4523-a8aa-944497fa7cbc"). InnerVolumeSpecName "kube-api-access-2lfzk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 19:10:28.134652 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:10:28.134634    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a117dcb0-50c7-4523-a8aa-944497fa7cbc-util" (OuterVolumeSpecName: "util") pod "a117dcb0-50c7-4523-a8aa-944497fa7cbc" (UID: "a117dcb0-50c7-4523-a8aa-944497fa7cbc"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 19:10:28.231253 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:10:28.231202    2582 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/a117dcb0-50c7-4523-a8aa-944497fa7cbc-bundle\") on node \"ip-10-0-131-232.ec2.internal\" DevicePath \"\""
Sep 17 19:10:28.231253 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:10:28.231245    2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2lfzk\" (UniqueName: \"kubernetes.io/projected/a117dcb0-50c7-4523-a8aa-944497fa7cbc-kube-api-access-2lfzk\") on node \"ip-10-0-131-232.ec2.internal\" DevicePath \"\""
Sep 17 19:10:28.231253 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:10:28.231256    2582 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/a117dcb0-50c7-4523-a8aa-944497fa7cbc-util\") on node \"ip-10-0-131-232.ec2.internal\" DevicePath \"\""
Sep 17 19:10:28.834896 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:10:28.834857    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1aqdjsk" event={"ID":"a117dcb0-50c7-4523-a8aa-944497fa7cbc","Type":"ContainerDied","Data":"92e27629239c8905db0c5e3619db255ea31eb3eb6851ead033637ee5eb4139fd"}
Sep 17 19:10:28.834896 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:10:28.834890    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1aqdjsk"
Sep 17 19:10:28.835102 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:10:28.834912    2582 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="92e27629239c8905db0c5e3619db255ea31eb3eb6851ead033637ee5eb4139fd"
Sep 17 19:10:31.985236 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:10:31.985202    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-keda/custom-metrics-autoscaler-operator-566c9cd69c-zl4z6"]
Sep 17 19:10:31.985717 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:10:31.985530    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a117dcb0-50c7-4523-a8aa-944497fa7cbc" containerName="util"
Sep 17 19:10:31.985717 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:10:31.985550    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="a117dcb0-50c7-4523-a8aa-944497fa7cbc" containerName="util"
Sep 17 19:10:31.985717 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:10:31.985573    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a117dcb0-50c7-4523-a8aa-944497fa7cbc" containerName="pull"
Sep 17 19:10:31.985717 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:10:31.985583    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="a117dcb0-50c7-4523-a8aa-944497fa7cbc" containerName="pull"
Sep 17 19:10:31.985717 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:10:31.985603    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a117dcb0-50c7-4523-a8aa-944497fa7cbc" containerName="extract"
Sep 17 19:10:31.985717 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:10:31.985611    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="a117dcb0-50c7-4523-a8aa-944497fa7cbc" containerName="extract"
Sep 17 19:10:31.985717 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:10:31.985667    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="a117dcb0-50c7-4523-a8aa-944497fa7cbc" containerName="extract"
Sep 17 19:10:31.989678 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:10:31.989658    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/custom-metrics-autoscaler-operator-566c9cd69c-zl4z6"
Sep 17 19:10:31.992246 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:10:31.992223    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-keda\"/\"kedaorg-certs\""
Sep 17 19:10:31.992372 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:10:31.992223    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-keda\"/\"custom-metrics-autoscaler-operator-dockercfg-qhnth\""
Sep 17 19:10:31.992372 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:10:31.992226    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-keda\"/\"openshift-service-ca.crt\""
Sep 17 19:10:31.992372 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:10:31.992226    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-keda\"/\"kube-root-ca.crt\""
Sep 17 19:10:32.003083 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:10:32.003060    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-keda/custom-metrics-autoscaler-operator-566c9cd69c-zl4z6"]
Sep 17 19:10:32.161115 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:10:32.161081    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t7mjd\" (UniqueName: \"kubernetes.io/projected/c4a5aaf5-b8d9-4c12-941b-e066b2e1da86-kube-api-access-t7mjd\") pod \"custom-metrics-autoscaler-operator-566c9cd69c-zl4z6\" (UID: \"c4a5aaf5-b8d9-4c12-941b-e066b2e1da86\") " pod="openshift-keda/custom-metrics-autoscaler-operator-566c9cd69c-zl4z6"
Sep 17 19:10:32.161257 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:10:32.161142    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/secret/c4a5aaf5-b8d9-4c12-941b-e066b2e1da86-certificates\") pod \"custom-metrics-autoscaler-operator-566c9cd69c-zl4z6\" (UID: \"c4a5aaf5-b8d9-4c12-941b-e066b2e1da86\") " pod="openshift-keda/custom-metrics-autoscaler-operator-566c9cd69c-zl4z6"
Sep 17 19:10:32.262062 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:10:32.261978    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/secret/c4a5aaf5-b8d9-4c12-941b-e066b2e1da86-certificates\") pod \"custom-metrics-autoscaler-operator-566c9cd69c-zl4z6\" (UID: \"c4a5aaf5-b8d9-4c12-941b-e066b2e1da86\") " pod="openshift-keda/custom-metrics-autoscaler-operator-566c9cd69c-zl4z6"
Sep 17 19:10:32.262062 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:10:32.262031    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t7mjd\" (UniqueName: \"kubernetes.io/projected/c4a5aaf5-b8d9-4c12-941b-e066b2e1da86-kube-api-access-t7mjd\") pod \"custom-metrics-autoscaler-operator-566c9cd69c-zl4z6\" (UID: \"c4a5aaf5-b8d9-4c12-941b-e066b2e1da86\") " pod="openshift-keda/custom-metrics-autoscaler-operator-566c9cd69c-zl4z6"
Sep 17 19:10:32.264527 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:10:32.264506    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"certificates\" (UniqueName: \"kubernetes.io/secret/c4a5aaf5-b8d9-4c12-941b-e066b2e1da86-certificates\") pod \"custom-metrics-autoscaler-operator-566c9cd69c-zl4z6\" (UID: \"c4a5aaf5-b8d9-4c12-941b-e066b2e1da86\") " pod="openshift-keda/custom-metrics-autoscaler-operator-566c9cd69c-zl4z6"
Sep 17 19:10:32.271265 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:10:32.271243    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-t7mjd\" (UniqueName: \"kubernetes.io/projected/c4a5aaf5-b8d9-4c12-941b-e066b2e1da86-kube-api-access-t7mjd\") pod \"custom-metrics-autoscaler-operator-566c9cd69c-zl4z6\" (UID: \"c4a5aaf5-b8d9-4c12-941b-e066b2e1da86\") " pod="openshift-keda/custom-metrics-autoscaler-operator-566c9cd69c-zl4z6"
Sep 17 19:10:32.299276 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:10:32.299239    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/custom-metrics-autoscaler-operator-566c9cd69c-zl4z6"
Sep 17 19:10:32.583291 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:10:32.583262    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-keda/custom-metrics-autoscaler-operator-566c9cd69c-zl4z6"]
Sep 17 19:10:32.585163 ip-10-0-131-232 kubenswrapper[2582]: W0917 19:10:32.585135    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc4a5aaf5_b8d9_4c12_941b_e066b2e1da86.slice/crio-872a110f12d4fdddefa91007a71eee27e1b7cd0e7e9a27434c31f1a5b490fc52 WatchSource:0}: Error finding container 872a110f12d4fdddefa91007a71eee27e1b7cd0e7e9a27434c31f1a5b490fc52: Status 404 returned error can't find the container with id 872a110f12d4fdddefa91007a71eee27e1b7cd0e7e9a27434c31f1a5b490fc52
Sep 17 19:10:32.849563 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:10:32.849482    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-keda/custom-metrics-autoscaler-operator-566c9cd69c-zl4z6" event={"ID":"c4a5aaf5-b8d9-4c12-941b-e066b2e1da86","Type":"ContainerStarted","Data":"872a110f12d4fdddefa91007a71eee27e1b7cd0e7e9a27434c31f1a5b490fc52"}
Sep 17 19:10:35.862293 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:10:35.862247    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-keda/custom-metrics-autoscaler-operator-566c9cd69c-zl4z6" event={"ID":"c4a5aaf5-b8d9-4c12-941b-e066b2e1da86","Type":"ContainerStarted","Data":"9cf5919dc3a2fda62fd23847c7e9f84c2dcd3857f6353d907a9b15ec286f6154"}
Sep 17 19:10:35.862690 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:10:35.862375    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-keda/custom-metrics-autoscaler-operator-566c9cd69c-zl4z6"
Sep 17 19:10:35.888554 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:10:35.888461    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-keda/custom-metrics-autoscaler-operator-566c9cd69c-zl4z6" podStartSLOduration=1.79091173 podStartE2EDuration="4.888444365s" totalImagesPullingTime="3.097532635s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 19:10:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 19:10:35.886410058 +0000 UTC m=+251.658081120" watchObservedRunningTime="2026-09-17 19:10:35.888444365 +0000 UTC m=+251.660115426"
Sep 17 19:10:36.455929 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:10:36.455882    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-keda/keda-operator-579c8958f7-msqcg"]
Sep 17 19:10:36.477698 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:10:36.477671    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-keda/keda-operator-579c8958f7-msqcg"]
Sep 17 19:10:36.477899 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:10:36.477779    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/keda-operator-579c8958f7-msqcg"
Sep 17 19:10:36.480681 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:10:36.480654    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-keda\"/\"keda-operator-dockercfg-v9x2p\""
Sep 17 19:10:36.480681 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:10:36.480672    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-keda\"/\"keda-ocp-cabundle\""
Sep 17 19:10:36.480852 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:10:36.480672    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-keda\"/\"keda-operator-certs\""
Sep 17 19:10:36.599887 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:10:36.599851    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d45zl\" (UniqueName: \"kubernetes.io/projected/fbcaf1e2-2166-4794-a97d-55f4b9dac869-kube-api-access-d45zl\") pod \"keda-operator-579c8958f7-msqcg\" (UID: \"fbcaf1e2-2166-4794-a97d-55f4b9dac869\") " pod="openshift-keda/keda-operator-579c8958f7-msqcg"
Sep 17 19:10:36.599887 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:10:36.599891    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle0\" (UniqueName: \"kubernetes.io/configmap/fbcaf1e2-2166-4794-a97d-55f4b9dac869-cabundle0\") pod \"keda-operator-579c8958f7-msqcg\" (UID: \"fbcaf1e2-2166-4794-a97d-55f4b9dac869\") " pod="openshift-keda/keda-operator-579c8958f7-msqcg"
Sep 17 19:10:36.600100 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:10:36.599913    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/fbcaf1e2-2166-4794-a97d-55f4b9dac869-certificates\") pod \"keda-operator-579c8958f7-msqcg\" (UID: \"fbcaf1e2-2166-4794-a97d-55f4b9dac869\") " pod="openshift-keda/keda-operator-579c8958f7-msqcg"
Sep 17 19:10:36.701326 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:10:36.701281    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d45zl\" (UniqueName: \"kubernetes.io/projected/fbcaf1e2-2166-4794-a97d-55f4b9dac869-kube-api-access-d45zl\") pod \"keda-operator-579c8958f7-msqcg\" (UID: \"fbcaf1e2-2166-4794-a97d-55f4b9dac869\") " pod="openshift-keda/keda-operator-579c8958f7-msqcg"
Sep 17 19:10:36.701518 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:10:36.701335    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle0\" (UniqueName: \"kubernetes.io/configmap/fbcaf1e2-2166-4794-a97d-55f4b9dac869-cabundle0\") pod \"keda-operator-579c8958f7-msqcg\" (UID: \"fbcaf1e2-2166-4794-a97d-55f4b9dac869\") " pod="openshift-keda/keda-operator-579c8958f7-msqcg"
Sep 17 19:10:36.701518 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:10:36.701374    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/fbcaf1e2-2166-4794-a97d-55f4b9dac869-certificates\") pod \"keda-operator-579c8958f7-msqcg\" (UID: \"fbcaf1e2-2166-4794-a97d-55f4b9dac869\") " pod="openshift-keda/keda-operator-579c8958f7-msqcg"
Sep 17 19:10:36.701518 ip-10-0-131-232 kubenswrapper[2582]: E0917 19:10:36.701484    2582 secret.go:281] references non-existent secret key: ca.crt
Sep 17 19:10:36.701518 ip-10-0-131-232 kubenswrapper[2582]: E0917 19:10:36.701503    2582 projected.go:279] Couldn't get secret payload openshift-keda/kedaorg-certs: references non-existent secret key: ca.crt
Sep 17 19:10:36.701518 ip-10-0-131-232 kubenswrapper[2582]: E0917 19:10:36.701515    2582 projected.go:196] Error preparing data for projected volume certificates for pod openshift-keda/keda-operator-579c8958f7-msqcg: references non-existent secret key: ca.crt
Sep 17 19:10:36.701734 ip-10-0-131-232 kubenswrapper[2582]: E0917 19:10:36.701582    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/fbcaf1e2-2166-4794-a97d-55f4b9dac869-certificates podName:fbcaf1e2-2166-4794-a97d-55f4b9dac869 nodeName:}" failed. No retries permitted until 2026-09-17 19:10:37.20156048 +0000 UTC m=+252.973231535 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "certificates" (UniqueName: "kubernetes.io/projected/fbcaf1e2-2166-4794-a97d-55f4b9dac869-certificates") pod "keda-operator-579c8958f7-msqcg" (UID: "fbcaf1e2-2166-4794-a97d-55f4b9dac869") : references non-existent secret key: ca.crt
Sep 17 19:10:36.702007 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:10:36.701976    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle0\" (UniqueName: \"kubernetes.io/configmap/fbcaf1e2-2166-4794-a97d-55f4b9dac869-cabundle0\") pod \"keda-operator-579c8958f7-msqcg\" (UID: \"fbcaf1e2-2166-4794-a97d-55f4b9dac869\") " pod="openshift-keda/keda-operator-579c8958f7-msqcg"
Sep 17 19:10:36.714489 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:10:36.714417    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-d45zl\" (UniqueName: \"kubernetes.io/projected/fbcaf1e2-2166-4794-a97d-55f4b9dac869-kube-api-access-d45zl\") pod \"keda-operator-579c8958f7-msqcg\" (UID: \"fbcaf1e2-2166-4794-a97d-55f4b9dac869\") " pod="openshift-keda/keda-operator-579c8958f7-msqcg"
Sep 17 19:10:37.204877 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:10:37.204842    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/fbcaf1e2-2166-4794-a97d-55f4b9dac869-certificates\") pod \"keda-operator-579c8958f7-msqcg\" (UID: \"fbcaf1e2-2166-4794-a97d-55f4b9dac869\") " pod="openshift-keda/keda-operator-579c8958f7-msqcg"
Sep 17 19:10:37.205254 ip-10-0-131-232 kubenswrapper[2582]: E0917 19:10:37.204982    2582 secret.go:281] references non-existent secret key: ca.crt
Sep 17 19:10:37.205254 ip-10-0-131-232 kubenswrapper[2582]: E0917 19:10:37.204996    2582 projected.go:279] Couldn't get secret payload openshift-keda/kedaorg-certs: references non-existent secret key: ca.crt
Sep 17 19:10:37.205254 ip-10-0-131-232 kubenswrapper[2582]: E0917 19:10:37.205005    2582 projected.go:196] Error preparing data for projected volume certificates for pod openshift-keda/keda-operator-579c8958f7-msqcg: references non-existent secret key: ca.crt
Sep 17 19:10:37.205254 ip-10-0-131-232 kubenswrapper[2582]: E0917 19:10:37.205072    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/fbcaf1e2-2166-4794-a97d-55f4b9dac869-certificates podName:fbcaf1e2-2166-4794-a97d-55f4b9dac869 nodeName:}" failed. No retries permitted until 2026-09-17 19:10:38.205057202 +0000 UTC m=+253.976728241 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "certificates" (UniqueName: "kubernetes.io/projected/fbcaf1e2-2166-4794-a97d-55f4b9dac869-certificates") pod "keda-operator-579c8958f7-msqcg" (UID: "fbcaf1e2-2166-4794-a97d-55f4b9dac869") : references non-existent secret key: ca.crt
Sep 17 19:10:38.211928 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:10:38.211894    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/fbcaf1e2-2166-4794-a97d-55f4b9dac869-certificates\") pod \"keda-operator-579c8958f7-msqcg\" (UID: \"fbcaf1e2-2166-4794-a97d-55f4b9dac869\") " pod="openshift-keda/keda-operator-579c8958f7-msqcg"
Sep 17 19:10:38.214575 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:10:38.214553    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/fbcaf1e2-2166-4794-a97d-55f4b9dac869-certificates\") pod \"keda-operator-579c8958f7-msqcg\" (UID: \"fbcaf1e2-2166-4794-a97d-55f4b9dac869\") " pod="openshift-keda/keda-operator-579c8958f7-msqcg"
Sep 17 19:10:38.288070 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:10:38.288029    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/keda-operator-579c8958f7-msqcg"
Sep 17 19:10:38.600582 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:10:38.600488    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-keda/keda-operator-579c8958f7-msqcg"]
Sep 17 19:10:38.604532 ip-10-0-131-232 kubenswrapper[2582]: W0917 19:10:38.604504    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfbcaf1e2_2166_4794_a97d_55f4b9dac869.slice/crio-c50501fcb2993b181ab51464dc971d5d317fbf3076f79088eb8caa7c2dcbef99 WatchSource:0}: Error finding container c50501fcb2993b181ab51464dc971d5d317fbf3076f79088eb8caa7c2dcbef99: Status 404 returned error can't find the container with id c50501fcb2993b181ab51464dc971d5d317fbf3076f79088eb8caa7c2dcbef99
Sep 17 19:10:38.874930 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:10:38.874856    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-keda/keda-operator-579c8958f7-msqcg" event={"ID":"fbcaf1e2-2166-4794-a97d-55f4b9dac869","Type":"ContainerStarted","Data":"c50501fcb2993b181ab51464dc971d5d317fbf3076f79088eb8caa7c2dcbef99"}
Sep 17 19:10:43.894972 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:10:43.894937    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-keda/keda-operator-579c8958f7-msqcg" event={"ID":"fbcaf1e2-2166-4794-a97d-55f4b9dac869","Type":"ContainerStarted","Data":"13b7a187a87bc7d0b1bc93b98f8e3cbbf3fa7fbc0b0e744b87836f989d421686"}
Sep 17 19:10:43.895418 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:10:43.895080    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-keda/keda-operator-579c8958f7-msqcg"
Sep 17 19:10:43.914610 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:10:43.914555    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-keda/keda-operator-579c8958f7-msqcg" podStartSLOduration=3.089787418 podStartE2EDuration="7.914538755s" totalImagesPullingTime="4.824751337s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 19:10:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 19:10:43.912523221 +0000 UTC m=+259.684194284" watchObservedRunningTime="2026-09-17 19:10:43.914538755 +0000 UTC m=+259.686209828"
Sep 17 19:10:56.868175 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:10:56.868092    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-keda/custom-metrics-autoscaler-operator-566c9cd69c-zl4z6"
Sep 17 19:11:04.900160 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:11:04.900129    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-keda/keda-operator-579c8958f7-msqcg"
Sep 17 19:11:24.643425 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:11:24.643393    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-kdm8s_c1b79c18-774f-4b78-a6a9-7ea8a229aaae/ovn-acl-logging/0.log"
Sep 17 19:11:24.650740 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:11:24.650713    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-kdm8s_c1b79c18-774f-4b78-a6a9-7ea8a229aaae/ovn-acl-logging/0.log"
Sep 17 19:11:24.650931 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:11:24.650765    2582 kubelet.go:1648] "Image garbage collection succeeded"
Sep 17 19:11:57.832536 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:11:57.832493    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/kserve-controller-manager-66576f5db6-rpr9j"]
Sep 17 19:11:57.835674 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:11:57.835654    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/kserve-controller-manager-66576f5db6-rpr9j"
Sep 17 19:11:57.838355 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:11:57.838331    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"kserve-webhook-server-cert\""
Sep 17 19:11:57.838482 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:11:57.838331    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve\"/\"openshift-service-ca.crt\""
Sep 17 19:11:57.839304 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:11:57.839279    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"kserve-controller-manager-dockercfg-s4gjc\""
Sep 17 19:11:57.840451 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:11:57.840433    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve\"/\"kube-root-ca.crt\""
Sep 17 19:11:57.847396 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:11:57.847372    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/kserve-controller-manager-66576f5db6-rpr9j"]
Sep 17 19:11:57.869215 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:11:57.869185    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/seaweedfs-64568bcd49-nvdj7"]
Sep 17 19:11:57.872176 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:11:57.872159    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/seaweedfs-64568bcd49-nvdj7"
Sep 17 19:11:57.874776 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:11:57.874753    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"default-dockercfg-jwhxb\""
Sep 17 19:11:57.874776 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:11:57.874771    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"mlpipeline-s3-artifact\""
Sep 17 19:11:57.887175 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:11:57.887150    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/seaweedfs-64568bcd49-nvdj7"]
Sep 17 19:11:58.002734 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:11:58.002691    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hlck7\" (UniqueName: \"kubernetes.io/projected/abf49362-4adb-43dd-8cd4-f863c86c4f12-kube-api-access-hlck7\") pod \"kserve-controller-manager-66576f5db6-rpr9j\" (UID: \"abf49362-4adb-43dd-8cd4-f863c86c4f12\") " pod="kserve/kserve-controller-manager-66576f5db6-rpr9j"
Sep 17 19:11:58.002734 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:11:58.002736    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/ce9e3abb-c2ec-4228-b7ca-c1ac82737c59-data\") pod \"seaweedfs-64568bcd49-nvdj7\" (UID: \"ce9e3abb-c2ec-4228-b7ca-c1ac82737c59\") " pod="kserve/seaweedfs-64568bcd49-nvdj7"
Sep 17 19:11:58.002980 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:11:58.002879    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/abf49362-4adb-43dd-8cd4-f863c86c4f12-cert\") pod \"kserve-controller-manager-66576f5db6-rpr9j\" (UID: \"abf49362-4adb-43dd-8cd4-f863c86c4f12\") " pod="kserve/kserve-controller-manager-66576f5db6-rpr9j"
Sep 17 19:11:58.002980 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:11:58.002913    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wh2rb\" (UniqueName: \"kubernetes.io/projected/ce9e3abb-c2ec-4228-b7ca-c1ac82737c59-kube-api-access-wh2rb\") pod \"seaweedfs-64568bcd49-nvdj7\" (UID: \"ce9e3abb-c2ec-4228-b7ca-c1ac82737c59\") " pod="kserve/seaweedfs-64568bcd49-nvdj7"
Sep 17 19:11:58.103689 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:11:58.103612    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/abf49362-4adb-43dd-8cd4-f863c86c4f12-cert\") pod \"kserve-controller-manager-66576f5db6-rpr9j\" (UID: \"abf49362-4adb-43dd-8cd4-f863c86c4f12\") " pod="kserve/kserve-controller-manager-66576f5db6-rpr9j"
Sep 17 19:11:58.103689 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:11:58.103649    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wh2rb\" (UniqueName: \"kubernetes.io/projected/ce9e3abb-c2ec-4228-b7ca-c1ac82737c59-kube-api-access-wh2rb\") pod \"seaweedfs-64568bcd49-nvdj7\" (UID: \"ce9e3abb-c2ec-4228-b7ca-c1ac82737c59\") " pod="kserve/seaweedfs-64568bcd49-nvdj7"
Sep 17 19:11:58.103689 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:11:58.103681    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hlck7\" (UniqueName: \"kubernetes.io/projected/abf49362-4adb-43dd-8cd4-f863c86c4f12-kube-api-access-hlck7\") pod \"kserve-controller-manager-66576f5db6-rpr9j\" (UID: \"abf49362-4adb-43dd-8cd4-f863c86c4f12\") " pod="kserve/kserve-controller-manager-66576f5db6-rpr9j"
Sep 17 19:11:58.103922 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:11:58.103715    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/ce9e3abb-c2ec-4228-b7ca-c1ac82737c59-data\") pod \"seaweedfs-64568bcd49-nvdj7\" (UID: \"ce9e3abb-c2ec-4228-b7ca-c1ac82737c59\") " pod="kserve/seaweedfs-64568bcd49-nvdj7"
Sep 17 19:11:58.104148 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:11:58.104126    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/ce9e3abb-c2ec-4228-b7ca-c1ac82737c59-data\") pod \"seaweedfs-64568bcd49-nvdj7\" (UID: \"ce9e3abb-c2ec-4228-b7ca-c1ac82737c59\") " pod="kserve/seaweedfs-64568bcd49-nvdj7"
Sep 17 19:11:58.106230 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:11:58.106208    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/abf49362-4adb-43dd-8cd4-f863c86c4f12-cert\") pod \"kserve-controller-manager-66576f5db6-rpr9j\" (UID: \"abf49362-4adb-43dd-8cd4-f863c86c4f12\") " pod="kserve/kserve-controller-manager-66576f5db6-rpr9j"
Sep 17 19:11:58.112452 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:11:58.112431    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hlck7\" (UniqueName: \"kubernetes.io/projected/abf49362-4adb-43dd-8cd4-f863c86c4f12-kube-api-access-hlck7\") pod \"kserve-controller-manager-66576f5db6-rpr9j\" (UID: \"abf49362-4adb-43dd-8cd4-f863c86c4f12\") " pod="kserve/kserve-controller-manager-66576f5db6-rpr9j"
Sep 17 19:11:58.112676 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:11:58.112656    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wh2rb\" (UniqueName: \"kubernetes.io/projected/ce9e3abb-c2ec-4228-b7ca-c1ac82737c59-kube-api-access-wh2rb\") pod \"seaweedfs-64568bcd49-nvdj7\" (UID: \"ce9e3abb-c2ec-4228-b7ca-c1ac82737c59\") " pod="kserve/seaweedfs-64568bcd49-nvdj7"
Sep 17 19:11:58.149398 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:11:58.149367    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/kserve-controller-manager-66576f5db6-rpr9j"
Sep 17 19:11:58.182879 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:11:58.182852    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/seaweedfs-64568bcd49-nvdj7"
Sep 17 19:11:58.418918 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:11:58.418720    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/kserve-controller-manager-66576f5db6-rpr9j"]
Sep 17 19:11:58.421549 ip-10-0-131-232 kubenswrapper[2582]: W0917 19:11:58.421518    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podabf49362_4adb_43dd_8cd4_f863c86c4f12.slice/crio-1007f19b2da27add0f9d5524dc4875c8b504e727e44bca1411e86b17373e433d WatchSource:0}: Error finding container 1007f19b2da27add0f9d5524dc4875c8b504e727e44bca1411e86b17373e433d: Status 404 returned error can't find the container with id 1007f19b2da27add0f9d5524dc4875c8b504e727e44bca1411e86b17373e433d
Sep 17 19:11:58.422815 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:11:58.422797    2582 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 17 19:11:58.455164 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:11:58.455120    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/seaweedfs-64568bcd49-nvdj7"]
Sep 17 19:11:58.458173 ip-10-0-131-232 kubenswrapper[2582]: W0917 19:11:58.458147    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podce9e3abb_c2ec_4228_b7ca_c1ac82737c59.slice/crio-c17491f1e93314375016eb6857cb022bc13d1fcc6434f181f64ebcf7ab654367 WatchSource:0}: Error finding container c17491f1e93314375016eb6857cb022bc13d1fcc6434f181f64ebcf7ab654367: Status 404 returned error can't find the container with id c17491f1e93314375016eb6857cb022bc13d1fcc6434f181f64ebcf7ab654367
Sep 17 19:11:59.183655 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:11:59.183614    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/kserve-controller-manager-66576f5db6-rpr9j" event={"ID":"abf49362-4adb-43dd-8cd4-f863c86c4f12","Type":"ContainerStarted","Data":"1007f19b2da27add0f9d5524dc4875c8b504e727e44bca1411e86b17373e433d"}
Sep 17 19:11:59.187802 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:11:59.187763    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/seaweedfs-64568bcd49-nvdj7" event={"ID":"ce9e3abb-c2ec-4228-b7ca-c1ac82737c59","Type":"ContainerStarted","Data":"c17491f1e93314375016eb6857cb022bc13d1fcc6434f181f64ebcf7ab654367"}
Sep 17 19:12:02.202014 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:12:02.201898    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/seaweedfs-64568bcd49-nvdj7" event={"ID":"ce9e3abb-c2ec-4228-b7ca-c1ac82737c59","Type":"ContainerStarted","Data":"196c5dec5ff549c71684e1e732e24b0f50d3c0f38e10326ccb9d4439a4c251c7"}
Sep 17 19:12:02.202014 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:12:02.202009    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve/seaweedfs-64568bcd49-nvdj7"
Sep 17 19:12:02.203399 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:12:02.203363    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/kserve-controller-manager-66576f5db6-rpr9j" event={"ID":"abf49362-4adb-43dd-8cd4-f863c86c4f12","Type":"ContainerStarted","Data":"92acf165ffd17b1e38f3f1d3a7cf80929da7987298ea92a83bba63b625898a1b"}
Sep 17 19:12:02.203514 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:12:02.203492    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve/kserve-controller-manager-66576f5db6-rpr9j"
Sep 17 19:12:02.219484 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:12:02.219435    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/seaweedfs-64568bcd49-nvdj7" podStartSLOduration=1.722669036 podStartE2EDuration="5.219420714s" totalImagesPullingTime="3.496751678s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 19:11:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 19:12:02.219169798 +0000 UTC m=+337.990840870" watchObservedRunningTime="2026-09-17 19:12:02.219420714 +0000 UTC m=+337.991091774"
Sep 17 19:12:02.238309 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:12:02.238260    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/kserve-controller-manager-66576f5db6-rpr9j" podStartSLOduration=1.791710959 podStartE2EDuration="5.238243938s" totalImagesPullingTime="3.446532979s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 19:11:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 19:12:02.237191835 +0000 UTC m=+338.008862909" watchObservedRunningTime="2026-09-17 19:12:02.238243938 +0000 UTC m=+338.009914997"
Sep 17 19:12:08.209465 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:12:08.209433    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve/seaweedfs-64568bcd49-nvdj7"
Sep 17 19:12:33.211974 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:12:33.211900    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve/kserve-controller-manager-66576f5db6-rpr9j"
Sep 17 19:12:41.347991 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:12:41.347957    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve/kserve-controller-manager-66576f5db6-rpr9j"]
Sep 17 19:12:41.348429 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:12:41.348175    2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve/kserve-controller-manager-66576f5db6-rpr9j" podUID="abf49362-4adb-43dd-8cd4-f863c86c4f12" containerName="manager" containerID="cri-o://92acf165ffd17b1e38f3f1d3a7cf80929da7987298ea92a83bba63b625898a1b" gracePeriod=10
Sep 17 19:12:41.375762 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:12:41.375731    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/kserve-controller-manager-66576f5db6-rs7mf"]
Sep 17 19:12:41.377812 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:12:41.377797    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/kserve-controller-manager-66576f5db6-rs7mf"
Sep 17 19:12:41.391589 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:12:41.391553    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/kserve-controller-manager-66576f5db6-rs7mf"]
Sep 17 19:12:41.426103 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:12:41.426061    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sqvt8\" (UniqueName: \"kubernetes.io/projected/138aebae-3002-4042-b6d4-9bef95f88da9-kube-api-access-sqvt8\") pod \"kserve-controller-manager-66576f5db6-rs7mf\" (UID: \"138aebae-3002-4042-b6d4-9bef95f88da9\") " pod="kserve/kserve-controller-manager-66576f5db6-rs7mf"
Sep 17 19:12:41.426276 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:12:41.426115    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/138aebae-3002-4042-b6d4-9bef95f88da9-cert\") pod \"kserve-controller-manager-66576f5db6-rs7mf\" (UID: \"138aebae-3002-4042-b6d4-9bef95f88da9\") " pod="kserve/kserve-controller-manager-66576f5db6-rs7mf"
Sep 17 19:12:41.527356 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:12:41.527319    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/138aebae-3002-4042-b6d4-9bef95f88da9-cert\") pod \"kserve-controller-manager-66576f5db6-rs7mf\" (UID: \"138aebae-3002-4042-b6d4-9bef95f88da9\") " pod="kserve/kserve-controller-manager-66576f5db6-rs7mf"
Sep 17 19:12:41.527514 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:12:41.527411    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sqvt8\" (UniqueName: \"kubernetes.io/projected/138aebae-3002-4042-b6d4-9bef95f88da9-kube-api-access-sqvt8\") pod \"kserve-controller-manager-66576f5db6-rs7mf\" (UID: \"138aebae-3002-4042-b6d4-9bef95f88da9\") " pod="kserve/kserve-controller-manager-66576f5db6-rs7mf"
Sep 17 19:12:41.530461 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:12:41.530425    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/138aebae-3002-4042-b6d4-9bef95f88da9-cert\") pod \"kserve-controller-manager-66576f5db6-rs7mf\" (UID: \"138aebae-3002-4042-b6d4-9bef95f88da9\") " pod="kserve/kserve-controller-manager-66576f5db6-rs7mf"
Sep 17 19:12:41.536810 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:12:41.536777    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sqvt8\" (UniqueName: \"kubernetes.io/projected/138aebae-3002-4042-b6d4-9bef95f88da9-kube-api-access-sqvt8\") pod \"kserve-controller-manager-66576f5db6-rs7mf\" (UID: \"138aebae-3002-4042-b6d4-9bef95f88da9\") " pod="kserve/kserve-controller-manager-66576f5db6-rs7mf"
Sep 17 19:12:41.588249 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:12:41.588221    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/kserve-controller-manager-66576f5db6-rpr9j"
Sep 17 19:12:41.627785 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:12:41.627699    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hlck7\" (UniqueName: \"kubernetes.io/projected/abf49362-4adb-43dd-8cd4-f863c86c4f12-kube-api-access-hlck7\") pod \"abf49362-4adb-43dd-8cd4-f863c86c4f12\" (UID: \"abf49362-4adb-43dd-8cd4-f863c86c4f12\") "
Sep 17 19:12:41.627785 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:12:41.627746    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/abf49362-4adb-43dd-8cd4-f863c86c4f12-cert\") pod \"abf49362-4adb-43dd-8cd4-f863c86c4f12\" (UID: \"abf49362-4adb-43dd-8cd4-f863c86c4f12\") "
Sep 17 19:12:41.630134 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:12:41.630104    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/abf49362-4adb-43dd-8cd4-f863c86c4f12-cert" (OuterVolumeSpecName: "cert") pod "abf49362-4adb-43dd-8cd4-f863c86c4f12" (UID: "abf49362-4adb-43dd-8cd4-f863c86c4f12"). InnerVolumeSpecName "cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 19:12:41.630248 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:12:41.630151    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/abf49362-4adb-43dd-8cd4-f863c86c4f12-kube-api-access-hlck7" (OuterVolumeSpecName: "kube-api-access-hlck7") pod "abf49362-4adb-43dd-8cd4-f863c86c4f12" (UID: "abf49362-4adb-43dd-8cd4-f863c86c4f12"). InnerVolumeSpecName "kube-api-access-hlck7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 19:12:41.728433 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:12:41.728397    2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hlck7\" (UniqueName: \"kubernetes.io/projected/abf49362-4adb-43dd-8cd4-f863c86c4f12-kube-api-access-hlck7\") on node \"ip-10-0-131-232.ec2.internal\" DevicePath \"\""
Sep 17 19:12:41.728433 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:12:41.728429    2582 reconciler_common.go:299] "Volume detached for volume \"cert\" (UniqueName: \"kubernetes.io/secret/abf49362-4adb-43dd-8cd4-f863c86c4f12-cert\") on node \"ip-10-0-131-232.ec2.internal\" DevicePath \"\""
Sep 17 19:12:41.741411 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:12:41.741385    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/kserve-controller-manager-66576f5db6-rs7mf"
Sep 17 19:12:42.014120 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:12:42.014091    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/kserve-controller-manager-66576f5db6-rs7mf"]
Sep 17 19:12:42.016888 ip-10-0-131-232 kubenswrapper[2582]: W0917 19:12:42.016859    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod138aebae_3002_4042_b6d4_9bef95f88da9.slice/crio-e440d6bdce494182794216ef6c0140c502256ab8591aac0e77d7c9068b6de7a5 WatchSource:0}: Error finding container e440d6bdce494182794216ef6c0140c502256ab8591aac0e77d7c9068b6de7a5: Status 404 returned error can't find the container with id e440d6bdce494182794216ef6c0140c502256ab8591aac0e77d7c9068b6de7a5
Sep 17 19:12:42.349214 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:12:42.349115    2582 generic.go:356] "Generic (PLEG): container finished" podID="abf49362-4adb-43dd-8cd4-f863c86c4f12" containerID="92acf165ffd17b1e38f3f1d3a7cf80929da7987298ea92a83bba63b625898a1b" exitCode=0
Sep 17 19:12:42.349214 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:12:42.349159    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/kserve-controller-manager-66576f5db6-rpr9j"
Sep 17 19:12:42.349214 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:12:42.349203    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/kserve-controller-manager-66576f5db6-rpr9j" event={"ID":"abf49362-4adb-43dd-8cd4-f863c86c4f12","Type":"ContainerDied","Data":"92acf165ffd17b1e38f3f1d3a7cf80929da7987298ea92a83bba63b625898a1b"}
Sep 17 19:12:42.349706 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:12:42.349257    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/kserve-controller-manager-66576f5db6-rpr9j" event={"ID":"abf49362-4adb-43dd-8cd4-f863c86c4f12","Type":"ContainerDied","Data":"1007f19b2da27add0f9d5524dc4875c8b504e727e44bca1411e86b17373e433d"}
Sep 17 19:12:42.349706 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:12:42.349281    2582 scope.go:117] "RemoveContainer" containerID="92acf165ffd17b1e38f3f1d3a7cf80929da7987298ea92a83bba63b625898a1b"
Sep 17 19:12:42.350584 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:12:42.350452    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/kserve-controller-manager-66576f5db6-rs7mf" event={"ID":"138aebae-3002-4042-b6d4-9bef95f88da9","Type":"ContainerStarted","Data":"e440d6bdce494182794216ef6c0140c502256ab8591aac0e77d7c9068b6de7a5"}
Sep 17 19:12:42.357438 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:12:42.357417    2582 scope.go:117] "RemoveContainer" containerID="92acf165ffd17b1e38f3f1d3a7cf80929da7987298ea92a83bba63b625898a1b"
Sep 17 19:12:42.357736 ip-10-0-131-232 kubenswrapper[2582]: E0917 19:12:42.357717    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"92acf165ffd17b1e38f3f1d3a7cf80929da7987298ea92a83bba63b625898a1b\": container with ID starting with 92acf165ffd17b1e38f3f1d3a7cf80929da7987298ea92a83bba63b625898a1b not found: ID does not exist" containerID="92acf165ffd17b1e38f3f1d3a7cf80929da7987298ea92a83bba63b625898a1b"
Sep 17 19:12:42.357850 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:12:42.357745    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"92acf165ffd17b1e38f3f1d3a7cf80929da7987298ea92a83bba63b625898a1b"} err="failed to get container status \"92acf165ffd17b1e38f3f1d3a7cf80929da7987298ea92a83bba63b625898a1b\": rpc error: code = NotFound desc = could not find container \"92acf165ffd17b1e38f3f1d3a7cf80929da7987298ea92a83bba63b625898a1b\": container with ID starting with 92acf165ffd17b1e38f3f1d3a7cf80929da7987298ea92a83bba63b625898a1b not found: ID does not exist"
Sep 17 19:12:42.370541 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:12:42.370513    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve/kserve-controller-manager-66576f5db6-rpr9j"]
Sep 17 19:12:42.374213 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:12:42.374188    2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve/kserve-controller-manager-66576f5db6-rpr9j"]
Sep 17 19:12:42.767864 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:12:42.767814    2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="abf49362-4adb-43dd-8cd4-f863c86c4f12" path="/var/lib/kubelet/pods/abf49362-4adb-43dd-8cd4-f863c86c4f12/volumes"
Sep 17 19:12:43.357646 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:12:43.357610    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/kserve-controller-manager-66576f5db6-rs7mf" event={"ID":"138aebae-3002-4042-b6d4-9bef95f88da9","Type":"ContainerStarted","Data":"7fb7569cedae58c62702ce3a699c84ca2cd6f769adfadc4364a9b55cba692553"}
Sep 17 19:12:43.358061 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:12:43.357718    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve/kserve-controller-manager-66576f5db6-rs7mf"
Sep 17 19:12:43.375909 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:12:43.375845    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/kserve-controller-manager-66576f5db6-rs7mf" podStartSLOduration=1.734591008 podStartE2EDuration="2.375810187s" totalImagesPullingTime="641.219179ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 19:12:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 19:12:43.374272802 +0000 UTC m=+379.145943865" watchObservedRunningTime="2026-09-17 19:12:43.375810187 +0000 UTC m=+379.147481248"
Sep 17 19:13:03.737439 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:13:03.737405    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-5c5887565b-f5gjx"]
Sep 17 19:13:03.737938 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:13:03.737704    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="abf49362-4adb-43dd-8cd4-f863c86c4f12" containerName="manager"
Sep 17 19:13:03.737938 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:13:03.737716    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="abf49362-4adb-43dd-8cd4-f863c86c4f12" containerName="manager"
Sep 17 19:13:03.737938 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:13:03.737767    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="abf49362-4adb-43dd-8cd4-f863c86c4f12" containerName="manager"
Sep 17 19:13:03.740523 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:13:03.740504    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5c5887565b-f5gjx"
Sep 17 19:13:03.743884 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:13:03.743863    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\""
Sep 17 19:13:03.743991 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:13:03.743863    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\""
Sep 17 19:13:03.743991 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:13:03.743870    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\""
Sep 17 19:13:03.744214 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:13:03.744196    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\""
Sep 17 19:13:03.744276 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:13:03.744253    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\""
Sep 17 19:13:03.744948 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:13:03.744931    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-rlcgb\""
Sep 17 19:13:03.744948 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:13:03.744947    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\""
Sep 17 19:13:03.745101 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:13:03.744973    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\""
Sep 17 19:13:03.749534 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:13:03.749506    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\""
Sep 17 19:13:03.750448 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:13:03.750425    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-5c5887565b-f5gjx"]
Sep 17 19:13:03.802045 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:13:03.802004    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/cbbdc2a7-0708-4445-97a8-2afd903120e3-service-ca\") pod \"console-5c5887565b-f5gjx\" (UID: \"cbbdc2a7-0708-4445-97a8-2afd903120e3\") " pod="openshift-console/console-5c5887565b-f5gjx"
Sep 17 19:13:03.802045 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:13:03.802049    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gqp74\" (UniqueName: \"kubernetes.io/projected/cbbdc2a7-0708-4445-97a8-2afd903120e3-kube-api-access-gqp74\") pod \"console-5c5887565b-f5gjx\" (UID: \"cbbdc2a7-0708-4445-97a8-2afd903120e3\") " pod="openshift-console/console-5c5887565b-f5gjx"
Sep 17 19:13:03.802262 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:13:03.802087    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/cbbdc2a7-0708-4445-97a8-2afd903120e3-console-config\") pod \"console-5c5887565b-f5gjx\" (UID: \"cbbdc2a7-0708-4445-97a8-2afd903120e3\") " pod="openshift-console/console-5c5887565b-f5gjx"
Sep 17 19:13:03.802262 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:13:03.802125    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/cbbdc2a7-0708-4445-97a8-2afd903120e3-console-serving-cert\") pod \"console-5c5887565b-f5gjx\" (UID: \"cbbdc2a7-0708-4445-97a8-2afd903120e3\") " pod="openshift-console/console-5c5887565b-f5gjx"
Sep 17 19:13:03.802262 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:13:03.802189    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/cbbdc2a7-0708-4445-97a8-2afd903120e3-console-oauth-config\") pod \"console-5c5887565b-f5gjx\" (UID: \"cbbdc2a7-0708-4445-97a8-2afd903120e3\") " pod="openshift-console/console-5c5887565b-f5gjx"
Sep 17 19:13:03.802262 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:13:03.802233    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/cbbdc2a7-0708-4445-97a8-2afd903120e3-trusted-ca-bundle\") pod \"console-5c5887565b-f5gjx\" (UID: \"cbbdc2a7-0708-4445-97a8-2afd903120e3\") " pod="openshift-console/console-5c5887565b-f5gjx"
Sep 17 19:13:03.802262 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:13:03.802251    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/cbbdc2a7-0708-4445-97a8-2afd903120e3-oauth-serving-cert\") pod \"console-5c5887565b-f5gjx\" (UID: \"cbbdc2a7-0708-4445-97a8-2afd903120e3\") " pod="openshift-console/console-5c5887565b-f5gjx"
Sep 17 19:13:03.903440 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:13:03.903401    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/cbbdc2a7-0708-4445-97a8-2afd903120e3-console-config\") pod \"console-5c5887565b-f5gjx\" (UID: \"cbbdc2a7-0708-4445-97a8-2afd903120e3\") " pod="openshift-console/console-5c5887565b-f5gjx"
Sep 17 19:13:03.903440 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:13:03.903442    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/cbbdc2a7-0708-4445-97a8-2afd903120e3-console-serving-cert\") pod \"console-5c5887565b-f5gjx\" (UID: \"cbbdc2a7-0708-4445-97a8-2afd903120e3\") " pod="openshift-console/console-5c5887565b-f5gjx"
Sep 17 19:13:03.903653 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:13:03.903470    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/cbbdc2a7-0708-4445-97a8-2afd903120e3-console-oauth-config\") pod \"console-5c5887565b-f5gjx\" (UID: \"cbbdc2a7-0708-4445-97a8-2afd903120e3\") " pod="openshift-console/console-5c5887565b-f5gjx"
Sep 17 19:13:03.903653 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:13:03.903498    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/cbbdc2a7-0708-4445-97a8-2afd903120e3-trusted-ca-bundle\") pod \"console-5c5887565b-f5gjx\" (UID: \"cbbdc2a7-0708-4445-97a8-2afd903120e3\") " pod="openshift-console/console-5c5887565b-f5gjx"
Sep 17 19:13:03.903653 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:13:03.903519    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/cbbdc2a7-0708-4445-97a8-2afd903120e3-oauth-serving-cert\") pod \"console-5c5887565b-f5gjx\" (UID: \"cbbdc2a7-0708-4445-97a8-2afd903120e3\") " pod="openshift-console/console-5c5887565b-f5gjx"
Sep 17 19:13:03.903653 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:13:03.903546    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/cbbdc2a7-0708-4445-97a8-2afd903120e3-service-ca\") pod \"console-5c5887565b-f5gjx\" (UID: \"cbbdc2a7-0708-4445-97a8-2afd903120e3\") " pod="openshift-console/console-5c5887565b-f5gjx"
Sep 17 19:13:03.903653 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:13:03.903574    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gqp74\" (UniqueName: \"kubernetes.io/projected/cbbdc2a7-0708-4445-97a8-2afd903120e3-kube-api-access-gqp74\") pod \"console-5c5887565b-f5gjx\" (UID: \"cbbdc2a7-0708-4445-97a8-2afd903120e3\") " pod="openshift-console/console-5c5887565b-f5gjx"
Sep 17 19:13:03.904263 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:13:03.904234    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/cbbdc2a7-0708-4445-97a8-2afd903120e3-console-config\") pod \"console-5c5887565b-f5gjx\" (UID: \"cbbdc2a7-0708-4445-97a8-2afd903120e3\") " pod="openshift-console/console-5c5887565b-f5gjx"
Sep 17 19:13:03.904358 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:13:03.904346    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/cbbdc2a7-0708-4445-97a8-2afd903120e3-trusted-ca-bundle\") pod \"console-5c5887565b-f5gjx\" (UID: \"cbbdc2a7-0708-4445-97a8-2afd903120e3\") " pod="openshift-console/console-5c5887565b-f5gjx"
Sep 17 19:13:03.904398 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:13:03.904370    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/cbbdc2a7-0708-4445-97a8-2afd903120e3-oauth-serving-cert\") pod \"console-5c5887565b-f5gjx\" (UID: \"cbbdc2a7-0708-4445-97a8-2afd903120e3\") " pod="openshift-console/console-5c5887565b-f5gjx"
Sep 17 19:13:03.904433 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:13:03.904400    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/cbbdc2a7-0708-4445-97a8-2afd903120e3-service-ca\") pod \"console-5c5887565b-f5gjx\" (UID: \"cbbdc2a7-0708-4445-97a8-2afd903120e3\") " pod="openshift-console/console-5c5887565b-f5gjx"
Sep 17 19:13:03.906540 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:13:03.906516    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/cbbdc2a7-0708-4445-97a8-2afd903120e3-console-oauth-config\") pod \"console-5c5887565b-f5gjx\" (UID: \"cbbdc2a7-0708-4445-97a8-2afd903120e3\") " pod="openshift-console/console-5c5887565b-f5gjx"
Sep 17 19:13:03.906644 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:13:03.906625    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/cbbdc2a7-0708-4445-97a8-2afd903120e3-console-serving-cert\") pod \"console-5c5887565b-f5gjx\" (UID: \"cbbdc2a7-0708-4445-97a8-2afd903120e3\") " pod="openshift-console/console-5c5887565b-f5gjx"
Sep 17 19:13:03.914566 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:13:03.914536    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gqp74\" (UniqueName: \"kubernetes.io/projected/cbbdc2a7-0708-4445-97a8-2afd903120e3-kube-api-access-gqp74\") pod \"console-5c5887565b-f5gjx\" (UID: \"cbbdc2a7-0708-4445-97a8-2afd903120e3\") " pod="openshift-console/console-5c5887565b-f5gjx"
Sep 17 19:13:04.050465 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:13:04.050380    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5c5887565b-f5gjx"
Sep 17 19:13:04.325498 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:13:04.325409    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-5c5887565b-f5gjx"]
Sep 17 19:13:04.328081 ip-10-0-131-232 kubenswrapper[2582]: W0917 19:13:04.328038    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcbbdc2a7_0708_4445_97a8_2afd903120e3.slice/crio-d0b091f3c0c97e5b339cd7c8b1047ba891125a227dfe9cb718ab8cec6f6e37dd WatchSource:0}: Error finding container d0b091f3c0c97e5b339cd7c8b1047ba891125a227dfe9cb718ab8cec6f6e37dd: Status 404 returned error can't find the container with id d0b091f3c0c97e5b339cd7c8b1047ba891125a227dfe9cb718ab8cec6f6e37dd
Sep 17 19:13:04.436295 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:13:04.436254    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5c5887565b-f5gjx" event={"ID":"cbbdc2a7-0708-4445-97a8-2afd903120e3","Type":"ContainerStarted","Data":"051422c7eca6515b2312431be084853da09cf0345cf1ad64a739e8e9b805134a"}
Sep 17 19:13:04.436439 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:13:04.436320    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5c5887565b-f5gjx" event={"ID":"cbbdc2a7-0708-4445-97a8-2afd903120e3","Type":"ContainerStarted","Data":"d0b091f3c0c97e5b339cd7c8b1047ba891125a227dfe9cb718ab8cec6f6e37dd"}
Sep 17 19:13:04.454391 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:13:04.454329    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-5c5887565b-f5gjx" podStartSLOduration=1.454309908 podStartE2EDuration="1.454309908s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 19:13:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 19:13:04.453787841 +0000 UTC m=+400.225458900" watchObservedRunningTime="2026-09-17 19:13:04.454309908 +0000 UTC m=+400.225980968"
Sep 17 19:13:14.051561 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:13:14.051527    2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-5c5887565b-f5gjx"
Sep 17 19:13:14.051963 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:13:14.051599    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-5c5887565b-f5gjx"
Sep 17 19:13:14.056396 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:13:14.056372    2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-5c5887565b-f5gjx"
Sep 17 19:13:14.366960 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:13:14.366875    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve/kserve-controller-manager-66576f5db6-rs7mf"
Sep 17 19:13:14.480791 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:13:14.480758    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-5c5887565b-f5gjx"
Sep 17 19:13:15.707434 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:13:15.707398    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/model-serving-api-7b954ffc95-mbjd9"]
Sep 17 19:13:15.710886 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:13:15.710864    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/model-serving-api-7b954ffc95-mbjd9"
Sep 17 19:13:15.713757 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:13:15.713730    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"model-serving-api-tls\""
Sep 17 19:13:15.713909 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:13:15.713784    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"model-serving-api-dockercfg-v8z76\""
Sep 17 19:13:15.721236 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:13:15.721197    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/odh-model-controller-7b966d7984-ck5rg"]
Sep 17 19:13:15.724512 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:13:15.724491    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/model-serving-api-7b954ffc95-mbjd9"]
Sep 17 19:13:15.724670 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:13:15.724635    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/odh-model-controller-7b966d7984-ck5rg"
Sep 17 19:13:15.727356 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:13:15.727337    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"odh-model-controller-webhook-cert\""
Sep 17 19:13:15.727539 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:13:15.727514    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"odh-model-controller-dockercfg-tmdrd\""
Sep 17 19:13:15.727644 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:13:15.727420    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"odh-model-controller-metrics-tls\""
Sep 17 19:13:15.736948 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:13:15.736902    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/odh-model-controller-7b966d7984-ck5rg"]
Sep 17 19:13:15.801374 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:13:15.801335    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/1f09d228-0f0e-47c4-96d3-bde8bf84941b-cert\") pod \"odh-model-controller-7b966d7984-ck5rg\" (UID: \"1f09d228-0f0e-47c4-96d3-bde8bf84941b\") " pod="kserve/odh-model-controller-7b966d7984-ck5rg"
Sep 17 19:13:15.801580 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:13:15.801401    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vzj7g\" (UniqueName: \"kubernetes.io/projected/b55ebb64-9548-4169-b74c-936fc5290e50-kube-api-access-vzj7g\") pod \"model-serving-api-7b954ffc95-mbjd9\" (UID: \"b55ebb64-9548-4169-b74c-936fc5290e50\") " pod="kserve/model-serving-api-7b954ffc95-mbjd9"
Sep 17 19:13:15.801580 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:13:15.801491    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-frqjb\" (UniqueName: \"kubernetes.io/projected/1f09d228-0f0e-47c4-96d3-bde8bf84941b-kube-api-access-frqjb\") pod \"odh-model-controller-7b966d7984-ck5rg\" (UID: \"1f09d228-0f0e-47c4-96d3-bde8bf84941b\") " pod="kserve/odh-model-controller-7b966d7984-ck5rg"
Sep 17 19:13:15.801580 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:13:15.801540    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b55ebb64-9548-4169-b74c-936fc5290e50-tls-certs\") pod \"model-serving-api-7b954ffc95-mbjd9\" (UID: \"b55ebb64-9548-4169-b74c-936fc5290e50\") " pod="kserve/model-serving-api-7b954ffc95-mbjd9"
Sep 17 19:13:15.801710 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:13:15.801579    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1f09d228-0f0e-47c4-96d3-bde8bf84941b-metrics-certs\") pod \"odh-model-controller-7b966d7984-ck5rg\" (UID: \"1f09d228-0f0e-47c4-96d3-bde8bf84941b\") " pod="kserve/odh-model-controller-7b966d7984-ck5rg"
Sep 17 19:13:15.902540 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:13:15.902500    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vzj7g\" (UniqueName: \"kubernetes.io/projected/b55ebb64-9548-4169-b74c-936fc5290e50-kube-api-access-vzj7g\") pod \"model-serving-api-7b954ffc95-mbjd9\" (UID: \"b55ebb64-9548-4169-b74c-936fc5290e50\") " pod="kserve/model-serving-api-7b954ffc95-mbjd9"
Sep 17 19:13:15.902737 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:13:15.902550    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-frqjb\" (UniqueName: \"kubernetes.io/projected/1f09d228-0f0e-47c4-96d3-bde8bf84941b-kube-api-access-frqjb\") pod \"odh-model-controller-7b966d7984-ck5rg\" (UID: \"1f09d228-0f0e-47c4-96d3-bde8bf84941b\") " pod="kserve/odh-model-controller-7b966d7984-ck5rg"
Sep 17 19:13:15.902737 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:13:15.902584    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b55ebb64-9548-4169-b74c-936fc5290e50-tls-certs\") pod \"model-serving-api-7b954ffc95-mbjd9\" (UID: \"b55ebb64-9548-4169-b74c-936fc5290e50\") " pod="kserve/model-serving-api-7b954ffc95-mbjd9"
Sep 17 19:13:15.902737 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:13:15.902612    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1f09d228-0f0e-47c4-96d3-bde8bf84941b-metrics-certs\") pod \"odh-model-controller-7b966d7984-ck5rg\" (UID: \"1f09d228-0f0e-47c4-96d3-bde8bf84941b\") " pod="kserve/odh-model-controller-7b966d7984-ck5rg"
Sep 17 19:13:15.902737 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:13:15.902662    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/1f09d228-0f0e-47c4-96d3-bde8bf84941b-cert\") pod \"odh-model-controller-7b966d7984-ck5rg\" (UID: \"1f09d228-0f0e-47c4-96d3-bde8bf84941b\") " pod="kserve/odh-model-controller-7b966d7984-ck5rg"
Sep 17 19:13:15.903006 ip-10-0-131-232 kubenswrapper[2582]: E0917 19:13:15.902746    2582 secret.go:189] Couldn't get secret kserve/odh-model-controller-metrics-tls: secret "odh-model-controller-metrics-tls" not found
Sep 17 19:13:15.903006 ip-10-0-131-232 kubenswrapper[2582]: E0917 19:13:15.902815    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1f09d228-0f0e-47c4-96d3-bde8bf84941b-metrics-certs podName:1f09d228-0f0e-47c4-96d3-bde8bf84941b nodeName:}" failed. No retries permitted until 2026-09-17 19:13:16.402794965 +0000 UTC m=+412.174466006 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/1f09d228-0f0e-47c4-96d3-bde8bf84941b-metrics-certs") pod "odh-model-controller-7b966d7984-ck5rg" (UID: "1f09d228-0f0e-47c4-96d3-bde8bf84941b") : secret "odh-model-controller-metrics-tls" not found
Sep 17 19:13:15.905440 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:13:15.905409    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b55ebb64-9548-4169-b74c-936fc5290e50-tls-certs\") pod \"model-serving-api-7b954ffc95-mbjd9\" (UID: \"b55ebb64-9548-4169-b74c-936fc5290e50\") " pod="kserve/model-serving-api-7b954ffc95-mbjd9"
Sep 17 19:13:15.905544 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:13:15.905475    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/1f09d228-0f0e-47c4-96d3-bde8bf84941b-cert\") pod \"odh-model-controller-7b966d7984-ck5rg\" (UID: \"1f09d228-0f0e-47c4-96d3-bde8bf84941b\") " pod="kserve/odh-model-controller-7b966d7984-ck5rg"
Sep 17 19:13:15.917878 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:13:15.917847    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vzj7g\" (UniqueName: \"kubernetes.io/projected/b55ebb64-9548-4169-b74c-936fc5290e50-kube-api-access-vzj7g\") pod \"model-serving-api-7b954ffc95-mbjd9\" (UID: \"b55ebb64-9548-4169-b74c-936fc5290e50\") " pod="kserve/model-serving-api-7b954ffc95-mbjd9"
Sep 17 19:13:15.918066 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:13:15.918048    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-frqjb\" (UniqueName: \"kubernetes.io/projected/1f09d228-0f0e-47c4-96d3-bde8bf84941b-kube-api-access-frqjb\") pod \"odh-model-controller-7b966d7984-ck5rg\" (UID: \"1f09d228-0f0e-47c4-96d3-bde8bf84941b\") " pod="kserve/odh-model-controller-7b966d7984-ck5rg"
Sep 17 19:13:16.023382 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:13:16.023287    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/model-serving-api-7b954ffc95-mbjd9"
Sep 17 19:13:16.294063 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:13:16.294036    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/model-serving-api-7b954ffc95-mbjd9"]
Sep 17 19:13:16.296587 ip-10-0-131-232 kubenswrapper[2582]: W0917 19:13:16.296559    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb55ebb64_9548_4169_b74c_936fc5290e50.slice/crio-f5142922934dc9202cd3e213a239e98e28c0c81d5caf9c43442c05ef2a4ef936 WatchSource:0}: Error finding container f5142922934dc9202cd3e213a239e98e28c0c81d5caf9c43442c05ef2a4ef936: Status 404 returned error can't find the container with id f5142922934dc9202cd3e213a239e98e28c0c81d5caf9c43442c05ef2a4ef936
Sep 17 19:13:16.406397 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:13:16.406360    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1f09d228-0f0e-47c4-96d3-bde8bf84941b-metrics-certs\") pod \"odh-model-controller-7b966d7984-ck5rg\" (UID: \"1f09d228-0f0e-47c4-96d3-bde8bf84941b\") " pod="kserve/odh-model-controller-7b966d7984-ck5rg"
Sep 17 19:13:16.409033 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:13:16.409007    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1f09d228-0f0e-47c4-96d3-bde8bf84941b-metrics-certs\") pod \"odh-model-controller-7b966d7984-ck5rg\" (UID: \"1f09d228-0f0e-47c4-96d3-bde8bf84941b\") " pod="kserve/odh-model-controller-7b966d7984-ck5rg"
Sep 17 19:13:16.485923 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:13:16.485889    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/model-serving-api-7b954ffc95-mbjd9" event={"ID":"b55ebb64-9548-4169-b74c-936fc5290e50","Type":"ContainerStarted","Data":"f5142922934dc9202cd3e213a239e98e28c0c81d5caf9c43442c05ef2a4ef936"}
Sep 17 19:13:16.637951 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:13:16.637913    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/odh-model-controller-7b966d7984-ck5rg"
Sep 17 19:13:16.913977 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:13:16.913675    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/odh-model-controller-7b966d7984-ck5rg"]
Sep 17 19:13:16.916697 ip-10-0-131-232 kubenswrapper[2582]: W0917 19:13:16.916657    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1f09d228_0f0e_47c4_96d3_bde8bf84941b.slice/crio-77308ca35345190071036af97014ea441719101f6ca731aab33f1bbbd63d2917 WatchSource:0}: Error finding container 77308ca35345190071036af97014ea441719101f6ca731aab33f1bbbd63d2917: Status 404 returned error can't find the container with id 77308ca35345190071036af97014ea441719101f6ca731aab33f1bbbd63d2917
Sep 17 19:13:17.495056 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:13:17.495012    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/odh-model-controller-7b966d7984-ck5rg" event={"ID":"1f09d228-0f0e-47c4-96d3-bde8bf84941b","Type":"ContainerStarted","Data":"77308ca35345190071036af97014ea441719101f6ca731aab33f1bbbd63d2917"}
Sep 17 19:13:20.510792 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:13:20.510741    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/model-serving-api-7b954ffc95-mbjd9" event={"ID":"b55ebb64-9548-4169-b74c-936fc5290e50","Type":"ContainerStarted","Data":"b4bf269ea1fff5e394477a5471d8533ff2c0ce0f531f6a69d9da70084276e52d"}
Sep 17 19:13:20.511253 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:13:20.510844    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve/model-serving-api-7b954ffc95-mbjd9"
Sep 17 19:13:20.512292 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:13:20.512267    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/odh-model-controller-7b966d7984-ck5rg" event={"ID":"1f09d228-0f0e-47c4-96d3-bde8bf84941b","Type":"ContainerStarted","Data":"e886b7ef1293caeba8a303b2f7606ddb53fa41ccc45a9d76dbf57ddd4ee790bd"}
Sep 17 19:13:20.512419 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:13:20.512405    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve/odh-model-controller-7b966d7984-ck5rg"
Sep 17 19:13:20.530010 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:13:20.529944    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/model-serving-api-7b954ffc95-mbjd9" podStartSLOduration=1.826778638 podStartE2EDuration="5.529925601s" totalImagesPullingTime="3.703146963s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 19:13:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 19:13:20.528786064 +0000 UTC m=+416.300457128" watchObservedRunningTime="2026-09-17 19:13:20.529925601 +0000 UTC m=+416.301596664"
Sep 17 19:13:20.551580 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:13:20.551527    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/odh-model-controller-7b966d7984-ck5rg" podStartSLOduration=2.464730467 podStartE2EDuration="5.551508013s" totalImagesPullingTime="3.086777546s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 19:13:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 19:13:20.550497745 +0000 UTC m=+416.322168850" watchObservedRunningTime="2026-09-17 19:13:20.551508013 +0000 UTC m=+416.323179075"
Sep 17 19:13:31.519801 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:13:31.519769    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve/odh-model-controller-7b966d7984-ck5rg"
Sep 17 19:13:31.521541 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:13:31.521519    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve/model-serving-api-7b954ffc95-mbjd9"
Sep 17 19:15:05.767734 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:15:05.767695    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-raw-sklearn-batcher-72d03-predictor-d5dddf8d4-2fxrg"]
Sep 17 19:15:05.770953 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:15:05.770924    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-72d03-predictor-d5dddf8d4-2fxrg"
Sep 17 19:15:05.773913 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:15:05.773883    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Sep 17 19:15:05.774056 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:15:05.774038    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-raw-sklearn-batcher-72d03-predictor-serving-cert\""
Sep 17 19:15:05.774660 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:15:05.774641    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-vpx7x\""
Sep 17 19:15:05.774787 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:15:05.774685    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Sep 17 19:15:05.774787 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:15:05.774700    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Sep 17 19:15:05.774927 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:15:05.774789    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-raw-sklearn-batcher-72d03-kube-rbac-proxy-sar-config\""
Sep 17 19:15:05.774927 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:15:05.774897    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Sep 17 19:15:05.783700 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:15:05.783679    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-raw-sklearn-batcher-72d03-predictor-d5dddf8d4-2fxrg"]
Sep 17 19:15:05.848511 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:15:05.848475    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nr5ll\" (UniqueName: \"kubernetes.io/projected/6b10759f-c25e-4045-a992-7fb4bd1edc2f-kube-api-access-nr5ll\") pod \"isvc-raw-sklearn-batcher-72d03-predictor-d5dddf8d4-2fxrg\" (UID: \"6b10759f-c25e-4045-a992-7fb4bd1edc2f\") " pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-72d03-predictor-d5dddf8d4-2fxrg"
Sep 17 19:15:05.848714 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:15:05.848523    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/6b10759f-c25e-4045-a992-7fb4bd1edc2f-kserve-provision-location\") pod \"isvc-raw-sklearn-batcher-72d03-predictor-d5dddf8d4-2fxrg\" (UID: \"6b10759f-c25e-4045-a992-7fb4bd1edc2f\") " pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-72d03-predictor-d5dddf8d4-2fxrg"
Sep 17 19:15:05.848714 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:15:05.848589    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/6b10759f-c25e-4045-a992-7fb4bd1edc2f-cabundle-cert\") pod \"isvc-raw-sklearn-batcher-72d03-predictor-d5dddf8d4-2fxrg\" (UID: \"6b10759f-c25e-4045-a992-7fb4bd1edc2f\") " pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-72d03-predictor-d5dddf8d4-2fxrg"
Sep 17 19:15:05.848714 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:15:05.848626    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6b10759f-c25e-4045-a992-7fb4bd1edc2f-proxy-tls\") pod \"isvc-raw-sklearn-batcher-72d03-predictor-d5dddf8d4-2fxrg\" (UID: \"6b10759f-c25e-4045-a992-7fb4bd1edc2f\") " pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-72d03-predictor-d5dddf8d4-2fxrg"
Sep 17 19:15:05.848714 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:15:05.848649    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/6b10759f-c25e-4045-a992-7fb4bd1edc2f-kube-rbac-proxy-sar-config\") pod \"isvc-raw-sklearn-batcher-72d03-predictor-d5dddf8d4-2fxrg\" (UID: \"6b10759f-c25e-4045-a992-7fb4bd1edc2f\") " pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-72d03-predictor-d5dddf8d4-2fxrg"
Sep 17 19:15:05.949563 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:15:05.949527    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nr5ll\" (UniqueName: \"kubernetes.io/projected/6b10759f-c25e-4045-a992-7fb4bd1edc2f-kube-api-access-nr5ll\") pod \"isvc-raw-sklearn-batcher-72d03-predictor-d5dddf8d4-2fxrg\" (UID: \"6b10759f-c25e-4045-a992-7fb4bd1edc2f\") " pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-72d03-predictor-d5dddf8d4-2fxrg"
Sep 17 19:15:05.949728 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:15:05.949570    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/6b10759f-c25e-4045-a992-7fb4bd1edc2f-kserve-provision-location\") pod \"isvc-raw-sklearn-batcher-72d03-predictor-d5dddf8d4-2fxrg\" (UID: \"6b10759f-c25e-4045-a992-7fb4bd1edc2f\") " pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-72d03-predictor-d5dddf8d4-2fxrg"
Sep 17 19:15:05.949728 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:15:05.949599    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/6b10759f-c25e-4045-a992-7fb4bd1edc2f-cabundle-cert\") pod \"isvc-raw-sklearn-batcher-72d03-predictor-d5dddf8d4-2fxrg\" (UID: \"6b10759f-c25e-4045-a992-7fb4bd1edc2f\") " pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-72d03-predictor-d5dddf8d4-2fxrg"
Sep 17 19:15:05.949728 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:15:05.949629    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6b10759f-c25e-4045-a992-7fb4bd1edc2f-proxy-tls\") pod \"isvc-raw-sklearn-batcher-72d03-predictor-d5dddf8d4-2fxrg\" (UID: \"6b10759f-c25e-4045-a992-7fb4bd1edc2f\") " pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-72d03-predictor-d5dddf8d4-2fxrg"
Sep 17 19:15:05.949728 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:15:05.949660    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/6b10759f-c25e-4045-a992-7fb4bd1edc2f-kube-rbac-proxy-sar-config\") pod \"isvc-raw-sklearn-batcher-72d03-predictor-d5dddf8d4-2fxrg\" (UID: \"6b10759f-c25e-4045-a992-7fb4bd1edc2f\") " pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-72d03-predictor-d5dddf8d4-2fxrg"
Sep 17 19:15:05.950016 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:15:05.949996    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/6b10759f-c25e-4045-a992-7fb4bd1edc2f-kserve-provision-location\") pod \"isvc-raw-sklearn-batcher-72d03-predictor-d5dddf8d4-2fxrg\" (UID: \"6b10759f-c25e-4045-a992-7fb4bd1edc2f\") " pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-72d03-predictor-d5dddf8d4-2fxrg"
Sep 17 19:15:05.950308 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:15:05.950284    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/6b10759f-c25e-4045-a992-7fb4bd1edc2f-cabundle-cert\") pod \"isvc-raw-sklearn-batcher-72d03-predictor-d5dddf8d4-2fxrg\" (UID: \"6b10759f-c25e-4045-a992-7fb4bd1edc2f\") " pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-72d03-predictor-d5dddf8d4-2fxrg"
Sep 17 19:15:05.950422 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:15:05.950401    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/6b10759f-c25e-4045-a992-7fb4bd1edc2f-kube-rbac-proxy-sar-config\") pod \"isvc-raw-sklearn-batcher-72d03-predictor-d5dddf8d4-2fxrg\" (UID: \"6b10759f-c25e-4045-a992-7fb4bd1edc2f\") " pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-72d03-predictor-d5dddf8d4-2fxrg"
Sep 17 19:15:05.952338 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:15:05.952320    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6b10759f-c25e-4045-a992-7fb4bd1edc2f-proxy-tls\") pod \"isvc-raw-sklearn-batcher-72d03-predictor-d5dddf8d4-2fxrg\" (UID: \"6b10759f-c25e-4045-a992-7fb4bd1edc2f\") " pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-72d03-predictor-d5dddf8d4-2fxrg"
Sep 17 19:15:05.963252 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:15:05.963232    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nr5ll\" (UniqueName: \"kubernetes.io/projected/6b10759f-c25e-4045-a992-7fb4bd1edc2f-kube-api-access-nr5ll\") pod \"isvc-raw-sklearn-batcher-72d03-predictor-d5dddf8d4-2fxrg\" (UID: \"6b10759f-c25e-4045-a992-7fb4bd1edc2f\") " pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-72d03-predictor-d5dddf8d4-2fxrg"
Sep 17 19:15:06.081843 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:15:06.081742    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-72d03-predictor-d5dddf8d4-2fxrg"
Sep 17 19:15:06.360211 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:15:06.360186    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-raw-sklearn-batcher-72d03-predictor-d5dddf8d4-2fxrg"]
Sep 17 19:15:06.362320 ip-10-0-131-232 kubenswrapper[2582]: W0917 19:15:06.362287    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6b10759f_c25e_4045_a992_7fb4bd1edc2f.slice/crio-da6603e9674780129e434b89f579fc574d71c8e2d99f6a1b6d3b6656437a4903 WatchSource:0}: Error finding container da6603e9674780129e434b89f579fc574d71c8e2d99f6a1b6d3b6656437a4903: Status 404 returned error can't find the container with id da6603e9674780129e434b89f579fc574d71c8e2d99f6a1b6d3b6656437a4903
Sep 17 19:15:06.956420 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:15:06.956383    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-72d03-predictor-d5dddf8d4-2fxrg" event={"ID":"6b10759f-c25e-4045-a992-7fb4bd1edc2f","Type":"ContainerStarted","Data":"da6603e9674780129e434b89f579fc574d71c8e2d99f6a1b6d3b6656437a4903"}
Sep 17 19:15:09.972556 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:15:09.972518    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-72d03-predictor-d5dddf8d4-2fxrg" event={"ID":"6b10759f-c25e-4045-a992-7fb4bd1edc2f","Type":"ContainerStarted","Data":"b9885e3ff26e0ebd0c492c9fbfaa107622f8fa25e0e8658c2d042db982afe683"}
Sep 17 19:15:13.992112 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:15:13.992074    2582 generic.go:356] "Generic (PLEG): container finished" podID="6b10759f-c25e-4045-a992-7fb4bd1edc2f" containerID="b9885e3ff26e0ebd0c492c9fbfaa107622f8fa25e0e8658c2d042db982afe683" exitCode=0
Sep 17 19:15:13.992468 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:15:13.992138    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-72d03-predictor-d5dddf8d4-2fxrg" event={"ID":"6b10759f-c25e-4045-a992-7fb4bd1edc2f","Type":"ContainerDied","Data":"b9885e3ff26e0ebd0c492c9fbfaa107622f8fa25e0e8658c2d042db982afe683"}
Sep 17 19:15:28.070650 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:15:28.070609    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-72d03-predictor-d5dddf8d4-2fxrg" event={"ID":"6b10759f-c25e-4045-a992-7fb4bd1edc2f","Type":"ContainerStarted","Data":"472c8fdab3e7cc52a8e72eab7206f301d09627b2c83845dd2733ea8a95c8833f"}
Sep 17 19:15:30.082333 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:15:30.082299    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-72d03-predictor-d5dddf8d4-2fxrg" event={"ID":"6b10759f-c25e-4045-a992-7fb4bd1edc2f","Type":"ContainerStarted","Data":"3cf1fc44d873abd21cd136ddc54208962a53190d0a3949f7d333a1e94f6b24cd"}
Sep 17 19:15:32.093816 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:15:32.093781    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-72d03-predictor-d5dddf8d4-2fxrg" event={"ID":"6b10759f-c25e-4045-a992-7fb4bd1edc2f","Type":"ContainerStarted","Data":"70e6ad8898d69c6cdf85d70a7bf28c5bda8d0a5147fdc8a20f7f5e1a39fd6878"}
Sep 17 19:15:32.094241 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:15:32.094010    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-72d03-predictor-d5dddf8d4-2fxrg"
Sep 17 19:15:32.094241 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:15:32.094060    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-72d03-predictor-d5dddf8d4-2fxrg"
Sep 17 19:15:32.095451 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:15:32.095423    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-72d03-predictor-d5dddf8d4-2fxrg" podUID="6b10759f-c25e-4045-a992-7fb4bd1edc2f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.25:8080: connect: connection refused"
Sep 17 19:15:32.120800 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:15:32.120724    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-72d03-predictor-d5dddf8d4-2fxrg" podStartSLOduration=2.022873309 podStartE2EDuration="27.120710832s" totalImagesPullingTime="21.232002408s" totalInitContainerRuntime="3.865835115s" isStatefulPod=false podCreationTimestamp="2026-09-17 19:15:05 +0000 UTC" imagePullSessionsCount=4 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 19:15:32.118712191 +0000 UTC m=+547.890383252" watchObservedRunningTime="2026-09-17 19:15:32.120710832 +0000 UTC m=+547.892381921"
Sep 17 19:15:33.098266 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:15:33.098224    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-72d03-predictor-d5dddf8d4-2fxrg"
Sep 17 19:15:33.098643 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:15:33.098431    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-72d03-predictor-d5dddf8d4-2fxrg" podUID="6b10759f-c25e-4045-a992-7fb4bd1edc2f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.25:8080: connect: connection refused"
Sep 17 19:15:33.099598 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:15:33.099574    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-72d03-predictor-d5dddf8d4-2fxrg" podUID="6b10759f-c25e-4045-a992-7fb4bd1edc2f" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 19:15:34.102542 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:15:34.102501    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-72d03-predictor-d5dddf8d4-2fxrg" podUID="6b10759f-c25e-4045-a992-7fb4bd1edc2f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.25:8080: connect: connection refused"
Sep 17 19:15:34.102964 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:15:34.102877    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-72d03-predictor-d5dddf8d4-2fxrg" podUID="6b10759f-c25e-4045-a992-7fb4bd1edc2f" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 19:15:34.106333 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:15:34.106313    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-72d03-predictor-d5dddf8d4-2fxrg"
Sep 17 19:15:35.106477 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:15:35.106431    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-72d03-predictor-d5dddf8d4-2fxrg" podUID="6b10759f-c25e-4045-a992-7fb4bd1edc2f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.25:8080: connect: connection refused"
Sep 17 19:15:35.106900 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:15:35.106656    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-72d03-predictor-d5dddf8d4-2fxrg" podUID="6b10759f-c25e-4045-a992-7fb4bd1edc2f" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 19:15:45.106436 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:15:45.106394    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-72d03-predictor-d5dddf8d4-2fxrg" podUID="6b10759f-c25e-4045-a992-7fb4bd1edc2f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.25:8080: connect: connection refused"
Sep 17 19:15:45.106898 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:15:45.106755    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-72d03-predictor-d5dddf8d4-2fxrg" podUID="6b10759f-c25e-4045-a992-7fb4bd1edc2f" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 19:15:55.107377 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:15:55.107326    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-72d03-predictor-d5dddf8d4-2fxrg" podUID="6b10759f-c25e-4045-a992-7fb4bd1edc2f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.25:8080: connect: connection refused"
Sep 17 19:15:55.107789 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:15:55.107763    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-72d03-predictor-d5dddf8d4-2fxrg" podUID="6b10759f-c25e-4045-a992-7fb4bd1edc2f" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 19:16:05.106507 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:16:05.106456    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-72d03-predictor-d5dddf8d4-2fxrg" podUID="6b10759f-c25e-4045-a992-7fb4bd1edc2f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.25:8080: connect: connection refused"
Sep 17 19:16:05.106973 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:16:05.106889    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-72d03-predictor-d5dddf8d4-2fxrg" podUID="6b10759f-c25e-4045-a992-7fb4bd1edc2f" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 19:16:15.106706 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:16:15.106652    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-72d03-predictor-d5dddf8d4-2fxrg" podUID="6b10759f-c25e-4045-a992-7fb4bd1edc2f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.25:8080: connect: connection refused"
Sep 17 19:16:15.107109 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:16:15.107072    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-72d03-predictor-d5dddf8d4-2fxrg" podUID="6b10759f-c25e-4045-a992-7fb4bd1edc2f" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 19:16:24.670420 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:16:24.670387    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-kdm8s_c1b79c18-774f-4b78-a6a9-7ea8a229aaae/ovn-acl-logging/0.log"
Sep 17 19:16:24.675179 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:16:24.675153    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-kdm8s_c1b79c18-774f-4b78-a6a9-7ea8a229aaae/ovn-acl-logging/0.log"
Sep 17 19:16:25.107463 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:16:25.107360    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-72d03-predictor-d5dddf8d4-2fxrg" podUID="6b10759f-c25e-4045-a992-7fb4bd1edc2f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.25:8080: connect: connection refused"
Sep 17 19:16:25.107856 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:16:25.107804    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-72d03-predictor-d5dddf8d4-2fxrg" podUID="6b10759f-c25e-4045-a992-7fb4bd1edc2f" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 19:16:35.107395 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:16:35.107347    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-72d03-predictor-d5dddf8d4-2fxrg" podUID="6b10759f-c25e-4045-a992-7fb4bd1edc2f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.25:8080: connect: connection refused"
Sep 17 19:16:35.107944 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:16:35.107789    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-72d03-predictor-d5dddf8d4-2fxrg" podUID="6b10759f-c25e-4045-a992-7fb4bd1edc2f" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 19:16:45.107678 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:16:45.107643    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-72d03-predictor-d5dddf8d4-2fxrg"
Sep 17 19:16:45.108144 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:16:45.107917    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-72d03-predictor-d5dddf8d4-2fxrg"
Sep 17 19:16:55.347370 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:16:55.347284    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-predictor-97977b6b5-2fskk"]
Sep 17 19:16:55.350653 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:16:55.350629    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-97977b6b5-2fskk"
Sep 17 19:16:55.353090 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:16:55.353068    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-rollback-kube-rbac-proxy-sar-config\""
Sep 17 19:16:55.353185 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:16:55.353068    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-rollback-predictor-serving-cert\""
Sep 17 19:16:55.362531 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:16:55.362509    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-predictor-97977b6b5-2fskk"]
Sep 17 19:16:55.429157 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:16:55.429136    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ec74f400-7706-4361-99cc-b32b746af739-proxy-tls\") pod \"isvc-canary-rollback-predictor-97977b6b5-2fskk\" (UID: \"ec74f400-7706-4361-99cc-b32b746af739\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-97977b6b5-2fskk"
Sep 17 19:16:55.429268 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:16:55.429173    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ec74f400-7706-4361-99cc-b32b746af739-kserve-provision-location\") pod \"isvc-canary-rollback-predictor-97977b6b5-2fskk\" (UID: \"ec74f400-7706-4361-99cc-b32b746af739\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-97977b6b5-2fskk"
Sep 17 19:16:55.429268 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:16:55.429193    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ppjtz\" (UniqueName: \"kubernetes.io/projected/ec74f400-7706-4361-99cc-b32b746af739-kube-api-access-ppjtz\") pod \"isvc-canary-rollback-predictor-97977b6b5-2fskk\" (UID: \"ec74f400-7706-4361-99cc-b32b746af739\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-97977b6b5-2fskk"
Sep 17 19:16:55.429337 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:16:55.429273    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/ec74f400-7706-4361-99cc-b32b746af739-cabundle-cert\") pod \"isvc-canary-rollback-predictor-97977b6b5-2fskk\" (UID: \"ec74f400-7706-4361-99cc-b32b746af739\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-97977b6b5-2fskk"
Sep 17 19:16:55.429337 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:16:55.429302    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ec74f400-7706-4361-99cc-b32b746af739-kube-rbac-proxy-sar-config\") pod \"isvc-canary-rollback-predictor-97977b6b5-2fskk\" (UID: \"ec74f400-7706-4361-99cc-b32b746af739\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-97977b6b5-2fskk"
Sep 17 19:16:55.530366 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:16:55.530319    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ec74f400-7706-4361-99cc-b32b746af739-kube-rbac-proxy-sar-config\") pod \"isvc-canary-rollback-predictor-97977b6b5-2fskk\" (UID: \"ec74f400-7706-4361-99cc-b32b746af739\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-97977b6b5-2fskk"
Sep 17 19:16:55.530366 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:16:55.530373    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ec74f400-7706-4361-99cc-b32b746af739-proxy-tls\") pod \"isvc-canary-rollback-predictor-97977b6b5-2fskk\" (UID: \"ec74f400-7706-4361-99cc-b32b746af739\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-97977b6b5-2fskk"
Sep 17 19:16:55.530623 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:16:55.530412    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ec74f400-7706-4361-99cc-b32b746af739-kserve-provision-location\") pod \"isvc-canary-rollback-predictor-97977b6b5-2fskk\" (UID: \"ec74f400-7706-4361-99cc-b32b746af739\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-97977b6b5-2fskk"
Sep 17 19:16:55.530623 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:16:55.530431    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ppjtz\" (UniqueName: \"kubernetes.io/projected/ec74f400-7706-4361-99cc-b32b746af739-kube-api-access-ppjtz\") pod \"isvc-canary-rollback-predictor-97977b6b5-2fskk\" (UID: \"ec74f400-7706-4361-99cc-b32b746af739\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-97977b6b5-2fskk"
Sep 17 19:16:55.530623 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:16:55.530464    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/ec74f400-7706-4361-99cc-b32b746af739-cabundle-cert\") pod \"isvc-canary-rollback-predictor-97977b6b5-2fskk\" (UID: \"ec74f400-7706-4361-99cc-b32b746af739\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-97977b6b5-2fskk"
Sep 17 19:16:55.530919 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:16:55.530898    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ec74f400-7706-4361-99cc-b32b746af739-kserve-provision-location\") pod \"isvc-canary-rollback-predictor-97977b6b5-2fskk\" (UID: \"ec74f400-7706-4361-99cc-b32b746af739\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-97977b6b5-2fskk"
Sep 17 19:16:55.531147 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:16:55.531128    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/ec74f400-7706-4361-99cc-b32b746af739-cabundle-cert\") pod \"isvc-canary-rollback-predictor-97977b6b5-2fskk\" (UID: \"ec74f400-7706-4361-99cc-b32b746af739\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-97977b6b5-2fskk"
Sep 17 19:16:55.531220 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:16:55.531160    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ec74f400-7706-4361-99cc-b32b746af739-kube-rbac-proxy-sar-config\") pod \"isvc-canary-rollback-predictor-97977b6b5-2fskk\" (UID: \"ec74f400-7706-4361-99cc-b32b746af739\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-97977b6b5-2fskk"
Sep 17 19:16:55.533036 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:16:55.533019    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ec74f400-7706-4361-99cc-b32b746af739-proxy-tls\") pod \"isvc-canary-rollback-predictor-97977b6b5-2fskk\" (UID: \"ec74f400-7706-4361-99cc-b32b746af739\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-97977b6b5-2fskk"
Sep 17 19:16:55.541374 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:16:55.541354    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ppjtz\" (UniqueName: \"kubernetes.io/projected/ec74f400-7706-4361-99cc-b32b746af739-kube-api-access-ppjtz\") pod \"isvc-canary-rollback-predictor-97977b6b5-2fskk\" (UID: \"ec74f400-7706-4361-99cc-b32b746af739\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-97977b6b5-2fskk"
Sep 17 19:16:55.660896 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:16:55.660864    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-97977b6b5-2fskk"
Sep 17 19:16:55.935559 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:16:55.935524    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-predictor-97977b6b5-2fskk"]
Sep 17 19:16:55.936557 ip-10-0-131-232 kubenswrapper[2582]: W0917 19:16:55.936523    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podec74f400_7706_4361_99cc_b32b746af739.slice/crio-22ce2876c5c87dcd4fc707d0513a9a4c8e127f64b7b608e625e12c44102475eb WatchSource:0}: Error finding container 22ce2876c5c87dcd4fc707d0513a9a4c8e127f64b7b608e625e12c44102475eb: Status 404 returned error can't find the container with id 22ce2876c5c87dcd4fc707d0513a9a4c8e127f64b7b608e625e12c44102475eb
Sep 17 19:16:56.440004 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:16:56.439969    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-97977b6b5-2fskk" event={"ID":"ec74f400-7706-4361-99cc-b32b746af739","Type":"ContainerStarted","Data":"3948b9a708ece5af0a11915d49b73b28b41edefa7575844ce1b16ecec59b6aae"}
Sep 17 19:16:56.440377 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:16:56.440064    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-97977b6b5-2fskk" event={"ID":"ec74f400-7706-4361-99cc-b32b746af739","Type":"ContainerStarted","Data":"22ce2876c5c87dcd4fc707d0513a9a4c8e127f64b7b608e625e12c44102475eb"}
Sep 17 19:17:00.456928 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:17:00.456897    2582 generic.go:356] "Generic (PLEG): container finished" podID="ec74f400-7706-4361-99cc-b32b746af739" containerID="3948b9a708ece5af0a11915d49b73b28b41edefa7575844ce1b16ecec59b6aae" exitCode=0
Sep 17 19:17:00.457351 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:17:00.456950    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-97977b6b5-2fskk" event={"ID":"ec74f400-7706-4361-99cc-b32b746af739","Type":"ContainerDied","Data":"3948b9a708ece5af0a11915d49b73b28b41edefa7575844ce1b16ecec59b6aae"}
Sep 17 19:17:00.727101 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:17:00.727025    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-raw-sklearn-batcher-72d03-predictor-d5dddf8d4-2fxrg"]
Sep 17 19:17:00.727312 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:17:00.727282    2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-72d03-predictor-d5dddf8d4-2fxrg" podUID="6b10759f-c25e-4045-a992-7fb4bd1edc2f" containerName="agent" containerID="cri-o://70e6ad8898d69c6cdf85d70a7bf28c5bda8d0a5147fdc8a20f7f5e1a39fd6878" gracePeriod=30
Sep 17 19:17:00.727380 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:17:00.727316    2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-72d03-predictor-d5dddf8d4-2fxrg" podUID="6b10759f-c25e-4045-a992-7fb4bd1edc2f" containerName="kube-rbac-proxy" containerID="cri-o://3cf1fc44d873abd21cd136ddc54208962a53190d0a3949f7d333a1e94f6b24cd" gracePeriod=30
Sep 17 19:17:00.727437 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:17:00.727282    2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-72d03-predictor-d5dddf8d4-2fxrg" podUID="6b10759f-c25e-4045-a992-7fb4bd1edc2f" containerName="kserve-container" containerID="cri-o://472c8fdab3e7cc52a8e72eab7206f301d09627b2c83845dd2733ea8a95c8833f" gracePeriod=30
Sep 17 19:17:01.463938 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:17:01.463905    2582 generic.go:356] "Generic (PLEG): container finished" podID="6b10759f-c25e-4045-a992-7fb4bd1edc2f" containerID="3cf1fc44d873abd21cd136ddc54208962a53190d0a3949f7d333a1e94f6b24cd" exitCode=2
Sep 17 19:17:01.464372 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:17:01.463975    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-72d03-predictor-d5dddf8d4-2fxrg" event={"ID":"6b10759f-c25e-4045-a992-7fb4bd1edc2f","Type":"ContainerDied","Data":"3cf1fc44d873abd21cd136ddc54208962a53190d0a3949f7d333a1e94f6b24cd"}
Sep 17 19:17:01.466018 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:17:01.465994    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-97977b6b5-2fskk" event={"ID":"ec74f400-7706-4361-99cc-b32b746af739","Type":"ContainerStarted","Data":"67ca2c768e7be7479832185701c3562e6bd8dccec3d1ce9f55ee8e22eff10214"}
Sep 17 19:17:01.466180 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:17:01.466046    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-97977b6b5-2fskk" event={"ID":"ec74f400-7706-4361-99cc-b32b746af739","Type":"ContainerStarted","Data":"3d88e360bcb94d17383d5b5c09404b0527a40511bb403e42c9e157f9f69c22da"}
Sep 17 19:17:01.466239 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:17:01.466223    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-97977b6b5-2fskk"
Sep 17 19:17:01.486935 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:17:01.486895    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-97977b6b5-2fskk" podStartSLOduration=2.654750773 podStartE2EDuration="6.486881695s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.832130922s" isStatefulPod=false podCreationTimestamp="2026-09-17 19:16:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 19:17:01.485285014 +0000 UTC m=+637.256956078" watchObservedRunningTime="2026-09-17 19:17:01.486881695 +0000 UTC m=+637.258552755"
Sep 17 19:17:02.469924 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:17:02.469892    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-97977b6b5-2fskk"
Sep 17 19:17:02.471048 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:17:02.471020    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-97977b6b5-2fskk" podUID="ec74f400-7706-4361-99cc-b32b746af739" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.26:8080: connect: connection refused"
Sep 17 19:17:03.474384 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:17:03.474342    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-97977b6b5-2fskk" podUID="ec74f400-7706-4361-99cc-b32b746af739" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.26:8080: connect: connection refused"
Sep 17 19:17:04.103271 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:17:04.103234    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-72d03-predictor-d5dddf8d4-2fxrg" podUID="6b10759f-c25e-4045-a992-7fb4bd1edc2f" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.25:8643/healthz\": dial tcp 10.134.0.25:8643: connect: connection refused"
Sep 17 19:17:05.107028 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:17:05.106980    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-72d03-predictor-d5dddf8d4-2fxrg" podUID="6b10759f-c25e-4045-a992-7fb4bd1edc2f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.25:8080: connect: connection refused"
Sep 17 19:17:05.107425 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:17:05.107241    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-72d03-predictor-d5dddf8d4-2fxrg" podUID="6b10759f-c25e-4045-a992-7fb4bd1edc2f" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 19:17:06.489454 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:17:06.489419    2582 generic.go:356] "Generic (PLEG): container finished" podID="6b10759f-c25e-4045-a992-7fb4bd1edc2f" containerID="472c8fdab3e7cc52a8e72eab7206f301d09627b2c83845dd2733ea8a95c8833f" exitCode=0
Sep 17 19:17:06.489932 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:17:06.489466    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-72d03-predictor-d5dddf8d4-2fxrg" event={"ID":"6b10759f-c25e-4045-a992-7fb4bd1edc2f","Type":"ContainerDied","Data":"472c8fdab3e7cc52a8e72eab7206f301d09627b2c83845dd2733ea8a95c8833f"}
Sep 17 19:17:08.479243 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:17:08.479211    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-97977b6b5-2fskk"
Sep 17 19:17:08.479745 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:17:08.479717    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-97977b6b5-2fskk" podUID="ec74f400-7706-4361-99cc-b32b746af739" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.26:8080: connect: connection refused"
Sep 17 19:17:09.103157 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:17:09.103118    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-72d03-predictor-d5dddf8d4-2fxrg" podUID="6b10759f-c25e-4045-a992-7fb4bd1edc2f" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.25:8643/healthz\": dial tcp 10.134.0.25:8643: connect: connection refused"
Sep 17 19:17:14.103257 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:17:14.103216    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-72d03-predictor-d5dddf8d4-2fxrg" podUID="6b10759f-c25e-4045-a992-7fb4bd1edc2f" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.25:8643/healthz\": dial tcp 10.134.0.25:8643: connect: connection refused"
Sep 17 19:17:14.103654 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:17:14.103358    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-72d03-predictor-d5dddf8d4-2fxrg"
Sep 17 19:17:15.106526 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:17:15.106478    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-72d03-predictor-d5dddf8d4-2fxrg" podUID="6b10759f-c25e-4045-a992-7fb4bd1edc2f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.25:8080: connect: connection refused"
Sep 17 19:17:15.106938 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:17:15.106799    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-72d03-predictor-d5dddf8d4-2fxrg" podUID="6b10759f-c25e-4045-a992-7fb4bd1edc2f" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 19:17:18.479951 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:17:18.479907    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-97977b6b5-2fskk" podUID="ec74f400-7706-4361-99cc-b32b746af739" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.26:8080: connect: connection refused"
Sep 17 19:17:19.103492 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:17:19.103449    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-72d03-predictor-d5dddf8d4-2fxrg" podUID="6b10759f-c25e-4045-a992-7fb4bd1edc2f" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.25:8643/healthz\": dial tcp 10.134.0.25:8643: connect: connection refused"
Sep 17 19:17:24.103520 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:17:24.103480    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-72d03-predictor-d5dddf8d4-2fxrg" podUID="6b10759f-c25e-4045-a992-7fb4bd1edc2f" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.25:8643/healthz\": dial tcp 10.134.0.25:8643: connect: connection refused"
Sep 17 19:17:25.106660 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:17:25.106615    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-72d03-predictor-d5dddf8d4-2fxrg" podUID="6b10759f-c25e-4045-a992-7fb4bd1edc2f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.25:8080: connect: connection refused"
Sep 17 19:17:25.107136 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:17:25.106733    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-72d03-predictor-d5dddf8d4-2fxrg"
Sep 17 19:17:25.107136 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:17:25.106976    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-72d03-predictor-d5dddf8d4-2fxrg" podUID="6b10759f-c25e-4045-a992-7fb4bd1edc2f" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 19:17:25.107136 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:17:25.107067    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-72d03-predictor-d5dddf8d4-2fxrg"
Sep 17 19:17:28.480139 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:17:28.480098    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-97977b6b5-2fskk" podUID="ec74f400-7706-4361-99cc-b32b746af739" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.26:8080: connect: connection refused"
Sep 17 19:17:29.102764 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:17:29.102717    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-72d03-predictor-d5dddf8d4-2fxrg" podUID="6b10759f-c25e-4045-a992-7fb4bd1edc2f" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.25:8643/healthz\": dial tcp 10.134.0.25:8643: connect: connection refused"
Sep 17 19:17:30.882832 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:17:30.882792    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-72d03-predictor-d5dddf8d4-2fxrg"
Sep 17 19:17:30.913200 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:17:30.913171    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-nr5ll\" (UniqueName: \"kubernetes.io/projected/6b10759f-c25e-4045-a992-7fb4bd1edc2f-kube-api-access-nr5ll\") pod \"6b10759f-c25e-4045-a992-7fb4bd1edc2f\" (UID: \"6b10759f-c25e-4045-a992-7fb4bd1edc2f\") "
Sep 17 19:17:30.913383 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:17:30.913223    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/6b10759f-c25e-4045-a992-7fb4bd1edc2f-kube-rbac-proxy-sar-config\") pod \"6b10759f-c25e-4045-a992-7fb4bd1edc2f\" (UID: \"6b10759f-c25e-4045-a992-7fb4bd1edc2f\") "
Sep 17 19:17:30.913383 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:17:30.913261    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/6b10759f-c25e-4045-a992-7fb4bd1edc2f-kserve-provision-location\") pod \"6b10759f-c25e-4045-a992-7fb4bd1edc2f\" (UID: \"6b10759f-c25e-4045-a992-7fb4bd1edc2f\") "
Sep 17 19:17:30.913383 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:17:30.913280    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6b10759f-c25e-4045-a992-7fb4bd1edc2f-proxy-tls\") pod \"6b10759f-c25e-4045-a992-7fb4bd1edc2f\" (UID: \"6b10759f-c25e-4045-a992-7fb4bd1edc2f\") "
Sep 17 19:17:30.913383 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:17:30.913310    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/6b10759f-c25e-4045-a992-7fb4bd1edc2f-cabundle-cert\") pod \"6b10759f-c25e-4045-a992-7fb4bd1edc2f\" (UID: \"6b10759f-c25e-4045-a992-7fb4bd1edc2f\") "
Sep 17 19:17:30.913700 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:17:30.913657    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6b10759f-c25e-4045-a992-7fb4bd1edc2f-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "6b10759f-c25e-4045-a992-7fb4bd1edc2f" (UID: "6b10759f-c25e-4045-a992-7fb4bd1edc2f"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 19:17:30.913700 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:17:30.913650    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6b10759f-c25e-4045-a992-7fb4bd1edc2f-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "6b10759f-c25e-4045-a992-7fb4bd1edc2f" (UID: "6b10759f-c25e-4045-a992-7fb4bd1edc2f"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 19:17:30.913853 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:17:30.913730    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6b10759f-c25e-4045-a992-7fb4bd1edc2f-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "6b10759f-c25e-4045-a992-7fb4bd1edc2f" (UID: "6b10759f-c25e-4045-a992-7fb4bd1edc2f"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 19:17:30.915535 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:17:30.915503    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6b10759f-c25e-4045-a992-7fb4bd1edc2f-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "6b10759f-c25e-4045-a992-7fb4bd1edc2f" (UID: "6b10759f-c25e-4045-a992-7fb4bd1edc2f"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 19:17:30.915656 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:17:30.915635    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6b10759f-c25e-4045-a992-7fb4bd1edc2f-kube-api-access-nr5ll" (OuterVolumeSpecName: "kube-api-access-nr5ll") pod "6b10759f-c25e-4045-a992-7fb4bd1edc2f" (UID: "6b10759f-c25e-4045-a992-7fb4bd1edc2f"). InnerVolumeSpecName "kube-api-access-nr5ll". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 19:17:31.014077 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:17:31.013985    2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nr5ll\" (UniqueName: \"kubernetes.io/projected/6b10759f-c25e-4045-a992-7fb4bd1edc2f-kube-api-access-nr5ll\") on node \"ip-10-0-131-232.ec2.internal\" DevicePath \"\""
Sep 17 19:17:31.014077 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:17:31.014014    2582 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/6b10759f-c25e-4045-a992-7fb4bd1edc2f-kube-rbac-proxy-sar-config\") on node \"ip-10-0-131-232.ec2.internal\" DevicePath \"\""
Sep 17 19:17:31.014077 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:17:31.014026    2582 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/6b10759f-c25e-4045-a992-7fb4bd1edc2f-kserve-provision-location\") on node \"ip-10-0-131-232.ec2.internal\" DevicePath \"\""
Sep 17 19:17:31.014077 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:17:31.014037    2582 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6b10759f-c25e-4045-a992-7fb4bd1edc2f-proxy-tls\") on node \"ip-10-0-131-232.ec2.internal\" DevicePath \"\""
Sep 17 19:17:31.014077 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:17:31.014049    2582 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/6b10759f-c25e-4045-a992-7fb4bd1edc2f-cabundle-cert\") on node \"ip-10-0-131-232.ec2.internal\" DevicePath \"\""
Sep 17 19:17:31.598527 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:17:31.598497    2582 generic.go:356] "Generic (PLEG): container finished" podID="6b10759f-c25e-4045-a992-7fb4bd1edc2f" containerID="70e6ad8898d69c6cdf85d70a7bf28c5bda8d0a5147fdc8a20f7f5e1a39fd6878" exitCode=0
Sep 17 19:17:31.598733 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:17:31.598566    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-72d03-predictor-d5dddf8d4-2fxrg"
Sep 17 19:17:31.598733 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:17:31.598570    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-72d03-predictor-d5dddf8d4-2fxrg" event={"ID":"6b10759f-c25e-4045-a992-7fb4bd1edc2f","Type":"ContainerDied","Data":"70e6ad8898d69c6cdf85d70a7bf28c5bda8d0a5147fdc8a20f7f5e1a39fd6878"}
Sep 17 19:17:31.598733 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:17:31.598631    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-72d03-predictor-d5dddf8d4-2fxrg" event={"ID":"6b10759f-c25e-4045-a992-7fb4bd1edc2f","Type":"ContainerDied","Data":"da6603e9674780129e434b89f579fc574d71c8e2d99f6a1b6d3b6656437a4903"}
Sep 17 19:17:31.598733 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:17:31.598660    2582 scope.go:117] "RemoveContainer" containerID="70e6ad8898d69c6cdf85d70a7bf28c5bda8d0a5147fdc8a20f7f5e1a39fd6878"
Sep 17 19:17:31.607222 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:17:31.607200    2582 scope.go:117] "RemoveContainer" containerID="3cf1fc44d873abd21cd136ddc54208962a53190d0a3949f7d333a1e94f6b24cd"
Sep 17 19:17:31.615318 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:17:31.615296    2582 scope.go:117] "RemoveContainer" containerID="472c8fdab3e7cc52a8e72eab7206f301d09627b2c83845dd2733ea8a95c8833f"
Sep 17 19:17:31.620813 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:17:31.620785    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-raw-sklearn-batcher-72d03-predictor-d5dddf8d4-2fxrg"]
Sep 17 19:17:31.624406 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:17:31.624386    2582 scope.go:117] "RemoveContainer" containerID="b9885e3ff26e0ebd0c492c9fbfaa107622f8fa25e0e8658c2d042db982afe683"
Sep 17 19:17:31.624802 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:17:31.624780    2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-raw-sklearn-batcher-72d03-predictor-d5dddf8d4-2fxrg"]
Sep 17 19:17:31.632358 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:17:31.632336    2582 scope.go:117] "RemoveContainer" containerID="70e6ad8898d69c6cdf85d70a7bf28c5bda8d0a5147fdc8a20f7f5e1a39fd6878"
Sep 17 19:17:31.632699 ip-10-0-131-232 kubenswrapper[2582]: E0917 19:17:31.632676    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"70e6ad8898d69c6cdf85d70a7bf28c5bda8d0a5147fdc8a20f7f5e1a39fd6878\": container with ID starting with 70e6ad8898d69c6cdf85d70a7bf28c5bda8d0a5147fdc8a20f7f5e1a39fd6878 not found: ID does not exist" containerID="70e6ad8898d69c6cdf85d70a7bf28c5bda8d0a5147fdc8a20f7f5e1a39fd6878"
Sep 17 19:17:31.632776 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:17:31.632709    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"70e6ad8898d69c6cdf85d70a7bf28c5bda8d0a5147fdc8a20f7f5e1a39fd6878"} err="failed to get container status \"70e6ad8898d69c6cdf85d70a7bf28c5bda8d0a5147fdc8a20f7f5e1a39fd6878\": rpc error: code = NotFound desc = could not find container \"70e6ad8898d69c6cdf85d70a7bf28c5bda8d0a5147fdc8a20f7f5e1a39fd6878\": container with ID starting with 70e6ad8898d69c6cdf85d70a7bf28c5bda8d0a5147fdc8a20f7f5e1a39fd6878 not found: ID does not exist"
Sep 17 19:17:31.632776 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:17:31.632731    2582 scope.go:117] "RemoveContainer" containerID="3cf1fc44d873abd21cd136ddc54208962a53190d0a3949f7d333a1e94f6b24cd"
Sep 17 19:17:31.633057 ip-10-0-131-232 kubenswrapper[2582]: E0917 19:17:31.633037    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3cf1fc44d873abd21cd136ddc54208962a53190d0a3949f7d333a1e94f6b24cd\": container with ID starting with 3cf1fc44d873abd21cd136ddc54208962a53190d0a3949f7d333a1e94f6b24cd not found: ID does not exist" containerID="3cf1fc44d873abd21cd136ddc54208962a53190d0a3949f7d333a1e94f6b24cd"
Sep 17 19:17:31.633144 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:17:31.633064    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3cf1fc44d873abd21cd136ddc54208962a53190d0a3949f7d333a1e94f6b24cd"} err="failed to get container status \"3cf1fc44d873abd21cd136ddc54208962a53190d0a3949f7d333a1e94f6b24cd\": rpc error: code = NotFound desc = could not find container \"3cf1fc44d873abd21cd136ddc54208962a53190d0a3949f7d333a1e94f6b24cd\": container with ID starting with 3cf1fc44d873abd21cd136ddc54208962a53190d0a3949f7d333a1e94f6b24cd not found: ID does not exist"
Sep 17 19:17:31.633144 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:17:31.633081    2582 scope.go:117] "RemoveContainer" containerID="472c8fdab3e7cc52a8e72eab7206f301d09627b2c83845dd2733ea8a95c8833f"
Sep 17 19:17:31.633328 ip-10-0-131-232 kubenswrapper[2582]: E0917 19:17:31.633310    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"472c8fdab3e7cc52a8e72eab7206f301d09627b2c83845dd2733ea8a95c8833f\": container with ID starting with 472c8fdab3e7cc52a8e72eab7206f301d09627b2c83845dd2733ea8a95c8833f not found: ID does not exist" containerID="472c8fdab3e7cc52a8e72eab7206f301d09627b2c83845dd2733ea8a95c8833f"
Sep 17 19:17:31.633394 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:17:31.633330    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"472c8fdab3e7cc52a8e72eab7206f301d09627b2c83845dd2733ea8a95c8833f"} err="failed to get container status \"472c8fdab3e7cc52a8e72eab7206f301d09627b2c83845dd2733ea8a95c8833f\": rpc error: code = NotFound desc = could not find container \"472c8fdab3e7cc52a8e72eab7206f301d09627b2c83845dd2733ea8a95c8833f\": container with ID starting with 472c8fdab3e7cc52a8e72eab7206f301d09627b2c83845dd2733ea8a95c8833f not found: ID does not exist"
Sep 17 19:17:31.633394 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:17:31.633347    2582 scope.go:117] "RemoveContainer" containerID="b9885e3ff26e0ebd0c492c9fbfaa107622f8fa25e0e8658c2d042db982afe683"
Sep 17 19:17:31.633604 ip-10-0-131-232 kubenswrapper[2582]: E0917 19:17:31.633585    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b9885e3ff26e0ebd0c492c9fbfaa107622f8fa25e0e8658c2d042db982afe683\": container with ID starting with b9885e3ff26e0ebd0c492c9fbfaa107622f8fa25e0e8658c2d042db982afe683 not found: ID does not exist" containerID="b9885e3ff26e0ebd0c492c9fbfaa107622f8fa25e0e8658c2d042db982afe683"
Sep 17 19:17:31.633665 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:17:31.633608    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b9885e3ff26e0ebd0c492c9fbfaa107622f8fa25e0e8658c2d042db982afe683"} err="failed to get container status \"b9885e3ff26e0ebd0c492c9fbfaa107622f8fa25e0e8658c2d042db982afe683\": rpc error: code = NotFound desc = could not find container \"b9885e3ff26e0ebd0c492c9fbfaa107622f8fa25e0e8658c2d042db982afe683\": container with ID starting with b9885e3ff26e0ebd0c492c9fbfaa107622f8fa25e0e8658c2d042db982afe683 not found: ID does not exist"
Sep 17 19:17:32.768214 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:17:32.768180    2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6b10759f-c25e-4045-a992-7fb4bd1edc2f" path="/var/lib/kubelet/pods/6b10759f-c25e-4045-a992-7fb4bd1edc2f/volumes"
Sep 17 19:17:38.479774 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:17:38.479734    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-97977b6b5-2fskk" podUID="ec74f400-7706-4361-99cc-b32b746af739" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.26:8080: connect: connection refused"
Sep 17 19:17:48.480618 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:17:48.480574    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-97977b6b5-2fskk" podUID="ec74f400-7706-4361-99cc-b32b746af739" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.26:8080: connect: connection refused"
Sep 17 19:17:58.482761 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:17:58.482528    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-97977b6b5-2fskk" podUID="ec74f400-7706-4361-99cc-b32b746af739" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.26:8080: connect: connection refused"
Sep 17 19:18:08.480204 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:18:08.480161    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-97977b6b5-2fskk" podUID="ec74f400-7706-4361-99cc-b32b746af739" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.26:8080: connect: connection refused"
Sep 17 19:18:18.481195 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:18:18.481109    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-97977b6b5-2fskk"
Sep 17 19:18:25.318454 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:18:25.318417    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-predictor-97977b6b5-2fskk"]
Sep 17 19:18:25.318844 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:18:25.318676    2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-97977b6b5-2fskk" podUID="ec74f400-7706-4361-99cc-b32b746af739" containerName="kserve-container" containerID="cri-o://3d88e360bcb94d17383d5b5c09404b0527a40511bb403e42c9e157f9f69c22da" gracePeriod=30
Sep 17 19:18:25.318844 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:18:25.318701    2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-97977b6b5-2fskk" podUID="ec74f400-7706-4361-99cc-b32b746af739" containerName="kube-rbac-proxy" containerID="cri-o://67ca2c768e7be7479832185701c3562e6bd8dccec3d1ce9f55ee8e22eff10214" gracePeriod=30
Sep 17 19:18:25.830864 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:18:25.830814    2582 generic.go:356] "Generic (PLEG): container finished" podID="ec74f400-7706-4361-99cc-b32b746af739" containerID="67ca2c768e7be7479832185701c3562e6bd8dccec3d1ce9f55ee8e22eff10214" exitCode=2
Sep 17 19:18:25.831042 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:18:25.830889    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-97977b6b5-2fskk" event={"ID":"ec74f400-7706-4361-99cc-b32b746af739","Type":"ContainerDied","Data":"67ca2c768e7be7479832185701c3562e6bd8dccec3d1ce9f55ee8e22eff10214"}
Sep 17 19:18:28.475391 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:18:28.475350    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-97977b6b5-2fskk" podUID="ec74f400-7706-4361-99cc-b32b746af739" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.26:8643/healthz\": dial tcp 10.134.0.26:8643: connect: connection refused"
Sep 17 19:18:28.479779 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:18:28.479755    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-97977b6b5-2fskk" podUID="ec74f400-7706-4361-99cc-b32b746af739" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.26:8080: connect: connection refused"
Sep 17 19:18:29.849558 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:18:29.849525    2582 generic.go:356] "Generic (PLEG): container finished" podID="ec74f400-7706-4361-99cc-b32b746af739" containerID="3d88e360bcb94d17383d5b5c09404b0527a40511bb403e42c9e157f9f69c22da" exitCode=0
Sep 17 19:18:29.849930 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:18:29.849592    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-97977b6b5-2fskk" event={"ID":"ec74f400-7706-4361-99cc-b32b746af739","Type":"ContainerDied","Data":"3d88e360bcb94d17383d5b5c09404b0527a40511bb403e42c9e157f9f69c22da"}
Sep 17 19:18:29.966660 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:18:29.966638    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-97977b6b5-2fskk"
Sep 17 19:18:29.974179 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:18:29.974154    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/ec74f400-7706-4361-99cc-b32b746af739-cabundle-cert\") pod \"ec74f400-7706-4361-99cc-b32b746af739\" (UID: \"ec74f400-7706-4361-99cc-b32b746af739\") "
Sep 17 19:18:29.974287 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:18:29.974206    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ec74f400-7706-4361-99cc-b32b746af739-proxy-tls\") pod \"ec74f400-7706-4361-99cc-b32b746af739\" (UID: \"ec74f400-7706-4361-99cc-b32b746af739\") "
Sep 17 19:18:29.974287 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:18:29.974237    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ppjtz\" (UniqueName: \"kubernetes.io/projected/ec74f400-7706-4361-99cc-b32b746af739-kube-api-access-ppjtz\") pod \"ec74f400-7706-4361-99cc-b32b746af739\" (UID: \"ec74f400-7706-4361-99cc-b32b746af739\") "
Sep 17 19:18:29.974287 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:18:29.974268    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ec74f400-7706-4361-99cc-b32b746af739-kube-rbac-proxy-sar-config\") pod \"ec74f400-7706-4361-99cc-b32b746af739\" (UID: \"ec74f400-7706-4361-99cc-b32b746af739\") "
Sep 17 19:18:29.974399 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:18:29.974311    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ec74f400-7706-4361-99cc-b32b746af739-kserve-provision-location\") pod \"ec74f400-7706-4361-99cc-b32b746af739\" (UID: \"ec74f400-7706-4361-99cc-b32b746af739\") "
Sep 17 19:18:29.974543 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:18:29.974504    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ec74f400-7706-4361-99cc-b32b746af739-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "ec74f400-7706-4361-99cc-b32b746af739" (UID: "ec74f400-7706-4361-99cc-b32b746af739"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 19:18:29.974697 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:18:29.974666    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ec74f400-7706-4361-99cc-b32b746af739-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "ec74f400-7706-4361-99cc-b32b746af739" (UID: "ec74f400-7706-4361-99cc-b32b746af739"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 19:18:29.974774 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:18:29.974669    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ec74f400-7706-4361-99cc-b32b746af739-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "ec74f400-7706-4361-99cc-b32b746af739" (UID: "ec74f400-7706-4361-99cc-b32b746af739"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 19:18:29.976354 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:18:29.976334    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ec74f400-7706-4361-99cc-b32b746af739-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "ec74f400-7706-4361-99cc-b32b746af739" (UID: "ec74f400-7706-4361-99cc-b32b746af739"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 19:18:29.976510 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:18:29.976488    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ec74f400-7706-4361-99cc-b32b746af739-kube-api-access-ppjtz" (OuterVolumeSpecName: "kube-api-access-ppjtz") pod "ec74f400-7706-4361-99cc-b32b746af739" (UID: "ec74f400-7706-4361-99cc-b32b746af739"). InnerVolumeSpecName "kube-api-access-ppjtz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 19:18:30.075044 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:18:30.074974    2582 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ec74f400-7706-4361-99cc-b32b746af739-proxy-tls\") on node \"ip-10-0-131-232.ec2.internal\" DevicePath \"\""
Sep 17 19:18:30.075044 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:18:30.074999    2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ppjtz\" (UniqueName: \"kubernetes.io/projected/ec74f400-7706-4361-99cc-b32b746af739-kube-api-access-ppjtz\") on node \"ip-10-0-131-232.ec2.internal\" DevicePath \"\""
Sep 17 19:18:30.075044 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:18:30.075010    2582 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ec74f400-7706-4361-99cc-b32b746af739-kube-rbac-proxy-sar-config\") on node \"ip-10-0-131-232.ec2.internal\" DevicePath \"\""
Sep 17 19:18:30.075044 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:18:30.075019    2582 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ec74f400-7706-4361-99cc-b32b746af739-kserve-provision-location\") on node \"ip-10-0-131-232.ec2.internal\" DevicePath \"\""
Sep 17 19:18:30.075044 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:18:30.075028    2582 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/ec74f400-7706-4361-99cc-b32b746af739-cabundle-cert\") on node \"ip-10-0-131-232.ec2.internal\" DevicePath \"\""
Sep 17 19:18:30.592513 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:18:30.592477    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-canary-stop-predictor-55656fdf77-6k5vs"]
Sep 17 19:18:30.592914 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:18:30.592898    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ec74f400-7706-4361-99cc-b32b746af739" containerName="storage-initializer"
Sep 17 19:18:30.592963 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:18:30.592918    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="ec74f400-7706-4361-99cc-b32b746af739" containerName="storage-initializer"
Sep 17 19:18:30.592963 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:18:30.592937    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6b10759f-c25e-4045-a992-7fb4bd1edc2f" containerName="kserve-container"
Sep 17 19:18:30.592963 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:18:30.592947    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="6b10759f-c25e-4045-a992-7fb4bd1edc2f" containerName="kserve-container"
Sep 17 19:18:30.593053 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:18:30.592963    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6b10759f-c25e-4045-a992-7fb4bd1edc2f" containerName="kube-rbac-proxy"
Sep 17 19:18:30.593053 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:18:30.592974    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="6b10759f-c25e-4045-a992-7fb4bd1edc2f" containerName="kube-rbac-proxy"
Sep 17 19:18:30.593053 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:18:30.592986    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6b10759f-c25e-4045-a992-7fb4bd1edc2f" containerName="agent"
Sep 17 19:18:30.593053 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:18:30.592995    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="6b10759f-c25e-4045-a992-7fb4bd1edc2f" containerName="agent"
Sep 17 19:18:30.593053 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:18:30.593010    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6b10759f-c25e-4045-a992-7fb4bd1edc2f" containerName="storage-initializer"
Sep 17 19:18:30.593053 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:18:30.593018    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="6b10759f-c25e-4045-a992-7fb4bd1edc2f" containerName="storage-initializer"
Sep 17 19:18:30.593053 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:18:30.593031    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ec74f400-7706-4361-99cc-b32b746af739" containerName="kube-rbac-proxy"
Sep 17 19:18:30.593053 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:18:30.593038    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="ec74f400-7706-4361-99cc-b32b746af739" containerName="kube-rbac-proxy"
Sep 17 19:18:30.593053 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:18:30.593049    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ec74f400-7706-4361-99cc-b32b746af739" containerName="kserve-container"
Sep 17 19:18:30.593311 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:18:30.593057    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="ec74f400-7706-4361-99cc-b32b746af739" containerName="kserve-container"
Sep 17 19:18:30.593311 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:18:30.593120    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="6b10759f-c25e-4045-a992-7fb4bd1edc2f" containerName="kube-rbac-proxy"
Sep 17 19:18:30.593311 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:18:30.593129    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="6b10759f-c25e-4045-a992-7fb4bd1edc2f" containerName="agent"
Sep 17 19:18:30.593311 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:18:30.593136    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="ec74f400-7706-4361-99cc-b32b746af739" containerName="kube-rbac-proxy"
Sep 17 19:18:30.593311 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:18:30.593147    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="6b10759f-c25e-4045-a992-7fb4bd1edc2f" containerName="kserve-container"
Sep 17 19:18:30.593311 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:18:30.593155    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="ec74f400-7706-4361-99cc-b32b746af739" containerName="kserve-container"
Sep 17 19:18:30.596886 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:18:30.596862    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-55656fdf77-6k5vs"
Sep 17 19:18:30.599220 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:18:30.599202    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-stop-kube-rbac-proxy-sar-config\""
Sep 17 19:18:30.599793 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:18:30.599769    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-stop-predictor-serving-cert\""
Sep 17 19:18:30.607496 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:18:30.607473    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-stop-predictor-55656fdf77-6k5vs"]
Sep 17 19:18:30.679563 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:18:30.679535    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/51310d70-7568-4c17-beb5-7be4bee6764b-proxy-tls\") pod \"isvc-canary-stop-predictor-55656fdf77-6k5vs\" (UID: \"51310d70-7568-4c17-beb5-7be4bee6764b\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-55656fdf77-6k5vs"
Sep 17 19:18:30.679563 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:18:30.679568    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w5hll\" (UniqueName: \"kubernetes.io/projected/51310d70-7568-4c17-beb5-7be4bee6764b-kube-api-access-w5hll\") pod \"isvc-canary-stop-predictor-55656fdf77-6k5vs\" (UID: \"51310d70-7568-4c17-beb5-7be4bee6764b\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-55656fdf77-6k5vs"
Sep 17 19:18:30.679739 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:18:30.679585    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/51310d70-7568-4c17-beb5-7be4bee6764b-cabundle-cert\") pod \"isvc-canary-stop-predictor-55656fdf77-6k5vs\" (UID: \"51310d70-7568-4c17-beb5-7be4bee6764b\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-55656fdf77-6k5vs"
Sep 17 19:18:30.679739 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:18:30.679670    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/51310d70-7568-4c17-beb5-7be4bee6764b-kube-rbac-proxy-sar-config\") pod \"isvc-canary-stop-predictor-55656fdf77-6k5vs\" (UID: \"51310d70-7568-4c17-beb5-7be4bee6764b\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-55656fdf77-6k5vs"
Sep 17 19:18:30.679739 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:18:30.679702    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/51310d70-7568-4c17-beb5-7be4bee6764b-kserve-provision-location\") pod \"isvc-canary-stop-predictor-55656fdf77-6k5vs\" (UID: \"51310d70-7568-4c17-beb5-7be4bee6764b\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-55656fdf77-6k5vs"
Sep 17 19:18:30.780670 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:18:30.780647    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/51310d70-7568-4c17-beb5-7be4bee6764b-kube-rbac-proxy-sar-config\") pod \"isvc-canary-stop-predictor-55656fdf77-6k5vs\" (UID: \"51310d70-7568-4c17-beb5-7be4bee6764b\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-55656fdf77-6k5vs"
Sep 17 19:18:30.780753 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:18:30.780678    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/51310d70-7568-4c17-beb5-7be4bee6764b-kserve-provision-location\") pod \"isvc-canary-stop-predictor-55656fdf77-6k5vs\" (UID: \"51310d70-7568-4c17-beb5-7be4bee6764b\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-55656fdf77-6k5vs"
Sep 17 19:18:30.780854 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:18:30.780809    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/51310d70-7568-4c17-beb5-7be4bee6764b-proxy-tls\") pod \"isvc-canary-stop-predictor-55656fdf77-6k5vs\" (UID: \"51310d70-7568-4c17-beb5-7be4bee6764b\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-55656fdf77-6k5vs"
Sep 17 19:18:30.780905 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:18:30.780880    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w5hll\" (UniqueName: \"kubernetes.io/projected/51310d70-7568-4c17-beb5-7be4bee6764b-kube-api-access-w5hll\") pod \"isvc-canary-stop-predictor-55656fdf77-6k5vs\" (UID: \"51310d70-7568-4c17-beb5-7be4bee6764b\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-55656fdf77-6k5vs"
Sep 17 19:18:30.780905 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:18:30.780900    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/51310d70-7568-4c17-beb5-7be4bee6764b-cabundle-cert\") pod \"isvc-canary-stop-predictor-55656fdf77-6k5vs\" (UID: \"51310d70-7568-4c17-beb5-7be4bee6764b\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-55656fdf77-6k5vs"
Sep 17 19:18:30.781081 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:18:30.781066    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/51310d70-7568-4c17-beb5-7be4bee6764b-kserve-provision-location\") pod \"isvc-canary-stop-predictor-55656fdf77-6k5vs\" (UID: \"51310d70-7568-4c17-beb5-7be4bee6764b\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-55656fdf77-6k5vs"
Sep 17 19:18:30.781455 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:18:30.781426    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/51310d70-7568-4c17-beb5-7be4bee6764b-kube-rbac-proxy-sar-config\") pod \"isvc-canary-stop-predictor-55656fdf77-6k5vs\" (UID: \"51310d70-7568-4c17-beb5-7be4bee6764b\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-55656fdf77-6k5vs"
Sep 17 19:18:30.781560 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:18:30.781462    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/51310d70-7568-4c17-beb5-7be4bee6764b-cabundle-cert\") pod \"isvc-canary-stop-predictor-55656fdf77-6k5vs\" (UID: \"51310d70-7568-4c17-beb5-7be4bee6764b\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-55656fdf77-6k5vs"
Sep 17 19:18:30.783278 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:18:30.783256    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/51310d70-7568-4c17-beb5-7be4bee6764b-proxy-tls\") pod \"isvc-canary-stop-predictor-55656fdf77-6k5vs\" (UID: \"51310d70-7568-4c17-beb5-7be4bee6764b\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-55656fdf77-6k5vs"
Sep 17 19:18:30.789183 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:18:30.789162    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-w5hll\" (UniqueName: \"kubernetes.io/projected/51310d70-7568-4c17-beb5-7be4bee6764b-kube-api-access-w5hll\") pod \"isvc-canary-stop-predictor-55656fdf77-6k5vs\" (UID: \"51310d70-7568-4c17-beb5-7be4bee6764b\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-55656fdf77-6k5vs"
Sep 17 19:18:30.856155 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:18:30.856088    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-97977b6b5-2fskk"
Sep 17 19:18:30.856155 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:18:30.856098    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-97977b6b5-2fskk" event={"ID":"ec74f400-7706-4361-99cc-b32b746af739","Type":"ContainerDied","Data":"22ce2876c5c87dcd4fc707d0513a9a4c8e127f64b7b608e625e12c44102475eb"}
Sep 17 19:18:30.856564 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:18:30.856164    2582 scope.go:117] "RemoveContainer" containerID="67ca2c768e7be7479832185701c3562e6bd8dccec3d1ce9f55ee8e22eff10214"
Sep 17 19:18:30.863689 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:18:30.863673    2582 scope.go:117] "RemoveContainer" containerID="3d88e360bcb94d17383d5b5c09404b0527a40511bb403e42c9e157f9f69c22da"
Sep 17 19:18:30.870811 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:18:30.870791    2582 scope.go:117] "RemoveContainer" containerID="3948b9a708ece5af0a11915d49b73b28b41edefa7575844ce1b16ecec59b6aae"
Sep 17 19:18:30.874432 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:18:30.874411    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-predictor-97977b6b5-2fskk"]
Sep 17 19:18:30.878452 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:18:30.878375    2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-predictor-97977b6b5-2fskk"]
Sep 17 19:18:30.909627 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:18:30.909597    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-55656fdf77-6k5vs"
Sep 17 19:18:31.182676 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:18:31.182645    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-stop-predictor-55656fdf77-6k5vs"]
Sep 17 19:18:31.185669 ip-10-0-131-232 kubenswrapper[2582]: W0917 19:18:31.185647    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod51310d70_7568_4c17_beb5_7be4bee6764b.slice/crio-1abf83cd7f224117270d4c7a0b398d076b68122308a5b5b216432f311a63d415 WatchSource:0}: Error finding container 1abf83cd7f224117270d4c7a0b398d076b68122308a5b5b216432f311a63d415: Status 404 returned error can't find the container with id 1abf83cd7f224117270d4c7a0b398d076b68122308a5b5b216432f311a63d415
Sep 17 19:18:31.862575 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:18:31.862530    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-55656fdf77-6k5vs" event={"ID":"51310d70-7568-4c17-beb5-7be4bee6764b","Type":"ContainerStarted","Data":"7de04522aa61ed4d75b29eabc9ff17da519306deca9a33931c7e34e2100d184e"}
Sep 17 19:18:31.862963 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:18:31.862605    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-55656fdf77-6k5vs" event={"ID":"51310d70-7568-4c17-beb5-7be4bee6764b","Type":"ContainerStarted","Data":"1abf83cd7f224117270d4c7a0b398d076b68122308a5b5b216432f311a63d415"}
Sep 17 19:18:32.768717 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:18:32.768684    2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ec74f400-7706-4361-99cc-b32b746af739" path="/var/lib/kubelet/pods/ec74f400-7706-4361-99cc-b32b746af739/volumes"
Sep 17 19:18:35.880439 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:18:35.880400    2582 generic.go:356] "Generic (PLEG): container finished" podID="51310d70-7568-4c17-beb5-7be4bee6764b" containerID="7de04522aa61ed4d75b29eabc9ff17da519306deca9a33931c7e34e2100d184e" exitCode=0
Sep 17 19:18:35.880934 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:18:35.880444    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-55656fdf77-6k5vs" event={"ID":"51310d70-7568-4c17-beb5-7be4bee6764b","Type":"ContainerDied","Data":"7de04522aa61ed4d75b29eabc9ff17da519306deca9a33931c7e34e2100d184e"}
Sep 17 19:18:36.886891 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:18:36.886858    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-55656fdf77-6k5vs" event={"ID":"51310d70-7568-4c17-beb5-7be4bee6764b","Type":"ContainerStarted","Data":"96b017d9b9f61b6aa417572d4bf732cfc282a55631bf0016676bc7c40b6f68c7"}
Sep 17 19:18:36.887263 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:18:36.886922    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-55656fdf77-6k5vs" event={"ID":"51310d70-7568-4c17-beb5-7be4bee6764b","Type":"ContainerStarted","Data":"470a775e39db790bfb1f4c96041392d36a7b882040c93d19352a4c5a8c0f70fa"}
Sep 17 19:18:36.887263 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:18:36.887125    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-55656fdf77-6k5vs"
Sep 17 19:18:36.907720 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:18:36.907663    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-55656fdf77-6k5vs" podStartSLOduration=3.141773995 podStartE2EDuration="6.907643841s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.765869846s" isStatefulPod=false podCreationTimestamp="2026-09-17 19:18:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 19:18:36.907130647 +0000 UTC m=+732.678801709" watchObservedRunningTime="2026-09-17 19:18:36.907643841 +0000 UTC m=+732.679314904"
Sep 17 19:18:37.891624 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:18:37.891589    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-55656fdf77-6k5vs"
Sep 17 19:18:37.892900 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:18:37.892871    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-55656fdf77-6k5vs" podUID="51310d70-7568-4c17-beb5-7be4bee6764b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.27:8080: connect: connection refused"
Sep 17 19:18:38.895184 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:18:38.895137    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-55656fdf77-6k5vs" podUID="51310d70-7568-4c17-beb5-7be4bee6764b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.27:8080: connect: connection refused"
Sep 17 19:18:40.865860 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:18:40.865813    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/model-chainer-raw-e3e91-747b5c94b5-dvrfx"]
Sep 17 19:18:40.869201 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:18:40.869182    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/model-chainer-raw-e3e91-747b5c94b5-dvrfx"
Sep 17 19:18:40.871607 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:18:40.871580    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"model-chainer-raw-e3e91-kube-rbac-proxy-sar-config\""
Sep 17 19:18:40.871723 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:18:40.871581    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"model-chainer-raw-e3e91-serving-cert\""
Sep 17 19:18:40.880050 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:18:40.880028    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/model-chainer-raw-e3e91-747b5c94b5-dvrfx"]
Sep 17 19:18:40.954116 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:18:40.954070    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7e8e1795-b624-4746-9390-acfc36398c2a-openshift-service-ca-bundle\") pod \"model-chainer-raw-e3e91-747b5c94b5-dvrfx\" (UID: \"7e8e1795-b624-4746-9390-acfc36398c2a\") " pod="kserve-ci-e2e-test/model-chainer-raw-e3e91-747b5c94b5-dvrfx"
Sep 17 19:18:40.954297 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:18:40.954143    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/7e8e1795-b624-4746-9390-acfc36398c2a-proxy-tls\") pod \"model-chainer-raw-e3e91-747b5c94b5-dvrfx\" (UID: \"7e8e1795-b624-4746-9390-acfc36398c2a\") " pod="kserve-ci-e2e-test/model-chainer-raw-e3e91-747b5c94b5-dvrfx"
Sep 17 19:18:41.055153 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:18:41.055124    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7e8e1795-b624-4746-9390-acfc36398c2a-openshift-service-ca-bundle\") pod \"model-chainer-raw-e3e91-747b5c94b5-dvrfx\" (UID: \"7e8e1795-b624-4746-9390-acfc36398c2a\") " pod="kserve-ci-e2e-test/model-chainer-raw-e3e91-747b5c94b5-dvrfx"
Sep 17 19:18:41.055323 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:18:41.055183    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/7e8e1795-b624-4746-9390-acfc36398c2a-proxy-tls\") pod \"model-chainer-raw-e3e91-747b5c94b5-dvrfx\" (UID: \"7e8e1795-b624-4746-9390-acfc36398c2a\") " pod="kserve-ci-e2e-test/model-chainer-raw-e3e91-747b5c94b5-dvrfx"
Sep 17 19:18:41.055323 ip-10-0-131-232 kubenswrapper[2582]: E0917 19:18:41.055301    2582 secret.go:189] Couldn't get secret kserve-ci-e2e-test/model-chainer-raw-e3e91-serving-cert: secret "model-chainer-raw-e3e91-serving-cert" not found
Sep 17 19:18:41.055615 ip-10-0-131-232 kubenswrapper[2582]: E0917 19:18:41.055384    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7e8e1795-b624-4746-9390-acfc36398c2a-proxy-tls podName:7e8e1795-b624-4746-9390-acfc36398c2a nodeName:}" failed. No retries permitted until 2026-09-17 19:18:41.555361812 +0000 UTC m=+737.327032865 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/7e8e1795-b624-4746-9390-acfc36398c2a-proxy-tls") pod "model-chainer-raw-e3e91-747b5c94b5-dvrfx" (UID: "7e8e1795-b624-4746-9390-acfc36398c2a") : secret "model-chainer-raw-e3e91-serving-cert" not found
Sep 17 19:18:41.055861 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:18:41.055841    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7e8e1795-b624-4746-9390-acfc36398c2a-openshift-service-ca-bundle\") pod \"model-chainer-raw-e3e91-747b5c94b5-dvrfx\" (UID: \"7e8e1795-b624-4746-9390-acfc36398c2a\") " pod="kserve-ci-e2e-test/model-chainer-raw-e3e91-747b5c94b5-dvrfx"
Sep 17 19:18:41.558226 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:18:41.558190    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/7e8e1795-b624-4746-9390-acfc36398c2a-proxy-tls\") pod \"model-chainer-raw-e3e91-747b5c94b5-dvrfx\" (UID: \"7e8e1795-b624-4746-9390-acfc36398c2a\") " pod="kserve-ci-e2e-test/model-chainer-raw-e3e91-747b5c94b5-dvrfx"
Sep 17 19:18:41.560903 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:18:41.560878    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/7e8e1795-b624-4746-9390-acfc36398c2a-proxy-tls\") pod \"model-chainer-raw-e3e91-747b5c94b5-dvrfx\" (UID: \"7e8e1795-b624-4746-9390-acfc36398c2a\") " pod="kserve-ci-e2e-test/model-chainer-raw-e3e91-747b5c94b5-dvrfx"
Sep 17 19:18:41.779943 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:18:41.779907    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/model-chainer-raw-e3e91-747b5c94b5-dvrfx"
Sep 17 19:18:42.051421 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:18:42.051392    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/model-chainer-raw-e3e91-747b5c94b5-dvrfx"]
Sep 17 19:18:42.054424 ip-10-0-131-232 kubenswrapper[2582]: W0917 19:18:42.054385    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7e8e1795_b624_4746_9390_acfc36398c2a.slice/crio-3ff684f4544d99697aea4e8263bc077b55599dd5636d9c17bfe6f041ecf3de31 WatchSource:0}: Error finding container 3ff684f4544d99697aea4e8263bc077b55599dd5636d9c17bfe6f041ecf3de31: Status 404 returned error can't find the container with id 3ff684f4544d99697aea4e8263bc077b55599dd5636d9c17bfe6f041ecf3de31
Sep 17 19:18:42.056373 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:18:42.056354    2582 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 17 19:18:42.913572 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:18:42.913536    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/model-chainer-raw-e3e91-747b5c94b5-dvrfx" event={"ID":"7e8e1795-b624-4746-9390-acfc36398c2a","Type":"ContainerStarted","Data":"3ff684f4544d99697aea4e8263bc077b55599dd5636d9c17bfe6f041ecf3de31"}
Sep 17 19:18:43.900249 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:18:43.900212    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-55656fdf77-6k5vs"
Sep 17 19:18:43.901038 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:18:43.900993    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-55656fdf77-6k5vs" podUID="51310d70-7568-4c17-beb5-7be4bee6764b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.27:8080: connect: connection refused"
Sep 17 19:18:44.923183 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:18:44.923150    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/model-chainer-raw-e3e91-747b5c94b5-dvrfx" event={"ID":"7e8e1795-b624-4746-9390-acfc36398c2a","Type":"ContainerStarted","Data":"8f2fcf60072da64b820e71c31bf7316ff60169536f9df89ff667a6e2680414cd"}
Sep 17 19:18:44.923610 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:18:44.923211    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/model-chainer-raw-e3e91-747b5c94b5-dvrfx"
Sep 17 19:18:44.942566 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:18:44.942494    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/model-chainer-raw-e3e91-747b5c94b5-dvrfx" podStartSLOduration=2.766322727 podStartE2EDuration="4.942479257s" totalImagesPullingTime="2.17615653s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 19:18:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 19:18:44.941184498 +0000 UTC m=+740.712855559" watchObservedRunningTime="2026-09-17 19:18:44.942479257 +0000 UTC m=+740.714150318"
Sep 17 19:18:50.927634 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:18:50.927600    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/model-chainer-raw-e3e91-747b5c94b5-dvrfx"]
Sep 17 19:18:50.928033 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:18:50.927772    2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/model-chainer-raw-e3e91-747b5c94b5-dvrfx" podUID="7e8e1795-b624-4746-9390-acfc36398c2a" containerName="model-chainer-raw-e3e91" containerID="cri-o://8f2fcf60072da64b820e71c31bf7316ff60169536f9df89ff667a6e2680414cd" gracePeriod=30
Sep 17 19:18:50.933445 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:18:50.933391    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/model-chainer-raw-e3e91-747b5c94b5-dvrfx" podUID="7e8e1795-b624-4746-9390-acfc36398c2a" containerName="model-chainer-raw-e3e91" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 19:18:51.205498 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:18:51.205416    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-7eb14-predictor-7dbd4779b4-8l7vj"]
Sep 17 19:18:51.209226 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:18:51.209207    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-7eb14-predictor-7dbd4779b4-8l7vj"
Sep 17 19:18:51.211628 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:18:51.211607    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-graph-raw-hpa-7eb14-predictor-serving-cert\""
Sep 17 19:18:51.211731 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:18:51.211657    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-graph-raw-hpa-7eb14-kube-rbac-proxy-sar-config\""
Sep 17 19:18:51.220019 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:18:51.219999    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-7eb14-predictor-7dbd4779b4-8l7vj"]
Sep 17 19:18:51.238779 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:18:51.238748    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/8b82721b-ff56-417c-8171-517d0a81ea8b-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-graph-raw-hpa-7eb14-predictor-7dbd4779b4-8l7vj\" (UID: \"8b82721b-ff56-417c-8171-517d0a81ea8b\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-7eb14-predictor-7dbd4779b4-8l7vj"
Sep 17 19:18:51.239005 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:18:51.238976    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/8b82721b-ff56-417c-8171-517d0a81ea8b-cabundle-cert\") pod \"isvc-sklearn-graph-raw-hpa-7eb14-predictor-7dbd4779b4-8l7vj\" (UID: \"8b82721b-ff56-417c-8171-517d0a81ea8b\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-7eb14-predictor-7dbd4779b4-8l7vj"
Sep 17 19:18:51.239117 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:18:51.239023    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/8b82721b-ff56-417c-8171-517d0a81ea8b-proxy-tls\") pod \"isvc-sklearn-graph-raw-hpa-7eb14-predictor-7dbd4779b4-8l7vj\" (UID: \"8b82721b-ff56-417c-8171-517d0a81ea8b\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-7eb14-predictor-7dbd4779b4-8l7vj"
Sep 17 19:18:51.239117 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:18:51.239103    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8b82721b-ff56-417c-8171-517d0a81ea8b-kserve-provision-location\") pod \"isvc-sklearn-graph-raw-hpa-7eb14-predictor-7dbd4779b4-8l7vj\" (UID: \"8b82721b-ff56-417c-8171-517d0a81ea8b\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-7eb14-predictor-7dbd4779b4-8l7vj"
Sep 17 19:18:51.239237 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:18:51.239175    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5jd2p\" (UniqueName: \"kubernetes.io/projected/8b82721b-ff56-417c-8171-517d0a81ea8b-kube-api-access-5jd2p\") pod \"isvc-sklearn-graph-raw-hpa-7eb14-predictor-7dbd4779b4-8l7vj\" (UID: \"8b82721b-ff56-417c-8171-517d0a81ea8b\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-7eb14-predictor-7dbd4779b4-8l7vj"
Sep 17 19:18:51.340470 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:18:51.340430    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/8b82721b-ff56-417c-8171-517d0a81ea8b-proxy-tls\") pod \"isvc-sklearn-graph-raw-hpa-7eb14-predictor-7dbd4779b4-8l7vj\" (UID: \"8b82721b-ff56-417c-8171-517d0a81ea8b\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-7eb14-predictor-7dbd4779b4-8l7vj"
Sep 17 19:18:51.340641 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:18:51.340486    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8b82721b-ff56-417c-8171-517d0a81ea8b-kserve-provision-location\") pod \"isvc-sklearn-graph-raw-hpa-7eb14-predictor-7dbd4779b4-8l7vj\" (UID: \"8b82721b-ff56-417c-8171-517d0a81ea8b\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-7eb14-predictor-7dbd4779b4-8l7vj"
Sep 17 19:18:51.340641 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:18:51.340525    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5jd2p\" (UniqueName: \"kubernetes.io/projected/8b82721b-ff56-417c-8171-517d0a81ea8b-kube-api-access-5jd2p\") pod \"isvc-sklearn-graph-raw-hpa-7eb14-predictor-7dbd4779b4-8l7vj\" (UID: \"8b82721b-ff56-417c-8171-517d0a81ea8b\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-7eb14-predictor-7dbd4779b4-8l7vj"
Sep 17 19:18:51.340641 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:18:51.340602    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/8b82721b-ff56-417c-8171-517d0a81ea8b-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-graph-raw-hpa-7eb14-predictor-7dbd4779b4-8l7vj\" (UID: \"8b82721b-ff56-417c-8171-517d0a81ea8b\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-7eb14-predictor-7dbd4779b4-8l7vj"
Sep 17 19:18:51.340641 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:18:51.340628    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/8b82721b-ff56-417c-8171-517d0a81ea8b-cabundle-cert\") pod \"isvc-sklearn-graph-raw-hpa-7eb14-predictor-7dbd4779b4-8l7vj\" (UID: \"8b82721b-ff56-417c-8171-517d0a81ea8b\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-7eb14-predictor-7dbd4779b4-8l7vj"
Sep 17 19:18:51.340641 ip-10-0-131-232 kubenswrapper[2582]: E0917 19:18:51.340630    2582 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-7eb14-predictor-serving-cert: secret "isvc-sklearn-graph-raw-hpa-7eb14-predictor-serving-cert" not found
Sep 17 19:18:51.340942 ip-10-0-131-232 kubenswrapper[2582]: E0917 19:18:51.340708    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8b82721b-ff56-417c-8171-517d0a81ea8b-proxy-tls podName:8b82721b-ff56-417c-8171-517d0a81ea8b nodeName:}" failed. No retries permitted until 2026-09-17 19:18:51.840685628 +0000 UTC m=+747.612356668 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/8b82721b-ff56-417c-8171-517d0a81ea8b-proxy-tls") pod "isvc-sklearn-graph-raw-hpa-7eb14-predictor-7dbd4779b4-8l7vj" (UID: "8b82721b-ff56-417c-8171-517d0a81ea8b") : secret "isvc-sklearn-graph-raw-hpa-7eb14-predictor-serving-cert" not found
Sep 17 19:18:51.341011 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:18:51.340981    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8b82721b-ff56-417c-8171-517d0a81ea8b-kserve-provision-location\") pod \"isvc-sklearn-graph-raw-hpa-7eb14-predictor-7dbd4779b4-8l7vj\" (UID: \"8b82721b-ff56-417c-8171-517d0a81ea8b\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-7eb14-predictor-7dbd4779b4-8l7vj"
Sep 17 19:18:51.341337 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:18:51.341316    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/8b82721b-ff56-417c-8171-517d0a81ea8b-cabundle-cert\") pod \"isvc-sklearn-graph-raw-hpa-7eb14-predictor-7dbd4779b4-8l7vj\" (UID: \"8b82721b-ff56-417c-8171-517d0a81ea8b\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-7eb14-predictor-7dbd4779b4-8l7vj"
Sep 17 19:18:51.341376 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:18:51.341356    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/8b82721b-ff56-417c-8171-517d0a81ea8b-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-graph-raw-hpa-7eb14-predictor-7dbd4779b4-8l7vj\" (UID: \"8b82721b-ff56-417c-8171-517d0a81ea8b\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-7eb14-predictor-7dbd4779b4-8l7vj"
Sep 17 19:18:51.351694 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:18:51.351662    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5jd2p\" (UniqueName: \"kubernetes.io/projected/8b82721b-ff56-417c-8171-517d0a81ea8b-kube-api-access-5jd2p\") pod \"isvc-sklearn-graph-raw-hpa-7eb14-predictor-7dbd4779b4-8l7vj\" (UID: \"8b82721b-ff56-417c-8171-517d0a81ea8b\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-7eb14-predictor-7dbd4779b4-8l7vj"
Sep 17 19:18:51.850281 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:18:51.850245    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/8b82721b-ff56-417c-8171-517d0a81ea8b-proxy-tls\") pod \"isvc-sklearn-graph-raw-hpa-7eb14-predictor-7dbd4779b4-8l7vj\" (UID: \"8b82721b-ff56-417c-8171-517d0a81ea8b\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-7eb14-predictor-7dbd4779b4-8l7vj"
Sep 17 19:18:51.852749 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:18:51.852720    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/8b82721b-ff56-417c-8171-517d0a81ea8b-proxy-tls\") pod \"isvc-sklearn-graph-raw-hpa-7eb14-predictor-7dbd4779b4-8l7vj\" (UID: \"8b82721b-ff56-417c-8171-517d0a81ea8b\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-7eb14-predictor-7dbd4779b4-8l7vj"
Sep 17 19:18:52.121862 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:18:52.121756    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-7eb14-predictor-7dbd4779b4-8l7vj"
Sep 17 19:18:52.396639 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:18:52.396616    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-7eb14-predictor-7dbd4779b4-8l7vj"]
Sep 17 19:18:52.399005 ip-10-0-131-232 kubenswrapper[2582]: W0917 19:18:52.398974    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8b82721b_ff56_417c_8171_517d0a81ea8b.slice/crio-54f37bb80fd6082e6df8e6cd9b403839e3fcb733f25233ecd74e871e8e3ae358 WatchSource:0}: Error finding container 54f37bb80fd6082e6df8e6cd9b403839e3fcb733f25233ecd74e871e8e3ae358: Status 404 returned error can't find the container with id 54f37bb80fd6082e6df8e6cd9b403839e3fcb733f25233ecd74e871e8e3ae358
Sep 17 19:18:52.964590 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:18:52.964557    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-7eb14-predictor-7dbd4779b4-8l7vj" event={"ID":"8b82721b-ff56-417c-8171-517d0a81ea8b","Type":"ContainerStarted","Data":"b6fc6b5c4f24f2cf6c4507852573bbe837805cd21f395ec53bb5a224755c48b4"}
Sep 17 19:18:52.964771 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:18:52.964613    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-7eb14-predictor-7dbd4779b4-8l7vj" event={"ID":"8b82721b-ff56-417c-8171-517d0a81ea8b","Type":"ContainerStarted","Data":"54f37bb80fd6082e6df8e6cd9b403839e3fcb733f25233ecd74e871e8e3ae358"}
Sep 17 19:18:53.901151 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:18:53.901112    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-55656fdf77-6k5vs" podUID="51310d70-7568-4c17-beb5-7be4bee6764b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.27:8080: connect: connection refused"
Sep 17 19:18:55.932660 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:18:55.932623    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/model-chainer-raw-e3e91-747b5c94b5-dvrfx" podUID="7e8e1795-b624-4746-9390-acfc36398c2a" containerName="model-chainer-raw-e3e91" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 19:18:56.984344 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:18:56.984310    2582 generic.go:356] "Generic (PLEG): container finished" podID="8b82721b-ff56-417c-8171-517d0a81ea8b" containerID="b6fc6b5c4f24f2cf6c4507852573bbe837805cd21f395ec53bb5a224755c48b4" exitCode=0
Sep 17 19:18:56.984762 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:18:56.984369    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-7eb14-predictor-7dbd4779b4-8l7vj" event={"ID":"8b82721b-ff56-417c-8171-517d0a81ea8b","Type":"ContainerDied","Data":"b6fc6b5c4f24f2cf6c4507852573bbe837805cd21f395ec53bb5a224755c48b4"}
Sep 17 19:18:57.990171 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:18:57.990135    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-7eb14-predictor-7dbd4779b4-8l7vj" event={"ID":"8b82721b-ff56-417c-8171-517d0a81ea8b","Type":"ContainerStarted","Data":"ef27ac62036db3653fe171d401ecdbc392750839571ec9950713dd57a82d055c"}
Sep 17 19:18:57.990542 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:18:57.990199    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-7eb14-predictor-7dbd4779b4-8l7vj" event={"ID":"8b82721b-ff56-417c-8171-517d0a81ea8b","Type":"ContainerStarted","Data":"7893ba91e736e6a4e08ddf0773a642286f13a9524d3e8685e1532a29e025669e"}
Sep 17 19:18:57.990542 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:18:57.990389    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-7eb14-predictor-7dbd4779b4-8l7vj"
Sep 17 19:18:58.010847 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:18:58.010764    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-7eb14-predictor-7dbd4779b4-8l7vj" podStartSLOduration=3.128517697 podStartE2EDuration="7.010744888s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.882227191s" isStatefulPod=false podCreationTimestamp="2026-09-17 19:18:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 19:18:58.009986874 +0000 UTC m=+753.781657937" watchObservedRunningTime="2026-09-17 19:18:58.010744888 +0000 UTC m=+753.782415951"
Sep 17 19:18:58.994728 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:18:58.994700    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-7eb14-predictor-7dbd4779b4-8l7vj"
Sep 17 19:18:58.995963 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:18:58.995937    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-7eb14-predictor-7dbd4779b4-8l7vj" podUID="8b82721b-ff56-417c-8171-517d0a81ea8b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.29:8080: connect: connection refused"
Sep 17 19:18:59.999254 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:18:59.999212    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-7eb14-predictor-7dbd4779b4-8l7vj" podUID="8b82721b-ff56-417c-8171-517d0a81ea8b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.29:8080: connect: connection refused"
Sep 17 19:19:00.932029 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:19:00.931978    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/model-chainer-raw-e3e91-747b5c94b5-dvrfx" podUID="7e8e1795-b624-4746-9390-acfc36398c2a" containerName="model-chainer-raw-e3e91" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 19:19:03.900915 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:19:03.900872    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-55656fdf77-6k5vs" podUID="51310d70-7568-4c17-beb5-7be4bee6764b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.27:8080: connect: connection refused"
Sep 17 19:19:05.003388 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:19:05.003354    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-7eb14-predictor-7dbd4779b4-8l7vj"
Sep 17 19:19:05.003889 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:19:05.003866    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-7eb14-predictor-7dbd4779b4-8l7vj" podUID="8b82721b-ff56-417c-8171-517d0a81ea8b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.29:8080: connect: connection refused"
Sep 17 19:19:05.932141 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:19:05.932098    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/model-chainer-raw-e3e91-747b5c94b5-dvrfx" podUID="7e8e1795-b624-4746-9390-acfc36398c2a" containerName="model-chainer-raw-e3e91" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 19:19:10.933253 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:19:10.933196    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/model-chainer-raw-e3e91-747b5c94b5-dvrfx" podUID="7e8e1795-b624-4746-9390-acfc36398c2a" containerName="model-chainer-raw-e3e91" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 19:19:13.901433 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:19:13.901390    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-55656fdf77-6k5vs" podUID="51310d70-7568-4c17-beb5-7be4bee6764b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.27:8080: connect: connection refused"
Sep 17 19:19:15.004013 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:19:15.003979    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-7eb14-predictor-7dbd4779b4-8l7vj" podUID="8b82721b-ff56-417c-8171-517d0a81ea8b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.29:8080: connect: connection refused"
Sep 17 19:19:15.931412 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:19:15.931367    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/model-chainer-raw-e3e91-747b5c94b5-dvrfx" podUID="7e8e1795-b624-4746-9390-acfc36398c2a" containerName="model-chainer-raw-e3e91" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 19:19:20.931663 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:19:20.931606    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/model-chainer-raw-e3e91-747b5c94b5-dvrfx" podUID="7e8e1795-b624-4746-9390-acfc36398c2a" containerName="model-chainer-raw-e3e91" probeResult="failure" output="Get \"https://10.134.0.28:8080/readyz\": dial tcp 10.134.0.28:8080: connect: connection refused"
Sep 17 19:19:21.071708 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:19:21.071685    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/model-chainer-raw-e3e91-747b5c94b5-dvrfx"
Sep 17 19:19:21.093557 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:19:21.093526    2582 generic.go:356] "Generic (PLEG): container finished" podID="7e8e1795-b624-4746-9390-acfc36398c2a" containerID="8f2fcf60072da64b820e71c31bf7316ff60169536f9df89ff667a6e2680414cd" exitCode=0
Sep 17 19:19:21.093711 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:19:21.093585    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/model-chainer-raw-e3e91-747b5c94b5-dvrfx"
Sep 17 19:19:21.093711 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:19:21.093607    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/model-chainer-raw-e3e91-747b5c94b5-dvrfx" event={"ID":"7e8e1795-b624-4746-9390-acfc36398c2a","Type":"ContainerDied","Data":"8f2fcf60072da64b820e71c31bf7316ff60169536f9df89ff667a6e2680414cd"}
Sep 17 19:19:21.093711 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:19:21.093671    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/model-chainer-raw-e3e91-747b5c94b5-dvrfx" event={"ID":"7e8e1795-b624-4746-9390-acfc36398c2a","Type":"ContainerDied","Data":"3ff684f4544d99697aea4e8263bc077b55599dd5636d9c17bfe6f041ecf3de31"}
Sep 17 19:19:21.093711 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:19:21.093703    2582 scope.go:117] "RemoveContainer" containerID="8f2fcf60072da64b820e71c31bf7316ff60169536f9df89ff667a6e2680414cd"
Sep 17 19:19:21.103074 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:19:21.103057    2582 scope.go:117] "RemoveContainer" containerID="8f2fcf60072da64b820e71c31bf7316ff60169536f9df89ff667a6e2680414cd"
Sep 17 19:19:21.103370 ip-10-0-131-232 kubenswrapper[2582]: E0917 19:19:21.103350    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8f2fcf60072da64b820e71c31bf7316ff60169536f9df89ff667a6e2680414cd\": container with ID starting with 8f2fcf60072da64b820e71c31bf7316ff60169536f9df89ff667a6e2680414cd not found: ID does not exist" containerID="8f2fcf60072da64b820e71c31bf7316ff60169536f9df89ff667a6e2680414cd"
Sep 17 19:19:21.103423 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:19:21.103378    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8f2fcf60072da64b820e71c31bf7316ff60169536f9df89ff667a6e2680414cd"} err="failed to get container status \"8f2fcf60072da64b820e71c31bf7316ff60169536f9df89ff667a6e2680414cd\": rpc error: code = NotFound desc = could not find container \"8f2fcf60072da64b820e71c31bf7316ff60169536f9df89ff667a6e2680414cd\": container with ID starting with 8f2fcf60072da64b820e71c31bf7316ff60169536f9df89ff667a6e2680414cd not found: ID does not exist"
Sep 17 19:19:21.198914 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:19:21.198850    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7e8e1795-b624-4746-9390-acfc36398c2a-openshift-service-ca-bundle\") pod \"7e8e1795-b624-4746-9390-acfc36398c2a\" (UID: \"7e8e1795-b624-4746-9390-acfc36398c2a\") "
Sep 17 19:19:21.199050 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:19:21.198928    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/7e8e1795-b624-4746-9390-acfc36398c2a-proxy-tls\") pod \"7e8e1795-b624-4746-9390-acfc36398c2a\" (UID: \"7e8e1795-b624-4746-9390-acfc36398c2a\") "
Sep 17 19:19:21.199196 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:19:21.199172    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7e8e1795-b624-4746-9390-acfc36398c2a-openshift-service-ca-bundle" (OuterVolumeSpecName: "openshift-service-ca-bundle") pod "7e8e1795-b624-4746-9390-acfc36398c2a" (UID: "7e8e1795-b624-4746-9390-acfc36398c2a"). InnerVolumeSpecName "openshift-service-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 19:19:21.201103 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:19:21.201081    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7e8e1795-b624-4746-9390-acfc36398c2a-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "7e8e1795-b624-4746-9390-acfc36398c2a" (UID: "7e8e1795-b624-4746-9390-acfc36398c2a"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 19:19:21.299454 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:19:21.299429    2582 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7e8e1795-b624-4746-9390-acfc36398c2a-openshift-service-ca-bundle\") on node \"ip-10-0-131-232.ec2.internal\" DevicePath \"\""
Sep 17 19:19:21.299454 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:19:21.299452    2582 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/7e8e1795-b624-4746-9390-acfc36398c2a-proxy-tls\") on node \"ip-10-0-131-232.ec2.internal\" DevicePath \"\""
Sep 17 19:19:21.414314 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:19:21.414284    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/model-chainer-raw-e3e91-747b5c94b5-dvrfx"]
Sep 17 19:19:21.418383 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:19:21.418363    2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/model-chainer-raw-e3e91-747b5c94b5-dvrfx"]
Sep 17 19:19:22.768457 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:19:22.768424    2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7e8e1795-b624-4746-9390-acfc36398c2a" path="/var/lib/kubelet/pods/7e8e1795-b624-4746-9390-acfc36398c2a/volumes"
Sep 17 19:19:23.901095 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:19:23.901052    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-55656fdf77-6k5vs" podUID="51310d70-7568-4c17-beb5-7be4bee6764b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.27:8080: connect: connection refused"
Sep 17 19:19:25.004028 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:19:25.003989    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-7eb14-predictor-7dbd4779b4-8l7vj" podUID="8b82721b-ff56-417c-8171-517d0a81ea8b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.29:8080: connect: connection refused"
Sep 17 19:19:33.901003 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:19:33.900964    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-55656fdf77-6k5vs" podUID="51310d70-7568-4c17-beb5-7be4bee6764b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.27:8080: connect: connection refused"
Sep 17 19:19:35.004482 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:19:35.004444    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-7eb14-predictor-7dbd4779b4-8l7vj" podUID="8b82721b-ff56-417c-8171-517d0a81ea8b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.29:8080: connect: connection refused"
Sep 17 19:19:43.901847 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:19:43.901778    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-55656fdf77-6k5vs" podUID="51310d70-7568-4c17-beb5-7be4bee6764b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.27:8080: connect: connection refused"
Sep 17 19:19:45.004030 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:19:45.003983    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-7eb14-predictor-7dbd4779b4-8l7vj" podUID="8b82721b-ff56-417c-8171-517d0a81ea8b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.29:8080: connect: connection refused"
Sep 17 19:19:53.902295 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:19:53.902215    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-55656fdf77-6k5vs"
Sep 17 19:19:55.004224 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:19:55.004186    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-7eb14-predictor-7dbd4779b4-8l7vj" podUID="8b82721b-ff56-417c-8171-517d0a81ea8b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.29:8080: connect: connection refused"
Sep 17 19:20:05.003904 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:20:05.003862    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-7eb14-predictor-7dbd4779b4-8l7vj" podUID="8b82721b-ff56-417c-8171-517d0a81ea8b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.29:8080: connect: connection refused"
Sep 17 19:20:10.221082 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:20:10.221046    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-stop-predictor-55656fdf77-6k5vs"]
Sep 17 19:20:10.221500 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:20:10.221343    2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-55656fdf77-6k5vs" podUID="51310d70-7568-4c17-beb5-7be4bee6764b" containerName="kserve-container" containerID="cri-o://470a775e39db790bfb1f4c96041392d36a7b882040c93d19352a4c5a8c0f70fa" gracePeriod=30
Sep 17 19:20:10.221500 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:20:10.221392    2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-55656fdf77-6k5vs" podUID="51310d70-7568-4c17-beb5-7be4bee6764b" containerName="kube-rbac-proxy" containerID="cri-o://96b017d9b9f61b6aa417572d4bf732cfc282a55631bf0016676bc7c40b6f68c7" gracePeriod=30
Sep 17 19:20:11.314586 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:20:11.314552    2582 generic.go:356] "Generic (PLEG): container finished" podID="51310d70-7568-4c17-beb5-7be4bee6764b" containerID="96b017d9b9f61b6aa417572d4bf732cfc282a55631bf0016676bc7c40b6f68c7" exitCode=2
Sep 17 19:20:11.314586 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:20:11.314589    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-55656fdf77-6k5vs" event={"ID":"51310d70-7568-4c17-beb5-7be4bee6764b","Type":"ContainerDied","Data":"96b017d9b9f61b6aa417572d4bf732cfc282a55631bf0016676bc7c40b6f68c7"}
Sep 17 19:20:13.895560 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:20:13.895511    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-55656fdf77-6k5vs" podUID="51310d70-7568-4c17-beb5-7be4bee6764b" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.27:8643/healthz\": dial tcp 10.134.0.27:8643: connect: connection refused"
Sep 17 19:20:13.900912 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:20:13.900877    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-55656fdf77-6k5vs" podUID="51310d70-7568-4c17-beb5-7be4bee6764b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.27:8080: connect: connection refused"
Sep 17 19:20:15.005116 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:20:15.005088    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-7eb14-predictor-7dbd4779b4-8l7vj"
Sep 17 19:20:15.463698 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:20:15.463672    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-55656fdf77-6k5vs"
Sep 17 19:20:15.528372 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:20:15.528341    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-w5hll\" (UniqueName: \"kubernetes.io/projected/51310d70-7568-4c17-beb5-7be4bee6764b-kube-api-access-w5hll\") pod \"51310d70-7568-4c17-beb5-7be4bee6764b\" (UID: \"51310d70-7568-4c17-beb5-7be4bee6764b\") "
Sep 17 19:20:15.528372 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:20:15.528375    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/51310d70-7568-4c17-beb5-7be4bee6764b-kube-rbac-proxy-sar-config\") pod \"51310d70-7568-4c17-beb5-7be4bee6764b\" (UID: \"51310d70-7568-4c17-beb5-7be4bee6764b\") "
Sep 17 19:20:15.528581 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:20:15.528429    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/51310d70-7568-4c17-beb5-7be4bee6764b-cabundle-cert\") pod \"51310d70-7568-4c17-beb5-7be4bee6764b\" (UID: \"51310d70-7568-4c17-beb5-7be4bee6764b\") "
Sep 17 19:20:15.528581 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:20:15.528460    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/51310d70-7568-4c17-beb5-7be4bee6764b-kserve-provision-location\") pod \"51310d70-7568-4c17-beb5-7be4bee6764b\" (UID: \"51310d70-7568-4c17-beb5-7be4bee6764b\") "
Sep 17 19:20:15.528581 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:20:15.528512    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/51310d70-7568-4c17-beb5-7be4bee6764b-proxy-tls\") pod \"51310d70-7568-4c17-beb5-7be4bee6764b\" (UID: \"51310d70-7568-4c17-beb5-7be4bee6764b\") "
Sep 17 19:20:15.528882 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:20:15.528854    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/51310d70-7568-4c17-beb5-7be4bee6764b-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "51310d70-7568-4c17-beb5-7be4bee6764b" (UID: "51310d70-7568-4c17-beb5-7be4bee6764b"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 19:20:15.528950 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:20:15.528867    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/51310d70-7568-4c17-beb5-7be4bee6764b-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "51310d70-7568-4c17-beb5-7be4bee6764b" (UID: "51310d70-7568-4c17-beb5-7be4bee6764b"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 19:20:15.528950 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:20:15.528858    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/51310d70-7568-4c17-beb5-7be4bee6764b-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "51310d70-7568-4c17-beb5-7be4bee6764b" (UID: "51310d70-7568-4c17-beb5-7be4bee6764b"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 19:20:15.530966 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:20:15.530930    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/51310d70-7568-4c17-beb5-7be4bee6764b-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "51310d70-7568-4c17-beb5-7be4bee6764b" (UID: "51310d70-7568-4c17-beb5-7be4bee6764b"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 19:20:15.531126 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:20:15.531106    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/51310d70-7568-4c17-beb5-7be4bee6764b-kube-api-access-w5hll" (OuterVolumeSpecName: "kube-api-access-w5hll") pod "51310d70-7568-4c17-beb5-7be4bee6764b" (UID: "51310d70-7568-4c17-beb5-7be4bee6764b"). InnerVolumeSpecName "kube-api-access-w5hll". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 19:20:15.630125 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:20:15.630034    2582 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/51310d70-7568-4c17-beb5-7be4bee6764b-cabundle-cert\") on node \"ip-10-0-131-232.ec2.internal\" DevicePath \"\""
Sep 17 19:20:15.630125 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:20:15.630070    2582 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/51310d70-7568-4c17-beb5-7be4bee6764b-kserve-provision-location\") on node \"ip-10-0-131-232.ec2.internal\" DevicePath \"\""
Sep 17 19:20:15.630125 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:20:15.630080    2582 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/51310d70-7568-4c17-beb5-7be4bee6764b-proxy-tls\") on node \"ip-10-0-131-232.ec2.internal\" DevicePath \"\""
Sep 17 19:20:15.630125 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:20:15.630087    2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w5hll\" (UniqueName: \"kubernetes.io/projected/51310d70-7568-4c17-beb5-7be4bee6764b-kube-api-access-w5hll\") on node \"ip-10-0-131-232.ec2.internal\" DevicePath \"\""
Sep 17 19:20:15.630125 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:20:15.630096    2582 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/51310d70-7568-4c17-beb5-7be4bee6764b-kube-rbac-proxy-sar-config\") on node \"ip-10-0-131-232.ec2.internal\" DevicePath \"\""
Sep 17 19:20:16.338109 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:20:16.338074    2582 generic.go:356] "Generic (PLEG): container finished" podID="51310d70-7568-4c17-beb5-7be4bee6764b" containerID="470a775e39db790bfb1f4c96041392d36a7b882040c93d19352a4c5a8c0f70fa" exitCode=0
Sep 17 19:20:16.338607 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:20:16.338132    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-55656fdf77-6k5vs" event={"ID":"51310d70-7568-4c17-beb5-7be4bee6764b","Type":"ContainerDied","Data":"470a775e39db790bfb1f4c96041392d36a7b882040c93d19352a4c5a8c0f70fa"}
Sep 17 19:20:16.338607 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:20:16.338146    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-55656fdf77-6k5vs"
Sep 17 19:20:16.338607 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:20:16.338187    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-55656fdf77-6k5vs" event={"ID":"51310d70-7568-4c17-beb5-7be4bee6764b","Type":"ContainerDied","Data":"1abf83cd7f224117270d4c7a0b398d076b68122308a5b5b216432f311a63d415"}
Sep 17 19:20:16.338607 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:20:16.338216    2582 scope.go:117] "RemoveContainer" containerID="96b017d9b9f61b6aa417572d4bf732cfc282a55631bf0016676bc7c40b6f68c7"
Sep 17 19:20:16.347096 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:20:16.347072    2582 scope.go:117] "RemoveContainer" containerID="470a775e39db790bfb1f4c96041392d36a7b882040c93d19352a4c5a8c0f70fa"
Sep 17 19:20:16.354670 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:20:16.354653    2582 scope.go:117] "RemoveContainer" containerID="7de04522aa61ed4d75b29eabc9ff17da519306deca9a33931c7e34e2100d184e"
Sep 17 19:20:16.360378 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:20:16.360349    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-stop-predictor-55656fdf77-6k5vs"]
Sep 17 19:20:16.363766 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:20:16.363744    2582 scope.go:117] "RemoveContainer" containerID="96b017d9b9f61b6aa417572d4bf732cfc282a55631bf0016676bc7c40b6f68c7"
Sep 17 19:20:16.364161 ip-10-0-131-232 kubenswrapper[2582]: E0917 19:20:16.364143    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"96b017d9b9f61b6aa417572d4bf732cfc282a55631bf0016676bc7c40b6f68c7\": container with ID starting with 96b017d9b9f61b6aa417572d4bf732cfc282a55631bf0016676bc7c40b6f68c7 not found: ID does not exist" containerID="96b017d9b9f61b6aa417572d4bf732cfc282a55631bf0016676bc7c40b6f68c7"
Sep 17 19:20:16.364209 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:20:16.364172    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"96b017d9b9f61b6aa417572d4bf732cfc282a55631bf0016676bc7c40b6f68c7"} err="failed to get container status \"96b017d9b9f61b6aa417572d4bf732cfc282a55631bf0016676bc7c40b6f68c7\": rpc error: code = NotFound desc = could not find container \"96b017d9b9f61b6aa417572d4bf732cfc282a55631bf0016676bc7c40b6f68c7\": container with ID starting with 96b017d9b9f61b6aa417572d4bf732cfc282a55631bf0016676bc7c40b6f68c7 not found: ID does not exist"
Sep 17 19:20:16.364209 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:20:16.364190    2582 scope.go:117] "RemoveContainer" containerID="470a775e39db790bfb1f4c96041392d36a7b882040c93d19352a4c5a8c0f70fa"
Sep 17 19:20:16.364477 ip-10-0-131-232 kubenswrapper[2582]: E0917 19:20:16.364462    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"470a775e39db790bfb1f4c96041392d36a7b882040c93d19352a4c5a8c0f70fa\": container with ID starting with 470a775e39db790bfb1f4c96041392d36a7b882040c93d19352a4c5a8c0f70fa not found: ID does not exist" containerID="470a775e39db790bfb1f4c96041392d36a7b882040c93d19352a4c5a8c0f70fa"
Sep 17 19:20:16.364528 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:20:16.364480    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"470a775e39db790bfb1f4c96041392d36a7b882040c93d19352a4c5a8c0f70fa"} err="failed to get container status \"470a775e39db790bfb1f4c96041392d36a7b882040c93d19352a4c5a8c0f70fa\": rpc error: code = NotFound desc = could not find container \"470a775e39db790bfb1f4c96041392d36a7b882040c93d19352a4c5a8c0f70fa\": container with ID starting with 470a775e39db790bfb1f4c96041392d36a7b882040c93d19352a4c5a8c0f70fa not found: ID does not exist"
Sep 17 19:20:16.364528 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:20:16.364493    2582 scope.go:117] "RemoveContainer" containerID="7de04522aa61ed4d75b29eabc9ff17da519306deca9a33931c7e34e2100d184e"
Sep 17 19:20:16.364602 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:20:16.364525    2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-stop-predictor-55656fdf77-6k5vs"]
Sep 17 19:20:16.364747 ip-10-0-131-232 kubenswrapper[2582]: E0917 19:20:16.364731    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7de04522aa61ed4d75b29eabc9ff17da519306deca9a33931c7e34e2100d184e\": container with ID starting with 7de04522aa61ed4d75b29eabc9ff17da519306deca9a33931c7e34e2100d184e not found: ID does not exist" containerID="7de04522aa61ed4d75b29eabc9ff17da519306deca9a33931c7e34e2100d184e"
Sep 17 19:20:16.364786 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:20:16.364754    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7de04522aa61ed4d75b29eabc9ff17da519306deca9a33931c7e34e2100d184e"} err="failed to get container status \"7de04522aa61ed4d75b29eabc9ff17da519306deca9a33931c7e34e2100d184e\": rpc error: code = NotFound desc = could not find container \"7de04522aa61ed4d75b29eabc9ff17da519306deca9a33931c7e34e2100d184e\": container with ID starting with 7de04522aa61ed4d75b29eabc9ff17da519306deca9a33931c7e34e2100d184e not found: ID does not exist"
Sep 17 19:20:16.768369 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:20:16.768335    2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="51310d70-7568-4c17-beb5-7be4bee6764b" path="/var/lib/kubelet/pods/51310d70-7568-4c17-beb5-7be4bee6764b/volumes"
Sep 17 19:20:20.838185 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:20:20.838143    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-primary-5ad3e2-predictor-546d64f79c-9p2qf"]
Sep 17 19:20:20.838559 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:20:20.838435    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="51310d70-7568-4c17-beb5-7be4bee6764b" containerName="kube-rbac-proxy"
Sep 17 19:20:20.838559 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:20:20.838446    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="51310d70-7568-4c17-beb5-7be4bee6764b" containerName="kube-rbac-proxy"
Sep 17 19:20:20.838559 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:20:20.838462    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="51310d70-7568-4c17-beb5-7be4bee6764b" containerName="kserve-container"
Sep 17 19:20:20.838559 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:20:20.838467    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="51310d70-7568-4c17-beb5-7be4bee6764b" containerName="kserve-container"
Sep 17 19:20:20.838559 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:20:20.838475    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7e8e1795-b624-4746-9390-acfc36398c2a" containerName="model-chainer-raw-e3e91"
Sep 17 19:20:20.838559 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:20:20.838481    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="7e8e1795-b624-4746-9390-acfc36398c2a" containerName="model-chainer-raw-e3e91"
Sep 17 19:20:20.838559 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:20:20.838493    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="51310d70-7568-4c17-beb5-7be4bee6764b" containerName="storage-initializer"
Sep 17 19:20:20.838559 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:20:20.838498    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="51310d70-7568-4c17-beb5-7be4bee6764b" containerName="storage-initializer"
Sep 17 19:20:20.838559 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:20:20.838542    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="51310d70-7568-4c17-beb5-7be4bee6764b" containerName="kube-rbac-proxy"
Sep 17 19:20:20.838559 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:20:20.838549    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="7e8e1795-b624-4746-9390-acfc36398c2a" containerName="model-chainer-raw-e3e91"
Sep 17 19:20:20.838559 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:20:20.838555    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="51310d70-7568-4c17-beb5-7be4bee6764b" containerName="kserve-container"
Sep 17 19:20:20.841606 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:20:20.841583    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-primary-5ad3e2-predictor-546d64f79c-9p2qf"
Sep 17 19:20:20.844103 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:20:20.844076    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-primary-5ad3e2-kube-rbac-proxy-sar-config\""
Sep 17 19:20:20.844103 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:20:20.844094    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-primary-5ad3e2-predictor-serving-cert\""
Sep 17 19:20:20.853884 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:20:20.853859    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-primary-5ad3e2-predictor-546d64f79c-9p2qf"]
Sep 17 19:20:20.973113 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:20:20.973074    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/66ebb0a4-ce00-4698-b750-7272f5812bf5-cabundle-cert\") pod \"isvc-primary-5ad3e2-predictor-546d64f79c-9p2qf\" (UID: \"66ebb0a4-ce00-4698-b750-7272f5812bf5\") " pod="kserve-ci-e2e-test/isvc-primary-5ad3e2-predictor-546d64f79c-9p2qf"
Sep 17 19:20:20.973318 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:20:20.973123    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ltwcp\" (UniqueName: \"kubernetes.io/projected/66ebb0a4-ce00-4698-b750-7272f5812bf5-kube-api-access-ltwcp\") pod \"isvc-primary-5ad3e2-predictor-546d64f79c-9p2qf\" (UID: \"66ebb0a4-ce00-4698-b750-7272f5812bf5\") " pod="kserve-ci-e2e-test/isvc-primary-5ad3e2-predictor-546d64f79c-9p2qf"
Sep 17 19:20:20.973318 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:20:20.973213    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/66ebb0a4-ce00-4698-b750-7272f5812bf5-kube-rbac-proxy-sar-config\") pod \"isvc-primary-5ad3e2-predictor-546d64f79c-9p2qf\" (UID: \"66ebb0a4-ce00-4698-b750-7272f5812bf5\") " pod="kserve-ci-e2e-test/isvc-primary-5ad3e2-predictor-546d64f79c-9p2qf"
Sep 17 19:20:20.973318 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:20:20.973271    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/66ebb0a4-ce00-4698-b750-7272f5812bf5-proxy-tls\") pod \"isvc-primary-5ad3e2-predictor-546d64f79c-9p2qf\" (UID: \"66ebb0a4-ce00-4698-b750-7272f5812bf5\") " pod="kserve-ci-e2e-test/isvc-primary-5ad3e2-predictor-546d64f79c-9p2qf"
Sep 17 19:20:20.973464 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:20:20.973321    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/66ebb0a4-ce00-4698-b750-7272f5812bf5-kserve-provision-location\") pod \"isvc-primary-5ad3e2-predictor-546d64f79c-9p2qf\" (UID: \"66ebb0a4-ce00-4698-b750-7272f5812bf5\") " pod="kserve-ci-e2e-test/isvc-primary-5ad3e2-predictor-546d64f79c-9p2qf"
Sep 17 19:20:21.074707 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:20:21.074674    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/66ebb0a4-ce00-4698-b750-7272f5812bf5-proxy-tls\") pod \"isvc-primary-5ad3e2-predictor-546d64f79c-9p2qf\" (UID: \"66ebb0a4-ce00-4698-b750-7272f5812bf5\") " pod="kserve-ci-e2e-test/isvc-primary-5ad3e2-predictor-546d64f79c-9p2qf"
Sep 17 19:20:21.074707 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:20:21.074716    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/66ebb0a4-ce00-4698-b750-7272f5812bf5-kserve-provision-location\") pod \"isvc-primary-5ad3e2-predictor-546d64f79c-9p2qf\" (UID: \"66ebb0a4-ce00-4698-b750-7272f5812bf5\") " pod="kserve-ci-e2e-test/isvc-primary-5ad3e2-predictor-546d64f79c-9p2qf"
Sep 17 19:20:21.074975 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:20:21.074773    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/66ebb0a4-ce00-4698-b750-7272f5812bf5-cabundle-cert\") pod \"isvc-primary-5ad3e2-predictor-546d64f79c-9p2qf\" (UID: \"66ebb0a4-ce00-4698-b750-7272f5812bf5\") " pod="kserve-ci-e2e-test/isvc-primary-5ad3e2-predictor-546d64f79c-9p2qf"
Sep 17 19:20:21.074975 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:20:21.074806    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ltwcp\" (UniqueName: \"kubernetes.io/projected/66ebb0a4-ce00-4698-b750-7272f5812bf5-kube-api-access-ltwcp\") pod \"isvc-primary-5ad3e2-predictor-546d64f79c-9p2qf\" (UID: \"66ebb0a4-ce00-4698-b750-7272f5812bf5\") " pod="kserve-ci-e2e-test/isvc-primary-5ad3e2-predictor-546d64f79c-9p2qf"
Sep 17 19:20:21.074975 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:20:21.074865    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/66ebb0a4-ce00-4698-b750-7272f5812bf5-kube-rbac-proxy-sar-config\") pod \"isvc-primary-5ad3e2-predictor-546d64f79c-9p2qf\" (UID: \"66ebb0a4-ce00-4698-b750-7272f5812bf5\") " pod="kserve-ci-e2e-test/isvc-primary-5ad3e2-predictor-546d64f79c-9p2qf"
Sep 17 19:20:21.074975 ip-10-0-131-232 kubenswrapper[2582]: E0917 19:20:21.074873    2582 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-primary-5ad3e2-predictor-serving-cert: secret "isvc-primary-5ad3e2-predictor-serving-cert" not found
Sep 17 19:20:21.074975 ip-10-0-131-232 kubenswrapper[2582]: E0917 19:20:21.074940    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/66ebb0a4-ce00-4698-b750-7272f5812bf5-proxy-tls podName:66ebb0a4-ce00-4698-b750-7272f5812bf5 nodeName:}" failed. No retries permitted until 2026-09-17 19:20:21.574920749 +0000 UTC m=+837.346591802 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/66ebb0a4-ce00-4698-b750-7272f5812bf5-proxy-tls") pod "isvc-primary-5ad3e2-predictor-546d64f79c-9p2qf" (UID: "66ebb0a4-ce00-4698-b750-7272f5812bf5") : secret "isvc-primary-5ad3e2-predictor-serving-cert" not found
Sep 17 19:20:21.075235 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:20:21.075150    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/66ebb0a4-ce00-4698-b750-7272f5812bf5-kserve-provision-location\") pod \"isvc-primary-5ad3e2-predictor-546d64f79c-9p2qf\" (UID: \"66ebb0a4-ce00-4698-b750-7272f5812bf5\") " pod="kserve-ci-e2e-test/isvc-primary-5ad3e2-predictor-546d64f79c-9p2qf"
Sep 17 19:20:21.075498 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:20:21.075477    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/66ebb0a4-ce00-4698-b750-7272f5812bf5-kube-rbac-proxy-sar-config\") pod \"isvc-primary-5ad3e2-predictor-546d64f79c-9p2qf\" (UID: \"66ebb0a4-ce00-4698-b750-7272f5812bf5\") " pod="kserve-ci-e2e-test/isvc-primary-5ad3e2-predictor-546d64f79c-9p2qf"
Sep 17 19:20:21.075555 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:20:21.075480    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/66ebb0a4-ce00-4698-b750-7272f5812bf5-cabundle-cert\") pod \"isvc-primary-5ad3e2-predictor-546d64f79c-9p2qf\" (UID: \"66ebb0a4-ce00-4698-b750-7272f5812bf5\") " pod="kserve-ci-e2e-test/isvc-primary-5ad3e2-predictor-546d64f79c-9p2qf"
Sep 17 19:20:21.084219 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:20:21.084191    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ltwcp\" (UniqueName: \"kubernetes.io/projected/66ebb0a4-ce00-4698-b750-7272f5812bf5-kube-api-access-ltwcp\") pod \"isvc-primary-5ad3e2-predictor-546d64f79c-9p2qf\" (UID: \"66ebb0a4-ce00-4698-b750-7272f5812bf5\") " pod="kserve-ci-e2e-test/isvc-primary-5ad3e2-predictor-546d64f79c-9p2qf"
Sep 17 19:20:21.579652 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:20:21.579615    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/66ebb0a4-ce00-4698-b750-7272f5812bf5-proxy-tls\") pod \"isvc-primary-5ad3e2-predictor-546d64f79c-9p2qf\" (UID: \"66ebb0a4-ce00-4698-b750-7272f5812bf5\") " pod="kserve-ci-e2e-test/isvc-primary-5ad3e2-predictor-546d64f79c-9p2qf"
Sep 17 19:20:21.582232 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:20:21.582200    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/66ebb0a4-ce00-4698-b750-7272f5812bf5-proxy-tls\") pod \"isvc-primary-5ad3e2-predictor-546d64f79c-9p2qf\" (UID: \"66ebb0a4-ce00-4698-b750-7272f5812bf5\") " pod="kserve-ci-e2e-test/isvc-primary-5ad3e2-predictor-546d64f79c-9p2qf"
Sep 17 19:20:21.751764 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:20:21.751729    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-primary-5ad3e2-predictor-546d64f79c-9p2qf"
Sep 17 19:20:22.025651 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:20:22.025621    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-primary-5ad3e2-predictor-546d64f79c-9p2qf"]
Sep 17 19:20:22.028207 ip-10-0-131-232 kubenswrapper[2582]: W0917 19:20:22.028182    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod66ebb0a4_ce00_4698_b750_7272f5812bf5.slice/crio-864bb094fd732b8ea43d685382162bf77dc97444dcad5faf97d7089cf1f763ff WatchSource:0}: Error finding container 864bb094fd732b8ea43d685382162bf77dc97444dcad5faf97d7089cf1f763ff: Status 404 returned error can't find the container with id 864bb094fd732b8ea43d685382162bf77dc97444dcad5faf97d7089cf1f763ff
Sep 17 19:20:22.365127 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:20:22.365038    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-primary-5ad3e2-predictor-546d64f79c-9p2qf" event={"ID":"66ebb0a4-ce00-4698-b750-7272f5812bf5","Type":"ContainerStarted","Data":"efd0235f10f38dc8682514048bb7bd9ec01e029d4addd4b0b4303b5249baeea8"}
Sep 17 19:20:22.365127 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:20:22.365093    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-primary-5ad3e2-predictor-546d64f79c-9p2qf" event={"ID":"66ebb0a4-ce00-4698-b750-7272f5812bf5","Type":"ContainerStarted","Data":"864bb094fd732b8ea43d685382162bf77dc97444dcad5faf97d7089cf1f763ff"}
Sep 17 19:20:26.384665 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:20:26.384633    2582 generic.go:356] "Generic (PLEG): container finished" podID="66ebb0a4-ce00-4698-b750-7272f5812bf5" containerID="efd0235f10f38dc8682514048bb7bd9ec01e029d4addd4b0b4303b5249baeea8" exitCode=0
Sep 17 19:20:26.385013 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:20:26.384707    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-primary-5ad3e2-predictor-546d64f79c-9p2qf" event={"ID":"66ebb0a4-ce00-4698-b750-7272f5812bf5","Type":"ContainerDied","Data":"efd0235f10f38dc8682514048bb7bd9ec01e029d4addd4b0b4303b5249baeea8"}
Sep 17 19:20:27.390909 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:20:27.390875    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-primary-5ad3e2-predictor-546d64f79c-9p2qf" event={"ID":"66ebb0a4-ce00-4698-b750-7272f5812bf5","Type":"ContainerStarted","Data":"d53c0f469969227ea32b6892917d632d2597d6bb7a326b418d596eda3a38d403"}
Sep 17 19:20:27.391246 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:20:27.390938    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-primary-5ad3e2-predictor-546d64f79c-9p2qf" event={"ID":"66ebb0a4-ce00-4698-b750-7272f5812bf5","Type":"ContainerStarted","Data":"075f52c965827d655948290468e4b2bb8351ddfa2d65ffd7d16b51bd3fa21c5e"}
Sep 17 19:20:27.391246 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:20:27.391124    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-primary-5ad3e2-predictor-546d64f79c-9p2qf"
Sep 17 19:20:27.410702 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:20:27.410661    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-primary-5ad3e2-predictor-546d64f79c-9p2qf" podStartSLOduration=3.34533775 podStartE2EDuration="7.410647072s" totalImagesPullingTime="0s" totalInitContainerRuntime="4.065309322s" isStatefulPod=false podCreationTimestamp="2026-09-17 19:20:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 19:20:27.408738988 +0000 UTC m=+843.180410047" watchObservedRunningTime="2026-09-17 19:20:27.410647072 +0000 UTC m=+843.182318132"
Sep 17 19:20:28.395174 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:20:28.395143    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-primary-5ad3e2-predictor-546d64f79c-9p2qf"
Sep 17 19:20:28.396261 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:20:28.396238    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-5ad3e2-predictor-546d64f79c-9p2qf" podUID="66ebb0a4-ce00-4698-b750-7272f5812bf5" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.30:8080: connect: connection refused"
Sep 17 19:20:29.400011 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:20:29.399970    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-5ad3e2-predictor-546d64f79c-9p2qf" podUID="66ebb0a4-ce00-4698-b750-7272f5812bf5" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.30:8080: connect: connection refused"
Sep 17 19:20:31.188463 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:20:31.188428    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/model-chainer-raw-hpa-7eb14-7c964d587c-skw74"]
Sep 17 19:20:31.191423 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:20:31.191399    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/model-chainer-raw-hpa-7eb14-7c964d587c-skw74"
Sep 17 19:20:31.193726 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:20:31.193704    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"model-chainer-raw-hpa-7eb14-serving-cert\""
Sep 17 19:20:31.194004 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:20:31.193985    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"model-chainer-raw-hpa-7eb14-kube-rbac-proxy-sar-config\""
Sep 17 19:20:31.204695 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:20:31.204663    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/model-chainer-raw-hpa-7eb14-7c964d587c-skw74"]
Sep 17 19:20:31.359659 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:20:31.359617    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/cee3f39c-6bb7-43df-9d53-85d1298a14ab-openshift-service-ca-bundle\") pod \"model-chainer-raw-hpa-7eb14-7c964d587c-skw74\" (UID: \"cee3f39c-6bb7-43df-9d53-85d1298a14ab\") " pod="kserve-ci-e2e-test/model-chainer-raw-hpa-7eb14-7c964d587c-skw74"
Sep 17 19:20:31.359807 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:20:31.359668    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/cee3f39c-6bb7-43df-9d53-85d1298a14ab-proxy-tls\") pod \"model-chainer-raw-hpa-7eb14-7c964d587c-skw74\" (UID: \"cee3f39c-6bb7-43df-9d53-85d1298a14ab\") " pod="kserve-ci-e2e-test/model-chainer-raw-hpa-7eb14-7c964d587c-skw74"
Sep 17 19:20:31.460987 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:20:31.460893    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/cee3f39c-6bb7-43df-9d53-85d1298a14ab-openshift-service-ca-bundle\") pod \"model-chainer-raw-hpa-7eb14-7c964d587c-skw74\" (UID: \"cee3f39c-6bb7-43df-9d53-85d1298a14ab\") " pod="kserve-ci-e2e-test/model-chainer-raw-hpa-7eb14-7c964d587c-skw74"
Sep 17 19:20:31.460987 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:20:31.460943    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/cee3f39c-6bb7-43df-9d53-85d1298a14ab-proxy-tls\") pod \"model-chainer-raw-hpa-7eb14-7c964d587c-skw74\" (UID: \"cee3f39c-6bb7-43df-9d53-85d1298a14ab\") " pod="kserve-ci-e2e-test/model-chainer-raw-hpa-7eb14-7c964d587c-skw74"
Sep 17 19:20:31.461520 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:20:31.461500    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/cee3f39c-6bb7-43df-9d53-85d1298a14ab-openshift-service-ca-bundle\") pod \"model-chainer-raw-hpa-7eb14-7c964d587c-skw74\" (UID: \"cee3f39c-6bb7-43df-9d53-85d1298a14ab\") " pod="kserve-ci-e2e-test/model-chainer-raw-hpa-7eb14-7c964d587c-skw74"
Sep 17 19:20:31.463414 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:20:31.463397    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/cee3f39c-6bb7-43df-9d53-85d1298a14ab-proxy-tls\") pod \"model-chainer-raw-hpa-7eb14-7c964d587c-skw74\" (UID: \"cee3f39c-6bb7-43df-9d53-85d1298a14ab\") " pod="kserve-ci-e2e-test/model-chainer-raw-hpa-7eb14-7c964d587c-skw74"
Sep 17 19:20:31.500945 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:20:31.500908    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/model-chainer-raw-hpa-7eb14-7c964d587c-skw74"
Sep 17 19:20:31.776240 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:20:31.776160    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/model-chainer-raw-hpa-7eb14-7c964d587c-skw74"]
Sep 17 19:20:31.779286 ip-10-0-131-232 kubenswrapper[2582]: W0917 19:20:31.779260    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcee3f39c_6bb7_43df_9d53_85d1298a14ab.slice/crio-bb75abfdb7b5bce42c4d93768e156489406e47956ad31072286aad0b35c171ab WatchSource:0}: Error finding container bb75abfdb7b5bce42c4d93768e156489406e47956ad31072286aad0b35c171ab: Status 404 returned error can't find the container with id bb75abfdb7b5bce42c4d93768e156489406e47956ad31072286aad0b35c171ab
Sep 17 19:20:32.414464 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:20:32.414427    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/model-chainer-raw-hpa-7eb14-7c964d587c-skw74" event={"ID":"cee3f39c-6bb7-43df-9d53-85d1298a14ab","Type":"ContainerStarted","Data":"0956fa3be64762ea0145ba0fbef7a1fea11c1fd694f750f7ea0428ed3d1669bb"}
Sep 17 19:20:32.414868 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:20:32.414485    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/model-chainer-raw-hpa-7eb14-7c964d587c-skw74" event={"ID":"cee3f39c-6bb7-43df-9d53-85d1298a14ab","Type":"ContainerStarted","Data":"bb75abfdb7b5bce42c4d93768e156489406e47956ad31072286aad0b35c171ab"}
Sep 17 19:20:32.414868 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:20:32.414525    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/model-chainer-raw-hpa-7eb14-7c964d587c-skw74"
Sep 17 19:20:32.437359 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:20:32.437300    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/model-chainer-raw-hpa-7eb14-7c964d587c-skw74" podStartSLOduration=1.437283555 podStartE2EDuration="1.437283555s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 19:20:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 19:20:32.436173642 +0000 UTC m=+848.207844702" watchObservedRunningTime="2026-09-17 19:20:32.437283555 +0000 UTC m=+848.208954615"
Sep 17 19:20:34.405130 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:20:34.405102    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-primary-5ad3e2-predictor-546d64f79c-9p2qf"
Sep 17 19:20:34.405671 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:20:34.405641    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-5ad3e2-predictor-546d64f79c-9p2qf" podUID="66ebb0a4-ce00-4698-b750-7272f5812bf5" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.30:8080: connect: connection refused"
Sep 17 19:20:38.424394 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:20:38.424363    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/model-chainer-raw-hpa-7eb14-7c964d587c-skw74"
Sep 17 19:20:41.240443 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:20:41.240404    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/model-chainer-raw-hpa-7eb14-7c964d587c-skw74"]
Sep 17 19:20:41.240922 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:20:41.240856    2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/model-chainer-raw-hpa-7eb14-7c964d587c-skw74" podUID="cee3f39c-6bb7-43df-9d53-85d1298a14ab" containerName="model-chainer-raw-hpa-7eb14" containerID="cri-o://0956fa3be64762ea0145ba0fbef7a1fea11c1fd694f750f7ea0428ed3d1669bb" gracePeriod=30
Sep 17 19:20:41.347870 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:20:41.347805    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-7eb14-predictor-7dbd4779b4-8l7vj"]
Sep 17 19:20:41.348277 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:20:41.348247    2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-7eb14-predictor-7dbd4779b4-8l7vj" podUID="8b82721b-ff56-417c-8171-517d0a81ea8b" containerName="kserve-container" containerID="cri-o://7893ba91e736e6a4e08ddf0773a642286f13a9524d3e8685e1532a29e025669e" gracePeriod=30
Sep 17 19:20:41.349066 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:20:41.348799    2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-7eb14-predictor-7dbd4779b4-8l7vj" podUID="8b82721b-ff56-417c-8171-517d0a81ea8b" containerName="kube-rbac-proxy" containerID="cri-o://ef27ac62036db3653fe171d401ecdbc392750839571ec9950713dd57a82d055c" gracePeriod=30
Sep 17 19:20:42.465136 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:20:42.465101    2582 generic.go:356] "Generic (PLEG): container finished" podID="8b82721b-ff56-417c-8171-517d0a81ea8b" containerID="ef27ac62036db3653fe171d401ecdbc392750839571ec9950713dd57a82d055c" exitCode=2
Sep 17 19:20:42.465136 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:20:42.465137    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-7eb14-predictor-7dbd4779b4-8l7vj" event={"ID":"8b82721b-ff56-417c-8171-517d0a81ea8b","Type":"ContainerDied","Data":"ef27ac62036db3653fe171d401ecdbc392750839571ec9950713dd57a82d055c"}
Sep 17 19:20:43.423479 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:20:43.423444    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/model-chainer-raw-hpa-7eb14-7c964d587c-skw74" podUID="cee3f39c-6bb7-43df-9d53-85d1298a14ab" containerName="model-chainer-raw-hpa-7eb14" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 19:20:44.405878 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:20:44.405839    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-5ad3e2-predictor-546d64f79c-9p2qf" podUID="66ebb0a4-ce00-4698-b750-7272f5812bf5" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.30:8080: connect: connection refused"
Sep 17 19:20:44.999856 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:20:44.999794    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-7eb14-predictor-7dbd4779b4-8l7vj" podUID="8b82721b-ff56-417c-8171-517d0a81ea8b" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.29:8643/healthz\": dial tcp 10.134.0.29:8643: connect: connection refused"
Sep 17 19:20:45.004081 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:20:45.004051    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-7eb14-predictor-7dbd4779b4-8l7vj" podUID="8b82721b-ff56-417c-8171-517d0a81ea8b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.29:8080: connect: connection refused"
Sep 17 19:20:45.786419 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:20:45.786399    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-7eb14-predictor-7dbd4779b4-8l7vj"
Sep 17 19:20:45.877888 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:20:45.877859    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5jd2p\" (UniqueName: \"kubernetes.io/projected/8b82721b-ff56-417c-8171-517d0a81ea8b-kube-api-access-5jd2p\") pod \"8b82721b-ff56-417c-8171-517d0a81ea8b\" (UID: \"8b82721b-ff56-417c-8171-517d0a81ea8b\") "
Sep 17 19:20:45.878038 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:20:45.877906    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/8b82721b-ff56-417c-8171-517d0a81ea8b-proxy-tls\") pod \"8b82721b-ff56-417c-8171-517d0a81ea8b\" (UID: \"8b82721b-ff56-417c-8171-517d0a81ea8b\") "
Sep 17 19:20:45.878038 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:20:45.877932    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/8b82721b-ff56-417c-8171-517d0a81ea8b-kube-rbac-proxy-sar-config\") pod \"8b82721b-ff56-417c-8171-517d0a81ea8b\" (UID: \"8b82721b-ff56-417c-8171-517d0a81ea8b\") "
Sep 17 19:20:45.878038 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:20:45.877972    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/8b82721b-ff56-417c-8171-517d0a81ea8b-cabundle-cert\") pod \"8b82721b-ff56-417c-8171-517d0a81ea8b\" (UID: \"8b82721b-ff56-417c-8171-517d0a81ea8b\") "
Sep 17 19:20:45.878038 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:20:45.877993    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8b82721b-ff56-417c-8171-517d0a81ea8b-kserve-provision-location\") pod \"8b82721b-ff56-417c-8171-517d0a81ea8b\" (UID: \"8b82721b-ff56-417c-8171-517d0a81ea8b\") "
Sep 17 19:20:45.878336 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:20:45.878304    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8b82721b-ff56-417c-8171-517d0a81ea8b-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "8b82721b-ff56-417c-8171-517d0a81ea8b" (UID: "8b82721b-ff56-417c-8171-517d0a81ea8b"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 19:20:45.878336 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:20:45.878324    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8b82721b-ff56-417c-8171-517d0a81ea8b-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "8b82721b-ff56-417c-8171-517d0a81ea8b" (UID: "8b82721b-ff56-417c-8171-517d0a81ea8b"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 19:20:45.878466 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:20:45.878344    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8b82721b-ff56-417c-8171-517d0a81ea8b-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "8b82721b-ff56-417c-8171-517d0a81ea8b" (UID: "8b82721b-ff56-417c-8171-517d0a81ea8b"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 19:20:45.880127 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:20:45.880104    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8b82721b-ff56-417c-8171-517d0a81ea8b-kube-api-access-5jd2p" (OuterVolumeSpecName: "kube-api-access-5jd2p") pod "8b82721b-ff56-417c-8171-517d0a81ea8b" (UID: "8b82721b-ff56-417c-8171-517d0a81ea8b"). InnerVolumeSpecName "kube-api-access-5jd2p". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 19:20:45.880221 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:20:45.880107    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8b82721b-ff56-417c-8171-517d0a81ea8b-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "8b82721b-ff56-417c-8171-517d0a81ea8b" (UID: "8b82721b-ff56-417c-8171-517d0a81ea8b"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 19:20:45.978596 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:20:45.978575    2582 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/8b82721b-ff56-417c-8171-517d0a81ea8b-kube-rbac-proxy-sar-config\") on node \"ip-10-0-131-232.ec2.internal\" DevicePath \"\""
Sep 17 19:20:45.978596 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:20:45.978595    2582 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/8b82721b-ff56-417c-8171-517d0a81ea8b-cabundle-cert\") on node \"ip-10-0-131-232.ec2.internal\" DevicePath \"\""
Sep 17 19:20:45.978716 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:20:45.978605    2582 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8b82721b-ff56-417c-8171-517d0a81ea8b-kserve-provision-location\") on node \"ip-10-0-131-232.ec2.internal\" DevicePath \"\""
Sep 17 19:20:45.978716 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:20:45.978614    2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5jd2p\" (UniqueName: \"kubernetes.io/projected/8b82721b-ff56-417c-8171-517d0a81ea8b-kube-api-access-5jd2p\") on node \"ip-10-0-131-232.ec2.internal\" DevicePath \"\""
Sep 17 19:20:45.978716 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:20:45.978622    2582 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/8b82721b-ff56-417c-8171-517d0a81ea8b-proxy-tls\") on node \"ip-10-0-131-232.ec2.internal\" DevicePath \"\""
Sep 17 19:20:46.483551 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:20:46.483514    2582 generic.go:356] "Generic (PLEG): container finished" podID="8b82721b-ff56-417c-8171-517d0a81ea8b" containerID="7893ba91e736e6a4e08ddf0773a642286f13a9524d3e8685e1532a29e025669e" exitCode=0
Sep 17 19:20:46.483713 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:20:46.483575    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-7eb14-predictor-7dbd4779b4-8l7vj"
Sep 17 19:20:46.483713 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:20:46.483589    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-7eb14-predictor-7dbd4779b4-8l7vj" event={"ID":"8b82721b-ff56-417c-8171-517d0a81ea8b","Type":"ContainerDied","Data":"7893ba91e736e6a4e08ddf0773a642286f13a9524d3e8685e1532a29e025669e"}
Sep 17 19:20:46.483713 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:20:46.483646    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-7eb14-predictor-7dbd4779b4-8l7vj" event={"ID":"8b82721b-ff56-417c-8171-517d0a81ea8b","Type":"ContainerDied","Data":"54f37bb80fd6082e6df8e6cd9b403839e3fcb733f25233ecd74e871e8e3ae358"}
Sep 17 19:20:46.483713 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:20:46.483673    2582 scope.go:117] "RemoveContainer" containerID="ef27ac62036db3653fe171d401ecdbc392750839571ec9950713dd57a82d055c"
Sep 17 19:20:46.498249 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:20:46.498231    2582 scope.go:117] "RemoveContainer" containerID="7893ba91e736e6a4e08ddf0773a642286f13a9524d3e8685e1532a29e025669e"
Sep 17 19:20:46.504941 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:20:46.504924    2582 scope.go:117] "RemoveContainer" containerID="b6fc6b5c4f24f2cf6c4507852573bbe837805cd21f395ec53bb5a224755c48b4"
Sep 17 19:20:46.510854 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:20:46.510816    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-7eb14-predictor-7dbd4779b4-8l7vj"]
Sep 17 19:20:46.511842 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:20:46.511813    2582 scope.go:117] "RemoveContainer" containerID="ef27ac62036db3653fe171d401ecdbc392750839571ec9950713dd57a82d055c"
Sep 17 19:20:46.512110 ip-10-0-131-232 kubenswrapper[2582]: E0917 19:20:46.512092    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ef27ac62036db3653fe171d401ecdbc392750839571ec9950713dd57a82d055c\": container with ID starting with ef27ac62036db3653fe171d401ecdbc392750839571ec9950713dd57a82d055c not found: ID does not exist" containerID="ef27ac62036db3653fe171d401ecdbc392750839571ec9950713dd57a82d055c"
Sep 17 19:20:46.512181 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:20:46.512121    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ef27ac62036db3653fe171d401ecdbc392750839571ec9950713dd57a82d055c"} err="failed to get container status \"ef27ac62036db3653fe171d401ecdbc392750839571ec9950713dd57a82d055c\": rpc error: code = NotFound desc = could not find container \"ef27ac62036db3653fe171d401ecdbc392750839571ec9950713dd57a82d055c\": container with ID starting with ef27ac62036db3653fe171d401ecdbc392750839571ec9950713dd57a82d055c not found: ID does not exist"
Sep 17 19:20:46.512181 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:20:46.512145    2582 scope.go:117] "RemoveContainer" containerID="7893ba91e736e6a4e08ddf0773a642286f13a9524d3e8685e1532a29e025669e"
Sep 17 19:20:46.512380 ip-10-0-131-232 kubenswrapper[2582]: E0917 19:20:46.512364    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7893ba91e736e6a4e08ddf0773a642286f13a9524d3e8685e1532a29e025669e\": container with ID starting with 7893ba91e736e6a4e08ddf0773a642286f13a9524d3e8685e1532a29e025669e not found: ID does not exist" containerID="7893ba91e736e6a4e08ddf0773a642286f13a9524d3e8685e1532a29e025669e"
Sep 17 19:20:46.512420 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:20:46.512387    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7893ba91e736e6a4e08ddf0773a642286f13a9524d3e8685e1532a29e025669e"} err="failed to get container status \"7893ba91e736e6a4e08ddf0773a642286f13a9524d3e8685e1532a29e025669e\": rpc error: code = NotFound desc = could not find container \"7893ba91e736e6a4e08ddf0773a642286f13a9524d3e8685e1532a29e025669e\": container with ID starting with 7893ba91e736e6a4e08ddf0773a642286f13a9524d3e8685e1532a29e025669e not found: ID does not exist"
Sep 17 19:20:46.512420 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:20:46.512402    2582 scope.go:117] "RemoveContainer" containerID="b6fc6b5c4f24f2cf6c4507852573bbe837805cd21f395ec53bb5a224755c48b4"
Sep 17 19:20:46.512667 ip-10-0-131-232 kubenswrapper[2582]: E0917 19:20:46.512651    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b6fc6b5c4f24f2cf6c4507852573bbe837805cd21f395ec53bb5a224755c48b4\": container with ID starting with b6fc6b5c4f24f2cf6c4507852573bbe837805cd21f395ec53bb5a224755c48b4 not found: ID does not exist" containerID="b6fc6b5c4f24f2cf6c4507852573bbe837805cd21f395ec53bb5a224755c48b4"
Sep 17 19:20:46.512706 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:20:46.512673    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b6fc6b5c4f24f2cf6c4507852573bbe837805cd21f395ec53bb5a224755c48b4"} err="failed to get container status \"b6fc6b5c4f24f2cf6c4507852573bbe837805cd21f395ec53bb5a224755c48b4\": rpc error: code = NotFound desc = could not find container \"b6fc6b5c4f24f2cf6c4507852573bbe837805cd21f395ec53bb5a224755c48b4\": container with ID starting with b6fc6b5c4f24f2cf6c4507852573bbe837805cd21f395ec53bb5a224755c48b4 not found: ID does not exist"
Sep 17 19:20:46.514620 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:20:46.514602    2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-7eb14-predictor-7dbd4779b4-8l7vj"]
Sep 17 19:20:46.768182 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:20:46.768104    2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8b82721b-ff56-417c-8171-517d0a81ea8b" path="/var/lib/kubelet/pods/8b82721b-ff56-417c-8171-517d0a81ea8b/volumes"
Sep 17 19:20:48.423329 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:20:48.423285    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/model-chainer-raw-hpa-7eb14-7c964d587c-skw74" podUID="cee3f39c-6bb7-43df-9d53-85d1298a14ab" containerName="model-chainer-raw-hpa-7eb14" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 19:20:51.513808 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:20:51.513774    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-logger-raw-09f87-predictor-f88dfc6f9-vzbn2"]
Sep 17 19:20:51.514278 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:20:51.514202    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8b82721b-ff56-417c-8171-517d0a81ea8b" containerName="storage-initializer"
Sep 17 19:20:51.514278 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:20:51.514221    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="8b82721b-ff56-417c-8171-517d0a81ea8b" containerName="storage-initializer"
Sep 17 19:20:51.514278 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:20:51.514250    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8b82721b-ff56-417c-8171-517d0a81ea8b" containerName="kserve-container"
Sep 17 19:20:51.514278 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:20:51.514258    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="8b82721b-ff56-417c-8171-517d0a81ea8b" containerName="kserve-container"
Sep 17 19:20:51.514278 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:20:51.514270    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8b82721b-ff56-417c-8171-517d0a81ea8b" containerName="kube-rbac-proxy"
Sep 17 19:20:51.514278 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:20:51.514278    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="8b82721b-ff56-417c-8171-517d0a81ea8b" containerName="kube-rbac-proxy"
Sep 17 19:20:51.514590 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:20:51.514350    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="8b82721b-ff56-417c-8171-517d0a81ea8b" containerName="kserve-container"
Sep 17 19:20:51.514590 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:20:51.514363    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="8b82721b-ff56-417c-8171-517d0a81ea8b" containerName="kube-rbac-proxy"
Sep 17 19:20:51.518730 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:20:51.518693    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-logger-raw-09f87-predictor-f88dfc6f9-vzbn2"
Sep 17 19:20:51.522222 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:20:51.522089    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-logger-raw-09f87-predictor-serving-cert\""
Sep 17 19:20:51.522430 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:20:51.522408    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-logger-raw-09f87-kube-rbac-proxy-sar-config\""
Sep 17 19:20:51.531206 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:20:51.531181    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-logger-raw-09f87-predictor-f88dfc6f9-vzbn2"]
Sep 17 19:20:51.620704 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:20:51.620666    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/085b71f3-788e-4449-92b8-1364b16d4479-proxy-tls\") pod \"isvc-logger-raw-09f87-predictor-f88dfc6f9-vzbn2\" (UID: \"085b71f3-788e-4449-92b8-1364b16d4479\") " pod="kserve-ci-e2e-test/isvc-logger-raw-09f87-predictor-f88dfc6f9-vzbn2"
Sep 17 19:20:51.620882 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:20:51.620750    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/085b71f3-788e-4449-92b8-1364b16d4479-kube-rbac-proxy-sar-config\") pod \"isvc-logger-raw-09f87-predictor-f88dfc6f9-vzbn2\" (UID: \"085b71f3-788e-4449-92b8-1364b16d4479\") " pod="kserve-ci-e2e-test/isvc-logger-raw-09f87-predictor-f88dfc6f9-vzbn2"
Sep 17 19:20:51.620882 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:20:51.620786    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/085b71f3-788e-4449-92b8-1364b16d4479-kserve-provision-location\") pod \"isvc-logger-raw-09f87-predictor-f88dfc6f9-vzbn2\" (UID: \"085b71f3-788e-4449-92b8-1364b16d4479\") " pod="kserve-ci-e2e-test/isvc-logger-raw-09f87-predictor-f88dfc6f9-vzbn2"
Sep 17 19:20:51.620882 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:20:51.620809    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/085b71f3-788e-4449-92b8-1364b16d4479-cabundle-cert\") pod \"isvc-logger-raw-09f87-predictor-f88dfc6f9-vzbn2\" (UID: \"085b71f3-788e-4449-92b8-1364b16d4479\") " pod="kserve-ci-e2e-test/isvc-logger-raw-09f87-predictor-f88dfc6f9-vzbn2"
Sep 17 19:20:51.621028 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:20:51.620888    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hm274\" (UniqueName: \"kubernetes.io/projected/085b71f3-788e-4449-92b8-1364b16d4479-kube-api-access-hm274\") pod \"isvc-logger-raw-09f87-predictor-f88dfc6f9-vzbn2\" (UID: \"085b71f3-788e-4449-92b8-1364b16d4479\") " pod="kserve-ci-e2e-test/isvc-logger-raw-09f87-predictor-f88dfc6f9-vzbn2"
Sep 17 19:20:51.721873 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:20:51.721806    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/085b71f3-788e-4449-92b8-1364b16d4479-kube-rbac-proxy-sar-config\") pod \"isvc-logger-raw-09f87-predictor-f88dfc6f9-vzbn2\" (UID: \"085b71f3-788e-4449-92b8-1364b16d4479\") " pod="kserve-ci-e2e-test/isvc-logger-raw-09f87-predictor-f88dfc6f9-vzbn2"
Sep 17 19:20:51.722048 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:20:51.721879    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/085b71f3-788e-4449-92b8-1364b16d4479-kserve-provision-location\") pod \"isvc-logger-raw-09f87-predictor-f88dfc6f9-vzbn2\" (UID: \"085b71f3-788e-4449-92b8-1364b16d4479\") " pod="kserve-ci-e2e-test/isvc-logger-raw-09f87-predictor-f88dfc6f9-vzbn2"
Sep 17 19:20:51.722048 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:20:51.721904    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/085b71f3-788e-4449-92b8-1364b16d4479-cabundle-cert\") pod \"isvc-logger-raw-09f87-predictor-f88dfc6f9-vzbn2\" (UID: \"085b71f3-788e-4449-92b8-1364b16d4479\") " pod="kserve-ci-e2e-test/isvc-logger-raw-09f87-predictor-f88dfc6f9-vzbn2"
Sep 17 19:20:51.722048 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:20:51.721960    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hm274\" (UniqueName: \"kubernetes.io/projected/085b71f3-788e-4449-92b8-1364b16d4479-kube-api-access-hm274\") pod \"isvc-logger-raw-09f87-predictor-f88dfc6f9-vzbn2\" (UID: \"085b71f3-788e-4449-92b8-1364b16d4479\") " pod="kserve-ci-e2e-test/isvc-logger-raw-09f87-predictor-f88dfc6f9-vzbn2"
Sep 17 19:20:51.722048 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:20:51.721994    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/085b71f3-788e-4449-92b8-1364b16d4479-proxy-tls\") pod \"isvc-logger-raw-09f87-predictor-f88dfc6f9-vzbn2\" (UID: \"085b71f3-788e-4449-92b8-1364b16d4479\") " pod="kserve-ci-e2e-test/isvc-logger-raw-09f87-predictor-f88dfc6f9-vzbn2"
Sep 17 19:20:51.722248 ip-10-0-131-232 kubenswrapper[2582]: E0917 19:20:51.722126    2582 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-logger-raw-09f87-predictor-serving-cert: secret "isvc-logger-raw-09f87-predictor-serving-cert" not found
Sep 17 19:20:51.722248 ip-10-0-131-232 kubenswrapper[2582]: E0917 19:20:51.722202    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/085b71f3-788e-4449-92b8-1364b16d4479-proxy-tls podName:085b71f3-788e-4449-92b8-1364b16d4479 nodeName:}" failed. No retries permitted until 2026-09-17 19:20:52.222182501 +0000 UTC m=+867.993853557 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/085b71f3-788e-4449-92b8-1364b16d4479-proxy-tls") pod "isvc-logger-raw-09f87-predictor-f88dfc6f9-vzbn2" (UID: "085b71f3-788e-4449-92b8-1364b16d4479") : secret "isvc-logger-raw-09f87-predictor-serving-cert" not found
Sep 17 19:20:51.722367 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:20:51.722341    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/085b71f3-788e-4449-92b8-1364b16d4479-kserve-provision-location\") pod \"isvc-logger-raw-09f87-predictor-f88dfc6f9-vzbn2\" (UID: \"085b71f3-788e-4449-92b8-1364b16d4479\") " pod="kserve-ci-e2e-test/isvc-logger-raw-09f87-predictor-f88dfc6f9-vzbn2"
Sep 17 19:20:51.722554 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:20:51.722537    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/085b71f3-788e-4449-92b8-1364b16d4479-kube-rbac-proxy-sar-config\") pod \"isvc-logger-raw-09f87-predictor-f88dfc6f9-vzbn2\" (UID: \"085b71f3-788e-4449-92b8-1364b16d4479\") " pod="kserve-ci-e2e-test/isvc-logger-raw-09f87-predictor-f88dfc6f9-vzbn2"
Sep 17 19:20:51.722607 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:20:51.722587    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/085b71f3-788e-4449-92b8-1364b16d4479-cabundle-cert\") pod \"isvc-logger-raw-09f87-predictor-f88dfc6f9-vzbn2\" (UID: \"085b71f3-788e-4449-92b8-1364b16d4479\") " pod="kserve-ci-e2e-test/isvc-logger-raw-09f87-predictor-f88dfc6f9-vzbn2"
Sep 17 19:20:51.732946 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:20:51.732925    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hm274\" (UniqueName: \"kubernetes.io/projected/085b71f3-788e-4449-92b8-1364b16d4479-kube-api-access-hm274\") pod \"isvc-logger-raw-09f87-predictor-f88dfc6f9-vzbn2\" (UID: \"085b71f3-788e-4449-92b8-1364b16d4479\") " pod="kserve-ci-e2e-test/isvc-logger-raw-09f87-predictor-f88dfc6f9-vzbn2"
Sep 17 19:20:52.225606 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:20:52.225574    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/085b71f3-788e-4449-92b8-1364b16d4479-proxy-tls\") pod \"isvc-logger-raw-09f87-predictor-f88dfc6f9-vzbn2\" (UID: \"085b71f3-788e-4449-92b8-1364b16d4479\") " pod="kserve-ci-e2e-test/isvc-logger-raw-09f87-predictor-f88dfc6f9-vzbn2"
Sep 17 19:20:52.228045 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:20:52.228015    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/085b71f3-788e-4449-92b8-1364b16d4479-proxy-tls\") pod \"isvc-logger-raw-09f87-predictor-f88dfc6f9-vzbn2\" (UID: \"085b71f3-788e-4449-92b8-1364b16d4479\") " pod="kserve-ci-e2e-test/isvc-logger-raw-09f87-predictor-f88dfc6f9-vzbn2"
Sep 17 19:20:52.433279 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:20:52.433247    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-logger-raw-09f87-predictor-f88dfc6f9-vzbn2"
Sep 17 19:20:52.711345 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:20:52.711242    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-logger-raw-09f87-predictor-f88dfc6f9-vzbn2"]
Sep 17 19:20:52.713771 ip-10-0-131-232 kubenswrapper[2582]: W0917 19:20:52.713745    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod085b71f3_788e_4449_92b8_1364b16d4479.slice/crio-257590b1ff83764538cdd38b9cef2370b3eae224b5b0fa121476ff65722a13dc WatchSource:0}: Error finding container 257590b1ff83764538cdd38b9cef2370b3eae224b5b0fa121476ff65722a13dc: Status 404 returned error can't find the container with id 257590b1ff83764538cdd38b9cef2370b3eae224b5b0fa121476ff65722a13dc
Sep 17 19:20:53.423966 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:20:53.423928    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/model-chainer-raw-hpa-7eb14-7c964d587c-skw74" podUID="cee3f39c-6bb7-43df-9d53-85d1298a14ab" containerName="model-chainer-raw-hpa-7eb14" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 19:20:53.424172 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:20:53.424044    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/model-chainer-raw-hpa-7eb14-7c964d587c-skw74"
Sep 17 19:20:53.516428 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:20:53.516393    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-raw-09f87-predictor-f88dfc6f9-vzbn2" event={"ID":"085b71f3-788e-4449-92b8-1364b16d4479","Type":"ContainerStarted","Data":"59de8c1ea1cc75e1094daeee8890a8d899f014bc1d44bbf22690464f93ee2c40"}
Sep 17 19:20:53.516620 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:20:53.516466    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-raw-09f87-predictor-f88dfc6f9-vzbn2" event={"ID":"085b71f3-788e-4449-92b8-1364b16d4479","Type":"ContainerStarted","Data":"257590b1ff83764538cdd38b9cef2370b3eae224b5b0fa121476ff65722a13dc"}
Sep 17 19:20:54.406173 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:20:54.406138    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-5ad3e2-predictor-546d64f79c-9p2qf" podUID="66ebb0a4-ce00-4698-b750-7272f5812bf5" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.30:8080: connect: connection refused"
Sep 17 19:20:57.534763 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:20:57.534728    2582 generic.go:356] "Generic (PLEG): container finished" podID="085b71f3-788e-4449-92b8-1364b16d4479" containerID="59de8c1ea1cc75e1094daeee8890a8d899f014bc1d44bbf22690464f93ee2c40" exitCode=0
Sep 17 19:20:57.535154 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:20:57.534792    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-raw-09f87-predictor-f88dfc6f9-vzbn2" event={"ID":"085b71f3-788e-4449-92b8-1364b16d4479","Type":"ContainerDied","Data":"59de8c1ea1cc75e1094daeee8890a8d899f014bc1d44bbf22690464f93ee2c40"}
Sep 17 19:20:58.422913 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:20:58.422872    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/model-chainer-raw-hpa-7eb14-7c964d587c-skw74" podUID="cee3f39c-6bb7-43df-9d53-85d1298a14ab" containerName="model-chainer-raw-hpa-7eb14" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 19:20:58.541922 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:20:58.541890    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-raw-09f87-predictor-f88dfc6f9-vzbn2" event={"ID":"085b71f3-788e-4449-92b8-1364b16d4479","Type":"ContainerStarted","Data":"76267d157572cd4b3c18c32c356ba75721217475f8921725ef4699159e605b09"}
Sep 17 19:20:58.542305 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:20:58.541962    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-raw-09f87-predictor-f88dfc6f9-vzbn2" event={"ID":"085b71f3-788e-4449-92b8-1364b16d4479","Type":"ContainerStarted","Data":"b8d88df8486d7e0532b744e2859d632d6a2c60dd93d1f29d3f2a12c33b5f6484"}
Sep 17 19:20:58.542305 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:20:58.541986    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-raw-09f87-predictor-f88dfc6f9-vzbn2" event={"ID":"085b71f3-788e-4449-92b8-1364b16d4479","Type":"ContainerStarted","Data":"2e7dd268337dab771b542d7a743c6caa58d24f1f16babc6c2b94dc10a9de9bf1"}
Sep 17 19:20:58.542422 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:20:58.542348    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-logger-raw-09f87-predictor-f88dfc6f9-vzbn2"
Sep 17 19:20:58.542477 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:20:58.542431    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-logger-raw-09f87-predictor-f88dfc6f9-vzbn2"
Sep 17 19:20:58.542477 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:20:58.542473    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-logger-raw-09f87-predictor-f88dfc6f9-vzbn2"
Sep 17 19:20:58.543701 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:20:58.543664    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-09f87-predictor-f88dfc6f9-vzbn2" podUID="085b71f3-788e-4449-92b8-1364b16d4479" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.32:8080: connect: connection refused"
Sep 17 19:20:58.544327 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:20:58.544306    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-09f87-predictor-f88dfc6f9-vzbn2" podUID="085b71f3-788e-4449-92b8-1364b16d4479" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 19:20:58.565301 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:20:58.565255    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-logger-raw-09f87-predictor-f88dfc6f9-vzbn2" podStartSLOduration=3.582663163 podStartE2EDuration="7.565241691s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.982578528s" isStatefulPod=false podCreationTimestamp="2026-09-17 19:20:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 19:20:58.564039705 +0000 UTC m=+874.335710792" watchObservedRunningTime="2026-09-17 19:20:58.565241691 +0000 UTC m=+874.336912747"
Sep 17 19:20:59.546268 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:20:59.546232    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-09f87-predictor-f88dfc6f9-vzbn2" podUID="085b71f3-788e-4449-92b8-1364b16d4479" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.32:8080: connect: connection refused"
Sep 17 19:20:59.546713 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:20:59.546622    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-09f87-predictor-f88dfc6f9-vzbn2" podUID="085b71f3-788e-4449-92b8-1364b16d4479" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 19:21:03.422993 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:21:03.422950    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/model-chainer-raw-hpa-7eb14-7c964d587c-skw74" podUID="cee3f39c-6bb7-43df-9d53-85d1298a14ab" containerName="model-chainer-raw-hpa-7eb14" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 19:21:04.405998 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:21:04.405954    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-5ad3e2-predictor-546d64f79c-9p2qf" podUID="66ebb0a4-ce00-4698-b750-7272f5812bf5" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.30:8080: connect: connection refused"
Sep 17 19:21:04.550397 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:21:04.550369    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-logger-raw-09f87-predictor-f88dfc6f9-vzbn2"
Sep 17 19:21:04.550835 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:21:04.550794    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-09f87-predictor-f88dfc6f9-vzbn2" podUID="085b71f3-788e-4449-92b8-1364b16d4479" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.32:8080: connect: connection refused"
Sep 17 19:21:04.551384 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:21:04.551361    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-09f87-predictor-f88dfc6f9-vzbn2" podUID="085b71f3-788e-4449-92b8-1364b16d4479" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 19:21:08.422905 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:21:08.422867    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/model-chainer-raw-hpa-7eb14-7c964d587c-skw74" podUID="cee3f39c-6bb7-43df-9d53-85d1298a14ab" containerName="model-chainer-raw-hpa-7eb14" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 19:21:11.383551 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:21:11.383531    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/model-chainer-raw-hpa-7eb14-7c964d587c-skw74"
Sep 17 19:21:11.481573 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:21:11.481535    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/cee3f39c-6bb7-43df-9d53-85d1298a14ab-openshift-service-ca-bundle\") pod \"cee3f39c-6bb7-43df-9d53-85d1298a14ab\" (UID: \"cee3f39c-6bb7-43df-9d53-85d1298a14ab\") "
Sep 17 19:21:11.481708 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:21:11.481613    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/cee3f39c-6bb7-43df-9d53-85d1298a14ab-proxy-tls\") pod \"cee3f39c-6bb7-43df-9d53-85d1298a14ab\" (UID: \"cee3f39c-6bb7-43df-9d53-85d1298a14ab\") "
Sep 17 19:21:11.481977 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:21:11.481939    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cee3f39c-6bb7-43df-9d53-85d1298a14ab-openshift-service-ca-bundle" (OuterVolumeSpecName: "openshift-service-ca-bundle") pod "cee3f39c-6bb7-43df-9d53-85d1298a14ab" (UID: "cee3f39c-6bb7-43df-9d53-85d1298a14ab"). InnerVolumeSpecName "openshift-service-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 19:21:11.483656 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:21:11.483640    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cee3f39c-6bb7-43df-9d53-85d1298a14ab-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "cee3f39c-6bb7-43df-9d53-85d1298a14ab" (UID: "cee3f39c-6bb7-43df-9d53-85d1298a14ab"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 19:21:11.582672 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:21:11.582599    2582 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/cee3f39c-6bb7-43df-9d53-85d1298a14ab-proxy-tls\") on node \"ip-10-0-131-232.ec2.internal\" DevicePath \"\""
Sep 17 19:21:11.582672 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:21:11.582626    2582 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/cee3f39c-6bb7-43df-9d53-85d1298a14ab-openshift-service-ca-bundle\") on node \"ip-10-0-131-232.ec2.internal\" DevicePath \"\""
Sep 17 19:21:11.599366 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:21:11.599332    2582 generic.go:356] "Generic (PLEG): container finished" podID="cee3f39c-6bb7-43df-9d53-85d1298a14ab" containerID="0956fa3be64762ea0145ba0fbef7a1fea11c1fd694f750f7ea0428ed3d1669bb" exitCode=0
Sep 17 19:21:11.599492 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:21:11.599381    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/model-chainer-raw-hpa-7eb14-7c964d587c-skw74"
Sep 17 19:21:11.599492 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:21:11.599414    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/model-chainer-raw-hpa-7eb14-7c964d587c-skw74" event={"ID":"cee3f39c-6bb7-43df-9d53-85d1298a14ab","Type":"ContainerDied","Data":"0956fa3be64762ea0145ba0fbef7a1fea11c1fd694f750f7ea0428ed3d1669bb"}
Sep 17 19:21:11.599492 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:21:11.599466    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/model-chainer-raw-hpa-7eb14-7c964d587c-skw74" event={"ID":"cee3f39c-6bb7-43df-9d53-85d1298a14ab","Type":"ContainerDied","Data":"bb75abfdb7b5bce42c4d93768e156489406e47956ad31072286aad0b35c171ab"}
Sep 17 19:21:11.599492 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:21:11.599488    2582 scope.go:117] "RemoveContainer" containerID="0956fa3be64762ea0145ba0fbef7a1fea11c1fd694f750f7ea0428ed3d1669bb"
Sep 17 19:21:11.607508 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:21:11.607495    2582 scope.go:117] "RemoveContainer" containerID="0956fa3be64762ea0145ba0fbef7a1fea11c1fd694f750f7ea0428ed3d1669bb"
Sep 17 19:21:11.607761 ip-10-0-131-232 kubenswrapper[2582]: E0917 19:21:11.607739    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0956fa3be64762ea0145ba0fbef7a1fea11c1fd694f750f7ea0428ed3d1669bb\": container with ID starting with 0956fa3be64762ea0145ba0fbef7a1fea11c1fd694f750f7ea0428ed3d1669bb not found: ID does not exist" containerID="0956fa3be64762ea0145ba0fbef7a1fea11c1fd694f750f7ea0428ed3d1669bb"
Sep 17 19:21:11.607850 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:21:11.607773    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0956fa3be64762ea0145ba0fbef7a1fea11c1fd694f750f7ea0428ed3d1669bb"} err="failed to get container status \"0956fa3be64762ea0145ba0fbef7a1fea11c1fd694f750f7ea0428ed3d1669bb\": rpc error: code = NotFound desc = could not find container \"0956fa3be64762ea0145ba0fbef7a1fea11c1fd694f750f7ea0428ed3d1669bb\": container with ID starting with 0956fa3be64762ea0145ba0fbef7a1fea11c1fd694f750f7ea0428ed3d1669bb not found: ID does not exist"
Sep 17 19:21:11.619599 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:21:11.619573    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/model-chainer-raw-hpa-7eb14-7c964d587c-skw74"]
Sep 17 19:21:11.623196 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:21:11.623175    2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/model-chainer-raw-hpa-7eb14-7c964d587c-skw74"]
Sep 17 19:21:12.768682 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:21:12.768650    2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="cee3f39c-6bb7-43df-9d53-85d1298a14ab" path="/var/lib/kubelet/pods/cee3f39c-6bb7-43df-9d53-85d1298a14ab/volumes"
Sep 17 19:21:14.405589 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:21:14.405557    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-5ad3e2-predictor-546d64f79c-9p2qf" podUID="66ebb0a4-ce00-4698-b750-7272f5812bf5" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.30:8080: connect: connection refused"
Sep 17 19:21:14.551492 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:21:14.551456    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-09f87-predictor-f88dfc6f9-vzbn2" podUID="085b71f3-788e-4449-92b8-1364b16d4479" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.32:8080: connect: connection refused"
Sep 17 19:21:14.551960 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:21:14.551936    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-09f87-predictor-f88dfc6f9-vzbn2" podUID="085b71f3-788e-4449-92b8-1364b16d4479" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 19:21:24.406566 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:21:24.406488    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-5ad3e2-predictor-546d64f79c-9p2qf" podUID="66ebb0a4-ce00-4698-b750-7272f5812bf5" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.30:8080: connect: connection refused"
Sep 17 19:21:24.550963 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:21:24.550921    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-09f87-predictor-f88dfc6f9-vzbn2" podUID="085b71f3-788e-4449-92b8-1364b16d4479" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.32:8080: connect: connection refused"
Sep 17 19:21:24.551351 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:21:24.551327    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-09f87-predictor-f88dfc6f9-vzbn2" podUID="085b71f3-788e-4449-92b8-1364b16d4479" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 19:21:24.699951 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:21:24.699873    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-kdm8s_c1b79c18-774f-4b78-a6a9-7ea8a229aaae/ovn-acl-logging/0.log"
Sep 17 19:21:24.701079 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:21:24.701055    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-kdm8s_c1b79c18-774f-4b78-a6a9-7ea8a229aaae/ovn-acl-logging/0.log"
Sep 17 19:21:34.406031 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:21:34.405988    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-5ad3e2-predictor-546d64f79c-9p2qf" podUID="66ebb0a4-ce00-4698-b750-7272f5812bf5" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.30:8080: connect: connection refused"
Sep 17 19:21:34.550980 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:21:34.550926    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-09f87-predictor-f88dfc6f9-vzbn2" podUID="085b71f3-788e-4449-92b8-1364b16d4479" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.32:8080: connect: connection refused"
Sep 17 19:21:34.551348 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:21:34.551323    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-09f87-predictor-f88dfc6f9-vzbn2" podUID="085b71f3-788e-4449-92b8-1364b16d4479" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 19:21:43.764833 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:21:43.764800    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-primary-5ad3e2-predictor-546d64f79c-9p2qf"
Sep 17 19:21:44.551857 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:21:44.551798    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-09f87-predictor-f88dfc6f9-vzbn2" podUID="085b71f3-788e-4449-92b8-1364b16d4479" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.32:8080: connect: connection refused"
Sep 17 19:21:44.552286 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:21:44.552265    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-09f87-predictor-f88dfc6f9-vzbn2" podUID="085b71f3-788e-4449-92b8-1364b16d4479" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 19:21:54.550836 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:21:54.550787    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-09f87-predictor-f88dfc6f9-vzbn2" podUID="085b71f3-788e-4449-92b8-1364b16d4479" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.32:8080: connect: connection refused"
Sep 17 19:21:54.551226 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:21:54.551198    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-09f87-predictor-f88dfc6f9-vzbn2" podUID="085b71f3-788e-4449-92b8-1364b16d4479" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 19:22:04.551502 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:22:04.551463    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-09f87-predictor-f88dfc6f9-vzbn2" podUID="085b71f3-788e-4449-92b8-1364b16d4479" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.32:8080: connect: connection refused"
Sep 17 19:22:04.551983 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:22:04.551960    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-09f87-predictor-f88dfc6f9-vzbn2" podUID="085b71f3-788e-4449-92b8-1364b16d4479" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 19:22:14.551444 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:22:14.551414    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-logger-raw-09f87-predictor-f88dfc6f9-vzbn2"
Sep 17 19:22:14.551851 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:22:14.551651    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-logger-raw-09f87-predictor-f88dfc6f9-vzbn2"
Sep 17 19:22:24.144603 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:22:24.144559    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-primary-5ad3e2-predictor-546d64f79c-9p2qf"]
Sep 17 19:22:24.145178 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:22:24.144872    2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-primary-5ad3e2-predictor-546d64f79c-9p2qf" podUID="66ebb0a4-ce00-4698-b750-7272f5812bf5" containerName="kserve-container" containerID="cri-o://075f52c965827d655948290468e4b2bb8351ddfa2d65ffd7d16b51bd3fa21c5e" gracePeriod=30
Sep 17 19:22:24.145178 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:22:24.144964    2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-primary-5ad3e2-predictor-546d64f79c-9p2qf" podUID="66ebb0a4-ce00-4698-b750-7272f5812bf5" containerName="kube-rbac-proxy" containerID="cri-o://d53c0f469969227ea32b6892917d632d2597d6bb7a326b418d596eda3a38d403" gracePeriod=30
Sep 17 19:22:24.400788 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:22:24.400695    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-5ad3e2-predictor-546d64f79c-9p2qf" podUID="66ebb0a4-ce00-4698-b750-7272f5812bf5" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.30:8643/healthz\": dial tcp 10.134.0.30:8643: connect: connection refused"
Sep 17 19:22:24.903142 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:22:24.903112    2582 generic.go:356] "Generic (PLEG): container finished" podID="66ebb0a4-ce00-4698-b750-7272f5812bf5" containerID="d53c0f469969227ea32b6892917d632d2597d6bb7a326b418d596eda3a38d403" exitCode=2
Sep 17 19:22:24.903379 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:22:24.903186    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-primary-5ad3e2-predictor-546d64f79c-9p2qf" event={"ID":"66ebb0a4-ce00-4698-b750-7272f5812bf5","Type":"ContainerDied","Data":"d53c0f469969227ea32b6892917d632d2597d6bb7a326b418d596eda3a38d403"}
Sep 17 19:22:27.071808 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:22:27.071768    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-logger-raw-09f87-predictor-f88dfc6f9-vzbn2"]
Sep 17 19:22:27.072280 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:22:27.072141    2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-logger-raw-09f87-predictor-f88dfc6f9-vzbn2" podUID="085b71f3-788e-4449-92b8-1364b16d4479" containerName="kube-rbac-proxy" containerID="cri-o://b8d88df8486d7e0532b744e2859d632d6a2c60dd93d1f29d3f2a12c33b5f6484" gracePeriod=30
Sep 17 19:22:27.072280 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:22:27.072142    2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-logger-raw-09f87-predictor-f88dfc6f9-vzbn2" podUID="085b71f3-788e-4449-92b8-1364b16d4479" containerName="agent" containerID="cri-o://76267d157572cd4b3c18c32c356ba75721217475f8921725ef4699159e605b09" gracePeriod=30
Sep 17 19:22:27.072280 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:22:27.072122    2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-logger-raw-09f87-predictor-f88dfc6f9-vzbn2" podUID="085b71f3-788e-4449-92b8-1364b16d4479" containerName="kserve-container" containerID="cri-o://2e7dd268337dab771b542d7a743c6caa58d24f1f16babc6c2b94dc10a9de9bf1" gracePeriod=30
Sep 17 19:22:27.919420 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:22:27.919385    2582 generic.go:356] "Generic (PLEG): container finished" podID="085b71f3-788e-4449-92b8-1364b16d4479" containerID="b8d88df8486d7e0532b744e2859d632d6a2c60dd93d1f29d3f2a12c33b5f6484" exitCode=2
Sep 17 19:22:27.919587 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:22:27.919458    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-raw-09f87-predictor-f88dfc6f9-vzbn2" event={"ID":"085b71f3-788e-4449-92b8-1364b16d4479","Type":"ContainerDied","Data":"b8d88df8486d7e0532b744e2859d632d6a2c60dd93d1f29d3f2a12c33b5f6484"}
Sep 17 19:22:29.296797 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:22:29.296774    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-primary-5ad3e2-predictor-546d64f79c-9p2qf"
Sep 17 19:22:29.399158 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:22:29.399059    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/66ebb0a4-ce00-4698-b750-7272f5812bf5-proxy-tls\") pod \"66ebb0a4-ce00-4698-b750-7272f5812bf5\" (UID: \"66ebb0a4-ce00-4698-b750-7272f5812bf5\") "
Sep 17 19:22:29.399158 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:22:29.399099    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/66ebb0a4-ce00-4698-b750-7272f5812bf5-cabundle-cert\") pod \"66ebb0a4-ce00-4698-b750-7272f5812bf5\" (UID: \"66ebb0a4-ce00-4698-b750-7272f5812bf5\") "
Sep 17 19:22:29.399158 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:22:29.399121    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ltwcp\" (UniqueName: \"kubernetes.io/projected/66ebb0a4-ce00-4698-b750-7272f5812bf5-kube-api-access-ltwcp\") pod \"66ebb0a4-ce00-4698-b750-7272f5812bf5\" (UID: \"66ebb0a4-ce00-4698-b750-7272f5812bf5\") "
Sep 17 19:22:29.399158 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:22:29.399157    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/66ebb0a4-ce00-4698-b750-7272f5812bf5-kube-rbac-proxy-sar-config\") pod \"66ebb0a4-ce00-4698-b750-7272f5812bf5\" (UID: \"66ebb0a4-ce00-4698-b750-7272f5812bf5\") "
Sep 17 19:22:29.399477 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:22:29.399217    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/66ebb0a4-ce00-4698-b750-7272f5812bf5-kserve-provision-location\") pod \"66ebb0a4-ce00-4698-b750-7272f5812bf5\" (UID: \"66ebb0a4-ce00-4698-b750-7272f5812bf5\") "
Sep 17 19:22:29.399925 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:22:29.399814    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/66ebb0a4-ce00-4698-b750-7272f5812bf5-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "66ebb0a4-ce00-4698-b750-7272f5812bf5" (UID: "66ebb0a4-ce00-4698-b750-7272f5812bf5"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 19:22:29.399925 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:22:29.399881    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/66ebb0a4-ce00-4698-b750-7272f5812bf5-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "66ebb0a4-ce00-4698-b750-7272f5812bf5" (UID: "66ebb0a4-ce00-4698-b750-7272f5812bf5"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 19:22:29.400095 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:22:29.399929    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/66ebb0a4-ce00-4698-b750-7272f5812bf5-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "66ebb0a4-ce00-4698-b750-7272f5812bf5" (UID: "66ebb0a4-ce00-4698-b750-7272f5812bf5"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 19:22:29.402365 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:22:29.402334    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/66ebb0a4-ce00-4698-b750-7272f5812bf5-kube-api-access-ltwcp" (OuterVolumeSpecName: "kube-api-access-ltwcp") pod "66ebb0a4-ce00-4698-b750-7272f5812bf5" (UID: "66ebb0a4-ce00-4698-b750-7272f5812bf5"). InnerVolumeSpecName "kube-api-access-ltwcp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 19:22:29.409304 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:22:29.409276    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/66ebb0a4-ce00-4698-b750-7272f5812bf5-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "66ebb0a4-ce00-4698-b750-7272f5812bf5" (UID: "66ebb0a4-ce00-4698-b750-7272f5812bf5"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 19:22:29.501112 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:22:29.501068    2582 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/66ebb0a4-ce00-4698-b750-7272f5812bf5-proxy-tls\") on node \"ip-10-0-131-232.ec2.internal\" DevicePath \"\""
Sep 17 19:22:29.501277 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:22:29.501124    2582 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/66ebb0a4-ce00-4698-b750-7272f5812bf5-cabundle-cert\") on node \"ip-10-0-131-232.ec2.internal\" DevicePath \"\""
Sep 17 19:22:29.501277 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:22:29.501134    2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ltwcp\" (UniqueName: \"kubernetes.io/projected/66ebb0a4-ce00-4698-b750-7272f5812bf5-kube-api-access-ltwcp\") on node \"ip-10-0-131-232.ec2.internal\" DevicePath \"\""
Sep 17 19:22:29.501277 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:22:29.501145    2582 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/66ebb0a4-ce00-4698-b750-7272f5812bf5-kube-rbac-proxy-sar-config\") on node \"ip-10-0-131-232.ec2.internal\" DevicePath \"\""
Sep 17 19:22:29.501277 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:22:29.501154    2582 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/66ebb0a4-ce00-4698-b750-7272f5812bf5-kserve-provision-location\") on node \"ip-10-0-131-232.ec2.internal\" DevicePath \"\""
Sep 17 19:22:29.546448 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:22:29.546400    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-09f87-predictor-f88dfc6f9-vzbn2" podUID="085b71f3-788e-4449-92b8-1364b16d4479" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.32:8643/healthz\": dial tcp 10.134.0.32:8643: connect: connection refused"
Sep 17 19:22:29.930310 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:22:29.930272    2582 generic.go:356] "Generic (PLEG): container finished" podID="66ebb0a4-ce00-4698-b750-7272f5812bf5" containerID="075f52c965827d655948290468e4b2bb8351ddfa2d65ffd7d16b51bd3fa21c5e" exitCode=0
Sep 17 19:22:29.930513 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:22:29.930345    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-primary-5ad3e2-predictor-546d64f79c-9p2qf"
Sep 17 19:22:29.930513 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:22:29.930358    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-primary-5ad3e2-predictor-546d64f79c-9p2qf" event={"ID":"66ebb0a4-ce00-4698-b750-7272f5812bf5","Type":"ContainerDied","Data":"075f52c965827d655948290468e4b2bb8351ddfa2d65ffd7d16b51bd3fa21c5e"}
Sep 17 19:22:29.930513 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:22:29.930433    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-primary-5ad3e2-predictor-546d64f79c-9p2qf" event={"ID":"66ebb0a4-ce00-4698-b750-7272f5812bf5","Type":"ContainerDied","Data":"864bb094fd732b8ea43d685382162bf77dc97444dcad5faf97d7089cf1f763ff"}
Sep 17 19:22:29.930513 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:22:29.930472    2582 scope.go:117] "RemoveContainer" containerID="d53c0f469969227ea32b6892917d632d2597d6bb7a326b418d596eda3a38d403"
Sep 17 19:22:29.939619 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:22:29.939601    2582 scope.go:117] "RemoveContainer" containerID="075f52c965827d655948290468e4b2bb8351ddfa2d65ffd7d16b51bd3fa21c5e"
Sep 17 19:22:29.947566 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:22:29.947546    2582 scope.go:117] "RemoveContainer" containerID="efd0235f10f38dc8682514048bb7bd9ec01e029d4addd4b0b4303b5249baeea8"
Sep 17 19:22:29.955504 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:22:29.955481    2582 scope.go:117] "RemoveContainer" containerID="d53c0f469969227ea32b6892917d632d2597d6bb7a326b418d596eda3a38d403"
Sep 17 19:22:29.955815 ip-10-0-131-232 kubenswrapper[2582]: E0917 19:22:29.955792    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d53c0f469969227ea32b6892917d632d2597d6bb7a326b418d596eda3a38d403\": container with ID starting with d53c0f469969227ea32b6892917d632d2597d6bb7a326b418d596eda3a38d403 not found: ID does not exist" containerID="d53c0f469969227ea32b6892917d632d2597d6bb7a326b418d596eda3a38d403"
Sep 17 19:22:29.955912 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:22:29.955850    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d53c0f469969227ea32b6892917d632d2597d6bb7a326b418d596eda3a38d403"} err="failed to get container status \"d53c0f469969227ea32b6892917d632d2597d6bb7a326b418d596eda3a38d403\": rpc error: code = NotFound desc = could not find container \"d53c0f469969227ea32b6892917d632d2597d6bb7a326b418d596eda3a38d403\": container with ID starting with d53c0f469969227ea32b6892917d632d2597d6bb7a326b418d596eda3a38d403 not found: ID does not exist"
Sep 17 19:22:29.955912 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:22:29.955872    2582 scope.go:117] "RemoveContainer" containerID="075f52c965827d655948290468e4b2bb8351ddfa2d65ffd7d16b51bd3fa21c5e"
Sep 17 19:22:29.956395 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:22:29.956368    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-primary-5ad3e2-predictor-546d64f79c-9p2qf"]
Sep 17 19:22:29.956723 ip-10-0-131-232 kubenswrapper[2582]: E0917 19:22:29.956696    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"075f52c965827d655948290468e4b2bb8351ddfa2d65ffd7d16b51bd3fa21c5e\": container with ID starting with 075f52c965827d655948290468e4b2bb8351ddfa2d65ffd7d16b51bd3fa21c5e not found: ID does not exist" containerID="075f52c965827d655948290468e4b2bb8351ddfa2d65ffd7d16b51bd3fa21c5e"
Sep 17 19:22:29.956855 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:22:29.956731    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"075f52c965827d655948290468e4b2bb8351ddfa2d65ffd7d16b51bd3fa21c5e"} err="failed to get container status \"075f52c965827d655948290468e4b2bb8351ddfa2d65ffd7d16b51bd3fa21c5e\": rpc error: code = NotFound desc = could not find container \"075f52c965827d655948290468e4b2bb8351ddfa2d65ffd7d16b51bd3fa21c5e\": container with ID starting with 075f52c965827d655948290468e4b2bb8351ddfa2d65ffd7d16b51bd3fa21c5e not found: ID does not exist"
Sep 17 19:22:29.956855 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:22:29.956770    2582 scope.go:117] "RemoveContainer" containerID="efd0235f10f38dc8682514048bb7bd9ec01e029d4addd4b0b4303b5249baeea8"
Sep 17 19:22:29.957206 ip-10-0-131-232 kubenswrapper[2582]: E0917 19:22:29.957180    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"efd0235f10f38dc8682514048bb7bd9ec01e029d4addd4b0b4303b5249baeea8\": container with ID starting with efd0235f10f38dc8682514048bb7bd9ec01e029d4addd4b0b4303b5249baeea8 not found: ID does not exist" containerID="efd0235f10f38dc8682514048bb7bd9ec01e029d4addd4b0b4303b5249baeea8"
Sep 17 19:22:29.957426 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:22:29.957400    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"efd0235f10f38dc8682514048bb7bd9ec01e029d4addd4b0b4303b5249baeea8"} err="failed to get container status \"efd0235f10f38dc8682514048bb7bd9ec01e029d4addd4b0b4303b5249baeea8\": rpc error: code = NotFound desc = could not find container \"efd0235f10f38dc8682514048bb7bd9ec01e029d4addd4b0b4303b5249baeea8\": container with ID starting with efd0235f10f38dc8682514048bb7bd9ec01e029d4addd4b0b4303b5249baeea8 not found: ID does not exist"
Sep 17 19:22:29.958418 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:22:29.958397    2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-primary-5ad3e2-predictor-546d64f79c-9p2qf"]
Sep 17 19:22:30.767983 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:22:30.767944    2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="66ebb0a4-ce00-4698-b750-7272f5812bf5" path="/var/lib/kubelet/pods/66ebb0a4-ce00-4698-b750-7272f5812bf5/volumes"
Sep 17 19:22:32.946515 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:22:32.946484    2582 generic.go:356] "Generic (PLEG): container finished" podID="085b71f3-788e-4449-92b8-1364b16d4479" containerID="2e7dd268337dab771b542d7a743c6caa58d24f1f16babc6c2b94dc10a9de9bf1" exitCode=0
Sep 17 19:22:32.946918 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:22:32.946559    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-raw-09f87-predictor-f88dfc6f9-vzbn2" event={"ID":"085b71f3-788e-4449-92b8-1364b16d4479","Type":"ContainerDied","Data":"2e7dd268337dab771b542d7a743c6caa58d24f1f16babc6c2b94dc10a9de9bf1"}
Sep 17 19:22:34.546444 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:22:34.546403    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-09f87-predictor-f88dfc6f9-vzbn2" podUID="085b71f3-788e-4449-92b8-1364b16d4479" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.32:8643/healthz\": dial tcp 10.134.0.32:8643: connect: connection refused"
Sep 17 19:22:34.551884 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:22:34.551809    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-09f87-predictor-f88dfc6f9-vzbn2" podUID="085b71f3-788e-4449-92b8-1364b16d4479" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.32:8080: connect: connection refused"
Sep 17 19:22:34.552218 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:22:34.552192    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-09f87-predictor-f88dfc6f9-vzbn2" podUID="085b71f3-788e-4449-92b8-1364b16d4479" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 19:22:39.546978 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:22:39.546939    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-09f87-predictor-f88dfc6f9-vzbn2" podUID="085b71f3-788e-4449-92b8-1364b16d4479" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.32:8643/healthz\": dial tcp 10.134.0.32:8643: connect: connection refused"
Sep 17 19:22:39.547342 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:22:39.547062    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-logger-raw-09f87-predictor-f88dfc6f9-vzbn2"
Sep 17 19:22:44.547313 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:22:44.547257    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-09f87-predictor-f88dfc6f9-vzbn2" podUID="085b71f3-788e-4449-92b8-1364b16d4479" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.32:8643/healthz\": dial tcp 10.134.0.32:8643: connect: connection refused"
Sep 17 19:22:44.551718 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:22:44.551685    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-09f87-predictor-f88dfc6f9-vzbn2" podUID="085b71f3-788e-4449-92b8-1364b16d4479" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.32:8080: connect: connection refused"
Sep 17 19:22:44.552099 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:22:44.552058    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-09f87-predictor-f88dfc6f9-vzbn2" podUID="085b71f3-788e-4449-92b8-1364b16d4479" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 19:22:49.546442 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:22:49.546358    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-09f87-predictor-f88dfc6f9-vzbn2" podUID="085b71f3-788e-4449-92b8-1364b16d4479" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.32:8643/healthz\": dial tcp 10.134.0.32:8643: connect: connection refused"
Sep 17 19:22:54.546925 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:22:54.546885    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-09f87-predictor-f88dfc6f9-vzbn2" podUID="085b71f3-788e-4449-92b8-1364b16d4479" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.32:8643/healthz\": dial tcp 10.134.0.32:8643: connect: connection refused"
Sep 17 19:22:54.551255 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:22:54.551222    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-09f87-predictor-f88dfc6f9-vzbn2" podUID="085b71f3-788e-4449-92b8-1364b16d4479" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.32:8080: connect: connection refused"
Sep 17 19:22:54.551393 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:22:54.551382    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-logger-raw-09f87-predictor-f88dfc6f9-vzbn2"
Sep 17 19:22:54.551547 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:22:54.551526    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-09f87-predictor-f88dfc6f9-vzbn2" podUID="085b71f3-788e-4449-92b8-1364b16d4479" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 19:22:54.551672 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:22:54.551617    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-logger-raw-09f87-predictor-f88dfc6f9-vzbn2"
Sep 17 19:22:57.221637 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:22:57.221610    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-logger-raw-09f87-predictor-f88dfc6f9-vzbn2"
Sep 17 19:22:57.325944 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:22:57.325910    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/085b71f3-788e-4449-92b8-1364b16d4479-cabundle-cert\") pod \"085b71f3-788e-4449-92b8-1364b16d4479\" (UID: \"085b71f3-788e-4449-92b8-1364b16d4479\") "
Sep 17 19:22:57.326121 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:22:57.325972    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hm274\" (UniqueName: \"kubernetes.io/projected/085b71f3-788e-4449-92b8-1364b16d4479-kube-api-access-hm274\") pod \"085b71f3-788e-4449-92b8-1364b16d4479\" (UID: \"085b71f3-788e-4449-92b8-1364b16d4479\") "
Sep 17 19:22:57.326121 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:22:57.326013    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/085b71f3-788e-4449-92b8-1364b16d4479-kserve-provision-location\") pod \"085b71f3-788e-4449-92b8-1364b16d4479\" (UID: \"085b71f3-788e-4449-92b8-1364b16d4479\") "
Sep 17 19:22:57.326121 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:22:57.326031    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/085b71f3-788e-4449-92b8-1364b16d4479-kube-rbac-proxy-sar-config\") pod \"085b71f3-788e-4449-92b8-1364b16d4479\" (UID: \"085b71f3-788e-4449-92b8-1364b16d4479\") "
Sep 17 19:22:57.326121 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:22:57.326067    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/085b71f3-788e-4449-92b8-1364b16d4479-proxy-tls\") pod \"085b71f3-788e-4449-92b8-1364b16d4479\" (UID: \"085b71f3-788e-4449-92b8-1364b16d4479\") "
Sep 17 19:22:57.326374 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:22:57.326346    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/085b71f3-788e-4449-92b8-1364b16d4479-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "085b71f3-788e-4449-92b8-1364b16d4479" (UID: "085b71f3-788e-4449-92b8-1364b16d4479"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 19:22:57.326435 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:22:57.326386    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/085b71f3-788e-4449-92b8-1364b16d4479-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "085b71f3-788e-4449-92b8-1364b16d4479" (UID: "085b71f3-788e-4449-92b8-1364b16d4479"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 19:22:57.326435 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:22:57.326401    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/085b71f3-788e-4449-92b8-1364b16d4479-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "085b71f3-788e-4449-92b8-1364b16d4479" (UID: "085b71f3-788e-4449-92b8-1364b16d4479"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 19:22:57.328300 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:22:57.328244    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/085b71f3-788e-4449-92b8-1364b16d4479-kube-api-access-hm274" (OuterVolumeSpecName: "kube-api-access-hm274") pod "085b71f3-788e-4449-92b8-1364b16d4479" (UID: "085b71f3-788e-4449-92b8-1364b16d4479"). InnerVolumeSpecName "kube-api-access-hm274". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 19:22:57.328300 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:22:57.328266    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/085b71f3-788e-4449-92b8-1364b16d4479-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "085b71f3-788e-4449-92b8-1364b16d4479" (UID: "085b71f3-788e-4449-92b8-1364b16d4479"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 19:22:57.426732 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:22:57.426702    2582 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/085b71f3-788e-4449-92b8-1364b16d4479-cabundle-cert\") on node \"ip-10-0-131-232.ec2.internal\" DevicePath \"\""
Sep 17 19:22:57.426732 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:22:57.426727    2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hm274\" (UniqueName: \"kubernetes.io/projected/085b71f3-788e-4449-92b8-1364b16d4479-kube-api-access-hm274\") on node \"ip-10-0-131-232.ec2.internal\" DevicePath \"\""
Sep 17 19:22:57.426732 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:22:57.426736    2582 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/085b71f3-788e-4449-92b8-1364b16d4479-kserve-provision-location\") on node \"ip-10-0-131-232.ec2.internal\" DevicePath \"\""
Sep 17 19:22:57.426979 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:22:57.426745    2582 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/085b71f3-788e-4449-92b8-1364b16d4479-kube-rbac-proxy-sar-config\") on node \"ip-10-0-131-232.ec2.internal\" DevicePath \"\""
Sep 17 19:22:57.426979 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:22:57.426755    2582 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/085b71f3-788e-4449-92b8-1364b16d4479-proxy-tls\") on node \"ip-10-0-131-232.ec2.internal\" DevicePath \"\""
Sep 17 19:22:58.049474 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:22:58.049439    2582 generic.go:356] "Generic (PLEG): container finished" podID="085b71f3-788e-4449-92b8-1364b16d4479" containerID="76267d157572cd4b3c18c32c356ba75721217475f8921725ef4699159e605b09" exitCode=0
Sep 17 19:22:58.049634 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:22:58.049497    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-logger-raw-09f87-predictor-f88dfc6f9-vzbn2"
Sep 17 19:22:58.049634 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:22:58.049517    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-raw-09f87-predictor-f88dfc6f9-vzbn2" event={"ID":"085b71f3-788e-4449-92b8-1364b16d4479","Type":"ContainerDied","Data":"76267d157572cd4b3c18c32c356ba75721217475f8921725ef4699159e605b09"}
Sep 17 19:22:58.049634 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:22:58.049582    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-raw-09f87-predictor-f88dfc6f9-vzbn2" event={"ID":"085b71f3-788e-4449-92b8-1364b16d4479","Type":"ContainerDied","Data":"257590b1ff83764538cdd38b9cef2370b3eae224b5b0fa121476ff65722a13dc"}
Sep 17 19:22:58.049634 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:22:58.049613    2582 scope.go:117] "RemoveContainer" containerID="76267d157572cd4b3c18c32c356ba75721217475f8921725ef4699159e605b09"
Sep 17 19:22:58.058437 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:22:58.058422    2582 scope.go:117] "RemoveContainer" containerID="b8d88df8486d7e0532b744e2859d632d6a2c60dd93d1f29d3f2a12c33b5f6484"
Sep 17 19:22:58.065648 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:22:58.065629    2582 scope.go:117] "RemoveContainer" containerID="2e7dd268337dab771b542d7a743c6caa58d24f1f16babc6c2b94dc10a9de9bf1"
Sep 17 19:22:58.073539 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:22:58.073519    2582 scope.go:117] "RemoveContainer" containerID="59de8c1ea1cc75e1094daeee8890a8d899f014bc1d44bbf22690464f93ee2c40"
Sep 17 19:22:58.079592 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:22:58.079556    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-logger-raw-09f87-predictor-f88dfc6f9-vzbn2"]
Sep 17 19:22:58.079714 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:22:58.079618    2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-logger-raw-09f87-predictor-f88dfc6f9-vzbn2"]
Sep 17 19:22:58.085012 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:22:58.084918    2582 scope.go:117] "RemoveContainer" containerID="76267d157572cd4b3c18c32c356ba75721217475f8921725ef4699159e605b09"
Sep 17 19:22:58.085327 ip-10-0-131-232 kubenswrapper[2582]: E0917 19:22:58.085306    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"76267d157572cd4b3c18c32c356ba75721217475f8921725ef4699159e605b09\": container with ID starting with 76267d157572cd4b3c18c32c356ba75721217475f8921725ef4699159e605b09 not found: ID does not exist" containerID="76267d157572cd4b3c18c32c356ba75721217475f8921725ef4699159e605b09"
Sep 17 19:22:58.085396 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:22:58.085335    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"76267d157572cd4b3c18c32c356ba75721217475f8921725ef4699159e605b09"} err="failed to get container status \"76267d157572cd4b3c18c32c356ba75721217475f8921725ef4699159e605b09\": rpc error: code = NotFound desc = could not find container \"76267d157572cd4b3c18c32c356ba75721217475f8921725ef4699159e605b09\": container with ID starting with 76267d157572cd4b3c18c32c356ba75721217475f8921725ef4699159e605b09 not found: ID does not exist"
Sep 17 19:22:58.085396 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:22:58.085353    2582 scope.go:117] "RemoveContainer" containerID="b8d88df8486d7e0532b744e2859d632d6a2c60dd93d1f29d3f2a12c33b5f6484"
Sep 17 19:22:58.085627 ip-10-0-131-232 kubenswrapper[2582]: E0917 19:22:58.085608    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b8d88df8486d7e0532b744e2859d632d6a2c60dd93d1f29d3f2a12c33b5f6484\": container with ID starting with b8d88df8486d7e0532b744e2859d632d6a2c60dd93d1f29d3f2a12c33b5f6484 not found: ID does not exist" containerID="b8d88df8486d7e0532b744e2859d632d6a2c60dd93d1f29d3f2a12c33b5f6484"
Sep 17 19:22:58.085738 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:22:58.085637    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b8d88df8486d7e0532b744e2859d632d6a2c60dd93d1f29d3f2a12c33b5f6484"} err="failed to get container status \"b8d88df8486d7e0532b744e2859d632d6a2c60dd93d1f29d3f2a12c33b5f6484\": rpc error: code = NotFound desc = could not find container \"b8d88df8486d7e0532b744e2859d632d6a2c60dd93d1f29d3f2a12c33b5f6484\": container with ID starting with b8d88df8486d7e0532b744e2859d632d6a2c60dd93d1f29d3f2a12c33b5f6484 not found: ID does not exist"
Sep 17 19:22:58.085738 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:22:58.085659    2582 scope.go:117] "RemoveContainer" containerID="2e7dd268337dab771b542d7a743c6caa58d24f1f16babc6c2b94dc10a9de9bf1"
Sep 17 19:22:58.085939 ip-10-0-131-232 kubenswrapper[2582]: E0917 19:22:58.085922    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2e7dd268337dab771b542d7a743c6caa58d24f1f16babc6c2b94dc10a9de9bf1\": container with ID starting with 2e7dd268337dab771b542d7a743c6caa58d24f1f16babc6c2b94dc10a9de9bf1 not found: ID does not exist" containerID="2e7dd268337dab771b542d7a743c6caa58d24f1f16babc6c2b94dc10a9de9bf1"
Sep 17 19:22:58.086008 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:22:58.085955    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2e7dd268337dab771b542d7a743c6caa58d24f1f16babc6c2b94dc10a9de9bf1"} err="failed to get container status \"2e7dd268337dab771b542d7a743c6caa58d24f1f16babc6c2b94dc10a9de9bf1\": rpc error: code = NotFound desc = could not find container \"2e7dd268337dab771b542d7a743c6caa58d24f1f16babc6c2b94dc10a9de9bf1\": container with ID starting with 2e7dd268337dab771b542d7a743c6caa58d24f1f16babc6c2b94dc10a9de9bf1 not found: ID does not exist"
Sep 17 19:22:58.086008 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:22:58.085974    2582 scope.go:117] "RemoveContainer" containerID="59de8c1ea1cc75e1094daeee8890a8d899f014bc1d44bbf22690464f93ee2c40"
Sep 17 19:22:58.086220 ip-10-0-131-232 kubenswrapper[2582]: E0917 19:22:58.086203    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"59de8c1ea1cc75e1094daeee8890a8d899f014bc1d44bbf22690464f93ee2c40\": container with ID starting with 59de8c1ea1cc75e1094daeee8890a8d899f014bc1d44bbf22690464f93ee2c40 not found: ID does not exist" containerID="59de8c1ea1cc75e1094daeee8890a8d899f014bc1d44bbf22690464f93ee2c40"
Sep 17 19:22:58.086280 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:22:58.086225    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"59de8c1ea1cc75e1094daeee8890a8d899f014bc1d44bbf22690464f93ee2c40"} err="failed to get container status \"59de8c1ea1cc75e1094daeee8890a8d899f014bc1d44bbf22690464f93ee2c40\": rpc error: code = NotFound desc = could not find container \"59de8c1ea1cc75e1094daeee8890a8d899f014bc1d44bbf22690464f93ee2c40\": container with ID starting with 59de8c1ea1cc75e1094daeee8890a8d899f014bc1d44bbf22690464f93ee2c40 not found: ID does not exist"
Sep 17 19:22:58.768406 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:22:58.768373    2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="085b71f3-788e-4449-92b8-1364b16d4479" path="/var/lib/kubelet/pods/085b71f3-788e-4449-92b8-1364b16d4479/volumes"
Sep 17 19:24:30.277464 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:24:30.277420    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/raw-sklearn-runtime-9534b-predictor-5476b9544d-l46q5"]
Sep 17 19:24:30.277899 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:24:30.277713    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cee3f39c-6bb7-43df-9d53-85d1298a14ab" containerName="model-chainer-raw-hpa-7eb14"
Sep 17 19:24:30.277899 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:24:30.277726    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="cee3f39c-6bb7-43df-9d53-85d1298a14ab" containerName="model-chainer-raw-hpa-7eb14"
Sep 17 19:24:30.277899 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:24:30.277740    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="66ebb0a4-ce00-4698-b750-7272f5812bf5" containerName="storage-initializer"
Sep 17 19:24:30.277899 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:24:30.277746    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="66ebb0a4-ce00-4698-b750-7272f5812bf5" containerName="storage-initializer"
Sep 17 19:24:30.277899 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:24:30.277753    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="085b71f3-788e-4449-92b8-1364b16d4479" containerName="agent"
Sep 17 19:24:30.277899 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:24:30.277758    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="085b71f3-788e-4449-92b8-1364b16d4479" containerName="agent"
Sep 17 19:24:30.277899 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:24:30.277763    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="66ebb0a4-ce00-4698-b750-7272f5812bf5" containerName="kube-rbac-proxy"
Sep 17 19:24:30.277899 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:24:30.277768    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="66ebb0a4-ce00-4698-b750-7272f5812bf5" containerName="kube-rbac-proxy"
Sep 17 19:24:30.277899 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:24:30.277775    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="085b71f3-788e-4449-92b8-1364b16d4479" containerName="kube-rbac-proxy"
Sep 17 19:24:30.277899 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:24:30.277780    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="085b71f3-788e-4449-92b8-1364b16d4479" containerName="kube-rbac-proxy"
Sep 17 19:24:30.277899 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:24:30.277790    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="66ebb0a4-ce00-4698-b750-7272f5812bf5" containerName="kserve-container"
Sep 17 19:24:30.277899 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:24:30.277795    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="66ebb0a4-ce00-4698-b750-7272f5812bf5" containerName="kserve-container"
Sep 17 19:24:30.277899 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:24:30.277800    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="085b71f3-788e-4449-92b8-1364b16d4479" containerName="kserve-container"
Sep 17 19:24:30.277899 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:24:30.277805    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="085b71f3-788e-4449-92b8-1364b16d4479" containerName="kserve-container"
Sep 17 19:24:30.277899 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:24:30.277814    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="085b71f3-788e-4449-92b8-1364b16d4479" containerName="storage-initializer"
Sep 17 19:24:30.277899 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:24:30.277834    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="085b71f3-788e-4449-92b8-1364b16d4479" containerName="storage-initializer"
Sep 17 19:24:30.277899 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:24:30.277880    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="cee3f39c-6bb7-43df-9d53-85d1298a14ab" containerName="model-chainer-raw-hpa-7eb14"
Sep 17 19:24:30.277899 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:24:30.277888    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="085b71f3-788e-4449-92b8-1364b16d4479" containerName="kserve-container"
Sep 17 19:24:30.277899 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:24:30.277893    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="085b71f3-788e-4449-92b8-1364b16d4479" containerName="kube-rbac-proxy"
Sep 17 19:24:30.277899 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:24:30.277900    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="085b71f3-788e-4449-92b8-1364b16d4479" containerName="agent"
Sep 17 19:24:30.277899 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:24:30.277906    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="66ebb0a4-ce00-4698-b750-7272f5812bf5" containerName="kube-rbac-proxy"
Sep 17 19:24:30.277899 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:24:30.277911    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="66ebb0a4-ce00-4698-b750-7272f5812bf5" containerName="kserve-container"
Sep 17 19:24:30.280925 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:24:30.280908    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/raw-sklearn-runtime-9534b-predictor-5476b9544d-l46q5"
Sep 17 19:24:30.283497 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:24:30.283472    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Sep 17 19:24:30.283622 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:24:30.283511    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-vpx7x\""
Sep 17 19:24:30.283622 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:24:30.283521    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"raw-sklearn-runtime-9534b-predictor-serving-cert\""
Sep 17 19:24:30.284433 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:24:30.284413    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"raw-sklearn-runtime-9534b-kube-rbac-proxy-sar-config\""
Sep 17 19:24:30.284571 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:24:30.284505    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Sep 17 19:24:30.284571 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:24:30.284523    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Sep 17 19:24:30.284571 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:24:30.284512    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Sep 17 19:24:30.292885 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:24:30.292865    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/raw-sklearn-runtime-9534b-predictor-5476b9544d-l46q5"]
Sep 17 19:24:30.433991 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:24:30.433964    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/9b932ceb-d635-4910-b20f-3ca89ddc6acc-cabundle-cert\") pod \"raw-sklearn-runtime-9534b-predictor-5476b9544d-l46q5\" (UID: \"9b932ceb-d635-4910-b20f-3ca89ddc6acc\") " pod="kserve-ci-e2e-test/raw-sklearn-runtime-9534b-predictor-5476b9544d-l46q5"
Sep 17 19:24:30.434164 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:24:30.434001    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/9b932ceb-d635-4910-b20f-3ca89ddc6acc-proxy-tls\") pod \"raw-sklearn-runtime-9534b-predictor-5476b9544d-l46q5\" (UID: \"9b932ceb-d635-4910-b20f-3ca89ddc6acc\") " pod="kserve-ci-e2e-test/raw-sklearn-runtime-9534b-predictor-5476b9544d-l46q5"
Sep 17 19:24:30.434164 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:24:30.434028    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/9b932ceb-d635-4910-b20f-3ca89ddc6acc-kube-rbac-proxy-sar-config\") pod \"raw-sklearn-runtime-9534b-predictor-5476b9544d-l46q5\" (UID: \"9b932ceb-d635-4910-b20f-3ca89ddc6acc\") " pod="kserve-ci-e2e-test/raw-sklearn-runtime-9534b-predictor-5476b9544d-l46q5"
Sep 17 19:24:30.434164 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:24:30.434129    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6ksng\" (UniqueName: \"kubernetes.io/projected/9b932ceb-d635-4910-b20f-3ca89ddc6acc-kube-api-access-6ksng\") pod \"raw-sklearn-runtime-9534b-predictor-5476b9544d-l46q5\" (UID: \"9b932ceb-d635-4910-b20f-3ca89ddc6acc\") " pod="kserve-ci-e2e-test/raw-sklearn-runtime-9534b-predictor-5476b9544d-l46q5"
Sep 17 19:24:30.434164 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:24:30.434156    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/9b932ceb-d635-4910-b20f-3ca89ddc6acc-kserve-provision-location\") pod \"raw-sklearn-runtime-9534b-predictor-5476b9544d-l46q5\" (UID: \"9b932ceb-d635-4910-b20f-3ca89ddc6acc\") " pod="kserve-ci-e2e-test/raw-sklearn-runtime-9534b-predictor-5476b9544d-l46q5"
Sep 17 19:24:30.534801 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:24:30.534742    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/9b932ceb-d635-4910-b20f-3ca89ddc6acc-kube-rbac-proxy-sar-config\") pod \"raw-sklearn-runtime-9534b-predictor-5476b9544d-l46q5\" (UID: \"9b932ceb-d635-4910-b20f-3ca89ddc6acc\") " pod="kserve-ci-e2e-test/raw-sklearn-runtime-9534b-predictor-5476b9544d-l46q5"
Sep 17 19:24:30.534801 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:24:30.534773    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6ksng\" (UniqueName: \"kubernetes.io/projected/9b932ceb-d635-4910-b20f-3ca89ddc6acc-kube-api-access-6ksng\") pod \"raw-sklearn-runtime-9534b-predictor-5476b9544d-l46q5\" (UID: \"9b932ceb-d635-4910-b20f-3ca89ddc6acc\") " pod="kserve-ci-e2e-test/raw-sklearn-runtime-9534b-predictor-5476b9544d-l46q5"
Sep 17 19:24:30.534801 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:24:30.534793    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/9b932ceb-d635-4910-b20f-3ca89ddc6acc-kserve-provision-location\") pod \"raw-sklearn-runtime-9534b-predictor-5476b9544d-l46q5\" (UID: \"9b932ceb-d635-4910-b20f-3ca89ddc6acc\") " pod="kserve-ci-e2e-test/raw-sklearn-runtime-9534b-predictor-5476b9544d-l46q5"
Sep 17 19:24:30.535076 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:24:30.535055    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/9b932ceb-d635-4910-b20f-3ca89ddc6acc-cabundle-cert\") pod \"raw-sklearn-runtime-9534b-predictor-5476b9544d-l46q5\" (UID: \"9b932ceb-d635-4910-b20f-3ca89ddc6acc\") " pod="kserve-ci-e2e-test/raw-sklearn-runtime-9534b-predictor-5476b9544d-l46q5"
Sep 17 19:24:30.535148 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:24:30.535115    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/9b932ceb-d635-4910-b20f-3ca89ddc6acc-proxy-tls\") pod \"raw-sklearn-runtime-9534b-predictor-5476b9544d-l46q5\" (UID: \"9b932ceb-d635-4910-b20f-3ca89ddc6acc\") " pod="kserve-ci-e2e-test/raw-sklearn-runtime-9534b-predictor-5476b9544d-l46q5"
Sep 17 19:24:30.535148 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:24:30.535142    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/9b932ceb-d635-4910-b20f-3ca89ddc6acc-kserve-provision-location\") pod \"raw-sklearn-runtime-9534b-predictor-5476b9544d-l46q5\" (UID: \"9b932ceb-d635-4910-b20f-3ca89ddc6acc\") " pod="kserve-ci-e2e-test/raw-sklearn-runtime-9534b-predictor-5476b9544d-l46q5"
Sep 17 19:24:30.535455 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:24:30.535434    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/9b932ceb-d635-4910-b20f-3ca89ddc6acc-kube-rbac-proxy-sar-config\") pod \"raw-sklearn-runtime-9534b-predictor-5476b9544d-l46q5\" (UID: \"9b932ceb-d635-4910-b20f-3ca89ddc6acc\") " pod="kserve-ci-e2e-test/raw-sklearn-runtime-9534b-predictor-5476b9544d-l46q5"
Sep 17 19:24:30.535562 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:24:30.535546    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/9b932ceb-d635-4910-b20f-3ca89ddc6acc-cabundle-cert\") pod \"raw-sklearn-runtime-9534b-predictor-5476b9544d-l46q5\" (UID: \"9b932ceb-d635-4910-b20f-3ca89ddc6acc\") " pod="kserve-ci-e2e-test/raw-sklearn-runtime-9534b-predictor-5476b9544d-l46q5"
Sep 17 19:24:30.537577 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:24:30.537556    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/9b932ceb-d635-4910-b20f-3ca89ddc6acc-proxy-tls\") pod \"raw-sklearn-runtime-9534b-predictor-5476b9544d-l46q5\" (UID: \"9b932ceb-d635-4910-b20f-3ca89ddc6acc\") " pod="kserve-ci-e2e-test/raw-sklearn-runtime-9534b-predictor-5476b9544d-l46q5"
Sep 17 19:24:30.545183 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:24:30.545161    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6ksng\" (UniqueName: \"kubernetes.io/projected/9b932ceb-d635-4910-b20f-3ca89ddc6acc-kube-api-access-6ksng\") pod \"raw-sklearn-runtime-9534b-predictor-5476b9544d-l46q5\" (UID: \"9b932ceb-d635-4910-b20f-3ca89ddc6acc\") " pod="kserve-ci-e2e-test/raw-sklearn-runtime-9534b-predictor-5476b9544d-l46q5"
Sep 17 19:24:30.591295 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:24:30.591271    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/raw-sklearn-runtime-9534b-predictor-5476b9544d-l46q5"
Sep 17 19:24:30.864540 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:24:30.864516    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/raw-sklearn-runtime-9534b-predictor-5476b9544d-l46q5"]
Sep 17 19:24:30.866854 ip-10-0-131-232 kubenswrapper[2582]: W0917 19:24:30.866812    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9b932ceb_d635_4910_b20f_3ca89ddc6acc.slice/crio-7925de2b9e597bece4ce7f14712c166aaa28d721f759ff2814014771307dd679 WatchSource:0}: Error finding container 7925de2b9e597bece4ce7f14712c166aaa28d721f759ff2814014771307dd679: Status 404 returned error can't find the container with id 7925de2b9e597bece4ce7f14712c166aaa28d721f759ff2814014771307dd679
Sep 17 19:24:31.428960 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:24:31.428922    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/raw-sklearn-runtime-9534b-predictor-5476b9544d-l46q5" event={"ID":"9b932ceb-d635-4910-b20f-3ca89ddc6acc","Type":"ContainerStarted","Data":"fcafc9910b6cae748a423d7ea0b9602cc6c3efdbd4886bfef3f4c0b31b12f577"}
Sep 17 19:24:31.429339 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:24:31.429008    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/raw-sklearn-runtime-9534b-predictor-5476b9544d-l46q5" event={"ID":"9b932ceb-d635-4910-b20f-3ca89ddc6acc","Type":"ContainerStarted","Data":"7925de2b9e597bece4ce7f14712c166aaa28d721f759ff2814014771307dd679"}
Sep 17 19:24:35.446188 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:24:35.446158    2582 generic.go:356] "Generic (PLEG): container finished" podID="9b932ceb-d635-4910-b20f-3ca89ddc6acc" containerID="fcafc9910b6cae748a423d7ea0b9602cc6c3efdbd4886bfef3f4c0b31b12f577" exitCode=0
Sep 17 19:24:35.446556 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:24:35.446200    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/raw-sklearn-runtime-9534b-predictor-5476b9544d-l46q5" event={"ID":"9b932ceb-d635-4910-b20f-3ca89ddc6acc","Type":"ContainerDied","Data":"fcafc9910b6cae748a423d7ea0b9602cc6c3efdbd4886bfef3f4c0b31b12f577"}
Sep 17 19:24:36.453166 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:24:36.453136    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/raw-sklearn-runtime-9534b-predictor-5476b9544d-l46q5" event={"ID":"9b932ceb-d635-4910-b20f-3ca89ddc6acc","Type":"ContainerStarted","Data":"4abc1e73f9cd5e852885771d79f323064f6e2f765726eff4d94770c2ea1d6b01"}
Sep 17 19:24:36.453549 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:24:36.453195    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/raw-sklearn-runtime-9534b-predictor-5476b9544d-l46q5" event={"ID":"9b932ceb-d635-4910-b20f-3ca89ddc6acc","Type":"ContainerStarted","Data":"74913f810a43f83720e469d956efe7e7fb87880365d1e65a3e2604ce127e30c7"}
Sep 17 19:24:36.453549 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:24:36.453426    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/raw-sklearn-runtime-9534b-predictor-5476b9544d-l46q5"
Sep 17 19:24:36.474065 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:24:36.474017    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/raw-sklearn-runtime-9534b-predictor-5476b9544d-l46q5" podStartSLOduration=2.71258715 podStartE2EDuration="6.473999807s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.761412657s" isStatefulPod=false podCreationTimestamp="2026-09-17 19:24:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 19:24:36.47304658 +0000 UTC m=+1092.244717647" watchObservedRunningTime="2026-09-17 19:24:36.473999807 +0000 UTC m=+1092.245670868"
Sep 17 19:24:37.457767 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:24:37.457734    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/raw-sklearn-runtime-9534b-predictor-5476b9544d-l46q5"
Sep 17 19:24:37.459084 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:24:37.459055    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-runtime-9534b-predictor-5476b9544d-l46q5" podUID="9b932ceb-d635-4910-b20f-3ca89ddc6acc" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.33:8080: connect: connection refused"
Sep 17 19:24:38.461368 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:24:38.461332    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-runtime-9534b-predictor-5476b9544d-l46q5" podUID="9b932ceb-d635-4910-b20f-3ca89ddc6acc" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.33:8080: connect: connection refused"
Sep 17 19:24:43.465297 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:24:43.465255    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/raw-sklearn-runtime-9534b-predictor-5476b9544d-l46q5"
Sep 17 19:24:43.465847 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:24:43.465792    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-runtime-9534b-predictor-5476b9544d-l46q5" podUID="9b932ceb-d635-4910-b20f-3ca89ddc6acc" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.33:8080: connect: connection refused"
Sep 17 19:24:53.466178 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:24:53.466140    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-runtime-9534b-predictor-5476b9544d-l46q5" podUID="9b932ceb-d635-4910-b20f-3ca89ddc6acc" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.33:8080: connect: connection refused"
Sep 17 19:25:03.465706 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:25:03.465664    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-runtime-9534b-predictor-5476b9544d-l46q5" podUID="9b932ceb-d635-4910-b20f-3ca89ddc6acc" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.33:8080: connect: connection refused"
Sep 17 19:25:13.466070 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:25:13.466030    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-runtime-9534b-predictor-5476b9544d-l46q5" podUID="9b932ceb-d635-4910-b20f-3ca89ddc6acc" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.33:8080: connect: connection refused"
Sep 17 19:25:23.466110 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:25:23.466072    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-runtime-9534b-predictor-5476b9544d-l46q5" podUID="9b932ceb-d635-4910-b20f-3ca89ddc6acc" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.33:8080: connect: connection refused"
Sep 17 19:25:33.465846 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:25:33.465787    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-runtime-9534b-predictor-5476b9544d-l46q5" podUID="9b932ceb-d635-4910-b20f-3ca89ddc6acc" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.33:8080: connect: connection refused"
Sep 17 19:25:43.465692 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:25:43.465653    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-runtime-9534b-predictor-5476b9544d-l46q5" podUID="9b932ceb-d635-4910-b20f-3ca89ddc6acc" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.33:8080: connect: connection refused"
Sep 17 19:25:43.669968 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:25:43.669934    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-v2-predictor-bfcdb48dd-p6nxc"]
Sep 17 19:25:43.673639 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:25:43.673618    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-bfcdb48dd-p6nxc"
Sep 17 19:25:43.675856 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:25:43.675813    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-create-v2-predictor-serving-cert\""
Sep 17 19:25:43.676065 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:25:43.676009    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-create-kube-rbac-proxy-sar-config\""
Sep 17 19:25:43.684769 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:25:43.684744    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-v2-predictor-bfcdb48dd-p6nxc"]
Sep 17 19:25:43.715994 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:25:43.715920    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/34230187-dccf-4b5b-bacc-fb35753a12c2-proxy-tls\") pod \"isvc-canary-create-v2-predictor-bfcdb48dd-p6nxc\" (UID: \"34230187-dccf-4b5b-bacc-fb35753a12c2\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-bfcdb48dd-p6nxc"
Sep 17 19:25:43.715994 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:25:43.715958    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/34230187-dccf-4b5b-bacc-fb35753a12c2-cabundle-cert\") pod \"isvc-canary-create-v2-predictor-bfcdb48dd-p6nxc\" (UID: \"34230187-dccf-4b5b-bacc-fb35753a12c2\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-bfcdb48dd-p6nxc"
Sep 17 19:25:43.715994 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:25:43.715983    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dnvxq\" (UniqueName: \"kubernetes.io/projected/34230187-dccf-4b5b-bacc-fb35753a12c2-kube-api-access-dnvxq\") pod \"isvc-canary-create-v2-predictor-bfcdb48dd-p6nxc\" (UID: \"34230187-dccf-4b5b-bacc-fb35753a12c2\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-bfcdb48dd-p6nxc"
Sep 17 19:25:43.716207 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:25:43.716000    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/34230187-dccf-4b5b-bacc-fb35753a12c2-kserve-provision-location\") pod \"isvc-canary-create-v2-predictor-bfcdb48dd-p6nxc\" (UID: \"34230187-dccf-4b5b-bacc-fb35753a12c2\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-bfcdb48dd-p6nxc"
Sep 17 19:25:43.716207 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:25:43.716081    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/34230187-dccf-4b5b-bacc-fb35753a12c2-kube-rbac-proxy-sar-config\") pod \"isvc-canary-create-v2-predictor-bfcdb48dd-p6nxc\" (UID: \"34230187-dccf-4b5b-bacc-fb35753a12c2\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-bfcdb48dd-p6nxc"
Sep 17 19:25:43.816743 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:25:43.816709    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/34230187-dccf-4b5b-bacc-fb35753a12c2-kube-rbac-proxy-sar-config\") pod \"isvc-canary-create-v2-predictor-bfcdb48dd-p6nxc\" (UID: \"34230187-dccf-4b5b-bacc-fb35753a12c2\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-bfcdb48dd-p6nxc"
Sep 17 19:25:43.816945 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:25:43.816844    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/34230187-dccf-4b5b-bacc-fb35753a12c2-proxy-tls\") pod \"isvc-canary-create-v2-predictor-bfcdb48dd-p6nxc\" (UID: \"34230187-dccf-4b5b-bacc-fb35753a12c2\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-bfcdb48dd-p6nxc"
Sep 17 19:25:43.816945 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:25:43.816894    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/34230187-dccf-4b5b-bacc-fb35753a12c2-cabundle-cert\") pod \"isvc-canary-create-v2-predictor-bfcdb48dd-p6nxc\" (UID: \"34230187-dccf-4b5b-bacc-fb35753a12c2\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-bfcdb48dd-p6nxc"
Sep 17 19:25:43.816945 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:25:43.816932    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dnvxq\" (UniqueName: \"kubernetes.io/projected/34230187-dccf-4b5b-bacc-fb35753a12c2-kube-api-access-dnvxq\") pod \"isvc-canary-create-v2-predictor-bfcdb48dd-p6nxc\" (UID: \"34230187-dccf-4b5b-bacc-fb35753a12c2\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-bfcdb48dd-p6nxc"
Sep 17 19:25:43.817118 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:25:43.816957    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/34230187-dccf-4b5b-bacc-fb35753a12c2-kserve-provision-location\") pod \"isvc-canary-create-v2-predictor-bfcdb48dd-p6nxc\" (UID: \"34230187-dccf-4b5b-bacc-fb35753a12c2\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-bfcdb48dd-p6nxc"
Sep 17 19:25:43.817118 ip-10-0-131-232 kubenswrapper[2582]: E0917 19:25:43.816986    2582 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-canary-create-v2-predictor-serving-cert: secret "isvc-canary-create-v2-predictor-serving-cert" not found
Sep 17 19:25:43.817118 ip-10-0-131-232 kubenswrapper[2582]: E0917 19:25:43.817061    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/34230187-dccf-4b5b-bacc-fb35753a12c2-proxy-tls podName:34230187-dccf-4b5b-bacc-fb35753a12c2 nodeName:}" failed. No retries permitted until 2026-09-17 19:25:44.317039659 +0000 UTC m=+1160.088710704 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/34230187-dccf-4b5b-bacc-fb35753a12c2-proxy-tls") pod "isvc-canary-create-v2-predictor-bfcdb48dd-p6nxc" (UID: "34230187-dccf-4b5b-bacc-fb35753a12c2") : secret "isvc-canary-create-v2-predictor-serving-cert" not found
Sep 17 19:25:43.817442 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:25:43.817418    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/34230187-dccf-4b5b-bacc-fb35753a12c2-kserve-provision-location\") pod \"isvc-canary-create-v2-predictor-bfcdb48dd-p6nxc\" (UID: \"34230187-dccf-4b5b-bacc-fb35753a12c2\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-bfcdb48dd-p6nxc"
Sep 17 19:25:43.817604 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:25:43.817588    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/34230187-dccf-4b5b-bacc-fb35753a12c2-cabundle-cert\") pod \"isvc-canary-create-v2-predictor-bfcdb48dd-p6nxc\" (UID: \"34230187-dccf-4b5b-bacc-fb35753a12c2\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-bfcdb48dd-p6nxc"
Sep 17 19:25:43.817682 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:25:43.817665    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/34230187-dccf-4b5b-bacc-fb35753a12c2-kube-rbac-proxy-sar-config\") pod \"isvc-canary-create-v2-predictor-bfcdb48dd-p6nxc\" (UID: \"34230187-dccf-4b5b-bacc-fb35753a12c2\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-bfcdb48dd-p6nxc"
Sep 17 19:25:43.826278 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:25:43.826258    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dnvxq\" (UniqueName: \"kubernetes.io/projected/34230187-dccf-4b5b-bacc-fb35753a12c2-kube-api-access-dnvxq\") pod \"isvc-canary-create-v2-predictor-bfcdb48dd-p6nxc\" (UID: \"34230187-dccf-4b5b-bacc-fb35753a12c2\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-bfcdb48dd-p6nxc"
Sep 17 19:25:44.320328 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:25:44.320292    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/34230187-dccf-4b5b-bacc-fb35753a12c2-proxy-tls\") pod \"isvc-canary-create-v2-predictor-bfcdb48dd-p6nxc\" (UID: \"34230187-dccf-4b5b-bacc-fb35753a12c2\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-bfcdb48dd-p6nxc"
Sep 17 19:25:44.322734 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:25:44.322714    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/34230187-dccf-4b5b-bacc-fb35753a12c2-proxy-tls\") pod \"isvc-canary-create-v2-predictor-bfcdb48dd-p6nxc\" (UID: \"34230187-dccf-4b5b-bacc-fb35753a12c2\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-bfcdb48dd-p6nxc"
Sep 17 19:25:44.586281 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:25:44.586207    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-bfcdb48dd-p6nxc"
Sep 17 19:25:44.865455 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:25:44.865312    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-v2-predictor-bfcdb48dd-p6nxc"]
Sep 17 19:25:44.867880 ip-10-0-131-232 kubenswrapper[2582]: W0917 19:25:44.867852    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod34230187_dccf_4b5b_bacc_fb35753a12c2.slice/crio-d53619fffb2de1e90da79ff5b793fd1a4543f063a8668b30dde7b33ded0a9561 WatchSource:0}: Error finding container d53619fffb2de1e90da79ff5b793fd1a4543f063a8668b30dde7b33ded0a9561: Status 404 returned error can't find the container with id d53619fffb2de1e90da79ff5b793fd1a4543f063a8668b30dde7b33ded0a9561
Sep 17 19:25:45.733911 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:25:45.733811    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-bfcdb48dd-p6nxc" event={"ID":"34230187-dccf-4b5b-bacc-fb35753a12c2","Type":"ContainerStarted","Data":"163fcc73f7c1bc38f0dcbe3c2429608334edfe8e2615c4f2ddb71c9bbe182245"}
Sep 17 19:25:45.733911 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:25:45.733902    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-bfcdb48dd-p6nxc" event={"ID":"34230187-dccf-4b5b-bacc-fb35753a12c2","Type":"ContainerStarted","Data":"d53619fffb2de1e90da79ff5b793fd1a4543f063a8668b30dde7b33ded0a9561"}
Sep 17 19:25:48.747018 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:25:48.746974    2582 generic.go:356] "Generic (PLEG): container finished" podID="34230187-dccf-4b5b-bacc-fb35753a12c2" containerID="163fcc73f7c1bc38f0dcbe3c2429608334edfe8e2615c4f2ddb71c9bbe182245" exitCode=0
Sep 17 19:25:48.747396 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:25:48.747047    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-bfcdb48dd-p6nxc" event={"ID":"34230187-dccf-4b5b-bacc-fb35753a12c2","Type":"ContainerDied","Data":"163fcc73f7c1bc38f0dcbe3c2429608334edfe8e2615c4f2ddb71c9bbe182245"}
Sep 17 19:25:49.753379 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:25:49.753341    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-bfcdb48dd-p6nxc" event={"ID":"34230187-dccf-4b5b-bacc-fb35753a12c2","Type":"ContainerStarted","Data":"78b47abc71851974278a5b58c7bebf21a0d4536dc5ddbb18c2b9f162ea713589"}
Sep 17 19:25:49.753740 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:25:49.753414    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-bfcdb48dd-p6nxc" event={"ID":"34230187-dccf-4b5b-bacc-fb35753a12c2","Type":"ContainerStarted","Data":"b06a53fb24bde27b59dc378d36d273ded43739eaf246a641673cdc83f3194562"}
Sep 17 19:25:49.753740 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:25:49.753613    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-bfcdb48dd-p6nxc"
Sep 17 19:25:49.775431 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:25:49.775385    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-bfcdb48dd-p6nxc" podStartSLOduration=3.081976023 podStartE2EDuration="6.775369614s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.693393591s" isStatefulPod=false podCreationTimestamp="2026-09-17 19:25:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 19:25:49.773905751 +0000 UTC m=+1165.545576810" watchObservedRunningTime="2026-09-17 19:25:49.775369614 +0000 UTC m=+1165.547040711"
Sep 17 19:25:50.757768 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:25:50.757738    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-bfcdb48dd-p6nxc"
Sep 17 19:25:50.758876 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:25:50.758845    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-bfcdb48dd-p6nxc" podUID="34230187-dccf-4b5b-bacc-fb35753a12c2" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.34:8080: connect: connection refused"
Sep 17 19:25:51.761610 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:25:51.761574    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-bfcdb48dd-p6nxc" podUID="34230187-dccf-4b5b-bacc-fb35753a12c2" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.34:8080: connect: connection refused"
Sep 17 19:25:53.466550 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:25:53.466519    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/raw-sklearn-runtime-9534b-predictor-5476b9544d-l46q5"
Sep 17 19:25:56.768738 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:25:56.768713    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-bfcdb48dd-p6nxc"
Sep 17 19:25:56.769272 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:25:56.769246    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-bfcdb48dd-p6nxc" podUID="34230187-dccf-4b5b-bacc-fb35753a12c2" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.34:8080: connect: connection refused"
Sep 17 19:26:00.769020 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:26:00.768992    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/raw-sklearn-runtime-9534b-predictor-5476b9544d-l46q5"]
Sep 17 19:26:00.769384 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:26:00.769197    2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/raw-sklearn-runtime-9534b-predictor-5476b9544d-l46q5" podUID="9b932ceb-d635-4910-b20f-3ca89ddc6acc" containerName="kserve-container" containerID="cri-o://74913f810a43f83720e469d956efe7e7fb87880365d1e65a3e2604ce127e30c7" gracePeriod=30
Sep 17 19:26:00.769384 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:26:00.769221    2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/raw-sklearn-runtime-9534b-predictor-5476b9544d-l46q5" podUID="9b932ceb-d635-4910-b20f-3ca89ddc6acc" containerName="kube-rbac-proxy" containerID="cri-o://4abc1e73f9cd5e852885771d79f323064f6e2f765726eff4d94770c2ea1d6b01" gracePeriod=30
Sep 17 19:26:01.805594 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:26:01.805561    2582 generic.go:356] "Generic (PLEG): container finished" podID="9b932ceb-d635-4910-b20f-3ca89ddc6acc" containerID="4abc1e73f9cd5e852885771d79f323064f6e2f765726eff4d94770c2ea1d6b01" exitCode=2
Sep 17 19:26:01.806003 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:26:01.805638    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/raw-sklearn-runtime-9534b-predictor-5476b9544d-l46q5" event={"ID":"9b932ceb-d635-4910-b20f-3ca89ddc6acc","Type":"ContainerDied","Data":"4abc1e73f9cd5e852885771d79f323064f6e2f765726eff4d94770c2ea1d6b01"}
Sep 17 19:26:03.462293 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:26:03.462255    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-runtime-9534b-predictor-5476b9544d-l46q5" podUID="9b932ceb-d635-4910-b20f-3ca89ddc6acc" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.33:8643/healthz\": dial tcp 10.134.0.33:8643: connect: connection refused"
Sep 17 19:26:03.466743 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:26:03.466711    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-runtime-9534b-predictor-5476b9544d-l46q5" podUID="9b932ceb-d635-4910-b20f-3ca89ddc6acc" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.33:8080: connect: connection refused"
Sep 17 19:26:05.610471 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:26:05.610444    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/raw-sklearn-runtime-9534b-predictor-5476b9544d-l46q5"
Sep 17 19:26:05.680128 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:26:05.680098    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/9b932ceb-d635-4910-b20f-3ca89ddc6acc-kserve-provision-location\") pod \"9b932ceb-d635-4910-b20f-3ca89ddc6acc\" (UID: \"9b932ceb-d635-4910-b20f-3ca89ddc6acc\") "
Sep 17 19:26:05.680313 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:26:05.680140    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/9b932ceb-d635-4910-b20f-3ca89ddc6acc-cabundle-cert\") pod \"9b932ceb-d635-4910-b20f-3ca89ddc6acc\" (UID: \"9b932ceb-d635-4910-b20f-3ca89ddc6acc\") "
Sep 17 19:26:05.680313 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:26:05.680160    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/9b932ceb-d635-4910-b20f-3ca89ddc6acc-proxy-tls\") pod \"9b932ceb-d635-4910-b20f-3ca89ddc6acc\" (UID: \"9b932ceb-d635-4910-b20f-3ca89ddc6acc\") "
Sep 17 19:26:05.680313 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:26:05.680185    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6ksng\" (UniqueName: \"kubernetes.io/projected/9b932ceb-d635-4910-b20f-3ca89ddc6acc-kube-api-access-6ksng\") pod \"9b932ceb-d635-4910-b20f-3ca89ddc6acc\" (UID: \"9b932ceb-d635-4910-b20f-3ca89ddc6acc\") "
Sep 17 19:26:05.680313 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:26:05.680212    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/9b932ceb-d635-4910-b20f-3ca89ddc6acc-kube-rbac-proxy-sar-config\") pod \"9b932ceb-d635-4910-b20f-3ca89ddc6acc\" (UID: \"9b932ceb-d635-4910-b20f-3ca89ddc6acc\") "
Sep 17 19:26:05.680526 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:26:05.680466    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9b932ceb-d635-4910-b20f-3ca89ddc6acc-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "9b932ceb-d635-4910-b20f-3ca89ddc6acc" (UID: "9b932ceb-d635-4910-b20f-3ca89ddc6acc"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 19:26:05.680583 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:26:05.680541    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9b932ceb-d635-4910-b20f-3ca89ddc6acc-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "9b932ceb-d635-4910-b20f-3ca89ddc6acc" (UID: "9b932ceb-d635-4910-b20f-3ca89ddc6acc"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 19:26:05.680681 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:26:05.680652    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9b932ceb-d635-4910-b20f-3ca89ddc6acc-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "9b932ceb-d635-4910-b20f-3ca89ddc6acc" (UID: "9b932ceb-d635-4910-b20f-3ca89ddc6acc"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 19:26:05.682355 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:26:05.682335    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9b932ceb-d635-4910-b20f-3ca89ddc6acc-kube-api-access-6ksng" (OuterVolumeSpecName: "kube-api-access-6ksng") pod "9b932ceb-d635-4910-b20f-3ca89ddc6acc" (UID: "9b932ceb-d635-4910-b20f-3ca89ddc6acc"). InnerVolumeSpecName "kube-api-access-6ksng". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 19:26:05.682497 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:26:05.682448    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9b932ceb-d635-4910-b20f-3ca89ddc6acc-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "9b932ceb-d635-4910-b20f-3ca89ddc6acc" (UID: "9b932ceb-d635-4910-b20f-3ca89ddc6acc"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 19:26:05.780801 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:26:05.780715    2582 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/9b932ceb-d635-4910-b20f-3ca89ddc6acc-kserve-provision-location\") on node \"ip-10-0-131-232.ec2.internal\" DevicePath \"\""
Sep 17 19:26:05.780801 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:26:05.780744    2582 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/9b932ceb-d635-4910-b20f-3ca89ddc6acc-cabundle-cert\") on node \"ip-10-0-131-232.ec2.internal\" DevicePath \"\""
Sep 17 19:26:05.780801 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:26:05.780754    2582 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/9b932ceb-d635-4910-b20f-3ca89ddc6acc-proxy-tls\") on node \"ip-10-0-131-232.ec2.internal\" DevicePath \"\""
Sep 17 19:26:05.780801 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:26:05.780762    2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6ksng\" (UniqueName: \"kubernetes.io/projected/9b932ceb-d635-4910-b20f-3ca89ddc6acc-kube-api-access-6ksng\") on node \"ip-10-0-131-232.ec2.internal\" DevicePath \"\""
Sep 17 19:26:05.780801 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:26:05.780770    2582 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/9b932ceb-d635-4910-b20f-3ca89ddc6acc-kube-rbac-proxy-sar-config\") on node \"ip-10-0-131-232.ec2.internal\" DevicePath \"\""
Sep 17 19:26:05.823023 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:26:05.822987    2582 generic.go:356] "Generic (PLEG): container finished" podID="9b932ceb-d635-4910-b20f-3ca89ddc6acc" containerID="74913f810a43f83720e469d956efe7e7fb87880365d1e65a3e2604ce127e30c7" exitCode=0
Sep 17 19:26:05.823155 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:26:05.823043    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/raw-sklearn-runtime-9534b-predictor-5476b9544d-l46q5"
Sep 17 19:26:05.823155 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:26:05.823054    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/raw-sklearn-runtime-9534b-predictor-5476b9544d-l46q5" event={"ID":"9b932ceb-d635-4910-b20f-3ca89ddc6acc","Type":"ContainerDied","Data":"74913f810a43f83720e469d956efe7e7fb87880365d1e65a3e2604ce127e30c7"}
Sep 17 19:26:05.823155 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:26:05.823102    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/raw-sklearn-runtime-9534b-predictor-5476b9544d-l46q5" event={"ID":"9b932ceb-d635-4910-b20f-3ca89ddc6acc","Type":"ContainerDied","Data":"7925de2b9e597bece4ce7f14712c166aaa28d721f759ff2814014771307dd679"}
Sep 17 19:26:05.823877 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:26:05.823127    2582 scope.go:117] "RemoveContainer" containerID="4abc1e73f9cd5e852885771d79f323064f6e2f765726eff4d94770c2ea1d6b01"
Sep 17 19:26:05.832112 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:26:05.832092    2582 scope.go:117] "RemoveContainer" containerID="74913f810a43f83720e469d956efe7e7fb87880365d1e65a3e2604ce127e30c7"
Sep 17 19:26:05.845256 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:26:05.845231    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/raw-sklearn-runtime-9534b-predictor-5476b9544d-l46q5"]
Sep 17 19:26:05.845869 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:26:05.845845    2582 scope.go:117] "RemoveContainer" containerID="fcafc9910b6cae748a423d7ea0b9602cc6c3efdbd4886bfef3f4c0b31b12f577"
Sep 17 19:26:05.849206 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:26:05.849180    2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/raw-sklearn-runtime-9534b-predictor-5476b9544d-l46q5"]
Sep 17 19:26:05.853260 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:26:05.853245    2582 scope.go:117] "RemoveContainer" containerID="4abc1e73f9cd5e852885771d79f323064f6e2f765726eff4d94770c2ea1d6b01"
Sep 17 19:26:05.853533 ip-10-0-131-232 kubenswrapper[2582]: E0917 19:26:05.853514    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4abc1e73f9cd5e852885771d79f323064f6e2f765726eff4d94770c2ea1d6b01\": container with ID starting with 4abc1e73f9cd5e852885771d79f323064f6e2f765726eff4d94770c2ea1d6b01 not found: ID does not exist" containerID="4abc1e73f9cd5e852885771d79f323064f6e2f765726eff4d94770c2ea1d6b01"
Sep 17 19:26:05.853612 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:26:05.853540    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4abc1e73f9cd5e852885771d79f323064f6e2f765726eff4d94770c2ea1d6b01"} err="failed to get container status \"4abc1e73f9cd5e852885771d79f323064f6e2f765726eff4d94770c2ea1d6b01\": rpc error: code = NotFound desc = could not find container \"4abc1e73f9cd5e852885771d79f323064f6e2f765726eff4d94770c2ea1d6b01\": container with ID starting with 4abc1e73f9cd5e852885771d79f323064f6e2f765726eff4d94770c2ea1d6b01 not found: ID does not exist"
Sep 17 19:26:05.853612 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:26:05.853557    2582 scope.go:117] "RemoveContainer" containerID="74913f810a43f83720e469d956efe7e7fb87880365d1e65a3e2604ce127e30c7"
Sep 17 19:26:05.853801 ip-10-0-131-232 kubenswrapper[2582]: E0917 19:26:05.853783    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"74913f810a43f83720e469d956efe7e7fb87880365d1e65a3e2604ce127e30c7\": container with ID starting with 74913f810a43f83720e469d956efe7e7fb87880365d1e65a3e2604ce127e30c7 not found: ID does not exist" containerID="74913f810a43f83720e469d956efe7e7fb87880365d1e65a3e2604ce127e30c7"
Sep 17 19:26:05.853891 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:26:05.853810    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"74913f810a43f83720e469d956efe7e7fb87880365d1e65a3e2604ce127e30c7"} err="failed to get container status \"74913f810a43f83720e469d956efe7e7fb87880365d1e65a3e2604ce127e30c7\": rpc error: code = NotFound desc = could not find container \"74913f810a43f83720e469d956efe7e7fb87880365d1e65a3e2604ce127e30c7\": container with ID starting with 74913f810a43f83720e469d956efe7e7fb87880365d1e65a3e2604ce127e30c7 not found: ID does not exist"
Sep 17 19:26:05.853891 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:26:05.853878    2582 scope.go:117] "RemoveContainer" containerID="fcafc9910b6cae748a423d7ea0b9602cc6c3efdbd4886bfef3f4c0b31b12f577"
Sep 17 19:26:05.854130 ip-10-0-131-232 kubenswrapper[2582]: E0917 19:26:05.854111    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fcafc9910b6cae748a423d7ea0b9602cc6c3efdbd4886bfef3f4c0b31b12f577\": container with ID starting with fcafc9910b6cae748a423d7ea0b9602cc6c3efdbd4886bfef3f4c0b31b12f577 not found: ID does not exist" containerID="fcafc9910b6cae748a423d7ea0b9602cc6c3efdbd4886bfef3f4c0b31b12f577"
Sep 17 19:26:05.854169 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:26:05.854135    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fcafc9910b6cae748a423d7ea0b9602cc6c3efdbd4886bfef3f4c0b31b12f577"} err="failed to get container status \"fcafc9910b6cae748a423d7ea0b9602cc6c3efdbd4886bfef3f4c0b31b12f577\": rpc error: code = NotFound desc = could not find container \"fcafc9910b6cae748a423d7ea0b9602cc6c3efdbd4886bfef3f4c0b31b12f577\": container with ID starting with fcafc9910b6cae748a423d7ea0b9602cc6c3efdbd4886bfef3f4c0b31b12f577 not found: ID does not exist"
Sep 17 19:26:06.768878 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:26:06.768847    2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9b932ceb-d635-4910-b20f-3ca89ddc6acc" path="/var/lib/kubelet/pods/9b932ceb-d635-4910-b20f-3ca89ddc6acc/volumes"
Sep 17 19:26:06.769269 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:26:06.769148    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-bfcdb48dd-p6nxc" podUID="34230187-dccf-4b5b-bacc-fb35753a12c2" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.34:8080: connect: connection refused"
Sep 17 19:26:16.769559 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:26:16.769524    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-bfcdb48dd-p6nxc" podUID="34230187-dccf-4b5b-bacc-fb35753a12c2" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.34:8080: connect: connection refused"
Sep 17 19:26:24.725784 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:26:24.725753    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-kdm8s_c1b79c18-774f-4b78-a6a9-7ea8a229aaae/ovn-acl-logging/0.log"
Sep 17 19:26:24.726231 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:26:24.726114    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-kdm8s_c1b79c18-774f-4b78-a6a9-7ea8a229aaae/ovn-acl-logging/0.log"
Sep 17 19:26:26.771294 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:26:26.771233    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-bfcdb48dd-p6nxc" podUID="34230187-dccf-4b5b-bacc-fb35753a12c2" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.34:8080: connect: connection refused"
Sep 17 19:26:36.769765 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:26:36.769722    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-bfcdb48dd-p6nxc" podUID="34230187-dccf-4b5b-bacc-fb35753a12c2" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.34:8080: connect: connection refused"
Sep 17 19:26:46.770004 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:26:46.769965    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-bfcdb48dd-p6nxc" podUID="34230187-dccf-4b5b-bacc-fb35753a12c2" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.34:8080: connect: connection refused"
Sep 17 19:26:56.769805 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:26:56.769761    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-bfcdb48dd-p6nxc" podUID="34230187-dccf-4b5b-bacc-fb35753a12c2" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.34:8080: connect: connection refused"
Sep 17 19:27:06.769946 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:27:06.769914    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-bfcdb48dd-p6nxc"
Sep 17 19:27:13.825837 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:27:13.825791    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-v2-predictor-bfcdb48dd-p6nxc"]
Sep 17 19:27:13.826208 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:27:13.826089    2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-bfcdb48dd-p6nxc" podUID="34230187-dccf-4b5b-bacc-fb35753a12c2" containerName="kube-rbac-proxy" containerID="cri-o://78b47abc71851974278a5b58c7bebf21a0d4536dc5ddbb18c2b9f162ea713589" gracePeriod=30
Sep 17 19:27:13.826208 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:27:13.826089    2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-bfcdb48dd-p6nxc" podUID="34230187-dccf-4b5b-bacc-fb35753a12c2" containerName="kserve-container" containerID="cri-o://b06a53fb24bde27b59dc378d36d273ded43739eaf246a641673cdc83f3194562" gracePeriod=30
Sep 17 19:27:14.105388 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:27:14.105301    2582 generic.go:356] "Generic (PLEG): container finished" podID="34230187-dccf-4b5b-bacc-fb35753a12c2" containerID="78b47abc71851974278a5b58c7bebf21a0d4536dc5ddbb18c2b9f162ea713589" exitCode=2
Sep 17 19:27:14.105388 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:27:14.105356    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-bfcdb48dd-p6nxc" event={"ID":"34230187-dccf-4b5b-bacc-fb35753a12c2","Type":"ContainerDied","Data":"78b47abc71851974278a5b58c7bebf21a0d4536dc5ddbb18c2b9f162ea713589"}
Sep 17 19:27:16.762439 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:27:16.762340    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-bfcdb48dd-p6nxc" podUID="34230187-dccf-4b5b-bacc-fb35753a12c2" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.34:8643/healthz\": dial tcp 10.134.0.34:8643: connect: connection refused"
Sep 17 19:27:16.770100 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:27:16.770070    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-bfcdb48dd-p6nxc" podUID="34230187-dccf-4b5b-bacc-fb35753a12c2" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.34:8080: connect: connection refused"
Sep 17 19:27:18.665568 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:27:18.665541    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-bfcdb48dd-p6nxc"
Sep 17 19:27:18.777910 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:27:18.777812    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/34230187-dccf-4b5b-bacc-fb35753a12c2-proxy-tls\") pod \"34230187-dccf-4b5b-bacc-fb35753a12c2\" (UID: \"34230187-dccf-4b5b-bacc-fb35753a12c2\") "
Sep 17 19:27:18.777910 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:27:18.777869    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/34230187-dccf-4b5b-bacc-fb35753a12c2-kserve-provision-location\") pod \"34230187-dccf-4b5b-bacc-fb35753a12c2\" (UID: \"34230187-dccf-4b5b-bacc-fb35753a12c2\") "
Sep 17 19:27:18.777910 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:27:18.777897    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/34230187-dccf-4b5b-bacc-fb35753a12c2-kube-rbac-proxy-sar-config\") pod \"34230187-dccf-4b5b-bacc-fb35753a12c2\" (UID: \"34230187-dccf-4b5b-bacc-fb35753a12c2\") "
Sep 17 19:27:18.778127 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:27:18.777915    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-dnvxq\" (UniqueName: \"kubernetes.io/projected/34230187-dccf-4b5b-bacc-fb35753a12c2-kube-api-access-dnvxq\") pod \"34230187-dccf-4b5b-bacc-fb35753a12c2\" (UID: \"34230187-dccf-4b5b-bacc-fb35753a12c2\") "
Sep 17 19:27:18.778127 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:27:18.777985    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/34230187-dccf-4b5b-bacc-fb35753a12c2-cabundle-cert\") pod \"34230187-dccf-4b5b-bacc-fb35753a12c2\" (UID: \"34230187-dccf-4b5b-bacc-fb35753a12c2\") "
Sep 17 19:27:18.778260 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:27:18.778228    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/34230187-dccf-4b5b-bacc-fb35753a12c2-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "34230187-dccf-4b5b-bacc-fb35753a12c2" (UID: "34230187-dccf-4b5b-bacc-fb35753a12c2"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 19:27:18.778324 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:27:18.778295    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/34230187-dccf-4b5b-bacc-fb35753a12c2-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "34230187-dccf-4b5b-bacc-fb35753a12c2" (UID: "34230187-dccf-4b5b-bacc-fb35753a12c2"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 19:27:18.778412 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:27:18.778394    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/34230187-dccf-4b5b-bacc-fb35753a12c2-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "34230187-dccf-4b5b-bacc-fb35753a12c2" (UID: "34230187-dccf-4b5b-bacc-fb35753a12c2"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 19:27:18.780128 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:27:18.780104    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/34230187-dccf-4b5b-bacc-fb35753a12c2-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "34230187-dccf-4b5b-bacc-fb35753a12c2" (UID: "34230187-dccf-4b5b-bacc-fb35753a12c2"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 19:27:18.780252 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:27:18.780235    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/34230187-dccf-4b5b-bacc-fb35753a12c2-kube-api-access-dnvxq" (OuterVolumeSpecName: "kube-api-access-dnvxq") pod "34230187-dccf-4b5b-bacc-fb35753a12c2" (UID: "34230187-dccf-4b5b-bacc-fb35753a12c2"). InnerVolumeSpecName "kube-api-access-dnvxq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 19:27:18.879238 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:27:18.879202    2582 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/34230187-dccf-4b5b-bacc-fb35753a12c2-cabundle-cert\") on node \"ip-10-0-131-232.ec2.internal\" DevicePath \"\""
Sep 17 19:27:18.879238 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:27:18.879237    2582 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/34230187-dccf-4b5b-bacc-fb35753a12c2-proxy-tls\") on node \"ip-10-0-131-232.ec2.internal\" DevicePath \"\""
Sep 17 19:27:18.879473 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:27:18.879250    2582 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/34230187-dccf-4b5b-bacc-fb35753a12c2-kserve-provision-location\") on node \"ip-10-0-131-232.ec2.internal\" DevicePath \"\""
Sep 17 19:27:18.879473 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:27:18.879264    2582 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/34230187-dccf-4b5b-bacc-fb35753a12c2-kube-rbac-proxy-sar-config\") on node \"ip-10-0-131-232.ec2.internal\" DevicePath \"\""
Sep 17 19:27:18.879473 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:27:18.879277    2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dnvxq\" (UniqueName: \"kubernetes.io/projected/34230187-dccf-4b5b-bacc-fb35753a12c2-kube-api-access-dnvxq\") on node \"ip-10-0-131-232.ec2.internal\" DevicePath \"\""
Sep 17 19:27:18.968571 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:27:18.968531    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-79d9bd5fdb-9n775"]
Sep 17 19:27:18.969075 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:27:18.969051    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="34230187-dccf-4b5b-bacc-fb35753a12c2" containerName="kserve-container"
Sep 17 19:27:18.969075 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:27:18.969073    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="34230187-dccf-4b5b-bacc-fb35753a12c2" containerName="kserve-container"
Sep 17 19:27:18.969303 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:27:18.969092    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9b932ceb-d635-4910-b20f-3ca89ddc6acc" containerName="storage-initializer"
Sep 17 19:27:18.969303 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:27:18.969100    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="9b932ceb-d635-4910-b20f-3ca89ddc6acc" containerName="storage-initializer"
Sep 17 19:27:18.969303 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:27:18.969110    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="34230187-dccf-4b5b-bacc-fb35753a12c2" containerName="kube-rbac-proxy"
Sep 17 19:27:18.969303 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:27:18.969119    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="34230187-dccf-4b5b-bacc-fb35753a12c2" containerName="kube-rbac-proxy"
Sep 17 19:27:18.969303 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:27:18.969130    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="34230187-dccf-4b5b-bacc-fb35753a12c2" containerName="storage-initializer"
Sep 17 19:27:18.969303 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:27:18.969139    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="34230187-dccf-4b5b-bacc-fb35753a12c2" containerName="storage-initializer"
Sep 17 19:27:18.969303 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:27:18.969167    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9b932ceb-d635-4910-b20f-3ca89ddc6acc" containerName="kube-rbac-proxy"
Sep 17 19:27:18.969303 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:27:18.969175    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="9b932ceb-d635-4910-b20f-3ca89ddc6acc" containerName="kube-rbac-proxy"
Sep 17 19:27:18.969303 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:27:18.969187    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9b932ceb-d635-4910-b20f-3ca89ddc6acc" containerName="kserve-container"
Sep 17 19:27:18.969303 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:27:18.969195    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="9b932ceb-d635-4910-b20f-3ca89ddc6acc" containerName="kserve-container"
Sep 17 19:27:18.969303 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:27:18.969266    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="34230187-dccf-4b5b-bacc-fb35753a12c2" containerName="kube-rbac-proxy"
Sep 17 19:27:18.969303 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:27:18.969275    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="9b932ceb-d635-4910-b20f-3ca89ddc6acc" containerName="kube-rbac-proxy"
Sep 17 19:27:18.969303 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:27:18.969285    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="9b932ceb-d635-4910-b20f-3ca89ddc6acc" containerName="kserve-container"
Sep 17 19:27:18.969303 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:27:18.969294    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="34230187-dccf-4b5b-bacc-fb35753a12c2" containerName="kserve-container"
Sep 17 19:27:18.972701 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:27:18.972683    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-79d9bd5fdb-9n775"
Sep 17 19:27:18.975086 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:27:18.975065    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-promote-v2-predictor-serving-cert\""
Sep 17 19:27:18.975211 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:27:18.975084    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-promote-kube-rbac-proxy-sar-config\""
Sep 17 19:27:18.977154 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:27:18.977133    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-predictor-78b4c78fb9-nbg4n"]
Sep 17 19:27:18.980091 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:27:18.980069    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-78b4c78fb9-nbg4n"
Sep 17 19:27:18.982117 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:27:18.982098    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-promote-predictor-serving-cert\""
Sep 17 19:27:18.986812 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:27:18.986789    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-79d9bd5fdb-9n775"]
Sep 17 19:27:19.004747 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:27:19.004721    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-predictor-78b4c78fb9-nbg4n"]
Sep 17 19:27:19.080567 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:27:19.080478    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/86a097c7-53f1-4020-a1b3-6f6c96deec67-kube-rbac-proxy-sar-config\") pod \"isvc-canary-promote-v2-predictor-79d9bd5fdb-9n775\" (UID: \"86a097c7-53f1-4020-a1b3-6f6c96deec67\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-79d9bd5fdb-9n775"
Sep 17 19:27:19.080567 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:27:19.080517    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d05fdd99-92a5-4625-8904-140482c6a53d-cabundle-cert\") pod \"isvc-canary-promote-predictor-78b4c78fb9-nbg4n\" (UID: \"d05fdd99-92a5-4625-8904-140482c6a53d\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-78b4c78fb9-nbg4n"
Sep 17 19:27:19.080567 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:27:19.080535    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/86a097c7-53f1-4020-a1b3-6f6c96deec67-cabundle-cert\") pod \"isvc-canary-promote-v2-predictor-79d9bd5fdb-9n775\" (UID: \"86a097c7-53f1-4020-a1b3-6f6c96deec67\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-79d9bd5fdb-9n775"
Sep 17 19:27:19.080567 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:27:19.080560    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nt8bb\" (UniqueName: \"kubernetes.io/projected/d05fdd99-92a5-4625-8904-140482c6a53d-kube-api-access-nt8bb\") pod \"isvc-canary-promote-predictor-78b4c78fb9-nbg4n\" (UID: \"d05fdd99-92a5-4625-8904-140482c6a53d\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-78b4c78fb9-nbg4n"
Sep 17 19:27:19.080931 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:27:19.080586    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tz4gr\" (UniqueName: \"kubernetes.io/projected/86a097c7-53f1-4020-a1b3-6f6c96deec67-kube-api-access-tz4gr\") pod \"isvc-canary-promote-v2-predictor-79d9bd5fdb-9n775\" (UID: \"86a097c7-53f1-4020-a1b3-6f6c96deec67\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-79d9bd5fdb-9n775"
Sep 17 19:27:19.080931 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:27:19.080638    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d05fdd99-92a5-4625-8904-140482c6a53d-kserve-provision-location\") pod \"isvc-canary-promote-predictor-78b4c78fb9-nbg4n\" (UID: \"d05fdd99-92a5-4625-8904-140482c6a53d\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-78b4c78fb9-nbg4n"
Sep 17 19:27:19.080931 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:27:19.080658    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/86a097c7-53f1-4020-a1b3-6f6c96deec67-proxy-tls\") pod \"isvc-canary-promote-v2-predictor-79d9bd5fdb-9n775\" (UID: \"86a097c7-53f1-4020-a1b3-6f6c96deec67\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-79d9bd5fdb-9n775"
Sep 17 19:27:19.080931 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:27:19.080676    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/86a097c7-53f1-4020-a1b3-6f6c96deec67-kserve-provision-location\") pod \"isvc-canary-promote-v2-predictor-79d9bd5fdb-9n775\" (UID: \"86a097c7-53f1-4020-a1b3-6f6c96deec67\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-79d9bd5fdb-9n775"
Sep 17 19:27:19.080931 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:27:19.080708    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d05fdd99-92a5-4625-8904-140482c6a53d-proxy-tls\") pod \"isvc-canary-promote-predictor-78b4c78fb9-nbg4n\" (UID: \"d05fdd99-92a5-4625-8904-140482c6a53d\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-78b4c78fb9-nbg4n"
Sep 17 19:27:19.080931 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:27:19.080739    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d05fdd99-92a5-4625-8904-140482c6a53d-kube-rbac-proxy-sar-config\") pod \"isvc-canary-promote-predictor-78b4c78fb9-nbg4n\" (UID: \"d05fdd99-92a5-4625-8904-140482c6a53d\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-78b4c78fb9-nbg4n"
Sep 17 19:27:19.127301 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:27:19.127258    2582 generic.go:356] "Generic (PLEG): container finished" podID="34230187-dccf-4b5b-bacc-fb35753a12c2" containerID="b06a53fb24bde27b59dc378d36d273ded43739eaf246a641673cdc83f3194562" exitCode=0
Sep 17 19:27:19.127468 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:27:19.127308    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-bfcdb48dd-p6nxc"
Sep 17 19:27:19.127468 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:27:19.127314    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-bfcdb48dd-p6nxc" event={"ID":"34230187-dccf-4b5b-bacc-fb35753a12c2","Type":"ContainerDied","Data":"b06a53fb24bde27b59dc378d36d273ded43739eaf246a641673cdc83f3194562"}
Sep 17 19:27:19.127468 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:27:19.127376    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-bfcdb48dd-p6nxc" event={"ID":"34230187-dccf-4b5b-bacc-fb35753a12c2","Type":"ContainerDied","Data":"d53619fffb2de1e90da79ff5b793fd1a4543f063a8668b30dde7b33ded0a9561"}
Sep 17 19:27:19.127468 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:27:19.127404    2582 scope.go:117] "RemoveContainer" containerID="78b47abc71851974278a5b58c7bebf21a0d4536dc5ddbb18c2b9f162ea713589"
Sep 17 19:27:19.136917 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:27:19.136897    2582 scope.go:117] "RemoveContainer" containerID="b06a53fb24bde27b59dc378d36d273ded43739eaf246a641673cdc83f3194562"
Sep 17 19:27:19.145993 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:27:19.145937    2582 scope.go:117] "RemoveContainer" containerID="163fcc73f7c1bc38f0dcbe3c2429608334edfe8e2615c4f2ddb71c9bbe182245"
Sep 17 19:27:19.151062 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:27:19.151039    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-v2-predictor-bfcdb48dd-p6nxc"]
Sep 17 19:27:19.153603 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:27:19.153582    2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-v2-predictor-bfcdb48dd-p6nxc"]
Sep 17 19:27:19.155308 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:27:19.155291    2582 scope.go:117] "RemoveContainer" containerID="78b47abc71851974278a5b58c7bebf21a0d4536dc5ddbb18c2b9f162ea713589"
Sep 17 19:27:19.155628 ip-10-0-131-232 kubenswrapper[2582]: E0917 19:27:19.155610    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"78b47abc71851974278a5b58c7bebf21a0d4536dc5ddbb18c2b9f162ea713589\": container with ID starting with 78b47abc71851974278a5b58c7bebf21a0d4536dc5ddbb18c2b9f162ea713589 not found: ID does not exist" containerID="78b47abc71851974278a5b58c7bebf21a0d4536dc5ddbb18c2b9f162ea713589"
Sep 17 19:27:19.155679 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:27:19.155637    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"78b47abc71851974278a5b58c7bebf21a0d4536dc5ddbb18c2b9f162ea713589"} err="failed to get container status \"78b47abc71851974278a5b58c7bebf21a0d4536dc5ddbb18c2b9f162ea713589\": rpc error: code = NotFound desc = could not find container \"78b47abc71851974278a5b58c7bebf21a0d4536dc5ddbb18c2b9f162ea713589\": container with ID starting with 78b47abc71851974278a5b58c7bebf21a0d4536dc5ddbb18c2b9f162ea713589 not found: ID does not exist"
Sep 17 19:27:19.155679 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:27:19.155653    2582 scope.go:117] "RemoveContainer" containerID="b06a53fb24bde27b59dc378d36d273ded43739eaf246a641673cdc83f3194562"
Sep 17 19:27:19.155924 ip-10-0-131-232 kubenswrapper[2582]: E0917 19:27:19.155907    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b06a53fb24bde27b59dc378d36d273ded43739eaf246a641673cdc83f3194562\": container with ID starting with b06a53fb24bde27b59dc378d36d273ded43739eaf246a641673cdc83f3194562 not found: ID does not exist" containerID="b06a53fb24bde27b59dc378d36d273ded43739eaf246a641673cdc83f3194562"
Sep 17 19:27:19.155988 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:27:19.155929    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b06a53fb24bde27b59dc378d36d273ded43739eaf246a641673cdc83f3194562"} err="failed to get container status \"b06a53fb24bde27b59dc378d36d273ded43739eaf246a641673cdc83f3194562\": rpc error: code = NotFound desc = could not find container \"b06a53fb24bde27b59dc378d36d273ded43739eaf246a641673cdc83f3194562\": container with ID starting with b06a53fb24bde27b59dc378d36d273ded43739eaf246a641673cdc83f3194562 not found: ID does not exist"
Sep 17 19:27:19.155988 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:27:19.155943    2582 scope.go:117] "RemoveContainer" containerID="163fcc73f7c1bc38f0dcbe3c2429608334edfe8e2615c4f2ddb71c9bbe182245"
Sep 17 19:27:19.156190 ip-10-0-131-232 kubenswrapper[2582]: E0917 19:27:19.156174    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"163fcc73f7c1bc38f0dcbe3c2429608334edfe8e2615c4f2ddb71c9bbe182245\": container with ID starting with 163fcc73f7c1bc38f0dcbe3c2429608334edfe8e2615c4f2ddb71c9bbe182245 not found: ID does not exist" containerID="163fcc73f7c1bc38f0dcbe3c2429608334edfe8e2615c4f2ddb71c9bbe182245"
Sep 17 19:27:19.156230 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:27:19.156194    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"163fcc73f7c1bc38f0dcbe3c2429608334edfe8e2615c4f2ddb71c9bbe182245"} err="failed to get container status \"163fcc73f7c1bc38f0dcbe3c2429608334edfe8e2615c4f2ddb71c9bbe182245\": rpc error: code = NotFound desc = could not find container \"163fcc73f7c1bc38f0dcbe3c2429608334edfe8e2615c4f2ddb71c9bbe182245\": container with ID starting with 163fcc73f7c1bc38f0dcbe3c2429608334edfe8e2615c4f2ddb71c9bbe182245 not found: ID does not exist"
Sep 17 19:27:19.181539 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:27:19.181510    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/86a097c7-53f1-4020-a1b3-6f6c96deec67-kube-rbac-proxy-sar-config\") pod \"isvc-canary-promote-v2-predictor-79d9bd5fdb-9n775\" (UID: \"86a097c7-53f1-4020-a1b3-6f6c96deec67\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-79d9bd5fdb-9n775"
Sep 17 19:27:19.181539 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:27:19.181544    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d05fdd99-92a5-4625-8904-140482c6a53d-cabundle-cert\") pod \"isvc-canary-promote-predictor-78b4c78fb9-nbg4n\" (UID: \"d05fdd99-92a5-4625-8904-140482c6a53d\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-78b4c78fb9-nbg4n"
Sep 17 19:27:19.181780 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:27:19.181559    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/86a097c7-53f1-4020-a1b3-6f6c96deec67-cabundle-cert\") pod \"isvc-canary-promote-v2-predictor-79d9bd5fdb-9n775\" (UID: \"86a097c7-53f1-4020-a1b3-6f6c96deec67\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-79d9bd5fdb-9n775"
Sep 17 19:27:19.181780 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:27:19.181760    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nt8bb\" (UniqueName: \"kubernetes.io/projected/d05fdd99-92a5-4625-8904-140482c6a53d-kube-api-access-nt8bb\") pod \"isvc-canary-promote-predictor-78b4c78fb9-nbg4n\" (UID: \"d05fdd99-92a5-4625-8904-140482c6a53d\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-78b4c78fb9-nbg4n"
Sep 17 19:27:19.181913 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:27:19.181789    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tz4gr\" (UniqueName: \"kubernetes.io/projected/86a097c7-53f1-4020-a1b3-6f6c96deec67-kube-api-access-tz4gr\") pod \"isvc-canary-promote-v2-predictor-79d9bd5fdb-9n775\" (UID: \"86a097c7-53f1-4020-a1b3-6f6c96deec67\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-79d9bd5fdb-9n775"
Sep 17 19:27:19.181913 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:27:19.181860    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d05fdd99-92a5-4625-8904-140482c6a53d-kserve-provision-location\") pod \"isvc-canary-promote-predictor-78b4c78fb9-nbg4n\" (UID: \"d05fdd99-92a5-4625-8904-140482c6a53d\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-78b4c78fb9-nbg4n"
Sep 17 19:27:19.181913 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:27:19.181894    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/86a097c7-53f1-4020-a1b3-6f6c96deec67-proxy-tls\") pod \"isvc-canary-promote-v2-predictor-79d9bd5fdb-9n775\" (UID: \"86a097c7-53f1-4020-a1b3-6f6c96deec67\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-79d9bd5fdb-9n775"
Sep 17 19:27:19.182073 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:27:19.181922    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/86a097c7-53f1-4020-a1b3-6f6c96deec67-kserve-provision-location\") pod \"isvc-canary-promote-v2-predictor-79d9bd5fdb-9n775\" (UID: \"86a097c7-53f1-4020-a1b3-6f6c96deec67\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-79d9bd5fdb-9n775"
Sep 17 19:27:19.182073 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:27:19.181971    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d05fdd99-92a5-4625-8904-140482c6a53d-proxy-tls\") pod \"isvc-canary-promote-predictor-78b4c78fb9-nbg4n\" (UID: \"d05fdd99-92a5-4625-8904-140482c6a53d\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-78b4c78fb9-nbg4n"
Sep 17 19:27:19.182073 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:27:19.181997    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d05fdd99-92a5-4625-8904-140482c6a53d-kube-rbac-proxy-sar-config\") pod \"isvc-canary-promote-predictor-78b4c78fb9-nbg4n\" (UID: \"d05fdd99-92a5-4625-8904-140482c6a53d\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-78b4c78fb9-nbg4n"
Sep 17 19:27:19.182073 ip-10-0-131-232 kubenswrapper[2582]: E0917 19:27:19.182049    2582 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-serving-cert: secret "isvc-canary-promote-v2-predictor-serving-cert" not found
Sep 17 19:27:19.182273 ip-10-0-131-232 kubenswrapper[2582]: E0917 19:27:19.182111    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/86a097c7-53f1-4020-a1b3-6f6c96deec67-proxy-tls podName:86a097c7-53f1-4020-a1b3-6f6c96deec67 nodeName:}" failed. No retries permitted until 2026-09-17 19:27:19.682092783 +0000 UTC m=+1255.453763830 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/86a097c7-53f1-4020-a1b3-6f6c96deec67-proxy-tls") pod "isvc-canary-promote-v2-predictor-79d9bd5fdb-9n775" (UID: "86a097c7-53f1-4020-a1b3-6f6c96deec67") : secret "isvc-canary-promote-v2-predictor-serving-cert" not found
Sep 17 19:27:19.182340 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:27:19.182289    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d05fdd99-92a5-4625-8904-140482c6a53d-cabundle-cert\") pod \"isvc-canary-promote-predictor-78b4c78fb9-nbg4n\" (UID: \"d05fdd99-92a5-4625-8904-140482c6a53d\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-78b4c78fb9-nbg4n"
Sep 17 19:27:19.182340 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:27:19.182331    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/86a097c7-53f1-4020-a1b3-6f6c96deec67-kube-rbac-proxy-sar-config\") pod \"isvc-canary-promote-v2-predictor-79d9bd5fdb-9n775\" (UID: \"86a097c7-53f1-4020-a1b3-6f6c96deec67\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-79d9bd5fdb-9n775"
Sep 17 19:27:19.182429 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:27:19.182358    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/86a097c7-53f1-4020-a1b3-6f6c96deec67-kserve-provision-location\") pod \"isvc-canary-promote-v2-predictor-79d9bd5fdb-9n775\" (UID: \"86a097c7-53f1-4020-a1b3-6f6c96deec67\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-79d9bd5fdb-9n775"
Sep 17 19:27:19.182464 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:27:19.182450    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/86a097c7-53f1-4020-a1b3-6f6c96deec67-cabundle-cert\") pod \"isvc-canary-promote-v2-predictor-79d9bd5fdb-9n775\" (UID: \"86a097c7-53f1-4020-a1b3-6f6c96deec67\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-79d9bd5fdb-9n775"
Sep 17 19:27:19.182559 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:27:19.182539    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d05fdd99-92a5-4625-8904-140482c6a53d-kserve-provision-location\") pod \"isvc-canary-promote-predictor-78b4c78fb9-nbg4n\" (UID: \"d05fdd99-92a5-4625-8904-140482c6a53d\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-78b4c78fb9-nbg4n"
Sep 17 19:27:19.182600 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:27:19.182586    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d05fdd99-92a5-4625-8904-140482c6a53d-kube-rbac-proxy-sar-config\") pod \"isvc-canary-promote-predictor-78b4c78fb9-nbg4n\" (UID: \"d05fdd99-92a5-4625-8904-140482c6a53d\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-78b4c78fb9-nbg4n"
Sep 17 19:27:19.184974 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:27:19.184955    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d05fdd99-92a5-4625-8904-140482c6a53d-proxy-tls\") pod \"isvc-canary-promote-predictor-78b4c78fb9-nbg4n\" (UID: \"d05fdd99-92a5-4625-8904-140482c6a53d\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-78b4c78fb9-nbg4n"
Sep 17 19:27:19.192599 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:27:19.192571    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tz4gr\" (UniqueName: \"kubernetes.io/projected/86a097c7-53f1-4020-a1b3-6f6c96deec67-kube-api-access-tz4gr\") pod \"isvc-canary-promote-v2-predictor-79d9bd5fdb-9n775\" (UID: \"86a097c7-53f1-4020-a1b3-6f6c96deec67\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-79d9bd5fdb-9n775"
Sep 17 19:27:19.193197 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:27:19.193177    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nt8bb\" (UniqueName: \"kubernetes.io/projected/d05fdd99-92a5-4625-8904-140482c6a53d-kube-api-access-nt8bb\") pod \"isvc-canary-promote-predictor-78b4c78fb9-nbg4n\" (UID: \"d05fdd99-92a5-4625-8904-140482c6a53d\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-78b4c78fb9-nbg4n"
Sep 17 19:27:19.295123 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:27:19.295081    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-78b4c78fb9-nbg4n"
Sep 17 19:27:19.572013 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:27:19.571991    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-predictor-78b4c78fb9-nbg4n"]
Sep 17 19:27:19.686338 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:27:19.686301    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/86a097c7-53f1-4020-a1b3-6f6c96deec67-proxy-tls\") pod \"isvc-canary-promote-v2-predictor-79d9bd5fdb-9n775\" (UID: \"86a097c7-53f1-4020-a1b3-6f6c96deec67\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-79d9bd5fdb-9n775"
Sep 17 19:27:19.688933 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:27:19.688902    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/86a097c7-53f1-4020-a1b3-6f6c96deec67-proxy-tls\") pod \"isvc-canary-promote-v2-predictor-79d9bd5fdb-9n775\" (UID: \"86a097c7-53f1-4020-a1b3-6f6c96deec67\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-79d9bd5fdb-9n775"
Sep 17 19:27:19.887063 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:27:19.887030    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-79d9bd5fdb-9n775"
Sep 17 19:27:20.133712 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:27:20.133678    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-78b4c78fb9-nbg4n" event={"ID":"d05fdd99-92a5-4625-8904-140482c6a53d","Type":"ContainerStarted","Data":"414f07cf51059c5d23f1fd303bcb615f3f27416267201c1d42edba9acb55cd26"}
Sep 17 19:27:20.133925 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:27:20.133743    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-78b4c78fb9-nbg4n" event={"ID":"d05fdd99-92a5-4625-8904-140482c6a53d","Type":"ContainerStarted","Data":"80891c16675e4ec2487a4a14d19b424330f16ed295f18368bf47a6cd9622d566"}
Sep 17 19:27:20.171519 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:27:20.171492    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-79d9bd5fdb-9n775"]
Sep 17 19:27:20.173963 ip-10-0-131-232 kubenswrapper[2582]: W0917 19:27:20.173928    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod86a097c7_53f1_4020_a1b3_6f6c96deec67.slice/crio-3ca1f51660248e283b46e78f369489dcd8428e45eac74f8603af126f281fbd2f WatchSource:0}: Error finding container 3ca1f51660248e283b46e78f369489dcd8428e45eac74f8603af126f281fbd2f: Status 404 returned error can't find the container with id 3ca1f51660248e283b46e78f369489dcd8428e45eac74f8603af126f281fbd2f
Sep 17 19:27:20.769124 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:27:20.769086    2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="34230187-dccf-4b5b-bacc-fb35753a12c2" path="/var/lib/kubelet/pods/34230187-dccf-4b5b-bacc-fb35753a12c2/volumes"
Sep 17 19:27:21.140759 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:27:21.140722    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-79d9bd5fdb-9n775" event={"ID":"86a097c7-53f1-4020-a1b3-6f6c96deec67","Type":"ContainerStarted","Data":"f97e49a1759309034aa18d5e469299c3f97b43829454c2cacb6598fe443c5a4c"}
Sep 17 19:27:21.140947 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:27:21.140799    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-79d9bd5fdb-9n775" event={"ID":"86a097c7-53f1-4020-a1b3-6f6c96deec67","Type":"ContainerStarted","Data":"3ca1f51660248e283b46e78f369489dcd8428e45eac74f8603af126f281fbd2f"}
Sep 17 19:27:24.154513 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:27:24.154474    2582 generic.go:356] "Generic (PLEG): container finished" podID="d05fdd99-92a5-4625-8904-140482c6a53d" containerID="414f07cf51059c5d23f1fd303bcb615f3f27416267201c1d42edba9acb55cd26" exitCode=0
Sep 17 19:27:24.154888 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:27:24.154549    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-78b4c78fb9-nbg4n" event={"ID":"d05fdd99-92a5-4625-8904-140482c6a53d","Type":"ContainerDied","Data":"414f07cf51059c5d23f1fd303bcb615f3f27416267201c1d42edba9acb55cd26"}
Sep 17 19:27:25.159952 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:27:25.159922    2582 generic.go:356] "Generic (PLEG): container finished" podID="86a097c7-53f1-4020-a1b3-6f6c96deec67" containerID="f97e49a1759309034aa18d5e469299c3f97b43829454c2cacb6598fe443c5a4c" exitCode=0
Sep 17 19:27:25.160430 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:27:25.160000    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-79d9bd5fdb-9n775" event={"ID":"86a097c7-53f1-4020-a1b3-6f6c96deec67","Type":"ContainerDied","Data":"f97e49a1759309034aa18d5e469299c3f97b43829454c2cacb6598fe443c5a4c"}
Sep 17 19:27:25.162144 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:27:25.162123    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-78b4c78fb9-nbg4n" event={"ID":"d05fdd99-92a5-4625-8904-140482c6a53d","Type":"ContainerStarted","Data":"f05665f54ba2cacac87861a78c5c144c229e224be1b1bd181d71f6d655d1fb6c"}
Sep 17 19:27:25.162249 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:27:25.162169    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-78b4c78fb9-nbg4n" event={"ID":"d05fdd99-92a5-4625-8904-140482c6a53d","Type":"ContainerStarted","Data":"4664dff98f3cf14ddc27ba231990e7a03ce7e3849ea641b48302de94e1be7317"}
Sep 17 19:27:25.162362 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:27:25.162344    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-78b4c78fb9-nbg4n"
Sep 17 19:27:25.162404 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:27:25.162399    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-78b4c78fb9-nbg4n"
Sep 17 19:27:25.163836 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:27:25.163795    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-78b4c78fb9-nbg4n" podUID="d05fdd99-92a5-4625-8904-140482c6a53d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.36:8080: connect: connection refused"
Sep 17 19:27:25.197539 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:27:25.197490    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-78b4c78fb9-nbg4n" podStartSLOduration=3.332164538 podStartE2EDuration="7.197477382s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.865312844s" isStatefulPod=false podCreationTimestamp="2026-09-17 19:27:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 19:27:25.195327721 +0000 UTC m=+1260.966998799" watchObservedRunningTime="2026-09-17 19:27:25.197477382 +0000 UTC m=+1260.969148443"
Sep 17 19:27:26.168494 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:27:26.168461    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-79d9bd5fdb-9n775" event={"ID":"86a097c7-53f1-4020-a1b3-6f6c96deec67","Type":"ContainerStarted","Data":"8ab1f050565ef8d3e8684d8a4ba1ef35719f6a4c139d14e6906374cbdd1e932c"}
Sep 17 19:27:26.168963 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:27:26.168515    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-79d9bd5fdb-9n775" event={"ID":"86a097c7-53f1-4020-a1b3-6f6c96deec67","Type":"ContainerStarted","Data":"4d0005fae7b19da455bb619cc2d87af6287a100e7065319f0cab66a02cb1d9c6"}
Sep 17 19:27:26.168963 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:27:26.168674    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-78b4c78fb9-nbg4n" podUID="d05fdd99-92a5-4625-8904-140482c6a53d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.36:8080: connect: connection refused"
Sep 17 19:27:26.168963 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:27:26.168772    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-79d9bd5fdb-9n775"
Sep 17 19:27:26.190816 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:27:26.190774    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-79d9bd5fdb-9n775" podStartSLOduration=4.201104886 podStartE2EDuration="8.190758272s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.989653386s" isStatefulPod=false podCreationTimestamp="2026-09-17 19:27:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 19:27:26.188524977 +0000 UTC m=+1261.960196037" watchObservedRunningTime="2026-09-17 19:27:26.190758272 +0000 UTC m=+1261.962429332"
Sep 17 19:27:27.173566 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:27:27.173529    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-79d9bd5fdb-9n775"
Sep 17 19:27:27.174991 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:27:27.174963    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-79d9bd5fdb-9n775" podUID="86a097c7-53f1-4020-a1b3-6f6c96deec67" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.35:8080: connect: connection refused"
Sep 17 19:27:28.176910 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:27:28.176871    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-79d9bd5fdb-9n775" podUID="86a097c7-53f1-4020-a1b3-6f6c96deec67" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.35:8080: connect: connection refused"
Sep 17 19:27:31.173728 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:27:31.173703    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-78b4c78fb9-nbg4n"
Sep 17 19:27:31.174304 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:27:31.174278    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-78b4c78fb9-nbg4n" podUID="d05fdd99-92a5-4625-8904-140482c6a53d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.36:8080: connect: connection refused"
Sep 17 19:27:33.181582 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:27:33.181546    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-79d9bd5fdb-9n775"
Sep 17 19:27:33.182136 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:27:33.182108    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-79d9bd5fdb-9n775" podUID="86a097c7-53f1-4020-a1b3-6f6c96deec67" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.35:8080: connect: connection refused"
Sep 17 19:27:41.175017 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:27:41.174937    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-78b4c78fb9-nbg4n" podUID="d05fdd99-92a5-4625-8904-140482c6a53d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.36:8080: connect: connection refused"
Sep 17 19:27:43.182152 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:27:43.182108    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-79d9bd5fdb-9n775" podUID="86a097c7-53f1-4020-a1b3-6f6c96deec67" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.35:8080: connect: connection refused"
Sep 17 19:27:51.174956 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:27:51.174914    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-78b4c78fb9-nbg4n" podUID="d05fdd99-92a5-4625-8904-140482c6a53d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.36:8080: connect: connection refused"
Sep 17 19:27:53.182480 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:27:53.182443    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-79d9bd5fdb-9n775" podUID="86a097c7-53f1-4020-a1b3-6f6c96deec67" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.35:8080: connect: connection refused"
Sep 17 19:28:01.174169 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:28:01.174120    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-78b4c78fb9-nbg4n" podUID="d05fdd99-92a5-4625-8904-140482c6a53d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.36:8080: connect: connection refused"
Sep 17 19:28:03.183009 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:28:03.182966    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-79d9bd5fdb-9n775" podUID="86a097c7-53f1-4020-a1b3-6f6c96deec67" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.35:8080: connect: connection refused"
Sep 17 19:28:11.175150 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:28:11.175112    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-78b4c78fb9-nbg4n" podUID="d05fdd99-92a5-4625-8904-140482c6a53d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.36:8080: connect: connection refused"
Sep 17 19:28:13.182252 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:28:13.182217    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-79d9bd5fdb-9n775" podUID="86a097c7-53f1-4020-a1b3-6f6c96deec67" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.35:8080: connect: connection refused"
Sep 17 19:28:21.174493 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:28:21.174457    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-78b4c78fb9-nbg4n" podUID="d05fdd99-92a5-4625-8904-140482c6a53d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.36:8080: connect: connection refused"
Sep 17 19:28:23.183021 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:28:23.182984    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-79d9bd5fdb-9n775" podUID="86a097c7-53f1-4020-a1b3-6f6c96deec67" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.35:8080: connect: connection refused"
Sep 17 19:28:31.175019 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:28:31.174983    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-78b4c78fb9-nbg4n" podUID="d05fdd99-92a5-4625-8904-140482c6a53d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.36:8080: connect: connection refused"
Sep 17 19:28:33.182498 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:28:33.182456    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-79d9bd5fdb-9n775" podUID="86a097c7-53f1-4020-a1b3-6f6c96deec67" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.35:8080: connect: connection refused"
Sep 17 19:28:41.174774 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:28:41.174743    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-78b4c78fb9-nbg4n"
Sep 17 19:28:41.764014 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:28:41.763982    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-79d9bd5fdb-9n775"
Sep 17 19:28:49.029469 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:28:49.029375    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-predictor-78b4c78fb9-nbg4n"]
Sep 17 19:28:49.029920 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:28:49.029653    2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-78b4c78fb9-nbg4n" podUID="d05fdd99-92a5-4625-8904-140482c6a53d" containerName="kserve-container" containerID="cri-o://4664dff98f3cf14ddc27ba231990e7a03ce7e3849ea641b48302de94e1be7317" gracePeriod=30
Sep 17 19:28:49.029920 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:28:49.029685    2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-78b4c78fb9-nbg4n" podUID="d05fdd99-92a5-4625-8904-140482c6a53d" containerName="kube-rbac-proxy" containerID="cri-o://f05665f54ba2cacac87861a78c5c144c229e224be1b1bd181d71f6d655d1fb6c" gracePeriod=30
Sep 17 19:28:49.522966 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:28:49.522931    2582 generic.go:356] "Generic (PLEG): container finished" podID="d05fdd99-92a5-4625-8904-140482c6a53d" containerID="f05665f54ba2cacac87861a78c5c144c229e224be1b1bd181d71f6d655d1fb6c" exitCode=2
Sep 17 19:28:49.523126 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:28:49.522999    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-78b4c78fb9-nbg4n" event={"ID":"d05fdd99-92a5-4625-8904-140482c6a53d","Type":"ContainerDied","Data":"f05665f54ba2cacac87861a78c5c144c229e224be1b1bd181d71f6d655d1fb6c"}
Sep 17 19:28:51.169678 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:28:51.169631    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-78b4c78fb9-nbg4n" podUID="d05fdd99-92a5-4625-8904-140482c6a53d" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.36:8643/healthz\": dial tcp 10.134.0.36:8643: connect: connection refused"
Sep 17 19:28:51.175052 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:28:51.175022    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-78b4c78fb9-nbg4n" podUID="d05fdd99-92a5-4625-8904-140482c6a53d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.36:8080: connect: connection refused"
Sep 17 19:28:53.668953 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:28:53.668932    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-78b4c78fb9-nbg4n"
Sep 17 19:28:53.726219 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:28:53.726150    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d05fdd99-92a5-4625-8904-140482c6a53d-kube-rbac-proxy-sar-config\") pod \"d05fdd99-92a5-4625-8904-140482c6a53d\" (UID: \"d05fdd99-92a5-4625-8904-140482c6a53d\") "
Sep 17 19:28:53.726219 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:28:53.726203    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d05fdd99-92a5-4625-8904-140482c6a53d-cabundle-cert\") pod \"d05fdd99-92a5-4625-8904-140482c6a53d\" (UID: \"d05fdd99-92a5-4625-8904-140482c6a53d\") "
Sep 17 19:28:53.726414 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:28:53.726243    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d05fdd99-92a5-4625-8904-140482c6a53d-kserve-provision-location\") pod \"d05fdd99-92a5-4625-8904-140482c6a53d\" (UID: \"d05fdd99-92a5-4625-8904-140482c6a53d\") "
Sep 17 19:28:53.726414 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:28:53.726284    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-nt8bb\" (UniqueName: \"kubernetes.io/projected/d05fdd99-92a5-4625-8904-140482c6a53d-kube-api-access-nt8bb\") pod \"d05fdd99-92a5-4625-8904-140482c6a53d\" (UID: \"d05fdd99-92a5-4625-8904-140482c6a53d\") "
Sep 17 19:28:53.726414 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:28:53.726317    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d05fdd99-92a5-4625-8904-140482c6a53d-proxy-tls\") pod \"d05fdd99-92a5-4625-8904-140482c6a53d\" (UID: \"d05fdd99-92a5-4625-8904-140482c6a53d\") "
Sep 17 19:28:53.726578 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:28:53.726556    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d05fdd99-92a5-4625-8904-140482c6a53d-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "d05fdd99-92a5-4625-8904-140482c6a53d" (UID: "d05fdd99-92a5-4625-8904-140482c6a53d"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 19:28:53.726641 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:28:53.726584    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d05fdd99-92a5-4625-8904-140482c6a53d-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "d05fdd99-92a5-4625-8904-140482c6a53d" (UID: "d05fdd99-92a5-4625-8904-140482c6a53d"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 19:28:53.726641 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:28:53.726591    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d05fdd99-92a5-4625-8904-140482c6a53d-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "d05fdd99-92a5-4625-8904-140482c6a53d" (UID: "d05fdd99-92a5-4625-8904-140482c6a53d"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 19:28:53.728472 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:28:53.728442    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d05fdd99-92a5-4625-8904-140482c6a53d-kube-api-access-nt8bb" (OuterVolumeSpecName: "kube-api-access-nt8bb") pod "d05fdd99-92a5-4625-8904-140482c6a53d" (UID: "d05fdd99-92a5-4625-8904-140482c6a53d"). InnerVolumeSpecName "kube-api-access-nt8bb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 19:28:53.728472 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:28:53.728445    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d05fdd99-92a5-4625-8904-140482c6a53d-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "d05fdd99-92a5-4625-8904-140482c6a53d" (UID: "d05fdd99-92a5-4625-8904-140482c6a53d"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 19:28:53.826995 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:28:53.826950    2582 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d05fdd99-92a5-4625-8904-140482c6a53d-kserve-provision-location\") on node \"ip-10-0-131-232.ec2.internal\" DevicePath \"\""
Sep 17 19:28:53.826995 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:28:53.826989    2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nt8bb\" (UniqueName: \"kubernetes.io/projected/d05fdd99-92a5-4625-8904-140482c6a53d-kube-api-access-nt8bb\") on node \"ip-10-0-131-232.ec2.internal\" DevicePath \"\""
Sep 17 19:28:53.826995 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:28:53.827000    2582 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d05fdd99-92a5-4625-8904-140482c6a53d-proxy-tls\") on node \"ip-10-0-131-232.ec2.internal\" DevicePath \"\""
Sep 17 19:28:53.826995 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:28:53.827009    2582 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d05fdd99-92a5-4625-8904-140482c6a53d-kube-rbac-proxy-sar-config\") on node \"ip-10-0-131-232.ec2.internal\" DevicePath \"\""
Sep 17 19:28:53.827244 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:28:53.827018    2582 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d05fdd99-92a5-4625-8904-140482c6a53d-cabundle-cert\") on node \"ip-10-0-131-232.ec2.internal\" DevicePath \"\""
Sep 17 19:28:54.547377 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:28:54.547345    2582 generic.go:356] "Generic (PLEG): container finished" podID="d05fdd99-92a5-4625-8904-140482c6a53d" containerID="4664dff98f3cf14ddc27ba231990e7a03ce7e3849ea641b48302de94e1be7317" exitCode=0
Sep 17 19:28:54.547545 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:28:54.547398    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-78b4c78fb9-nbg4n" event={"ID":"d05fdd99-92a5-4625-8904-140482c6a53d","Type":"ContainerDied","Data":"4664dff98f3cf14ddc27ba231990e7a03ce7e3849ea641b48302de94e1be7317"}
Sep 17 19:28:54.547545 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:28:54.547402    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-78b4c78fb9-nbg4n"
Sep 17 19:28:54.547545 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:28:54.547446    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-78b4c78fb9-nbg4n" event={"ID":"d05fdd99-92a5-4625-8904-140482c6a53d","Type":"ContainerDied","Data":"80891c16675e4ec2487a4a14d19b424330f16ed295f18368bf47a6cd9622d566"}
Sep 17 19:28:54.547545 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:28:54.547477    2582 scope.go:117] "RemoveContainer" containerID="f05665f54ba2cacac87861a78c5c144c229e224be1b1bd181d71f6d655d1fb6c"
Sep 17 19:28:54.555959 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:28:54.555941    2582 scope.go:117] "RemoveContainer" containerID="4664dff98f3cf14ddc27ba231990e7a03ce7e3849ea641b48302de94e1be7317"
Sep 17 19:28:54.563061 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:28:54.563045    2582 scope.go:117] "RemoveContainer" containerID="414f07cf51059c5d23f1fd303bcb615f3f27416267201c1d42edba9acb55cd26"
Sep 17 19:28:54.569708 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:28:54.569688    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-predictor-78b4c78fb9-nbg4n"]
Sep 17 19:28:54.570858 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:28:54.570843    2582 scope.go:117] "RemoveContainer" containerID="f05665f54ba2cacac87861a78c5c144c229e224be1b1bd181d71f6d655d1fb6c"
Sep 17 19:28:54.571115 ip-10-0-131-232 kubenswrapper[2582]: E0917 19:28:54.571097    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f05665f54ba2cacac87861a78c5c144c229e224be1b1bd181d71f6d655d1fb6c\": container with ID starting with f05665f54ba2cacac87861a78c5c144c229e224be1b1bd181d71f6d655d1fb6c not found: ID does not exist" containerID="f05665f54ba2cacac87861a78c5c144c229e224be1b1bd181d71f6d655d1fb6c"
Sep 17 19:28:54.571173 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:28:54.571123    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f05665f54ba2cacac87861a78c5c144c229e224be1b1bd181d71f6d655d1fb6c"} err="failed to get container status \"f05665f54ba2cacac87861a78c5c144c229e224be1b1bd181d71f6d655d1fb6c\": rpc error: code = NotFound desc = could not find container \"f05665f54ba2cacac87861a78c5c144c229e224be1b1bd181d71f6d655d1fb6c\": container with ID starting with f05665f54ba2cacac87861a78c5c144c229e224be1b1bd181d71f6d655d1fb6c not found: ID does not exist"
Sep 17 19:28:54.571173 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:28:54.571140    2582 scope.go:117] "RemoveContainer" containerID="4664dff98f3cf14ddc27ba231990e7a03ce7e3849ea641b48302de94e1be7317"
Sep 17 19:28:54.571391 ip-10-0-131-232 kubenswrapper[2582]: E0917 19:28:54.571374    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4664dff98f3cf14ddc27ba231990e7a03ce7e3849ea641b48302de94e1be7317\": container with ID starting with 4664dff98f3cf14ddc27ba231990e7a03ce7e3849ea641b48302de94e1be7317 not found: ID does not exist" containerID="4664dff98f3cf14ddc27ba231990e7a03ce7e3849ea641b48302de94e1be7317"
Sep 17 19:28:54.571430 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:28:54.571397    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4664dff98f3cf14ddc27ba231990e7a03ce7e3849ea641b48302de94e1be7317"} err="failed to get container status \"4664dff98f3cf14ddc27ba231990e7a03ce7e3849ea641b48302de94e1be7317\": rpc error: code = NotFound desc = could not find container \"4664dff98f3cf14ddc27ba231990e7a03ce7e3849ea641b48302de94e1be7317\": container with ID starting with 4664dff98f3cf14ddc27ba231990e7a03ce7e3849ea641b48302de94e1be7317 not found: ID does not exist"
Sep 17 19:28:54.571430 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:28:54.571411    2582 scope.go:117] "RemoveContainer" containerID="414f07cf51059c5d23f1fd303bcb615f3f27416267201c1d42edba9acb55cd26"
Sep 17 19:28:54.571647 ip-10-0-131-232 kubenswrapper[2582]: E0917 19:28:54.571630    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"414f07cf51059c5d23f1fd303bcb615f3f27416267201c1d42edba9acb55cd26\": container with ID starting with 414f07cf51059c5d23f1fd303bcb615f3f27416267201c1d42edba9acb55cd26 not found: ID does not exist" containerID="414f07cf51059c5d23f1fd303bcb615f3f27416267201c1d42edba9acb55cd26"
Sep 17 19:28:54.571711 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:28:54.571649    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"414f07cf51059c5d23f1fd303bcb615f3f27416267201c1d42edba9acb55cd26"} err="failed to get container status \"414f07cf51059c5d23f1fd303bcb615f3f27416267201c1d42edba9acb55cd26\": rpc error: code = NotFound desc = could not find container \"414f07cf51059c5d23f1fd303bcb615f3f27416267201c1d42edba9acb55cd26\": container with ID starting with 414f07cf51059c5d23f1fd303bcb615f3f27416267201c1d42edba9acb55cd26 not found: ID does not exist"
Sep 17 19:28:54.574203 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:28:54.574182    2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-predictor-78b4c78fb9-nbg4n"]
Sep 17 19:28:54.769151 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:28:54.769119    2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d05fdd99-92a5-4625-8904-140482c6a53d" path="/var/lib/kubelet/pods/d05fdd99-92a5-4625-8904-140482c6a53d/volumes"
Sep 17 19:30:19.241946 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:30:19.241853    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-79d9bd5fdb-9n775"]
Sep 17 19:30:19.242546 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:30:19.242127    2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-79d9bd5fdb-9n775" podUID="86a097c7-53f1-4020-a1b3-6f6c96deec67" containerName="kserve-container" containerID="cri-o://4d0005fae7b19da455bb619cc2d87af6287a100e7065319f0cab66a02cb1d9c6" gracePeriod=30
Sep 17 19:30:19.242546 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:30:19.242173    2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-79d9bd5fdb-9n775" podUID="86a097c7-53f1-4020-a1b3-6f6c96deec67" containerName="kube-rbac-proxy" containerID="cri-o://8ab1f050565ef8d3e8684d8a4ba1ef35719f6a4c139d14e6906374cbdd1e932c" gracePeriod=30
Sep 17 19:30:19.897526 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:30:19.897491    2582 generic.go:356] "Generic (PLEG): container finished" podID="86a097c7-53f1-4020-a1b3-6f6c96deec67" containerID="8ab1f050565ef8d3e8684d8a4ba1ef35719f6a4c139d14e6906374cbdd1e932c" exitCode=2
Sep 17 19:30:19.897526 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:30:19.897531    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-79d9bd5fdb-9n775" event={"ID":"86a097c7-53f1-4020-a1b3-6f6c96deec67","Type":"ContainerDied","Data":"8ab1f050565ef8d3e8684d8a4ba1ef35719f6a4c139d14e6906374cbdd1e932c"}
Sep 17 19:30:21.764137 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:30:21.764100    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-79d9bd5fdb-9n775" podUID="86a097c7-53f1-4020-a1b3-6f6c96deec67" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.35:8080: connect: connection refused"
Sep 17 19:30:23.178143 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:30:23.178099    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-79d9bd5fdb-9n775" podUID="86a097c7-53f1-4020-a1b3-6f6c96deec67" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.35:8643/healthz\": dial tcp 10.134.0.35:8643: connect: connection refused"
Sep 17 19:30:24.379002 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:30:24.378978    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-79d9bd5fdb-9n775"
Sep 17 19:30:24.531185 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:30:24.531097    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/86a097c7-53f1-4020-a1b3-6f6c96deec67-cabundle-cert\") pod \"86a097c7-53f1-4020-a1b3-6f6c96deec67\" (UID: \"86a097c7-53f1-4020-a1b3-6f6c96deec67\") "
Sep 17 19:30:24.531185 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:30:24.531135    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/86a097c7-53f1-4020-a1b3-6f6c96deec67-kserve-provision-location\") pod \"86a097c7-53f1-4020-a1b3-6f6c96deec67\" (UID: \"86a097c7-53f1-4020-a1b3-6f6c96deec67\") "
Sep 17 19:30:24.531185 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:30:24.531168    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-tz4gr\" (UniqueName: \"kubernetes.io/projected/86a097c7-53f1-4020-a1b3-6f6c96deec67-kube-api-access-tz4gr\") pod \"86a097c7-53f1-4020-a1b3-6f6c96deec67\" (UID: \"86a097c7-53f1-4020-a1b3-6f6c96deec67\") "
Sep 17 19:30:24.531578 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:30:24.531203    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/86a097c7-53f1-4020-a1b3-6f6c96deec67-kube-rbac-proxy-sar-config\") pod \"86a097c7-53f1-4020-a1b3-6f6c96deec67\" (UID: \"86a097c7-53f1-4020-a1b3-6f6c96deec67\") "
Sep 17 19:30:24.531578 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:30:24.531226    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/86a097c7-53f1-4020-a1b3-6f6c96deec67-proxy-tls\") pod \"86a097c7-53f1-4020-a1b3-6f6c96deec67\" (UID: \"86a097c7-53f1-4020-a1b3-6f6c96deec67\") "
Sep 17 19:30:24.531578 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:30:24.531389    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/86a097c7-53f1-4020-a1b3-6f6c96deec67-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "86a097c7-53f1-4020-a1b3-6f6c96deec67" (UID: "86a097c7-53f1-4020-a1b3-6f6c96deec67"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 19:30:24.531578 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:30:24.531513    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/86a097c7-53f1-4020-a1b3-6f6c96deec67-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "86a097c7-53f1-4020-a1b3-6f6c96deec67" (UID: "86a097c7-53f1-4020-a1b3-6f6c96deec67"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 19:30:24.531745 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:30:24.531635    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/86a097c7-53f1-4020-a1b3-6f6c96deec67-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "86a097c7-53f1-4020-a1b3-6f6c96deec67" (UID: "86a097c7-53f1-4020-a1b3-6f6c96deec67"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 19:30:24.533419 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:30:24.533392    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/86a097c7-53f1-4020-a1b3-6f6c96deec67-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "86a097c7-53f1-4020-a1b3-6f6c96deec67" (UID: "86a097c7-53f1-4020-a1b3-6f6c96deec67"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 19:30:24.533485 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:30:24.533432    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/86a097c7-53f1-4020-a1b3-6f6c96deec67-kube-api-access-tz4gr" (OuterVolumeSpecName: "kube-api-access-tz4gr") pod "86a097c7-53f1-4020-a1b3-6f6c96deec67" (UID: "86a097c7-53f1-4020-a1b3-6f6c96deec67"). InnerVolumeSpecName "kube-api-access-tz4gr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 19:30:24.632190 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:30:24.632156    2582 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/86a097c7-53f1-4020-a1b3-6f6c96deec67-cabundle-cert\") on node \"ip-10-0-131-232.ec2.internal\" DevicePath \"\""
Sep 17 19:30:24.632190 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:30:24.632182    2582 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/86a097c7-53f1-4020-a1b3-6f6c96deec67-kserve-provision-location\") on node \"ip-10-0-131-232.ec2.internal\" DevicePath \"\""
Sep 17 19:30:24.632190 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:30:24.632193    2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tz4gr\" (UniqueName: \"kubernetes.io/projected/86a097c7-53f1-4020-a1b3-6f6c96deec67-kube-api-access-tz4gr\") on node \"ip-10-0-131-232.ec2.internal\" DevicePath \"\""
Sep 17 19:30:24.632423 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:30:24.632201    2582 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/86a097c7-53f1-4020-a1b3-6f6c96deec67-kube-rbac-proxy-sar-config\") on node \"ip-10-0-131-232.ec2.internal\" DevicePath \"\""
Sep 17 19:30:24.632423 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:30:24.632209    2582 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/86a097c7-53f1-4020-a1b3-6f6c96deec67-proxy-tls\") on node \"ip-10-0-131-232.ec2.internal\" DevicePath \"\""
Sep 17 19:30:24.919516 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:30:24.919486    2582 generic.go:356] "Generic (PLEG): container finished" podID="86a097c7-53f1-4020-a1b3-6f6c96deec67" containerID="4d0005fae7b19da455bb619cc2d87af6287a100e7065319f0cab66a02cb1d9c6" exitCode=0
Sep 17 19:30:24.919692 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:30:24.919546    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-79d9bd5fdb-9n775" event={"ID":"86a097c7-53f1-4020-a1b3-6f6c96deec67","Type":"ContainerDied","Data":"4d0005fae7b19da455bb619cc2d87af6287a100e7065319f0cab66a02cb1d9c6"}
Sep 17 19:30:24.919692 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:30:24.919557    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-79d9bd5fdb-9n775"
Sep 17 19:30:24.919692 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:30:24.919592    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-79d9bd5fdb-9n775" event={"ID":"86a097c7-53f1-4020-a1b3-6f6c96deec67","Type":"ContainerDied","Data":"3ca1f51660248e283b46e78f369489dcd8428e45eac74f8603af126f281fbd2f"}
Sep 17 19:30:24.919692 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:30:24.919617    2582 scope.go:117] "RemoveContainer" containerID="8ab1f050565ef8d3e8684d8a4ba1ef35719f6a4c139d14e6906374cbdd1e932c"
Sep 17 19:30:24.927665 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:30:24.927645    2582 scope.go:117] "RemoveContainer" containerID="4d0005fae7b19da455bb619cc2d87af6287a100e7065319f0cab66a02cb1d9c6"
Sep 17 19:30:24.935563 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:30:24.935539    2582 scope.go:117] "RemoveContainer" containerID="f97e49a1759309034aa18d5e469299c3f97b43829454c2cacb6598fe443c5a4c"
Sep 17 19:30:24.937376 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:30:24.937358    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-79d9bd5fdb-9n775"]
Sep 17 19:30:24.939852 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:30:24.939814    2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-79d9bd5fdb-9n775"]
Sep 17 19:30:24.944187 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:30:24.944166    2582 scope.go:117] "RemoveContainer" containerID="8ab1f050565ef8d3e8684d8a4ba1ef35719f6a4c139d14e6906374cbdd1e932c"
Sep 17 19:30:24.944516 ip-10-0-131-232 kubenswrapper[2582]: E0917 19:30:24.944496    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8ab1f050565ef8d3e8684d8a4ba1ef35719f6a4c139d14e6906374cbdd1e932c\": container with ID starting with 8ab1f050565ef8d3e8684d8a4ba1ef35719f6a4c139d14e6906374cbdd1e932c not found: ID does not exist" containerID="8ab1f050565ef8d3e8684d8a4ba1ef35719f6a4c139d14e6906374cbdd1e932c"
Sep 17 19:30:24.944595 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:30:24.944530    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8ab1f050565ef8d3e8684d8a4ba1ef35719f6a4c139d14e6906374cbdd1e932c"} err="failed to get container status \"8ab1f050565ef8d3e8684d8a4ba1ef35719f6a4c139d14e6906374cbdd1e932c\": rpc error: code = NotFound desc = could not find container \"8ab1f050565ef8d3e8684d8a4ba1ef35719f6a4c139d14e6906374cbdd1e932c\": container with ID starting with 8ab1f050565ef8d3e8684d8a4ba1ef35719f6a4c139d14e6906374cbdd1e932c not found: ID does not exist"
Sep 17 19:30:24.944595 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:30:24.944555    2582 scope.go:117] "RemoveContainer" containerID="4d0005fae7b19da455bb619cc2d87af6287a100e7065319f0cab66a02cb1d9c6"
Sep 17 19:30:24.944868 ip-10-0-131-232 kubenswrapper[2582]: E0917 19:30:24.944846    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4d0005fae7b19da455bb619cc2d87af6287a100e7065319f0cab66a02cb1d9c6\": container with ID starting with 4d0005fae7b19da455bb619cc2d87af6287a100e7065319f0cab66a02cb1d9c6 not found: ID does not exist" containerID="4d0005fae7b19da455bb619cc2d87af6287a100e7065319f0cab66a02cb1d9c6"
Sep 17 19:30:24.944917 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:30:24.944875    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4d0005fae7b19da455bb619cc2d87af6287a100e7065319f0cab66a02cb1d9c6"} err="failed to get container status \"4d0005fae7b19da455bb619cc2d87af6287a100e7065319f0cab66a02cb1d9c6\": rpc error: code = NotFound desc = could not find container \"4d0005fae7b19da455bb619cc2d87af6287a100e7065319f0cab66a02cb1d9c6\": container with ID starting with 4d0005fae7b19da455bb619cc2d87af6287a100e7065319f0cab66a02cb1d9c6 not found: ID does not exist"
Sep 17 19:30:24.944917 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:30:24.944892    2582 scope.go:117] "RemoveContainer" containerID="f97e49a1759309034aa18d5e469299c3f97b43829454c2cacb6598fe443c5a4c"
Sep 17 19:30:24.945123 ip-10-0-131-232 kubenswrapper[2582]: E0917 19:30:24.945099    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f97e49a1759309034aa18d5e469299c3f97b43829454c2cacb6598fe443c5a4c\": container with ID starting with f97e49a1759309034aa18d5e469299c3f97b43829454c2cacb6598fe443c5a4c not found: ID does not exist" containerID="f97e49a1759309034aa18d5e469299c3f97b43829454c2cacb6598fe443c5a4c"
Sep 17 19:30:24.945224 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:30:24.945127    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f97e49a1759309034aa18d5e469299c3f97b43829454c2cacb6598fe443c5a4c"} err="failed to get container status \"f97e49a1759309034aa18d5e469299c3f97b43829454c2cacb6598fe443c5a4c\": rpc error: code = NotFound desc = could not find container \"f97e49a1759309034aa18d5e469299c3f97b43829454c2cacb6598fe443c5a4c\": container with ID starting with f97e49a1759309034aa18d5e469299c3f97b43829454c2cacb6598fe443c5a4c not found: ID does not exist"
Sep 17 19:30:26.768146 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:30:26.768115    2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="86a097c7-53f1-4020-a1b3-6f6c96deec67" path="/var/lib/kubelet/pods/86a097c7-53f1-4020-a1b3-6f6c96deec67/volumes"
Sep 17 19:30:43.808516 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:30:43.808475    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-797ff577-6z5xx"]
Sep 17 19:30:43.809013 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:30:43.808753    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d05fdd99-92a5-4625-8904-140482c6a53d" containerName="storage-initializer"
Sep 17 19:30:43.809013 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:30:43.808765    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="d05fdd99-92a5-4625-8904-140482c6a53d" containerName="storage-initializer"
Sep 17 19:30:43.809013 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:30:43.808775    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="86a097c7-53f1-4020-a1b3-6f6c96deec67" containerName="storage-initializer"
Sep 17 19:30:43.809013 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:30:43.808780    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="86a097c7-53f1-4020-a1b3-6f6c96deec67" containerName="storage-initializer"
Sep 17 19:30:43.809013 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:30:43.808794    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d05fdd99-92a5-4625-8904-140482c6a53d" containerName="kserve-container"
Sep 17 19:30:43.809013 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:30:43.808803    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="d05fdd99-92a5-4625-8904-140482c6a53d" containerName="kserve-container"
Sep 17 19:30:43.809013 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:30:43.808813    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d05fdd99-92a5-4625-8904-140482c6a53d" containerName="kube-rbac-proxy"
Sep 17 19:30:43.809013 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:30:43.808836    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="d05fdd99-92a5-4625-8904-140482c6a53d" containerName="kube-rbac-proxy"
Sep 17 19:30:43.809013 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:30:43.808847    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="86a097c7-53f1-4020-a1b3-6f6c96deec67" containerName="kserve-container"
Sep 17 19:30:43.809013 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:30:43.808855    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="86a097c7-53f1-4020-a1b3-6f6c96deec67" containerName="kserve-container"
Sep 17 19:30:43.809013 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:30:43.808868    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="86a097c7-53f1-4020-a1b3-6f6c96deec67" containerName="kube-rbac-proxy"
Sep 17 19:30:43.809013 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:30:43.808875    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="86a097c7-53f1-4020-a1b3-6f6c96deec67" containerName="kube-rbac-proxy"
Sep 17 19:30:43.809013 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:30:43.808963    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="d05fdd99-92a5-4625-8904-140482c6a53d" containerName="kube-rbac-proxy"
Sep 17 19:30:43.809013 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:30:43.808973    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="86a097c7-53f1-4020-a1b3-6f6c96deec67" containerName="kserve-container"
Sep 17 19:30:43.809013 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:30:43.808979    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="86a097c7-53f1-4020-a1b3-6f6c96deec67" containerName="kube-rbac-proxy"
Sep 17 19:30:43.809013 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:30:43.808986    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="d05fdd99-92a5-4625-8904-140482c6a53d" containerName="kserve-container"
Sep 17 19:30:43.814054 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:30:43.814027    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-797ff577-6z5xx"
Sep 17 19:30:43.817551 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:30:43.817529    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-logger-raw-cipn-predictor-serving-cert\""
Sep 17 19:30:43.817648 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:30:43.817560    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-vpx7x\""
Sep 17 19:30:43.817866 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:30:43.817847    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Sep 17 19:30:43.817948 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:30:43.817879    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Sep 17 19:30:43.818587 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:30:43.818524    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-logger-raw-cipn-kube-rbac-proxy-sar-config\""
Sep 17 19:30:43.818587 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:30:43.818546    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Sep 17 19:30:43.818587 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:30:43.818527    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Sep 17 19:30:43.824301 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:30:43.824275    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-797ff577-6z5xx"]
Sep 17 19:30:43.884454 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:30:43.884422    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-knqbf\" (UniqueName: \"kubernetes.io/projected/e218f413-d390-4255-a660-7b07886c785c-kube-api-access-knqbf\") pod \"isvc-logger-raw-cipn-predictor-797ff577-6z5xx\" (UID: \"e218f413-d390-4255-a660-7b07886c785c\") " pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-797ff577-6z5xx"
Sep 17 19:30:43.884454 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:30:43.884462    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e218f413-d390-4255-a660-7b07886c785c-kube-rbac-proxy-sar-config\") pod \"isvc-logger-raw-cipn-predictor-797ff577-6z5xx\" (UID: \"e218f413-d390-4255-a660-7b07886c785c\") " pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-797ff577-6z5xx"
Sep 17 19:30:43.884695 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:30:43.884489    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/e218f413-d390-4255-a660-7b07886c785c-cabundle-cert\") pod \"isvc-logger-raw-cipn-predictor-797ff577-6z5xx\" (UID: \"e218f413-d390-4255-a660-7b07886c785c\") " pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-797ff577-6z5xx"
Sep 17 19:30:43.884695 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:30:43.884539    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e218f413-d390-4255-a660-7b07886c785c-kserve-provision-location\") pod \"isvc-logger-raw-cipn-predictor-797ff577-6z5xx\" (UID: \"e218f413-d390-4255-a660-7b07886c785c\") " pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-797ff577-6z5xx"
Sep 17 19:30:43.884695 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:30:43.884620    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e218f413-d390-4255-a660-7b07886c785c-proxy-tls\") pod \"isvc-logger-raw-cipn-predictor-797ff577-6z5xx\" (UID: \"e218f413-d390-4255-a660-7b07886c785c\") " pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-797ff577-6z5xx"
Sep 17 19:30:43.985522 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:30:43.985482    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-knqbf\" (UniqueName: \"kubernetes.io/projected/e218f413-d390-4255-a660-7b07886c785c-kube-api-access-knqbf\") pod \"isvc-logger-raw-cipn-predictor-797ff577-6z5xx\" (UID: \"e218f413-d390-4255-a660-7b07886c785c\") " pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-797ff577-6z5xx"
Sep 17 19:30:43.985522 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:30:43.985518    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e218f413-d390-4255-a660-7b07886c785c-kube-rbac-proxy-sar-config\") pod \"isvc-logger-raw-cipn-predictor-797ff577-6z5xx\" (UID: \"e218f413-d390-4255-a660-7b07886c785c\") " pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-797ff577-6z5xx"
Sep 17 19:30:43.985785 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:30:43.985534    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/e218f413-d390-4255-a660-7b07886c785c-cabundle-cert\") pod \"isvc-logger-raw-cipn-predictor-797ff577-6z5xx\" (UID: \"e218f413-d390-4255-a660-7b07886c785c\") " pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-797ff577-6z5xx"
Sep 17 19:30:43.985785 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:30:43.985552    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e218f413-d390-4255-a660-7b07886c785c-kserve-provision-location\") pod \"isvc-logger-raw-cipn-predictor-797ff577-6z5xx\" (UID: \"e218f413-d390-4255-a660-7b07886c785c\") " pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-797ff577-6z5xx"
Sep 17 19:30:43.985785 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:30:43.985606    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e218f413-d390-4255-a660-7b07886c785c-proxy-tls\") pod \"isvc-logger-raw-cipn-predictor-797ff577-6z5xx\" (UID: \"e218f413-d390-4255-a660-7b07886c785c\") " pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-797ff577-6z5xx"
Sep 17 19:30:43.986081 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:30:43.986056    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e218f413-d390-4255-a660-7b07886c785c-kserve-provision-location\") pod \"isvc-logger-raw-cipn-predictor-797ff577-6z5xx\" (UID: \"e218f413-d390-4255-a660-7b07886c785c\") " pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-797ff577-6z5xx"
Sep 17 19:30:43.986276 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:30:43.986248    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e218f413-d390-4255-a660-7b07886c785c-kube-rbac-proxy-sar-config\") pod \"isvc-logger-raw-cipn-predictor-797ff577-6z5xx\" (UID: \"e218f413-d390-4255-a660-7b07886c785c\") " pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-797ff577-6z5xx"
Sep 17 19:30:43.986377 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:30:43.986257    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/e218f413-d390-4255-a660-7b07886c785c-cabundle-cert\") pod \"isvc-logger-raw-cipn-predictor-797ff577-6z5xx\" (UID: \"e218f413-d390-4255-a660-7b07886c785c\") " pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-797ff577-6z5xx"
Sep 17 19:30:43.988262 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:30:43.988241    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e218f413-d390-4255-a660-7b07886c785c-proxy-tls\") pod \"isvc-logger-raw-cipn-predictor-797ff577-6z5xx\" (UID: \"e218f413-d390-4255-a660-7b07886c785c\") " pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-797ff577-6z5xx"
Sep 17 19:30:43.993398 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:30:43.993375    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-knqbf\" (UniqueName: \"kubernetes.io/projected/e218f413-d390-4255-a660-7b07886c785c-kube-api-access-knqbf\") pod \"isvc-logger-raw-cipn-predictor-797ff577-6z5xx\" (UID: \"e218f413-d390-4255-a660-7b07886c785c\") " pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-797ff577-6z5xx"
Sep 17 19:30:44.125899 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:30:44.125789    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-797ff577-6z5xx"
Sep 17 19:30:44.400635 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:30:44.400601    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-797ff577-6z5xx"]
Sep 17 19:30:44.403690 ip-10-0-131-232 kubenswrapper[2582]: W0917 19:30:44.403660    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode218f413_d390_4255_a660_7b07886c785c.slice/crio-e9126fbf549da5a527a43cc9a51c2127fe1c3a9c468ea1981dc6b1224dafa9fa WatchSource:0}: Error finding container e9126fbf549da5a527a43cc9a51c2127fe1c3a9c468ea1981dc6b1224dafa9fa: Status 404 returned error can't find the container with id e9126fbf549da5a527a43cc9a51c2127fe1c3a9c468ea1981dc6b1224dafa9fa
Sep 17 19:30:45.007363 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:30:45.007328    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-797ff577-6z5xx" event={"ID":"e218f413-d390-4255-a660-7b07886c785c","Type":"ContainerStarted","Data":"cffafc977f94adb2bc2fe9b1128431faeefc7cc63350ef01a3cbe702133675ce"}
Sep 17 19:30:45.007750 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:30:45.007434    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-797ff577-6z5xx" event={"ID":"e218f413-d390-4255-a660-7b07886c785c","Type":"ContainerStarted","Data":"e9126fbf549da5a527a43cc9a51c2127fe1c3a9c468ea1981dc6b1224dafa9fa"}
Sep 17 19:30:49.026205 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:30:49.026173    2582 generic.go:356] "Generic (PLEG): container finished" podID="e218f413-d390-4255-a660-7b07886c785c" containerID="cffafc977f94adb2bc2fe9b1128431faeefc7cc63350ef01a3cbe702133675ce" exitCode=0
Sep 17 19:30:49.026604 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:30:49.026239    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-797ff577-6z5xx" event={"ID":"e218f413-d390-4255-a660-7b07886c785c","Type":"ContainerDied","Data":"cffafc977f94adb2bc2fe9b1128431faeefc7cc63350ef01a3cbe702133675ce"}
Sep 17 19:30:50.033562 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:30:50.033527    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-797ff577-6z5xx" event={"ID":"e218f413-d390-4255-a660-7b07886c785c","Type":"ContainerStarted","Data":"336d11f3221e43582779ff51e3c1b1bb26ba73fb31dce0441569b590eaae360a"}
Sep 17 19:30:50.034068 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:30:50.033596    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-797ff577-6z5xx" event={"ID":"e218f413-d390-4255-a660-7b07886c785c","Type":"ContainerStarted","Data":"537574cb3416c4feb9e57be9571c15a5bff216a771e41cddc1360fae18eb62c4"}
Sep 17 19:30:50.034068 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:30:50.033622    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-797ff577-6z5xx" event={"ID":"e218f413-d390-4255-a660-7b07886c785c","Type":"ContainerStarted","Data":"72ef4402fd9f877db3cd0b65c3017bed4288f4ba45b0bd93219b52873a4255d8"}
Sep 17 19:30:50.034068 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:30:50.033912    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-797ff577-6z5xx"
Sep 17 19:30:50.034068 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:30:50.033965    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-797ff577-6z5xx"
Sep 17 19:30:50.035422 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:30:50.035389    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-797ff577-6z5xx" podUID="e218f413-d390-4255-a660-7b07886c785c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.37:8080: connect: connection refused"
Sep 17 19:30:51.038542 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:30:51.038496    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-797ff577-6z5xx"
Sep 17 19:30:51.039056 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:30:51.038692    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-797ff577-6z5xx" podUID="e218f413-d390-4255-a660-7b07886c785c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.37:8080: connect: connection refused"
Sep 17 19:30:51.039971 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:30:51.039943    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-797ff577-6z5xx" podUID="e218f413-d390-4255-a660-7b07886c785c" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 19:30:52.043647 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:30:52.043611    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-797ff577-6z5xx" podUID="e218f413-d390-4255-a660-7b07886c785c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.37:8080: connect: connection refused"
Sep 17 19:30:52.044090 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:30:52.044067    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-797ff577-6z5xx" podUID="e218f413-d390-4255-a660-7b07886c785c" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 19:30:57.048038 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:30:57.047995    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-797ff577-6z5xx"
Sep 17 19:30:57.048615 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:30:57.048544    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-797ff577-6z5xx" podUID="e218f413-d390-4255-a660-7b07886c785c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.37:8080: connect: connection refused"
Sep 17 19:30:57.048787 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:30:57.048759    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-797ff577-6z5xx" podUID="e218f413-d390-4255-a660-7b07886c785c" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 19:30:57.077435 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:30:57.077381    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-797ff577-6z5xx" podStartSLOduration=9.910523584 podStartE2EDuration="14.07736526s" totalImagesPullingTime="0s" totalInitContainerRuntime="4.166841676s" isStatefulPod=false podCreationTimestamp="2026-09-17 19:30:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 19:30:50.058913686 +0000 UTC m=+1465.830584747" watchObservedRunningTime="2026-09-17 19:30:57.07736526 +0000 UTC m=+1472.849036320"
Sep 17 19:31:07.049280 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:31:07.049237    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-797ff577-6z5xx" podUID="e218f413-d390-4255-a660-7b07886c785c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.37:8080: connect: connection refused"
Sep 17 19:31:07.049714 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:31:07.049642    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-797ff577-6z5xx" podUID="e218f413-d390-4255-a660-7b07886c785c" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 19:31:17.048719 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:31:17.048675    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-797ff577-6z5xx" podUID="e218f413-d390-4255-a660-7b07886c785c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.37:8080: connect: connection refused"
Sep 17 19:31:17.049345 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:31:17.049274    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-797ff577-6z5xx" podUID="e218f413-d390-4255-a660-7b07886c785c" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 19:31:24.751298 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:31:24.751259    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-kdm8s_c1b79c18-774f-4b78-a6a9-7ea8a229aaae/ovn-acl-logging/0.log"
Sep 17 19:31:24.752416 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:31:24.752393    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-kdm8s_c1b79c18-774f-4b78-a6a9-7ea8a229aaae/ovn-acl-logging/0.log"
Sep 17 19:31:27.049205 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:31:27.049154    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-797ff577-6z5xx" podUID="e218f413-d390-4255-a660-7b07886c785c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.37:8080: connect: connection refused"
Sep 17 19:31:27.049619 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:31:27.049593    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-797ff577-6z5xx" podUID="e218f413-d390-4255-a660-7b07886c785c" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 19:31:37.048497 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:31:37.048454    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-797ff577-6z5xx" podUID="e218f413-d390-4255-a660-7b07886c785c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.37:8080: connect: connection refused"
Sep 17 19:31:37.048934 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:31:37.048868    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-797ff577-6z5xx" podUID="e218f413-d390-4255-a660-7b07886c785c" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 19:31:47.048877 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:31:47.048765    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-797ff577-6z5xx" podUID="e218f413-d390-4255-a660-7b07886c785c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.37:8080: connect: connection refused"
Sep 17 19:31:47.049258 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:31:47.049231    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-797ff577-6z5xx" podUID="e218f413-d390-4255-a660-7b07886c785c" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 19:31:57.048612 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:31:57.048561    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-797ff577-6z5xx" podUID="e218f413-d390-4255-a660-7b07886c785c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.37:8080: connect: connection refused"
Sep 17 19:31:57.049166 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:31:57.049067    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-797ff577-6z5xx" podUID="e218f413-d390-4255-a660-7b07886c785c" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 19:32:04.768974 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:32:04.768924    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-797ff577-6z5xx" podUID="e218f413-d390-4255-a660-7b07886c785c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.37:8080: connect: connection refused"
Sep 17 19:32:04.769450 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:32:04.769254    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-797ff577-6z5xx" podUID="e218f413-d390-4255-a660-7b07886c785c" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 19:32:14.773873 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:32:14.773842    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-797ff577-6z5xx"
Sep 17 19:32:14.776269 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:32:14.773950    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-797ff577-6z5xx"
Sep 17 19:32:29.628899 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:32:29.628863    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-797ff577-6z5xx"]
Sep 17 19:32:29.629408 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:32:29.629146    2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-797ff577-6z5xx" podUID="e218f413-d390-4255-a660-7b07886c785c" containerName="kserve-container" containerID="cri-o://72ef4402fd9f877db3cd0b65c3017bed4288f4ba45b0bd93219b52873a4255d8" gracePeriod=30
Sep 17 19:32:29.629408 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:32:29.629159    2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-797ff577-6z5xx" podUID="e218f413-d390-4255-a660-7b07886c785c" containerName="agent" containerID="cri-o://336d11f3221e43582779ff51e3c1b1bb26ba73fb31dce0441569b590eaae360a" gracePeriod=30
Sep 17 19:32:29.629408 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:32:29.629153    2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-797ff577-6z5xx" podUID="e218f413-d390-4255-a660-7b07886c785c" containerName="kube-rbac-proxy" containerID="cri-o://537574cb3416c4feb9e57be9571c15a5bff216a771e41cddc1360fae18eb62c4" gracePeriod=30
Sep 17 19:32:30.462332 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:32:30.462292    2582 generic.go:356] "Generic (PLEG): container finished" podID="e218f413-d390-4255-a660-7b07886c785c" containerID="537574cb3416c4feb9e57be9571c15a5bff216a771e41cddc1360fae18eb62c4" exitCode=2
Sep 17 19:32:30.462499 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:32:30.462371    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-797ff577-6z5xx" event={"ID":"e218f413-d390-4255-a660-7b07886c785c","Type":"ContainerDied","Data":"537574cb3416c4feb9e57be9571c15a5bff216a771e41cddc1360fae18eb62c4"}
Sep 17 19:32:32.044463 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:32:32.044424    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-797ff577-6z5xx" podUID="e218f413-d390-4255-a660-7b07886c785c" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.37:8643/healthz\": dial tcp 10.134.0.37:8643: connect: connection refused"
Sep 17 19:32:34.482126 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:32:34.482094    2582 generic.go:356] "Generic (PLEG): container finished" podID="e218f413-d390-4255-a660-7b07886c785c" containerID="72ef4402fd9f877db3cd0b65c3017bed4288f4ba45b0bd93219b52873a4255d8" exitCode=0
Sep 17 19:32:34.482470 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:32:34.482131    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-797ff577-6z5xx" event={"ID":"e218f413-d390-4255-a660-7b07886c785c","Type":"ContainerDied","Data":"72ef4402fd9f877db3cd0b65c3017bed4288f4ba45b0bd93219b52873a4255d8"}
Sep 17 19:32:34.769682 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:32:34.769631    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-797ff577-6z5xx" podUID="e218f413-d390-4255-a660-7b07886c785c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.37:8080: connect: connection refused"
Sep 17 19:32:34.770016 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:32:34.769988    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-797ff577-6z5xx" podUID="e218f413-d390-4255-a660-7b07886c785c" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 19:32:37.044325 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:32:37.044278    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-797ff577-6z5xx" podUID="e218f413-d390-4255-a660-7b07886c785c" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.37:8643/healthz\": dial tcp 10.134.0.37:8643: connect: connection refused"
Sep 17 19:32:42.044190 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:32:42.044136    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-797ff577-6z5xx" podUID="e218f413-d390-4255-a660-7b07886c785c" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.37:8643/healthz\": dial tcp 10.134.0.37:8643: connect: connection refused"
Sep 17 19:32:42.044618 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:32:42.044285    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-797ff577-6z5xx"
Sep 17 19:32:44.769186 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:32:44.769146    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-797ff577-6z5xx" podUID="e218f413-d390-4255-a660-7b07886c785c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.37:8080: connect: connection refused"
Sep 17 19:32:44.769605 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:32:44.769470    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-797ff577-6z5xx" podUID="e218f413-d390-4255-a660-7b07886c785c" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 19:32:47.043902 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:32:47.043857    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-797ff577-6z5xx" podUID="e218f413-d390-4255-a660-7b07886c785c" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.37:8643/healthz\": dial tcp 10.134.0.37:8643: connect: connection refused"
Sep 17 19:32:47.046572 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:32:47.046548    2582 ???:1] "http: TLS handshake error from 10.0.140.225:40184: EOF"
Sep 17 19:32:47.049475 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:32:47.049448    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-797ff577-6z5xx_e218f413-d390-4255-a660-7b07886c785c/kserve-container/0.log"
Sep 17 19:32:47.074532 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:32:47.074506    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-797ff577-6z5xx_e218f413-d390-4255-a660-7b07886c785c/kube-rbac-proxy/0.log"
Sep 17 19:32:47.086223 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:32:47.086197    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-797ff577-6z5xx_e218f413-d390-4255-a660-7b07886c785c/agent/0.log"
Sep 17 19:32:47.097263 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:32:47.097243    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-797ff577-6z5xx_e218f413-d390-4255-a660-7b07886c785c/storage-initializer/0.log"
Sep 17 19:32:47.939243 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:32:47.939208    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-797ff577-6z5xx_e218f413-d390-4255-a660-7b07886c785c/kserve-container/0.log"
Sep 17 19:32:47.951307 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:32:47.951277    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-797ff577-6z5xx_e218f413-d390-4255-a660-7b07886c785c/kube-rbac-proxy/0.log"
Sep 17 19:32:47.960899 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:32:47.960877    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-797ff577-6z5xx_e218f413-d390-4255-a660-7b07886c785c/agent/0.log"
Sep 17 19:32:47.970708 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:32:47.970689    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-797ff577-6z5xx_e218f413-d390-4255-a660-7b07886c785c/storage-initializer/0.log"
Sep 17 19:32:48.803448 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:32:48.803418    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-797ff577-6z5xx_e218f413-d390-4255-a660-7b07886c785c/kserve-container/0.log"
Sep 17 19:32:48.816120 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:32:48.816092    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-797ff577-6z5xx_e218f413-d390-4255-a660-7b07886c785c/kube-rbac-proxy/0.log"
Sep 17 19:32:48.828377 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:32:48.828346    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-797ff577-6z5xx_e218f413-d390-4255-a660-7b07886c785c/agent/0.log"
Sep 17 19:32:48.839344 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:32:48.839318    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-797ff577-6z5xx_e218f413-d390-4255-a660-7b07886c785c/storage-initializer/0.log"
Sep 17 19:32:49.645001 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:32:49.644972    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-797ff577-6z5xx_e218f413-d390-4255-a660-7b07886c785c/kserve-container/0.log"
Sep 17 19:32:49.657080 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:32:49.657039    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-797ff577-6z5xx_e218f413-d390-4255-a660-7b07886c785c/kube-rbac-proxy/0.log"
Sep 17 19:32:49.666653 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:32:49.666614    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-797ff577-6z5xx_e218f413-d390-4255-a660-7b07886c785c/agent/0.log"
Sep 17 19:32:49.676462 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:32:49.676443    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-797ff577-6z5xx_e218f413-d390-4255-a660-7b07886c785c/storage-initializer/0.log"
Sep 17 19:32:50.501257 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:32:50.501229    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-797ff577-6z5xx_e218f413-d390-4255-a660-7b07886c785c/kserve-container/0.log"
Sep 17 19:32:50.528716 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:32:50.528692    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-797ff577-6z5xx_e218f413-d390-4255-a660-7b07886c785c/kube-rbac-proxy/0.log"
Sep 17 19:32:50.541752 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:32:50.541734    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-797ff577-6z5xx_e218f413-d390-4255-a660-7b07886c785c/agent/0.log"
Sep 17 19:32:50.556184 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:32:50.556159    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-797ff577-6z5xx_e218f413-d390-4255-a660-7b07886c785c/storage-initializer/0.log"
Sep 17 19:32:51.418549 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:32:51.418522    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-797ff577-6z5xx_e218f413-d390-4255-a660-7b07886c785c/kserve-container/0.log"
Sep 17 19:32:51.431015 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:32:51.430991    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-797ff577-6z5xx_e218f413-d390-4255-a660-7b07886c785c/kube-rbac-proxy/0.log"
Sep 17 19:32:51.440971 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:32:51.440949    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-797ff577-6z5xx_e218f413-d390-4255-a660-7b07886c785c/agent/0.log"
Sep 17 19:32:51.451576 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:32:51.451539    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-797ff577-6z5xx_e218f413-d390-4255-a660-7b07886c785c/storage-initializer/0.log"
Sep 17 19:32:52.044933 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:32:52.044894    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-797ff577-6z5xx" podUID="e218f413-d390-4255-a660-7b07886c785c" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.37:8643/healthz\": dial tcp 10.134.0.37:8643: connect: connection refused"
Sep 17 19:32:52.267317 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:32:52.267291    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-797ff577-6z5xx_e218f413-d390-4255-a660-7b07886c785c/kserve-container/0.log"
Sep 17 19:32:52.283419 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:32:52.283395    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-797ff577-6z5xx_e218f413-d390-4255-a660-7b07886c785c/kube-rbac-proxy/0.log"
Sep 17 19:32:52.301433 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:32:52.301353    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-797ff577-6z5xx_e218f413-d390-4255-a660-7b07886c785c/agent/0.log"
Sep 17 19:32:52.313484 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:32:52.313452    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-797ff577-6z5xx_e218f413-d390-4255-a660-7b07886c785c/storage-initializer/0.log"
Sep 17 19:32:53.144937 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:32:53.144911    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-797ff577-6z5xx_e218f413-d390-4255-a660-7b07886c785c/kserve-container/0.log"
Sep 17 19:32:53.158105 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:32:53.158079    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-797ff577-6z5xx_e218f413-d390-4255-a660-7b07886c785c/kube-rbac-proxy/0.log"
Sep 17 19:32:53.168786 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:32:53.168759    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-797ff577-6z5xx_e218f413-d390-4255-a660-7b07886c785c/agent/0.log"
Sep 17 19:32:53.179504 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:32:53.179475    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-797ff577-6z5xx_e218f413-d390-4255-a660-7b07886c785c/storage-initializer/0.log"
Sep 17 19:32:53.995816 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:32:53.995786    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-797ff577-6z5xx_e218f413-d390-4255-a660-7b07886c785c/kserve-container/0.log"
Sep 17 19:32:54.009411 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:32:54.009382    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-797ff577-6z5xx_e218f413-d390-4255-a660-7b07886c785c/kube-rbac-proxy/0.log"
Sep 17 19:32:54.024351 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:32:54.024322    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-797ff577-6z5xx_e218f413-d390-4255-a660-7b07886c785c/agent/0.log"
Sep 17 19:32:54.037118 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:32:54.037078    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-797ff577-6z5xx_e218f413-d390-4255-a660-7b07886c785c/storage-initializer/0.log"
Sep 17 19:32:54.768801 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:32:54.768757    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-797ff577-6z5xx" podUID="e218f413-d390-4255-a660-7b07886c785c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.37:8080: connect: connection refused"
Sep 17 19:32:54.769190 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:32:54.768911    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-797ff577-6z5xx"
Sep 17 19:32:54.769277 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:32:54.769256    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-797ff577-6z5xx" podUID="e218f413-d390-4255-a660-7b07886c785c" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 19:32:54.769359 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:32:54.769347    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-797ff577-6z5xx"
Sep 17 19:32:54.867529 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:32:54.867503    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-797ff577-6z5xx_e218f413-d390-4255-a660-7b07886c785c/kserve-container/0.log"
Sep 17 19:32:54.880181 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:32:54.880161    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-797ff577-6z5xx_e218f413-d390-4255-a660-7b07886c785c/kube-rbac-proxy/0.log"
Sep 17 19:32:54.890449 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:32:54.890426    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-797ff577-6z5xx_e218f413-d390-4255-a660-7b07886c785c/agent/0.log"
Sep 17 19:32:54.900880 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:32:54.900853    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-797ff577-6z5xx_e218f413-d390-4255-a660-7b07886c785c/storage-initializer/0.log"
Sep 17 19:32:55.714145 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:32:55.714118    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-797ff577-6z5xx_e218f413-d390-4255-a660-7b07886c785c/kserve-container/0.log"
Sep 17 19:32:55.726598 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:32:55.726561    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-797ff577-6z5xx_e218f413-d390-4255-a660-7b07886c785c/kube-rbac-proxy/0.log"
Sep 17 19:32:55.736912 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:32:55.736879    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-797ff577-6z5xx_e218f413-d390-4255-a660-7b07886c785c/agent/0.log"
Sep 17 19:32:55.748142 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:32:55.748126    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-797ff577-6z5xx_e218f413-d390-4255-a660-7b07886c785c/storage-initializer/0.log"
Sep 17 19:32:56.564443 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:32:56.564417    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-797ff577-6z5xx_e218f413-d390-4255-a660-7b07886c785c/kserve-container/0.log"
Sep 17 19:32:56.576468 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:32:56.576444    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-797ff577-6z5xx_e218f413-d390-4255-a660-7b07886c785c/kube-rbac-proxy/0.log"
Sep 17 19:32:56.588145 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:32:56.588110    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-797ff577-6z5xx_e218f413-d390-4255-a660-7b07886c785c/agent/0.log"
Sep 17 19:32:56.598616 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:32:56.598600    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-797ff577-6z5xx_e218f413-d390-4255-a660-7b07886c785c/storage-initializer/0.log"
Sep 17 19:32:57.044425 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:32:57.044388    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-797ff577-6z5xx" podUID="e218f413-d390-4255-a660-7b07886c785c" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.37:8643/healthz\": dial tcp 10.134.0.37:8643: connect: connection refused"
Sep 17 19:32:57.452323 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:32:57.452299    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-797ff577-6z5xx_e218f413-d390-4255-a660-7b07886c785c/kserve-container/0.log"
Sep 17 19:32:57.464023 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:32:57.464000    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-797ff577-6z5xx_e218f413-d390-4255-a660-7b07886c785c/kube-rbac-proxy/0.log"
Sep 17 19:32:57.474184 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:32:57.474150    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-797ff577-6z5xx_e218f413-d390-4255-a660-7b07886c785c/agent/0.log"
Sep 17 19:32:57.484868 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:32:57.484843    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-797ff577-6z5xx_e218f413-d390-4255-a660-7b07886c785c/storage-initializer/0.log"
Sep 17 19:32:58.367444 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:32:58.367382    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-797ff577-6z5xx_e218f413-d390-4255-a660-7b07886c785c/kserve-container/0.log"
Sep 17 19:32:58.379282 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:32:58.379259    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-797ff577-6z5xx_e218f413-d390-4255-a660-7b07886c785c/kube-rbac-proxy/0.log"
Sep 17 19:32:58.388972 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:32:58.388950    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-797ff577-6z5xx_e218f413-d390-4255-a660-7b07886c785c/agent/0.log"
Sep 17 19:32:58.399153 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:32:58.399135    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-797ff577-6z5xx_e218f413-d390-4255-a660-7b07886c785c/storage-initializer/0.log"
Sep 17 19:32:59.281065 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:32:59.281035    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-797ff577-6z5xx_e218f413-d390-4255-a660-7b07886c785c/kserve-container/0.log"
Sep 17 19:32:59.293209 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:32:59.293162    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-797ff577-6z5xx_e218f413-d390-4255-a660-7b07886c785c/kube-rbac-proxy/0.log"
Sep 17 19:32:59.304456 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:32:59.304429    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-797ff577-6z5xx_e218f413-d390-4255-a660-7b07886c785c/agent/0.log"
Sep 17 19:32:59.315019 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:32:59.314998    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-797ff577-6z5xx_e218f413-d390-4255-a660-7b07886c785c/storage-initializer/0.log"
Sep 17 19:32:59.771912 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:32:59.771890    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-797ff577-6z5xx"
Sep 17 19:32:59.930502 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:32:59.930417    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-knqbf\" (UniqueName: \"kubernetes.io/projected/e218f413-d390-4255-a660-7b07886c785c-kube-api-access-knqbf\") pod \"e218f413-d390-4255-a660-7b07886c785c\" (UID: \"e218f413-d390-4255-a660-7b07886c785c\") "
Sep 17 19:32:59.930502 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:32:59.930470    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/e218f413-d390-4255-a660-7b07886c785c-cabundle-cert\") pod \"e218f413-d390-4255-a660-7b07886c785c\" (UID: \"e218f413-d390-4255-a660-7b07886c785c\") "
Sep 17 19:32:59.930502 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:32:59.930491    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e218f413-d390-4255-a660-7b07886c785c-kube-rbac-proxy-sar-config\") pod \"e218f413-d390-4255-a660-7b07886c785c\" (UID: \"e218f413-d390-4255-a660-7b07886c785c\") "
Sep 17 19:32:59.930782 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:32:59.930529    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e218f413-d390-4255-a660-7b07886c785c-kserve-provision-location\") pod \"e218f413-d390-4255-a660-7b07886c785c\" (UID: \"e218f413-d390-4255-a660-7b07886c785c\") "
Sep 17 19:32:59.930782 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:32:59.930587    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e218f413-d390-4255-a660-7b07886c785c-proxy-tls\") pod \"e218f413-d390-4255-a660-7b07886c785c\" (UID: \"e218f413-d390-4255-a660-7b07886c785c\") "
Sep 17 19:32:59.930966 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:32:59.930931    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e218f413-d390-4255-a660-7b07886c785c-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "e218f413-d390-4255-a660-7b07886c785c" (UID: "e218f413-d390-4255-a660-7b07886c785c"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 19:32:59.931021 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:32:59.930979    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e218f413-d390-4255-a660-7b07886c785c-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "e218f413-d390-4255-a660-7b07886c785c" (UID: "e218f413-d390-4255-a660-7b07886c785c"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 19:32:59.931414 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:32:59.931392    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e218f413-d390-4255-a660-7b07886c785c-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "e218f413-d390-4255-a660-7b07886c785c" (UID: "e218f413-d390-4255-a660-7b07886c785c"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 19:32:59.937654 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:32:59.937602    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e218f413-d390-4255-a660-7b07886c785c-kube-api-access-knqbf" (OuterVolumeSpecName: "kube-api-access-knqbf") pod "e218f413-d390-4255-a660-7b07886c785c" (UID: "e218f413-d390-4255-a660-7b07886c785c"). InnerVolumeSpecName "kube-api-access-knqbf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 19:32:59.938237 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:32:59.938218    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e218f413-d390-4255-a660-7b07886c785c-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "e218f413-d390-4255-a660-7b07886c785c" (UID: "e218f413-d390-4255-a660-7b07886c785c"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 19:33:00.032122 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:33:00.032085    2582 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e218f413-d390-4255-a660-7b07886c785c-proxy-tls\") on node \"ip-10-0-131-232.ec2.internal\" DevicePath \"\""
Sep 17 19:33:00.032281 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:33:00.032138    2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-knqbf\" (UniqueName: \"kubernetes.io/projected/e218f413-d390-4255-a660-7b07886c785c-kube-api-access-knqbf\") on node \"ip-10-0-131-232.ec2.internal\" DevicePath \"\""
Sep 17 19:33:00.032281 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:33:00.032154    2582 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/e218f413-d390-4255-a660-7b07886c785c-cabundle-cert\") on node \"ip-10-0-131-232.ec2.internal\" DevicePath \"\""
Sep 17 19:33:00.032281 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:33:00.032167    2582 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e218f413-d390-4255-a660-7b07886c785c-kube-rbac-proxy-sar-config\") on node \"ip-10-0-131-232.ec2.internal\" DevicePath \"\""
Sep 17 19:33:00.032281 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:33:00.032177    2582 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e218f413-d390-4255-a660-7b07886c785c-kserve-provision-location\") on node \"ip-10-0-131-232.ec2.internal\" DevicePath \"\""
Sep 17 19:33:00.588206 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:33:00.588169    2582 generic.go:356] "Generic (PLEG): container finished" podID="e218f413-d390-4255-a660-7b07886c785c" containerID="336d11f3221e43582779ff51e3c1b1bb26ba73fb31dce0441569b590eaae360a" exitCode=0
Sep 17 19:33:00.588405 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:33:00.588271    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-797ff577-6z5xx" event={"ID":"e218f413-d390-4255-a660-7b07886c785c","Type":"ContainerDied","Data":"336d11f3221e43582779ff51e3c1b1bb26ba73fb31dce0441569b590eaae360a"}
Sep 17 19:33:00.588405 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:33:00.588299    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-797ff577-6z5xx"
Sep 17 19:33:00.588405 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:33:00.588363    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-797ff577-6z5xx" event={"ID":"e218f413-d390-4255-a660-7b07886c785c","Type":"ContainerDied","Data":"e9126fbf549da5a527a43cc9a51c2127fe1c3a9c468ea1981dc6b1224dafa9fa"}
Sep 17 19:33:00.588405 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:33:00.588396    2582 scope.go:117] "RemoveContainer" containerID="336d11f3221e43582779ff51e3c1b1bb26ba73fb31dce0441569b590eaae360a"
Sep 17 19:33:00.597498 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:33:00.597483    2582 scope.go:117] "RemoveContainer" containerID="537574cb3416c4feb9e57be9571c15a5bff216a771e41cddc1360fae18eb62c4"
Sep 17 19:33:00.605270 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:33:00.605254    2582 scope.go:117] "RemoveContainer" containerID="72ef4402fd9f877db3cd0b65c3017bed4288f4ba45b0bd93219b52873a4255d8"
Sep 17 19:33:00.612859 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:33:00.612812    2582 scope.go:117] "RemoveContainer" containerID="cffafc977f94adb2bc2fe9b1128431faeefc7cc63350ef01a3cbe702133675ce"
Sep 17 19:33:00.613394 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:33:00.613367    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-797ff577-6z5xx"]
Sep 17 19:33:00.620815 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:33:00.620789    2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-797ff577-6z5xx"]
Sep 17 19:33:00.622151 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:33:00.622133    2582 scope.go:117] "RemoveContainer" containerID="336d11f3221e43582779ff51e3c1b1bb26ba73fb31dce0441569b590eaae360a"
Sep 17 19:33:00.622696 ip-10-0-131-232 kubenswrapper[2582]: E0917 19:33:00.622674    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"336d11f3221e43582779ff51e3c1b1bb26ba73fb31dce0441569b590eaae360a\": container with ID starting with 336d11f3221e43582779ff51e3c1b1bb26ba73fb31dce0441569b590eaae360a not found: ID does not exist" containerID="336d11f3221e43582779ff51e3c1b1bb26ba73fb31dce0441569b590eaae360a"
Sep 17 19:33:00.622782 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:33:00.622705    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"336d11f3221e43582779ff51e3c1b1bb26ba73fb31dce0441569b590eaae360a"} err="failed to get container status \"336d11f3221e43582779ff51e3c1b1bb26ba73fb31dce0441569b590eaae360a\": rpc error: code = NotFound desc = could not find container \"336d11f3221e43582779ff51e3c1b1bb26ba73fb31dce0441569b590eaae360a\": container with ID starting with 336d11f3221e43582779ff51e3c1b1bb26ba73fb31dce0441569b590eaae360a not found: ID does not exist"
Sep 17 19:33:00.622782 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:33:00.622728    2582 scope.go:117] "RemoveContainer" containerID="537574cb3416c4feb9e57be9571c15a5bff216a771e41cddc1360fae18eb62c4"
Sep 17 19:33:00.623052 ip-10-0-131-232 kubenswrapper[2582]: E0917 19:33:00.623029    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"537574cb3416c4feb9e57be9571c15a5bff216a771e41cddc1360fae18eb62c4\": container with ID starting with 537574cb3416c4feb9e57be9571c15a5bff216a771e41cddc1360fae18eb62c4 not found: ID does not exist" containerID="537574cb3416c4feb9e57be9571c15a5bff216a771e41cddc1360fae18eb62c4"
Sep 17 19:33:00.623113 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:33:00.623058    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"537574cb3416c4feb9e57be9571c15a5bff216a771e41cddc1360fae18eb62c4"} err="failed to get container status \"537574cb3416c4feb9e57be9571c15a5bff216a771e41cddc1360fae18eb62c4\": rpc error: code = NotFound desc = could not find container \"537574cb3416c4feb9e57be9571c15a5bff216a771e41cddc1360fae18eb62c4\": container with ID starting with 537574cb3416c4feb9e57be9571c15a5bff216a771e41cddc1360fae18eb62c4 not found: ID does not exist"
Sep 17 19:33:00.623113 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:33:00.623074    2582 scope.go:117] "RemoveContainer" containerID="72ef4402fd9f877db3cd0b65c3017bed4288f4ba45b0bd93219b52873a4255d8"
Sep 17 19:33:00.623335 ip-10-0-131-232 kubenswrapper[2582]: E0917 19:33:00.623317    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"72ef4402fd9f877db3cd0b65c3017bed4288f4ba45b0bd93219b52873a4255d8\": container with ID starting with 72ef4402fd9f877db3cd0b65c3017bed4288f4ba45b0bd93219b52873a4255d8 not found: ID does not exist" containerID="72ef4402fd9f877db3cd0b65c3017bed4288f4ba45b0bd93219b52873a4255d8"
Sep 17 19:33:00.623410 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:33:00.623343    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"72ef4402fd9f877db3cd0b65c3017bed4288f4ba45b0bd93219b52873a4255d8"} err="failed to get container status \"72ef4402fd9f877db3cd0b65c3017bed4288f4ba45b0bd93219b52873a4255d8\": rpc error: code = NotFound desc = could not find container \"72ef4402fd9f877db3cd0b65c3017bed4288f4ba45b0bd93219b52873a4255d8\": container with ID starting with 72ef4402fd9f877db3cd0b65c3017bed4288f4ba45b0bd93219b52873a4255d8 not found: ID does not exist"
Sep 17 19:33:00.623410 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:33:00.623364    2582 scope.go:117] "RemoveContainer" containerID="cffafc977f94adb2bc2fe9b1128431faeefc7cc63350ef01a3cbe702133675ce"
Sep 17 19:33:00.623653 ip-10-0-131-232 kubenswrapper[2582]: E0917 19:33:00.623638    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cffafc977f94adb2bc2fe9b1128431faeefc7cc63350ef01a3cbe702133675ce\": container with ID starting with cffafc977f94adb2bc2fe9b1128431faeefc7cc63350ef01a3cbe702133675ce not found: ID does not exist" containerID="cffafc977f94adb2bc2fe9b1128431faeefc7cc63350ef01a3cbe702133675ce"
Sep 17 19:33:00.623705 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:33:00.623661    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cffafc977f94adb2bc2fe9b1128431faeefc7cc63350ef01a3cbe702133675ce"} err="failed to get container status \"cffafc977f94adb2bc2fe9b1128431faeefc7cc63350ef01a3cbe702133675ce\": rpc error: code = NotFound desc = could not find container \"cffafc977f94adb2bc2fe9b1128431faeefc7cc63350ef01a3cbe702133675ce\": container with ID starting with cffafc977f94adb2bc2fe9b1128431faeefc7cc63350ef01a3cbe702133675ce not found: ID does not exist"
Sep 17 19:33:00.768291 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:33:00.768257    2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e218f413-d390-4255-a660-7b07886c785c" path="/var/lib/kubelet/pods/e218f413-d390-4255-a660-7b07886c785c/volumes"
Sep 17 19:33:04.717899 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:33:04.717864    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-f6kjs_7cd887c0-2cc8-4363-b553-8d2b4e10f158/global-pull-secret-syncer/0.log"
Sep 17 19:33:04.870023 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:33:04.870000    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-qwhmv_fe7c72bc-812d-4272-8352-78f8fa2c19d1/konnectivity-agent/0.log"
Sep 17 19:33:04.892437 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:33:04.892412    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-131-232.ec2.internal_a1f9bd0034d8d01c3b2f70180b25c21f/haproxy/0.log"
Sep 17 19:33:09.289025 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:33:09.288992    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-rrzmt_53864523-604f-40d9-be99-cab48397a2f5/node-exporter/0.log"
Sep 17 19:33:09.309843 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:33:09.309808    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-rrzmt_53864523-604f-40d9-be99-cab48397a2f5/kube-rbac-proxy/0.log"
Sep 17 19:33:09.340655 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:33:09.340638    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-rrzmt_53864523-604f-40d9-be99-cab48397a2f5/init-textfile/0.log"
Sep 17 19:33:12.533344 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:33:12.533307    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_console-5c5887565b-f5gjx_cbbdc2a7-0708-4445-97a8-2afd903120e3/console/0.log"
Sep 17 19:33:13.207484 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:33:13.207443    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-n9b8n/perf-node-gather-daemonset-4ncwb"]
Sep 17 19:33:13.207728 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:33:13.207716    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e218f413-d390-4255-a660-7b07886c785c" containerName="kserve-container"
Sep 17 19:33:13.207772 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:33:13.207730    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="e218f413-d390-4255-a660-7b07886c785c" containerName="kserve-container"
Sep 17 19:33:13.207772 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:33:13.207739    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e218f413-d390-4255-a660-7b07886c785c" containerName="agent"
Sep 17 19:33:13.207772 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:33:13.207745    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="e218f413-d390-4255-a660-7b07886c785c" containerName="agent"
Sep 17 19:33:13.207772 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:33:13.207756    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e218f413-d390-4255-a660-7b07886c785c" containerName="storage-initializer"
Sep 17 19:33:13.207772 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:33:13.207761    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="e218f413-d390-4255-a660-7b07886c785c" containerName="storage-initializer"
Sep 17 19:33:13.207947 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:33:13.207775    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e218f413-d390-4255-a660-7b07886c785c" containerName="kube-rbac-proxy"
Sep 17 19:33:13.207947 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:33:13.207780    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="e218f413-d390-4255-a660-7b07886c785c" containerName="kube-rbac-proxy"
Sep 17 19:33:13.207947 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:33:13.207841    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="e218f413-d390-4255-a660-7b07886c785c" containerName="kserve-container"
Sep 17 19:33:13.207947 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:33:13.207851    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="e218f413-d390-4255-a660-7b07886c785c" containerName="kube-rbac-proxy"
Sep 17 19:33:13.207947 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:33:13.207857    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="e218f413-d390-4255-a660-7b07886c785c" containerName="agent"
Sep 17 19:33:13.212502 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:33:13.212480    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-n9b8n/perf-node-gather-daemonset-4ncwb"
Sep 17 19:33:13.214959 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:33:13.214938    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-n9b8n\"/\"kube-root-ca.crt\""
Sep 17 19:33:13.215085 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:33:13.214960    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-n9b8n\"/\"default-dockercfg-qplkf\""
Sep 17 19:33:13.215974 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:33:13.215959    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-n9b8n\"/\"openshift-service-ca.crt\""
Sep 17 19:33:13.218258 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:33:13.218236    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-n9b8n/perf-node-gather-daemonset-4ncwb"]
Sep 17 19:33:13.332553 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:33:13.332515    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/00aa0cb0-21a2-4f5c-b452-5794196038c3-proc\") pod \"perf-node-gather-daemonset-4ncwb\" (UID: \"00aa0cb0-21a2-4f5c-b452-5794196038c3\") " pod="openshift-must-gather-n9b8n/perf-node-gather-daemonset-4ncwb"
Sep 17 19:33:13.332717 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:33:13.332565    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/00aa0cb0-21a2-4f5c-b452-5794196038c3-podres\") pod \"perf-node-gather-daemonset-4ncwb\" (UID: \"00aa0cb0-21a2-4f5c-b452-5794196038c3\") " pod="openshift-must-gather-n9b8n/perf-node-gather-daemonset-4ncwb"
Sep 17 19:33:13.332717 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:33:13.332619    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/00aa0cb0-21a2-4f5c-b452-5794196038c3-lib-modules\") pod \"perf-node-gather-daemonset-4ncwb\" (UID: \"00aa0cb0-21a2-4f5c-b452-5794196038c3\") " pod="openshift-must-gather-n9b8n/perf-node-gather-daemonset-4ncwb"
Sep 17 19:33:13.332717 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:33:13.332654    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xjwf4\" (UniqueName: \"kubernetes.io/projected/00aa0cb0-21a2-4f5c-b452-5794196038c3-kube-api-access-xjwf4\") pod \"perf-node-gather-daemonset-4ncwb\" (UID: \"00aa0cb0-21a2-4f5c-b452-5794196038c3\") " pod="openshift-must-gather-n9b8n/perf-node-gather-daemonset-4ncwb"
Sep 17 19:33:13.332851 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:33:13.332752    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/00aa0cb0-21a2-4f5c-b452-5794196038c3-sys\") pod \"perf-node-gather-daemonset-4ncwb\" (UID: \"00aa0cb0-21a2-4f5c-b452-5794196038c3\") " pod="openshift-must-gather-n9b8n/perf-node-gather-daemonset-4ncwb"
Sep 17 19:33:13.434061 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:33:13.434036    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/00aa0cb0-21a2-4f5c-b452-5794196038c3-proc\") pod \"perf-node-gather-daemonset-4ncwb\" (UID: \"00aa0cb0-21a2-4f5c-b452-5794196038c3\") " pod="openshift-must-gather-n9b8n/perf-node-gather-daemonset-4ncwb"
Sep 17 19:33:13.434184 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:33:13.434068    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/00aa0cb0-21a2-4f5c-b452-5794196038c3-podres\") pod \"perf-node-gather-daemonset-4ncwb\" (UID: \"00aa0cb0-21a2-4f5c-b452-5794196038c3\") " pod="openshift-must-gather-n9b8n/perf-node-gather-daemonset-4ncwb"
Sep 17 19:33:13.434184 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:33:13.434094    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/00aa0cb0-21a2-4f5c-b452-5794196038c3-lib-modules\") pod \"perf-node-gather-daemonset-4ncwb\" (UID: \"00aa0cb0-21a2-4f5c-b452-5794196038c3\") " pod="openshift-must-gather-n9b8n/perf-node-gather-daemonset-4ncwb"
Sep 17 19:33:13.434184 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:33:13.434117    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xjwf4\" (UniqueName: \"kubernetes.io/projected/00aa0cb0-21a2-4f5c-b452-5794196038c3-kube-api-access-xjwf4\") pod \"perf-node-gather-daemonset-4ncwb\" (UID: \"00aa0cb0-21a2-4f5c-b452-5794196038c3\") " pod="openshift-must-gather-n9b8n/perf-node-gather-daemonset-4ncwb"
Sep 17 19:33:13.434184 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:33:13.434148    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/00aa0cb0-21a2-4f5c-b452-5794196038c3-sys\") pod \"perf-node-gather-daemonset-4ncwb\" (UID: \"00aa0cb0-21a2-4f5c-b452-5794196038c3\") " pod="openshift-must-gather-n9b8n/perf-node-gather-daemonset-4ncwb"
Sep 17 19:33:13.434184 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:33:13.434168    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/00aa0cb0-21a2-4f5c-b452-5794196038c3-proc\") pod \"perf-node-gather-daemonset-4ncwb\" (UID: \"00aa0cb0-21a2-4f5c-b452-5794196038c3\") " pod="openshift-must-gather-n9b8n/perf-node-gather-daemonset-4ncwb"
Sep 17 19:33:13.434347 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:33:13.434234    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/00aa0cb0-21a2-4f5c-b452-5794196038c3-sys\") pod \"perf-node-gather-daemonset-4ncwb\" (UID: \"00aa0cb0-21a2-4f5c-b452-5794196038c3\") " pod="openshift-must-gather-n9b8n/perf-node-gather-daemonset-4ncwb"
Sep 17 19:33:13.434347 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:33:13.434241    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/00aa0cb0-21a2-4f5c-b452-5794196038c3-lib-modules\") pod \"perf-node-gather-daemonset-4ncwb\" (UID: \"00aa0cb0-21a2-4f5c-b452-5794196038c3\") " pod="openshift-must-gather-n9b8n/perf-node-gather-daemonset-4ncwb"
Sep 17 19:33:13.434347 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:33:13.434241    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/00aa0cb0-21a2-4f5c-b452-5794196038c3-podres\") pod \"perf-node-gather-daemonset-4ncwb\" (UID: \"00aa0cb0-21a2-4f5c-b452-5794196038c3\") " pod="openshift-must-gather-n9b8n/perf-node-gather-daemonset-4ncwb"
Sep 17 19:33:13.444240 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:33:13.444214    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xjwf4\" (UniqueName: \"kubernetes.io/projected/00aa0cb0-21a2-4f5c-b452-5794196038c3-kube-api-access-xjwf4\") pod \"perf-node-gather-daemonset-4ncwb\" (UID: \"00aa0cb0-21a2-4f5c-b452-5794196038c3\") " pod="openshift-must-gather-n9b8n/perf-node-gather-daemonset-4ncwb"
Sep 17 19:33:13.523278 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:33:13.523185    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-n9b8n/perf-node-gather-daemonset-4ncwb"
Sep 17 19:33:13.793192 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:33:13.793110    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-n9b8n/perf-node-gather-daemonset-4ncwb"]
Sep 17 19:33:13.795787 ip-10-0-131-232 kubenswrapper[2582]: W0917 19:33:13.795756    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod00aa0cb0_21a2_4f5c_b452_5794196038c3.slice/crio-5913c945fb6e64f29758a7a63f1f3edd35d924ddb443bf422c69f06e74242e5f WatchSource:0}: Error finding container 5913c945fb6e64f29758a7a63f1f3edd35d924ddb443bf422c69f06e74242e5f: Status 404 returned error can't find the container with id 5913c945fb6e64f29758a7a63f1f3edd35d924ddb443bf422c69f06e74242e5f
Sep 17 19:33:14.008985 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:33:14.008955    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-67qx4_413bc45c-0bcb-421e-aa86-c1b75aea7d11/dns/0.log"
Sep 17 19:33:14.031477 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:33:14.031421    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-67qx4_413bc45c-0bcb-421e-aa86-c1b75aea7d11/kube-rbac-proxy/0.log"
Sep 17 19:33:14.182699 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:33:14.182670    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-cwtkf_46629fc1-07ce-45b8-9d73-453abfa383b9/dns-node-resolver/0.log"
Sep 17 19:33:14.647248 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:33:14.647213    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-n9b8n/perf-node-gather-daemonset-4ncwb" event={"ID":"00aa0cb0-21a2-4f5c-b452-5794196038c3","Type":"ContainerStarted","Data":"8352c1d6ccc6f082ad5e8c2f7fba85e31bbed8fbb336bd18471ee40309e9938c"}
Sep 17 19:33:14.647417 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:33:14.647262    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-n9b8n/perf-node-gather-daemonset-4ncwb" event={"ID":"00aa0cb0-21a2-4f5c-b452-5794196038c3","Type":"ContainerStarted","Data":"5913c945fb6e64f29758a7a63f1f3edd35d924ddb443bf422c69f06e74242e5f"}
Sep 17 19:33:14.647417 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:33:14.647328    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-n9b8n/perf-node-gather-daemonset-4ncwb"
Sep 17 19:33:14.665615 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:33:14.665575    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-n9b8n/perf-node-gather-daemonset-4ncwb" podStartSLOduration=1.665561958 podStartE2EDuration="1.665561958s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 19:33:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 19:33:14.665076962 +0000 UTC m=+1610.436748024" watchObservedRunningTime="2026-09-17 19:33:14.665561958 +0000 UTC m=+1610.437233019"
Sep 17 19:33:14.712181 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:33:14.712141    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-64g8v_271e6574-9235-474f-b0bc-06ec57117390/node-ca/0.log"
Sep 17 19:33:15.859109 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:33:15.859028    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-4b2jj_e6357f17-007b-4c5c-83c8-7a3d1c2c0e85/serve-healthcheck-canary/0.log"
Sep 17 19:33:16.407133 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:33:16.407103    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-kff2k_984299cc-0712-45a1-89b5-883a6508992c/kube-rbac-proxy/0.log"
Sep 17 19:33:16.428071 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:33:16.428028    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-kff2k_984299cc-0712-45a1-89b5-883a6508992c/exporter/0.log"
Sep 17 19:33:16.449159 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:33:16.449117    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-kff2k_984299cc-0712-45a1-89b5-883a6508992c/extractor/0.log"
Sep 17 19:33:18.731241 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:33:18.731202    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_kserve-controller-manager-66576f5db6-rs7mf_138aebae-3002-4042-b6d4-9bef95f88da9/manager/0.log"
Sep 17 19:33:18.843838 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:33:18.843803    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_model-serving-api-7b954ffc95-mbjd9_b55ebb64-9548-4169-b74c-936fc5290e50/server/0.log"
Sep 17 19:33:19.081987 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:33:19.081905    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_odh-model-controller-7b966d7984-ck5rg_1f09d228-0f0e-47c4-96d3-bde8bf84941b/manager/0.log"
Sep 17 19:33:19.165343 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:33:19.165318    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_seaweedfs-64568bcd49-nvdj7_ce9e3abb-c2ec-4228-b7ca-c1ac82737c59/seaweedfs/0.log"
Sep 17 19:33:20.660900 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:33:20.660871    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-n9b8n/perf-node-gather-daemonset-4ncwb"
Sep 17 19:33:23.898014 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:33:23.897969    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator-operator_kube-storage-version-migrator-operator-55889f7b47-2w6p2_7074d19e-e52e-460a-a702-44130277bea5/kube-storage-version-migrator-operator/1.log"
Sep 17 19:33:23.900077 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:33:23.900051    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator-operator_kube-storage-version-migrator-operator-55889f7b47-2w6p2_7074d19e-e52e-460a-a702-44130277bea5/kube-storage-version-migrator-operator/0.log"
Sep 17 19:33:25.268334 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:33:25.268305    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-wz7lk_03964cb9-d415-48a8-9a6a-834979450459/kube-multus-additional-cni-plugins/0.log"
Sep 17 19:33:25.290245 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:33:25.290207    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-wz7lk_03964cb9-d415-48a8-9a6a-834979450459/egress-router-binary-copy/0.log"
Sep 17 19:33:25.312935 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:33:25.312905    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-wz7lk_03964cb9-d415-48a8-9a6a-834979450459/cni-plugins/0.log"
Sep 17 19:33:25.336763 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:33:25.336705    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-wz7lk_03964cb9-d415-48a8-9a6a-834979450459/bond-cni-plugin/0.log"
Sep 17 19:33:25.358700 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:33:25.358674    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-wz7lk_03964cb9-d415-48a8-9a6a-834979450459/routeoverride-cni/0.log"
Sep 17 19:33:25.382710 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:33:25.382686    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-wz7lk_03964cb9-d415-48a8-9a6a-834979450459/whereabouts-cni-bincopy/0.log"
Sep 17 19:33:25.403194 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:33:25.403167    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-wz7lk_03964cb9-d415-48a8-9a6a-834979450459/whereabouts-cni/0.log"
Sep 17 19:33:25.475704 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:33:25.475673    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-fppr7_aa2ade5e-8564-46a7-819c-75c57cb0961d/kube-multus/0.log"
Sep 17 19:33:25.595488 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:33:25.595421    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-hj5r7_cc1b159b-69e7-404d-9a25-bcf0ff8148c5/network-metrics-daemon/0.log"
Sep 17 19:33:25.620747 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:33:25.620717    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-hj5r7_cc1b159b-69e7-404d-9a25-bcf0ff8148c5/kube-rbac-proxy/0.log"
Sep 17 19:33:26.581417 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:33:26.581385    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-kdm8s_c1b79c18-774f-4b78-a6a9-7ea8a229aaae/ovn-controller/0.log"
Sep 17 19:33:26.600310 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:33:26.600281    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-kdm8s_c1b79c18-774f-4b78-a6a9-7ea8a229aaae/ovn-acl-logging/1.log"
Sep 17 19:33:26.600465 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:33:26.600381    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-kdm8s_c1b79c18-774f-4b78-a6a9-7ea8a229aaae/ovn-acl-logging/0.log"
Sep 17 19:33:26.618727 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:33:26.618707    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-kdm8s_c1b79c18-774f-4b78-a6a9-7ea8a229aaae/kube-rbac-proxy-node/0.log"
Sep 17 19:33:26.643154 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:33:26.643136    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-kdm8s_c1b79c18-774f-4b78-a6a9-7ea8a229aaae/kube-rbac-proxy-ovn-metrics/0.log"
Sep 17 19:33:26.663615 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:33:26.663596    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-kdm8s_c1b79c18-774f-4b78-a6a9-7ea8a229aaae/northd/0.log"
Sep 17 19:33:26.684505 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:33:26.684473    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-kdm8s_c1b79c18-774f-4b78-a6a9-7ea8a229aaae/nbdb/0.log"
Sep 17 19:33:26.704743 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:33:26.704727    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-kdm8s_c1b79c18-774f-4b78-a6a9-7ea8a229aaae/sbdb/0.log"
Sep 17 19:33:26.761323 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:33:26.761297    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-kdm8s_c1b79c18-774f-4b78-a6a9-7ea8a229aaae/ovnkube-controller/0.log"
Sep 17 19:33:28.159559 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:33:28.159530    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-diagnostics_network-check-source-7cbc45fddc-h5bj4_8d44cde2-a7be-41dd-b5aa-f2eda8e86447/check-endpoints/0.log"
Sep 17 19:33:28.184336 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:33:28.184309    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-diagnostics_network-check-target-jhjrs_07daf4ff-9d55-4476-aefd-df39d86849a6/network-check-target-container/0.log"
Sep 17 19:33:29.227782 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:33:29.227754    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-operator_iptables-alerter-kzblg_f902d60d-52fe-4458-94ce-21fc761c4bb5/iptables-alerter/0.log"
Sep 17 19:33:29.929418 ip-10-0-131-232 kubenswrapper[2582]: I0917 19:33:29.929388    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cluster-node-tuning-operator_tuned-g8qm4_f661fc84-37f7-4cb9-a5b5-be6673fbd0dd/tuned/0.log"