Sep 09 19:53:21.248595 ip-10-0-136-183 systemd[1]: kubelet.service: Failed to load environment files: No such file or directory
Sep 09 19:53:21.248605 ip-10-0-136-183 systemd[1]: kubelet.service: Failed to run 'start-pre' task: No such file or directory
Sep 09 19:53:21.248612 ip-10-0-136-183 systemd[1]: kubelet.service: Failed with result 'resources'.
Sep 09 19:53:21.248763 ip-10-0-136-183 systemd[1]: Failed to start Kubernetes Kubelet.
Sep 09 19:53:31.456908 ip-10-0-136-183 systemd[1]: kubelet.service: Failed to schedule restart job: Unit crio.service not found.
Sep 09 19:53:31.456925 ip-10-0-136-183 systemd[1]: kubelet.service: Failed with result 'resources'.
-- Boot 8bd4ba6ecca44bd6847ff2f3f3defe02 --
Sep 09 19:55:14.600769 ip-10-0-136-183 systemd[1]: Starting Kubernetes Kubelet...
Sep 09 19:55:14.981145 ip-10-0-136-183 kubenswrapper[2574]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 09 19:55:14.981145 ip-10-0-136-183 kubenswrapper[2574]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Sep 09 19:55:14.981145 ip-10-0-136-183 kubenswrapper[2574]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 09 19:55:14.981145 ip-10-0-136-183 kubenswrapper[2574]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI.
Sep 09 19:55:14.981145 ip-10-0-136-183 kubenswrapper[2574]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 09 19:55:14.982129 ip-10-0-136-183 kubenswrapper[2574]: I0909 19:55:14.982033    2574 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime"
Sep 09 19:55:14.988145 ip-10-0-136-183 kubenswrapper[2574]: W0909 19:55:14.988119    2574 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 09 19:55:14.988145 ip-10-0-136-183 kubenswrapper[2574]: W0909 19:55:14.988139    2574 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 09 19:55:14.988145 ip-10-0-136-183 kubenswrapper[2574]: W0909 19:55:14.988144    2574 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 09 19:55:14.988145 ip-10-0-136-183 kubenswrapper[2574]: W0909 19:55:14.988148    2574 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 09 19:55:14.988145 ip-10-0-136-183 kubenswrapper[2574]: W0909 19:55:14.988153    2574 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 09 19:55:14.988442 ip-10-0-136-183 kubenswrapper[2574]: W0909 19:55:14.988157    2574 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 09 19:55:14.988442 ip-10-0-136-183 kubenswrapper[2574]: W0909 19:55:14.988163    2574 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 09 19:55:14.988442 ip-10-0-136-183 kubenswrapper[2574]: W0909 19:55:14.988167    2574 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 09 19:55:14.988442 ip-10-0-136-183 kubenswrapper[2574]: W0909 19:55:14.988171    2574 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 09 19:55:14.988442 ip-10-0-136-183 kubenswrapper[2574]: W0909 19:55:14.988175    2574 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 09 19:55:14.988442 ip-10-0-136-183 kubenswrapper[2574]: W0909 19:55:14.988179    2574 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 09 19:55:14.988442 ip-10-0-136-183 kubenswrapper[2574]: W0909 19:55:14.988182    2574 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 09 19:55:14.988442 ip-10-0-136-183 kubenswrapper[2574]: W0909 19:55:14.988187    2574 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 09 19:55:14.988442 ip-10-0-136-183 kubenswrapper[2574]: W0909 19:55:14.988192    2574 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 09 19:55:14.988442 ip-10-0-136-183 kubenswrapper[2574]: W0909 19:55:14.988197    2574 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 09 19:55:14.988442 ip-10-0-136-183 kubenswrapper[2574]: W0909 19:55:14.988201    2574 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 09 19:55:14.988442 ip-10-0-136-183 kubenswrapper[2574]: W0909 19:55:14.988204    2574 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 09 19:55:14.988442 ip-10-0-136-183 kubenswrapper[2574]: W0909 19:55:14.988210    2574 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 09 19:55:14.988442 ip-10-0-136-183 kubenswrapper[2574]: W0909 19:55:14.988214    2574 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 09 19:55:14.988442 ip-10-0-136-183 kubenswrapper[2574]: W0909 19:55:14.988218    2574 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 09 19:55:14.988442 ip-10-0-136-183 kubenswrapper[2574]: W0909 19:55:14.988223    2574 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 09 19:55:14.988442 ip-10-0-136-183 kubenswrapper[2574]: W0909 19:55:14.988226    2574 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 09 19:55:14.988442 ip-10-0-136-183 kubenswrapper[2574]: W0909 19:55:14.988230    2574 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 09 19:55:14.988442 ip-10-0-136-183 kubenswrapper[2574]: W0909 19:55:14.988234    2574 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 09 19:55:14.989230 ip-10-0-136-183 kubenswrapper[2574]: W0909 19:55:14.988238    2574 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 09 19:55:14.989230 ip-10-0-136-183 kubenswrapper[2574]: W0909 19:55:14.988242    2574 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 09 19:55:14.989230 ip-10-0-136-183 kubenswrapper[2574]: W0909 19:55:14.988248    2574 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 09 19:55:14.989230 ip-10-0-136-183 kubenswrapper[2574]: W0909 19:55:14.988252    2574 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 09 19:55:14.989230 ip-10-0-136-183 kubenswrapper[2574]: W0909 19:55:14.988256    2574 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 09 19:55:14.989230 ip-10-0-136-183 kubenswrapper[2574]: W0909 19:55:14.988259    2574 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 09 19:55:14.989230 ip-10-0-136-183 kubenswrapper[2574]: W0909 19:55:14.988263    2574 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 09 19:55:14.989230 ip-10-0-136-183 kubenswrapper[2574]: W0909 19:55:14.988267    2574 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 09 19:55:14.989230 ip-10-0-136-183 kubenswrapper[2574]: W0909 19:55:14.988271    2574 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 09 19:55:14.989230 ip-10-0-136-183 kubenswrapper[2574]: W0909 19:55:14.988275    2574 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 09 19:55:14.989230 ip-10-0-136-183 kubenswrapper[2574]: W0909 19:55:14.988279    2574 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 09 19:55:14.989230 ip-10-0-136-183 kubenswrapper[2574]: W0909 19:55:14.988283    2574 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 09 19:55:14.989230 ip-10-0-136-183 kubenswrapper[2574]: W0909 19:55:14.988287    2574 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 09 19:55:14.989230 ip-10-0-136-183 kubenswrapper[2574]: W0909 19:55:14.988291    2574 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 09 19:55:14.989230 ip-10-0-136-183 kubenswrapper[2574]: W0909 19:55:14.988296    2574 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 09 19:55:14.989230 ip-10-0-136-183 kubenswrapper[2574]: W0909 19:55:14.988300    2574 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 09 19:55:14.989230 ip-10-0-136-183 kubenswrapper[2574]: W0909 19:55:14.988304    2574 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 09 19:55:14.989230 ip-10-0-136-183 kubenswrapper[2574]: W0909 19:55:14.988308    2574 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 09 19:55:14.989230 ip-10-0-136-183 kubenswrapper[2574]: W0909 19:55:14.988312    2574 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 09 19:55:14.989230 ip-10-0-136-183 kubenswrapper[2574]: W0909 19:55:14.988316    2574 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 09 19:55:14.990104 ip-10-0-136-183 kubenswrapper[2574]: W0909 19:55:14.988321    2574 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 09 19:55:14.990104 ip-10-0-136-183 kubenswrapper[2574]: W0909 19:55:14.988325    2574 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 09 19:55:14.990104 ip-10-0-136-183 kubenswrapper[2574]: W0909 19:55:14.988329    2574 feature_gate.go:328] unrecognized feature gate: Example2
Sep 09 19:55:14.990104 ip-10-0-136-183 kubenswrapper[2574]: W0909 19:55:14.988333    2574 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 09 19:55:14.990104 ip-10-0-136-183 kubenswrapper[2574]: W0909 19:55:14.988337    2574 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 09 19:55:14.990104 ip-10-0-136-183 kubenswrapper[2574]: W0909 19:55:14.988341    2574 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 09 19:55:14.990104 ip-10-0-136-183 kubenswrapper[2574]: W0909 19:55:14.988355    2574 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 09 19:55:14.990104 ip-10-0-136-183 kubenswrapper[2574]: W0909 19:55:14.988359    2574 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 09 19:55:14.990104 ip-10-0-136-183 kubenswrapper[2574]: W0909 19:55:14.988363    2574 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 09 19:55:14.990104 ip-10-0-136-183 kubenswrapper[2574]: W0909 19:55:14.988368    2574 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 09 19:55:14.990104 ip-10-0-136-183 kubenswrapper[2574]: W0909 19:55:14.988372    2574 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 09 19:55:14.990104 ip-10-0-136-183 kubenswrapper[2574]: W0909 19:55:14.988376    2574 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 09 19:55:14.990104 ip-10-0-136-183 kubenswrapper[2574]: W0909 19:55:14.988381    2574 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 09 19:55:14.990104 ip-10-0-136-183 kubenswrapper[2574]: W0909 19:55:14.988386    2574 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 09 19:55:14.990104 ip-10-0-136-183 kubenswrapper[2574]: W0909 19:55:14.988390    2574 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 09 19:55:14.990104 ip-10-0-136-183 kubenswrapper[2574]: W0909 19:55:14.988394    2574 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 09 19:55:14.990104 ip-10-0-136-183 kubenswrapper[2574]: W0909 19:55:14.988399    2574 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 09 19:55:14.990104 ip-10-0-136-183 kubenswrapper[2574]: W0909 19:55:14.988403    2574 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 09 19:55:14.990104 ip-10-0-136-183 kubenswrapper[2574]: W0909 19:55:14.988407    2574 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 09 19:55:14.990104 ip-10-0-136-183 kubenswrapper[2574]: W0909 19:55:14.988411    2574 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 09 19:55:14.991012 ip-10-0-136-183 kubenswrapper[2574]: W0909 19:55:14.988416    2574 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 09 19:55:14.991012 ip-10-0-136-183 kubenswrapper[2574]: W0909 19:55:14.988420    2574 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 09 19:55:14.991012 ip-10-0-136-183 kubenswrapper[2574]: W0909 19:55:14.988424    2574 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 09 19:55:14.991012 ip-10-0-136-183 kubenswrapper[2574]: W0909 19:55:14.988428    2574 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 09 19:55:14.991012 ip-10-0-136-183 kubenswrapper[2574]: W0909 19:55:14.988433    2574 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 09 19:55:14.991012 ip-10-0-136-183 kubenswrapper[2574]: W0909 19:55:14.988437    2574 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 09 19:55:14.991012 ip-10-0-136-183 kubenswrapper[2574]: W0909 19:55:14.988442    2574 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 09 19:55:14.991012 ip-10-0-136-183 kubenswrapper[2574]: W0909 19:55:14.988446    2574 feature_gate.go:328] unrecognized feature gate: Example
Sep 09 19:55:14.991012 ip-10-0-136-183 kubenswrapper[2574]: W0909 19:55:14.988450    2574 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 09 19:55:14.991012 ip-10-0-136-183 kubenswrapper[2574]: W0909 19:55:14.988472    2574 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 09 19:55:14.991012 ip-10-0-136-183 kubenswrapper[2574]: W0909 19:55:14.988477    2574 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 09 19:55:14.991012 ip-10-0-136-183 kubenswrapper[2574]: W0909 19:55:14.988481    2574 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 09 19:55:14.991012 ip-10-0-136-183 kubenswrapper[2574]: W0909 19:55:14.988485    2574 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 09 19:55:14.991012 ip-10-0-136-183 kubenswrapper[2574]: W0909 19:55:14.988491    2574 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 09 19:55:14.991012 ip-10-0-136-183 kubenswrapper[2574]: W0909 19:55:14.988496    2574 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 09 19:55:14.991012 ip-10-0-136-183 kubenswrapper[2574]: W0909 19:55:14.988500    2574 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 09 19:55:14.991012 ip-10-0-136-183 kubenswrapper[2574]: W0909 19:55:14.988504    2574 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 09 19:55:14.991012 ip-10-0-136-183 kubenswrapper[2574]: W0909 19:55:14.988508    2574 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 09 19:55:14.991012 ip-10-0-136-183 kubenswrapper[2574]: W0909 19:55:14.988512    2574 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 09 19:55:14.991012 ip-10-0-136-183 kubenswrapper[2574]: W0909 19:55:14.988527    2574 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 09 19:55:14.991866 ip-10-0-136-183 kubenswrapper[2574]: W0909 19:55:14.988533    2574 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 09 19:55:14.991866 ip-10-0-136-183 kubenswrapper[2574]: W0909 19:55:14.988537    2574 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 09 19:55:14.991866 ip-10-0-136-183 kubenswrapper[2574]: W0909 19:55:14.988543    2574 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 09 19:55:14.991866 ip-10-0-136-183 kubenswrapper[2574]: W0909 19:55:14.988547    2574 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 09 19:55:14.991866 ip-10-0-136-183 kubenswrapper[2574]: W0909 19:55:14.988552    2574 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 09 19:55:14.991866 ip-10-0-136-183 kubenswrapper[2574]: W0909 19:55:14.988557    2574 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 09 19:55:14.991866 ip-10-0-136-183 kubenswrapper[2574]: W0909 19:55:14.988561    2574 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 09 19:55:14.991866 ip-10-0-136-183 kubenswrapper[2574]: W0909 19:55:14.988565    2574 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 09 19:55:14.991866 ip-10-0-136-183 kubenswrapper[2574]: W0909 19:55:14.988570    2574 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 09 19:55:14.991866 ip-10-0-136-183 kubenswrapper[2574]: W0909 19:55:14.988574    2574 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 09 19:55:14.991866 ip-10-0-136-183 kubenswrapper[2574]: W0909 19:55:14.988582    2574 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 09 19:55:14.991866 ip-10-0-136-183 kubenswrapper[2574]: W0909 19:55:14.988587    2574 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 09 19:55:14.991866 ip-10-0-136-183 kubenswrapper[2574]: W0909 19:55:14.988591    2574 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 09 19:55:14.991866 ip-10-0-136-183 kubenswrapper[2574]: W0909 19:55:14.988596    2574 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 09 19:55:14.991866 ip-10-0-136-183 kubenswrapper[2574]: W0909 19:55:14.988600    2574 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 09 19:55:14.991866 ip-10-0-136-183 kubenswrapper[2574]: W0909 19:55:14.989813    2574 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 09 19:55:14.991866 ip-10-0-136-183 kubenswrapper[2574]: W0909 19:55:14.989823    2574 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 09 19:55:14.991866 ip-10-0-136-183 kubenswrapper[2574]: W0909 19:55:14.989828    2574 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 09 19:55:14.991866 ip-10-0-136-183 kubenswrapper[2574]: W0909 19:55:14.989834    2574 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 09 19:55:14.991866 ip-10-0-136-183 kubenswrapper[2574]: W0909 19:55:14.989838    2574 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 09 19:55:14.992844 ip-10-0-136-183 kubenswrapper[2574]: W0909 19:55:14.989842    2574 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 09 19:55:14.992844 ip-10-0-136-183 kubenswrapper[2574]: W0909 19:55:14.989847    2574 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 09 19:55:14.992844 ip-10-0-136-183 kubenswrapper[2574]: W0909 19:55:14.989854    2574 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 09 19:55:14.992844 ip-10-0-136-183 kubenswrapper[2574]: W0909 19:55:14.989860    2574 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 09 19:55:14.992844 ip-10-0-136-183 kubenswrapper[2574]: W0909 19:55:14.989866    2574 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 09 19:55:14.992844 ip-10-0-136-183 kubenswrapper[2574]: W0909 19:55:14.989872    2574 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 09 19:55:14.992844 ip-10-0-136-183 kubenswrapper[2574]: W0909 19:55:14.989878    2574 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 09 19:55:14.992844 ip-10-0-136-183 kubenswrapper[2574]: W0909 19:55:14.989882    2574 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 09 19:55:14.992844 ip-10-0-136-183 kubenswrapper[2574]: W0909 19:55:14.989887    2574 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 09 19:55:14.992844 ip-10-0-136-183 kubenswrapper[2574]: W0909 19:55:14.989892    2574 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 09 19:55:14.992844 ip-10-0-136-183 kubenswrapper[2574]: W0909 19:55:14.989896    2574 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 09 19:55:14.992844 ip-10-0-136-183 kubenswrapper[2574]: W0909 19:55:14.989901    2574 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 09 19:55:14.992844 ip-10-0-136-183 kubenswrapper[2574]: W0909 19:55:14.989906    2574 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 09 19:55:14.992844 ip-10-0-136-183 kubenswrapper[2574]: W0909 19:55:14.989910    2574 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 09 19:55:14.992844 ip-10-0-136-183 kubenswrapper[2574]: W0909 19:55:14.989914    2574 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 09 19:55:14.992844 ip-10-0-136-183 kubenswrapper[2574]: W0909 19:55:14.989919    2574 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 09 19:55:14.992844 ip-10-0-136-183 kubenswrapper[2574]: W0909 19:55:14.989923    2574 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 09 19:55:14.992844 ip-10-0-136-183 kubenswrapper[2574]: W0909 19:55:14.989927    2574 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 09 19:55:14.992844 ip-10-0-136-183 kubenswrapper[2574]: W0909 19:55:14.989931    2574 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 09 19:55:14.993419 ip-10-0-136-183 kubenswrapper[2574]: W0909 19:55:14.989935    2574 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 09 19:55:14.993419 ip-10-0-136-183 kubenswrapper[2574]: W0909 19:55:14.989939    2574 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 09 19:55:14.993419 ip-10-0-136-183 kubenswrapper[2574]: W0909 19:55:14.989943    2574 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 09 19:55:14.993419 ip-10-0-136-183 kubenswrapper[2574]: W0909 19:55:14.989947    2574 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 09 19:55:14.993419 ip-10-0-136-183 kubenswrapper[2574]: W0909 19:55:14.989951    2574 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 09 19:55:14.993419 ip-10-0-136-183 kubenswrapper[2574]: W0909 19:55:14.989955    2574 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 09 19:55:14.993419 ip-10-0-136-183 kubenswrapper[2574]: W0909 19:55:14.989960    2574 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 09 19:55:14.993419 ip-10-0-136-183 kubenswrapper[2574]: W0909 19:55:14.989965    2574 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 09 19:55:14.993419 ip-10-0-136-183 kubenswrapper[2574]: W0909 19:55:14.989969    2574 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 09 19:55:14.993419 ip-10-0-136-183 kubenswrapper[2574]: W0909 19:55:14.989973    2574 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 09 19:55:14.993419 ip-10-0-136-183 kubenswrapper[2574]: W0909 19:55:14.989977    2574 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 09 19:55:14.993419 ip-10-0-136-183 kubenswrapper[2574]: W0909 19:55:14.989982    2574 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 09 19:55:14.993419 ip-10-0-136-183 kubenswrapper[2574]: W0909 19:55:14.989987    2574 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 09 19:55:14.993419 ip-10-0-136-183 kubenswrapper[2574]: W0909 19:55:14.989992    2574 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 09 19:55:14.993419 ip-10-0-136-183 kubenswrapper[2574]: W0909 19:55:14.989996    2574 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 09 19:55:14.993419 ip-10-0-136-183 kubenswrapper[2574]: W0909 19:55:14.990001    2574 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 09 19:55:14.993419 ip-10-0-136-183 kubenswrapper[2574]: W0909 19:55:14.990005    2574 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 09 19:55:14.993419 ip-10-0-136-183 kubenswrapper[2574]: W0909 19:55:14.990009    2574 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 09 19:55:14.993419 ip-10-0-136-183 kubenswrapper[2574]: W0909 19:55:14.990013    2574 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 09 19:55:14.993419 ip-10-0-136-183 kubenswrapper[2574]: W0909 19:55:14.990017    2574 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 09 19:55:14.994008 ip-10-0-136-183 kubenswrapper[2574]: W0909 19:55:14.990021    2574 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 09 19:55:14.994008 ip-10-0-136-183 kubenswrapper[2574]: W0909 19:55:14.990025    2574 feature_gate.go:328] unrecognized feature gate: Example
Sep 09 19:55:14.994008 ip-10-0-136-183 kubenswrapper[2574]: W0909 19:55:14.990029    2574 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 09 19:55:14.994008 ip-10-0-136-183 kubenswrapper[2574]: W0909 19:55:14.990034    2574 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 09 19:55:14.994008 ip-10-0-136-183 kubenswrapper[2574]: W0909 19:55:14.990038    2574 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 09 19:55:14.994008 ip-10-0-136-183 kubenswrapper[2574]: W0909 19:55:14.990042    2574 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 09 19:55:14.994008 ip-10-0-136-183 kubenswrapper[2574]: W0909 19:55:14.990047    2574 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 09 19:55:14.994008 ip-10-0-136-183 kubenswrapper[2574]: W0909 19:55:14.990051    2574 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 09 19:55:14.994008 ip-10-0-136-183 kubenswrapper[2574]: W0909 19:55:14.990055    2574 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 09 19:55:14.994008 ip-10-0-136-183 kubenswrapper[2574]: W0909 19:55:14.990059    2574 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 09 19:55:14.994008 ip-10-0-136-183 kubenswrapper[2574]: W0909 19:55:14.990063    2574 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 09 19:55:14.994008 ip-10-0-136-183 kubenswrapper[2574]: W0909 19:55:14.990067    2574 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 09 19:55:14.994008 ip-10-0-136-183 kubenswrapper[2574]: W0909 19:55:14.990071    2574 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 09 19:55:14.994008 ip-10-0-136-183 kubenswrapper[2574]: W0909 19:55:14.990075    2574 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 09 19:55:14.994008 ip-10-0-136-183 kubenswrapper[2574]: W0909 19:55:14.990079    2574 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 09 19:55:14.994008 ip-10-0-136-183 kubenswrapper[2574]: W0909 19:55:14.990083    2574 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 09 19:55:14.994008 ip-10-0-136-183 kubenswrapper[2574]: W0909 19:55:14.990087    2574 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 09 19:55:14.994008 ip-10-0-136-183 kubenswrapper[2574]: W0909 19:55:14.990091    2574 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 09 19:55:14.994008 ip-10-0-136-183 kubenswrapper[2574]: W0909 19:55:14.990095    2574 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 09 19:55:14.994008 ip-10-0-136-183 kubenswrapper[2574]: W0909 19:55:14.990103    2574 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 09 19:55:14.994534 ip-10-0-136-183 kubenswrapper[2574]: W0909 19:55:14.990108    2574 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 09 19:55:14.994534 ip-10-0-136-183 kubenswrapper[2574]: W0909 19:55:14.990112    2574 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 09 19:55:14.994534 ip-10-0-136-183 kubenswrapper[2574]: W0909 19:55:14.990117    2574 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 09 19:55:14.994534 ip-10-0-136-183 kubenswrapper[2574]: W0909 19:55:14.990121    2574 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 09 19:55:14.994534 ip-10-0-136-183 kubenswrapper[2574]: W0909 19:55:14.990126    2574 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 09 19:55:14.994534 ip-10-0-136-183 kubenswrapper[2574]: W0909 19:55:14.990130    2574 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 09 19:55:14.994534 ip-10-0-136-183 kubenswrapper[2574]: W0909 19:55:14.990134    2574 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 09 19:55:14.994534 ip-10-0-136-183 kubenswrapper[2574]: W0909 19:55:14.990138    2574 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 09 19:55:14.994534 ip-10-0-136-183 kubenswrapper[2574]: W0909 19:55:14.990143    2574 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 09 19:55:14.994534 ip-10-0-136-183 kubenswrapper[2574]: W0909 19:55:14.990147    2574 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 09 19:55:14.994534 ip-10-0-136-183 kubenswrapper[2574]: W0909 19:55:14.990151    2574 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 09 19:55:14.994534 ip-10-0-136-183 kubenswrapper[2574]: W0909 19:55:14.990155    2574 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 09 19:55:14.994534 ip-10-0-136-183 kubenswrapper[2574]: W0909 19:55:14.990159    2574 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 09 19:55:14.994534 ip-10-0-136-183 kubenswrapper[2574]: W0909 19:55:14.990164    2574 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 09 19:55:14.994534 ip-10-0-136-183 kubenswrapper[2574]: W0909 19:55:14.990168    2574 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 09 19:55:14.994534 ip-10-0-136-183 kubenswrapper[2574]: W0909 19:55:14.990172    2574 feature_gate.go:328] unrecognized feature gate: Example2
Sep 09 19:55:14.994534 ip-10-0-136-183 kubenswrapper[2574]: W0909 19:55:14.990176    2574 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 09 19:55:14.994534 ip-10-0-136-183 kubenswrapper[2574]: W0909 19:55:14.990180    2574 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 09 19:55:14.994534 ip-10-0-136-183 kubenswrapper[2574]: W0909 19:55:14.990184    2574 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 09 19:55:14.994534 ip-10-0-136-183 kubenswrapper[2574]: W0909 19:55:14.990192    2574 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 09 19:55:14.995052 ip-10-0-136-183 kubenswrapper[2574]: W0909 19:55:14.990196    2574 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 09 19:55:14.995052 ip-10-0-136-183 kubenswrapper[2574]: W0909 19:55:14.990201    2574 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 09 19:55:14.995052 ip-10-0-136-183 kubenswrapper[2574]: W0909 19:55:14.990205    2574 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 09 19:55:14.995052 ip-10-0-136-183 kubenswrapper[2574]: W0909 19:55:14.990209    2574 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 09 19:55:14.995052 ip-10-0-136-183 kubenswrapper[2574]: W0909 19:55:14.990213    2574 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 09 19:55:14.995052 ip-10-0-136-183 kubenswrapper[2574]: W0909 19:55:14.990217    2574 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 09 19:55:14.995052 ip-10-0-136-183 kubenswrapper[2574]: W0909 19:55:14.990222    2574 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 09 19:55:14.995052 ip-10-0-136-183 kubenswrapper[2574]: W0909 19:55:14.990226    2574 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 09 19:55:14.995052 ip-10-0-136-183 kubenswrapper[2574]: W0909 19:55:14.990231    2574 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 09 19:55:14.995052 ip-10-0-136-183 kubenswrapper[2574]: W0909 19:55:14.990236    2574 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 09 19:55:14.995052 ip-10-0-136-183 kubenswrapper[2574]: W0909 19:55:14.990240    2574 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 09 19:55:14.995052 ip-10-0-136-183 kubenswrapper[2574]: W0909 19:55:14.990245    2574 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 09 19:55:14.995052 ip-10-0-136-183 kubenswrapper[2574]: W0909 19:55:14.990249    2574 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 09 19:55:14.995052 ip-10-0-136-183 kubenswrapper[2574]: W0909 19:55:14.990253    2574 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 09 19:55:14.995052 ip-10-0-136-183 kubenswrapper[2574]: W0909 19:55:14.990258    2574 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 09 19:55:14.995052 ip-10-0-136-183 kubenswrapper[2574]: W0909 19:55:14.990952    2574 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 09 19:55:14.995052 ip-10-0-136-183 kubenswrapper[2574]: W0909 19:55:14.990963    2574 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 09 19:55:14.995052 ip-10-0-136-183 kubenswrapper[2574]: W0909 19:55:14.990968    2574 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 09 19:55:14.995052 ip-10-0-136-183 kubenswrapper[2574]: W0909 19:55:14.990973    2574 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 09 19:55:14.995594 ip-10-0-136-183 kubenswrapper[2574]: W0909 19:55:14.990978    2574 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 09 19:55:14.995594 ip-10-0-136-183 kubenswrapper[2574]: W0909 19:55:14.990982    2574 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 09 19:55:14.995594 ip-10-0-136-183 kubenswrapper[2574]: W0909 19:55:14.990987    2574 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 09 19:55:14.995594 ip-10-0-136-183 kubenswrapper[2574]: W0909 19:55:14.990991    2574 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 09 19:55:14.995594 ip-10-0-136-183 kubenswrapper[2574]: W0909 19:55:14.990995    2574 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 09 19:55:14.995594 ip-10-0-136-183 kubenswrapper[2574]: W0909 19:55:14.990999    2574 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 09 19:55:14.995594 ip-10-0-136-183 kubenswrapper[2574]: W0909 19:55:14.991004    2574 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 09 19:55:14.995594 ip-10-0-136-183 kubenswrapper[2574]: W0909 19:55:14.991007    2574 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 09 19:55:14.995594 ip-10-0-136-183 kubenswrapper[2574]: W0909 19:55:14.991011    2574 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 09 19:55:14.995594 ip-10-0-136-183 kubenswrapper[2574]: W0909 19:55:14.991015    2574 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 09 19:55:14.995594 ip-10-0-136-183 kubenswrapper[2574]: W0909 19:55:14.991019    2574 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 09 19:55:14.995594 ip-10-0-136-183 kubenswrapper[2574]: W0909 19:55:14.991024    2574 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 09 19:55:14.995594 ip-10-0-136-183 kubenswrapper[2574]: W0909 19:55:14.991028    2574 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 09 19:55:14.995594 ip-10-0-136-183 kubenswrapper[2574]: W0909 19:55:14.991033    2574 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 09 19:55:14.995594 ip-10-0-136-183 kubenswrapper[2574]: W0909 19:55:14.991037    2574 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 09 19:55:14.995594 ip-10-0-136-183 kubenswrapper[2574]: W0909 19:55:14.991042    2574 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 09 19:55:14.995594 ip-10-0-136-183 kubenswrapper[2574]: W0909 19:55:14.991046    2574 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 09 19:55:14.995594 ip-10-0-136-183 kubenswrapper[2574]: W0909 19:55:14.991050    2574 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 09 19:55:14.995594 ip-10-0-136-183 kubenswrapper[2574]: W0909 19:55:14.991054    2574 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 09 19:55:14.995594 ip-10-0-136-183 kubenswrapper[2574]: W0909 19:55:14.991059    2574 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 09 19:55:14.996254 ip-10-0-136-183 kubenswrapper[2574]: W0909 19:55:14.991063    2574 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 09 19:55:14.996254 ip-10-0-136-183 kubenswrapper[2574]: W0909 19:55:14.991068    2574 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 09 19:55:14.996254 ip-10-0-136-183 kubenswrapper[2574]: W0909 19:55:14.991072    2574 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 09 19:55:14.996254 ip-10-0-136-183 kubenswrapper[2574]: W0909 19:55:14.991077    2574 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 09 19:55:14.996254 ip-10-0-136-183 kubenswrapper[2574]: W0909 19:55:14.991081    2574 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 09 19:55:14.996254 ip-10-0-136-183 kubenswrapper[2574]: W0909 19:55:14.991086    2574 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 09 19:55:14.996254 ip-10-0-136-183 kubenswrapper[2574]: W0909 19:55:14.991090    2574 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 09 19:55:14.996254 ip-10-0-136-183 kubenswrapper[2574]: W0909 19:55:14.991094    2574 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 09 19:55:14.996254 ip-10-0-136-183 kubenswrapper[2574]: W0909 19:55:14.991098    2574 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 09 19:55:14.996254 ip-10-0-136-183 kubenswrapper[2574]: W0909 19:55:14.991102    2574 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 09 19:55:14.996254 ip-10-0-136-183 kubenswrapper[2574]: W0909 19:55:14.991106    2574 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 09 19:55:14.996254 ip-10-0-136-183 kubenswrapper[2574]: W0909 19:55:14.991114    2574 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 09 19:55:14.996254 ip-10-0-136-183 kubenswrapper[2574]: W0909 19:55:14.991119    2574 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 09 19:55:14.996254 ip-10-0-136-183 kubenswrapper[2574]: W0909 19:55:14.991126    2574 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 09 19:55:14.996254 ip-10-0-136-183 kubenswrapper[2574]: W0909 19:55:14.991130    2574 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 09 19:55:14.996254 ip-10-0-136-183 kubenswrapper[2574]: W0909 19:55:14.991135    2574 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 09 19:55:14.996254 ip-10-0-136-183 kubenswrapper[2574]: W0909 19:55:14.991140    2574 feature_gate.go:328] unrecognized feature gate: Example
Sep 09 19:55:14.996254 ip-10-0-136-183 kubenswrapper[2574]: W0909 19:55:14.991144    2574 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 09 19:55:14.996254 ip-10-0-136-183 kubenswrapper[2574]: W0909 19:55:14.991149    2574 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 09 19:55:14.996742 ip-10-0-136-183 kubenswrapper[2574]: W0909 19:55:14.991153    2574 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 09 19:55:14.996742 ip-10-0-136-183 kubenswrapper[2574]: W0909 19:55:14.991157    2574 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 09 19:55:14.996742 ip-10-0-136-183 kubenswrapper[2574]: W0909 19:55:14.991161    2574 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 09 19:55:14.996742 ip-10-0-136-183 kubenswrapper[2574]: W0909 19:55:14.991165    2574 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 09 19:55:14.996742 ip-10-0-136-183 kubenswrapper[2574]: W0909 19:55:14.991170    2574 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 09 19:55:14.996742 ip-10-0-136-183 kubenswrapper[2574]: W0909 19:55:14.991174    2574 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 09 19:55:14.996742 ip-10-0-136-183 kubenswrapper[2574]: W0909 19:55:14.991180    2574 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 09 19:55:14.996742 ip-10-0-136-183 kubenswrapper[2574]: W0909 19:55:14.991184    2574 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 09 19:55:14.996742 ip-10-0-136-183 kubenswrapper[2574]: W0909 19:55:14.991189    2574 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 09 19:55:14.996742 ip-10-0-136-183 kubenswrapper[2574]: W0909 19:55:14.991193    2574 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 09 19:55:14.996742 ip-10-0-136-183 kubenswrapper[2574]: W0909 19:55:14.991198    2574 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 09 19:55:14.996742 ip-10-0-136-183 kubenswrapper[2574]: W0909 19:55:14.991202    2574 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 09 19:55:14.996742 ip-10-0-136-183 kubenswrapper[2574]: W0909 19:55:14.991206    2574 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 09 19:55:14.996742 ip-10-0-136-183 kubenswrapper[2574]: W0909 19:55:14.991210    2574 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 09 19:55:14.996742 ip-10-0-136-183 kubenswrapper[2574]: W0909 19:55:14.991214    2574 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 09 19:55:14.996742 ip-10-0-136-183 kubenswrapper[2574]: W0909 19:55:14.991218    2574 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 09 19:55:14.996742 ip-10-0-136-183 kubenswrapper[2574]: W0909 19:55:14.991222    2574 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 09 19:55:14.996742 ip-10-0-136-183 kubenswrapper[2574]: W0909 19:55:14.991227    2574 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 09 19:55:14.996742 ip-10-0-136-183 kubenswrapper[2574]: W0909 19:55:14.991231    2574 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 09 19:55:14.996742 ip-10-0-136-183 kubenswrapper[2574]: W0909 19:55:14.991235    2574 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 09 19:55:14.997258 ip-10-0-136-183 kubenswrapper[2574]: W0909 19:55:14.991239    2574 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 09 19:55:14.997258 ip-10-0-136-183 kubenswrapper[2574]: W0909 19:55:14.991243    2574 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 09 19:55:14.997258 ip-10-0-136-183 kubenswrapper[2574]: W0909 19:55:14.991248    2574 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 09 19:55:14.997258 ip-10-0-136-183 kubenswrapper[2574]: W0909 19:55:14.991252    2574 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 09 19:55:14.997258 ip-10-0-136-183 kubenswrapper[2574]: W0909 19:55:14.991257    2574 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 09 19:55:14.997258 ip-10-0-136-183 kubenswrapper[2574]: W0909 19:55:14.991261    2574 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 09 19:55:14.997258 ip-10-0-136-183 kubenswrapper[2574]: W0909 19:55:14.991269    2574 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 09 19:55:14.997258 ip-10-0-136-183 kubenswrapper[2574]: W0909 19:55:14.991276    2574 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 09 19:55:14.997258 ip-10-0-136-183 kubenswrapper[2574]: W0909 19:55:14.991280    2574 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 09 19:55:14.997258 ip-10-0-136-183 kubenswrapper[2574]: W0909 19:55:14.991284    2574 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 09 19:55:14.997258 ip-10-0-136-183 kubenswrapper[2574]: W0909 19:55:14.991288    2574 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 09 19:55:14.997258 ip-10-0-136-183 kubenswrapper[2574]: W0909 19:55:14.991292    2574 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 09 19:55:14.997258 ip-10-0-136-183 kubenswrapper[2574]: W0909 19:55:14.991296    2574 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 09 19:55:14.997258 ip-10-0-136-183 kubenswrapper[2574]: W0909 19:55:14.991303    2574 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 09 19:55:14.997258 ip-10-0-136-183 kubenswrapper[2574]: W0909 19:55:14.991308    2574 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 09 19:55:14.997258 ip-10-0-136-183 kubenswrapper[2574]: W0909 19:55:14.991313    2574 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 09 19:55:14.997258 ip-10-0-136-183 kubenswrapper[2574]: W0909 19:55:14.991317    2574 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 09 19:55:14.997258 ip-10-0-136-183 kubenswrapper[2574]: W0909 19:55:14.991321    2574 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 09 19:55:14.997258 ip-10-0-136-183 kubenswrapper[2574]: W0909 19:55:14.991327    2574 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 09 19:55:14.997838 ip-10-0-136-183 kubenswrapper[2574]: W0909 19:55:14.991332    2574 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 09 19:55:14.997838 ip-10-0-136-183 kubenswrapper[2574]: W0909 19:55:14.991336    2574 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 09 19:55:14.997838 ip-10-0-136-183 kubenswrapper[2574]: W0909 19:55:14.991340    2574 feature_gate.go:328] unrecognized feature gate: Example2
Sep 09 19:55:14.997838 ip-10-0-136-183 kubenswrapper[2574]: W0909 19:55:14.991344    2574 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 09 19:55:14.997838 ip-10-0-136-183 kubenswrapper[2574]: W0909 19:55:14.991348    2574 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 09 19:55:14.997838 ip-10-0-136-183 kubenswrapper[2574]: W0909 19:55:14.991352    2574 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 09 19:55:14.997838 ip-10-0-136-183 kubenswrapper[2574]: W0909 19:55:14.991356    2574 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 09 19:55:14.997838 ip-10-0-136-183 kubenswrapper[2574]: W0909 19:55:14.991360    2574 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 09 19:55:14.997838 ip-10-0-136-183 kubenswrapper[2574]: W0909 19:55:14.991364    2574 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 09 19:55:14.997838 ip-10-0-136-183 kubenswrapper[2574]: W0909 19:55:14.991368    2574 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 09 19:55:14.997838 ip-10-0-136-183 kubenswrapper[2574]: W0909 19:55:14.991373    2574 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 09 19:55:14.997838 ip-10-0-136-183 kubenswrapper[2574]: W0909 19:55:14.991377    2574 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 09 19:55:14.997838 ip-10-0-136-183 kubenswrapper[2574]: W0909 19:55:14.991381    2574 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 09 19:55:14.997838 ip-10-0-136-183 kubenswrapper[2574]: W0909 19:55:14.991389    2574 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 09 19:55:14.997838 ip-10-0-136-183 kubenswrapper[2574]: W0909 19:55:14.991393    2574 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 09 19:55:14.997838 ip-10-0-136-183 kubenswrapper[2574]: W0909 19:55:14.991397    2574 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 09 19:55:14.997838 ip-10-0-136-183 kubenswrapper[2574]: W0909 19:55:14.991401    2574 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 09 19:55:14.997838 ip-10-0-136-183 kubenswrapper[2574]: I0909 19:55:14.991519    2574 flags.go:64] FLAG: --address="0.0.0.0"
Sep 09 19:55:14.997838 ip-10-0-136-183 kubenswrapper[2574]: I0909 19:55:14.991571    2574 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Sep 09 19:55:14.997838 ip-10-0-136-183 kubenswrapper[2574]: I0909 19:55:14.991583    2574 flags.go:64] FLAG: --anonymous-auth="true"
Sep 09 19:55:14.997838 ip-10-0-136-183 kubenswrapper[2574]: I0909 19:55:14.991590    2574 flags.go:64] FLAG: --application-metrics-count-limit="100"
Sep 09 19:55:14.998336 ip-10-0-136-183 kubenswrapper[2574]: I0909 19:55:14.991596    2574 flags.go:64] FLAG: --authentication-token-webhook="false"
Sep 09 19:55:14.998336 ip-10-0-136-183 kubenswrapper[2574]: I0909 19:55:14.991601    2574 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Sep 09 19:55:14.998336 ip-10-0-136-183 kubenswrapper[2574]: I0909 19:55:14.991608    2574 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Sep 09 19:55:14.998336 ip-10-0-136-183 kubenswrapper[2574]: I0909 19:55:14.991615    2574 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Sep 09 19:55:14.998336 ip-10-0-136-183 kubenswrapper[2574]: I0909 19:55:14.991620    2574 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Sep 09 19:55:14.998336 ip-10-0-136-183 kubenswrapper[2574]: I0909 19:55:14.991625    2574 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Sep 09 19:55:14.998336 ip-10-0-136-183 kubenswrapper[2574]: I0909 19:55:14.991630    2574 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Sep 09 19:55:14.998336 ip-10-0-136-183 kubenswrapper[2574]: I0909 19:55:14.991636    2574 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Sep 09 19:55:14.998336 ip-10-0-136-183 kubenswrapper[2574]: I0909 19:55:14.991641    2574 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Sep 09 19:55:14.998336 ip-10-0-136-183 kubenswrapper[2574]: I0909 19:55:14.991645    2574 flags.go:64] FLAG: --cgroup-root=""
Sep 09 19:55:14.998336 ip-10-0-136-183 kubenswrapper[2574]: I0909 19:55:14.991650    2574 flags.go:64] FLAG: --cgroups-per-qos="true"
Sep 09 19:55:14.998336 ip-10-0-136-183 kubenswrapper[2574]: I0909 19:55:14.991655    2574 flags.go:64] FLAG: --client-ca-file=""
Sep 09 19:55:14.998336 ip-10-0-136-183 kubenswrapper[2574]: I0909 19:55:14.991660    2574 flags.go:64] FLAG: --cloud-provider="external"
Sep 09 19:55:14.998336 ip-10-0-136-183 kubenswrapper[2574]: I0909 19:55:14.991665    2574 flags.go:64] FLAG: --cluster-dns="[]"
Sep 09 19:55:14.998336 ip-10-0-136-183 kubenswrapper[2574]: I0909 19:55:14.991676    2574 flags.go:64] FLAG: --cluster-domain=""
Sep 09 19:55:14.998336 ip-10-0-136-183 kubenswrapper[2574]: I0909 19:55:14.991680    2574 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Sep 09 19:55:14.998336 ip-10-0-136-183 kubenswrapper[2574]: I0909 19:55:14.991686    2574 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Sep 09 19:55:14.998336 ip-10-0-136-183 kubenswrapper[2574]: I0909 19:55:14.991691    2574 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Sep 09 19:55:14.998336 ip-10-0-136-183 kubenswrapper[2574]: I0909 19:55:14.991696    2574 flags.go:64] FLAG: --container-log-max-files="5"
Sep 09 19:55:14.998336 ip-10-0-136-183 kubenswrapper[2574]: I0909 19:55:14.991702    2574 flags.go:64] FLAG: --container-log-max-size="10Mi"
Sep 09 19:55:14.998336 ip-10-0-136-183 kubenswrapper[2574]: I0909 19:55:14.991707    2574 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Sep 09 19:55:14.998336 ip-10-0-136-183 kubenswrapper[2574]: I0909 19:55:14.991713    2574 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Sep 09 19:55:14.998336 ip-10-0-136-183 kubenswrapper[2574]: I0909 19:55:14.991719    2574 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Sep 09 19:55:14.998926 ip-10-0-136-183 kubenswrapper[2574]: I0909 19:55:14.991724    2574 flags.go:64] FLAG: --contention-profiling="false"
Sep 09 19:55:14.998926 ip-10-0-136-183 kubenswrapper[2574]: I0909 19:55:14.991734    2574 flags.go:64] FLAG: --cpu-cfs-quota="true"
Sep 09 19:55:14.998926 ip-10-0-136-183 kubenswrapper[2574]: I0909 19:55:14.991739    2574 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Sep 09 19:55:14.998926 ip-10-0-136-183 kubenswrapper[2574]: I0909 19:55:14.991744    2574 flags.go:64] FLAG: --cpu-manager-policy="none"
Sep 09 19:55:14.998926 ip-10-0-136-183 kubenswrapper[2574]: I0909 19:55:14.991749    2574 flags.go:64] FLAG: --cpu-manager-policy-options=""
Sep 09 19:55:14.998926 ip-10-0-136-183 kubenswrapper[2574]: I0909 19:55:14.991756    2574 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Sep 09 19:55:14.998926 ip-10-0-136-183 kubenswrapper[2574]: I0909 19:55:14.991761    2574 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Sep 09 19:55:14.998926 ip-10-0-136-183 kubenswrapper[2574]: I0909 19:55:14.991765    2574 flags.go:64] FLAG: --enable-debugging-handlers="true"
Sep 09 19:55:14.998926 ip-10-0-136-183 kubenswrapper[2574]: I0909 19:55:14.991770    2574 flags.go:64] FLAG: --enable-load-reader="false"
Sep 09 19:55:14.998926 ip-10-0-136-183 kubenswrapper[2574]: I0909 19:55:14.991775    2574 flags.go:64] FLAG: --enable-server="true"
Sep 09 19:55:14.998926 ip-10-0-136-183 kubenswrapper[2574]: I0909 19:55:14.991780    2574 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Sep 09 19:55:14.998926 ip-10-0-136-183 kubenswrapper[2574]: I0909 19:55:14.991792    2574 flags.go:64] FLAG: --event-burst="100"
Sep 09 19:55:14.998926 ip-10-0-136-183 kubenswrapper[2574]: I0909 19:55:14.991797    2574 flags.go:64] FLAG: --event-qps="50"
Sep 09 19:55:14.998926 ip-10-0-136-183 kubenswrapper[2574]: I0909 19:55:14.991802    2574 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Sep 09 19:55:14.998926 ip-10-0-136-183 kubenswrapper[2574]: I0909 19:55:14.991806    2574 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Sep 09 19:55:14.998926 ip-10-0-136-183 kubenswrapper[2574]: I0909 19:55:14.991811    2574 flags.go:64] FLAG: --eviction-hard=""
Sep 09 19:55:14.998926 ip-10-0-136-183 kubenswrapper[2574]: I0909 19:55:14.991817    2574 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Sep 09 19:55:14.998926 ip-10-0-136-183 kubenswrapper[2574]: I0909 19:55:14.991821    2574 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Sep 09 19:55:14.998926 ip-10-0-136-183 kubenswrapper[2574]: I0909 19:55:14.991830    2574 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Sep 09 19:55:14.998926 ip-10-0-136-183 kubenswrapper[2574]: I0909 19:55:14.991835    2574 flags.go:64] FLAG: --eviction-soft=""
Sep 09 19:55:14.998926 ip-10-0-136-183 kubenswrapper[2574]: I0909 19:55:14.991840    2574 flags.go:64] FLAG: --eviction-soft-grace-period=""
Sep 09 19:55:14.998926 ip-10-0-136-183 kubenswrapper[2574]: I0909 19:55:14.991845    2574 flags.go:64] FLAG: --exit-on-lock-contention="false"
Sep 09 19:55:14.998926 ip-10-0-136-183 kubenswrapper[2574]: I0909 19:55:14.991856    2574 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Sep 09 19:55:14.998926 ip-10-0-136-183 kubenswrapper[2574]: I0909 19:55:14.991861    2574 flags.go:64] FLAG: --experimental-mounter-path=""
Sep 09 19:55:14.998926 ip-10-0-136-183 kubenswrapper[2574]: I0909 19:55:14.991866    2574 flags.go:64] FLAG: --fail-cgroupv1="false"
Sep 09 19:55:14.999569 ip-10-0-136-183 kubenswrapper[2574]: I0909 19:55:14.991870    2574 flags.go:64] FLAG: --fail-swap-on="true"
Sep 09 19:55:14.999569 ip-10-0-136-183 kubenswrapper[2574]: I0909 19:55:14.991874    2574 flags.go:64] FLAG: --feature-gates=""
Sep 09 19:55:14.999569 ip-10-0-136-183 kubenswrapper[2574]: I0909 19:55:14.991880    2574 flags.go:64] FLAG: --file-check-frequency="20s"
Sep 09 19:55:14.999569 ip-10-0-136-183 kubenswrapper[2574]: I0909 19:55:14.991885    2574 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Sep 09 19:55:14.999569 ip-10-0-136-183 kubenswrapper[2574]: I0909 19:55:14.991890    2574 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Sep 09 19:55:14.999569 ip-10-0-136-183 kubenswrapper[2574]: I0909 19:55:14.991895    2574 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Sep 09 19:55:14.999569 ip-10-0-136-183 kubenswrapper[2574]: I0909 19:55:14.991900    2574 flags.go:64] FLAG: --healthz-port="10248"
Sep 09 19:55:14.999569 ip-10-0-136-183 kubenswrapper[2574]: I0909 19:55:14.991905    2574 flags.go:64] FLAG: --help="false"
Sep 09 19:55:14.999569 ip-10-0-136-183 kubenswrapper[2574]: I0909 19:55:14.991910    2574 flags.go:64] FLAG: --hostname-override="ip-10-0-136-183.ec2.internal"
Sep 09 19:55:14.999569 ip-10-0-136-183 kubenswrapper[2574]: I0909 19:55:14.991915    2574 flags.go:64] FLAG: --housekeeping-interval="10s"
Sep 09 19:55:14.999569 ip-10-0-136-183 kubenswrapper[2574]: I0909 19:55:14.991920    2574 flags.go:64] FLAG: --http-check-frequency="20s"
Sep 09 19:55:14.999569 ip-10-0-136-183 kubenswrapper[2574]: I0909 19:55:14.991925    2574 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Sep 09 19:55:14.999569 ip-10-0-136-183 kubenswrapper[2574]: I0909 19:55:14.991931    2574 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Sep 09 19:55:14.999569 ip-10-0-136-183 kubenswrapper[2574]: I0909 19:55:14.991936    2574 flags.go:64] FLAG: --image-gc-high-threshold="85"
Sep 09 19:55:14.999569 ip-10-0-136-183 kubenswrapper[2574]: I0909 19:55:14.991941    2574 flags.go:64] FLAG: --image-gc-low-threshold="80"
Sep 09 19:55:14.999569 ip-10-0-136-183 kubenswrapper[2574]: I0909 19:55:14.991945    2574 flags.go:64] FLAG: --image-service-endpoint=""
Sep 09 19:55:14.999569 ip-10-0-136-183 kubenswrapper[2574]: I0909 19:55:14.991949    2574 flags.go:64] FLAG: --kernel-memcg-notification="false"
Sep 09 19:55:14.999569 ip-10-0-136-183 kubenswrapper[2574]: I0909 19:55:14.991954    2574 flags.go:64] FLAG: --kube-api-burst="100"
Sep 09 19:55:14.999569 ip-10-0-136-183 kubenswrapper[2574]: I0909 19:55:14.991958    2574 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Sep 09 19:55:14.999569 ip-10-0-136-183 kubenswrapper[2574]: I0909 19:55:14.991962    2574 flags.go:64] FLAG: --kube-api-qps="50"
Sep 09 19:55:14.999569 ip-10-0-136-183 kubenswrapper[2574]: I0909 19:55:14.991967    2574 flags.go:64] FLAG: --kube-reserved=""
Sep 09 19:55:14.999569 ip-10-0-136-183 kubenswrapper[2574]: I0909 19:55:14.991972    2574 flags.go:64] FLAG: --kube-reserved-cgroup=""
Sep 09 19:55:14.999569 ip-10-0-136-183 kubenswrapper[2574]: I0909 19:55:14.991976    2574 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Sep 09 19:55:14.999569 ip-10-0-136-183 kubenswrapper[2574]: I0909 19:55:14.991981    2574 flags.go:64] FLAG: --kubelet-cgroups=""
Sep 09 19:55:15.000155 ip-10-0-136-183 kubenswrapper[2574]: I0909 19:55:14.991985    2574 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Sep 09 19:55:15.000155 ip-10-0-136-183 kubenswrapper[2574]: I0909 19:55:14.991990    2574 flags.go:64] FLAG: --lock-file=""
Sep 09 19:55:15.000155 ip-10-0-136-183 kubenswrapper[2574]: I0909 19:55:14.991994    2574 flags.go:64] FLAG: --log-cadvisor-usage="false"
Sep 09 19:55:15.000155 ip-10-0-136-183 kubenswrapper[2574]: I0909 19:55:14.991998    2574 flags.go:64] FLAG: --log-flush-frequency="5s"
Sep 09 19:55:15.000155 ip-10-0-136-183 kubenswrapper[2574]: I0909 19:55:14.992003    2574 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Sep 09 19:55:15.000155 ip-10-0-136-183 kubenswrapper[2574]: I0909 19:55:14.992010    2574 flags.go:64] FLAG: --log-json-split-stream="false"
Sep 09 19:55:15.000155 ip-10-0-136-183 kubenswrapper[2574]: I0909 19:55:14.992016    2574 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Sep 09 19:55:15.000155 ip-10-0-136-183 kubenswrapper[2574]: I0909 19:55:14.992021    2574 flags.go:64] FLAG: --log-text-split-stream="false"
Sep 09 19:55:15.000155 ip-10-0-136-183 kubenswrapper[2574]: I0909 19:55:14.992025    2574 flags.go:64] FLAG: --logging-format="text"
Sep 09 19:55:15.000155 ip-10-0-136-183 kubenswrapper[2574]: I0909 19:55:14.992030    2574 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Sep 09 19:55:15.000155 ip-10-0-136-183 kubenswrapper[2574]: I0909 19:55:14.992035    2574 flags.go:64] FLAG: --make-iptables-util-chains="true"
Sep 09 19:55:15.000155 ip-10-0-136-183 kubenswrapper[2574]: I0909 19:55:14.992039    2574 flags.go:64] FLAG: --manifest-url=""
Sep 09 19:55:15.000155 ip-10-0-136-183 kubenswrapper[2574]: I0909 19:55:14.992044    2574 flags.go:64] FLAG: --manifest-url-header=""
Sep 09 19:55:15.000155 ip-10-0-136-183 kubenswrapper[2574]: I0909 19:55:14.992050    2574 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Sep 09 19:55:15.000155 ip-10-0-136-183 kubenswrapper[2574]: I0909 19:55:14.992055    2574 flags.go:64] FLAG: --max-open-files="1000000"
Sep 09 19:55:15.000155 ip-10-0-136-183 kubenswrapper[2574]: I0909 19:55:14.992061    2574 flags.go:64] FLAG: --max-pods="110"
Sep 09 19:55:15.000155 ip-10-0-136-183 kubenswrapper[2574]: I0909 19:55:14.992067    2574 flags.go:64] FLAG: --maximum-dead-containers="-1"
Sep 09 19:55:15.000155 ip-10-0-136-183 kubenswrapper[2574]: I0909 19:55:14.992071    2574 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Sep 09 19:55:15.000155 ip-10-0-136-183 kubenswrapper[2574]: I0909 19:55:14.992076    2574 flags.go:64] FLAG: --memory-manager-policy="None"
Sep 09 19:55:15.000155 ip-10-0-136-183 kubenswrapper[2574]: I0909 19:55:14.992081    2574 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Sep 09 19:55:15.000155 ip-10-0-136-183 kubenswrapper[2574]: I0909 19:55:14.992086    2574 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Sep 09 19:55:15.000155 ip-10-0-136-183 kubenswrapper[2574]: I0909 19:55:14.992090    2574 flags.go:64] FLAG: --node-ip="0.0.0.0"
Sep 09 19:55:15.000155 ip-10-0-136-183 kubenswrapper[2574]: I0909 19:55:14.992095    2574 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Sep 09 19:55:15.000155 ip-10-0-136-183 kubenswrapper[2574]: I0909 19:55:14.992107    2574 flags.go:64] FLAG: --node-status-max-images="50"
Sep 09 19:55:15.000155 ip-10-0-136-183 kubenswrapper[2574]: I0909 19:55:14.992112    2574 flags.go:64] FLAG: --node-status-update-frequency="10s"
Sep 09 19:55:15.000884 ip-10-0-136-183 kubenswrapper[2574]: I0909 19:55:14.992116    2574 flags.go:64] FLAG: --oom-score-adj="-999"
Sep 09 19:55:15.000884 ip-10-0-136-183 kubenswrapper[2574]: I0909 19:55:14.992121    2574 flags.go:64] FLAG: --pod-cidr=""
Sep 09 19:55:15.000884 ip-10-0-136-183 kubenswrapper[2574]: I0909 19:55:14.992126    2574 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:8ccbacc999dbf9abad8f4009240188312b683f36ced4763eed166c1500c164b1"
Sep 09 19:55:15.000884 ip-10-0-136-183 kubenswrapper[2574]: I0909 19:55:14.992135    2574 flags.go:64] FLAG: --pod-manifest-path=""
Sep 09 19:55:15.000884 ip-10-0-136-183 kubenswrapper[2574]: I0909 19:55:14.992140    2574 flags.go:64] FLAG: --pod-max-pids="-1"
Sep 09 19:55:15.000884 ip-10-0-136-183 kubenswrapper[2574]: I0909 19:55:14.992145    2574 flags.go:64] FLAG: --pods-per-core="0"
Sep 09 19:55:15.000884 ip-10-0-136-183 kubenswrapper[2574]: I0909 19:55:14.992149    2574 flags.go:64] FLAG: --port="10250"
Sep 09 19:55:15.000884 ip-10-0-136-183 kubenswrapper[2574]: I0909 19:55:14.992154    2574 flags.go:64] FLAG: --protect-kernel-defaults="false"
Sep 09 19:55:15.000884 ip-10-0-136-183 kubenswrapper[2574]: I0909 19:55:14.992158    2574 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-0c2013b5b51cc56f3"
Sep 09 19:55:15.000884 ip-10-0-136-183 kubenswrapper[2574]: I0909 19:55:14.992164    2574 flags.go:64] FLAG: --qos-reserved=""
Sep 09 19:55:15.000884 ip-10-0-136-183 kubenswrapper[2574]: I0909 19:55:14.992169    2574 flags.go:64] FLAG: --read-only-port="10255"
Sep 09 19:55:15.000884 ip-10-0-136-183 kubenswrapper[2574]: I0909 19:55:14.992174    2574 flags.go:64] FLAG: --register-node="true"
Sep 09 19:55:15.000884 ip-10-0-136-183 kubenswrapper[2574]: I0909 19:55:14.992179    2574 flags.go:64] FLAG: --register-with-taints=""
Sep 09 19:55:15.000884 ip-10-0-136-183 kubenswrapper[2574]: I0909 19:55:14.992184    2574 flags.go:64] FLAG: --registry-burst="10"
Sep 09 19:55:15.000884 ip-10-0-136-183 kubenswrapper[2574]: I0909 19:55:14.992197    2574 flags.go:64] FLAG: --registry-qps="5"
Sep 09 19:55:15.000884 ip-10-0-136-183 kubenswrapper[2574]: I0909 19:55:14.992202    2574 flags.go:64] FLAG: --reserved-cpus=""
Sep 09 19:55:15.000884 ip-10-0-136-183 kubenswrapper[2574]: I0909 19:55:14.992206    2574 flags.go:64] FLAG: --reserved-memory=""
Sep 09 19:55:15.000884 ip-10-0-136-183 kubenswrapper[2574]: I0909 19:55:14.992213    2574 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Sep 09 19:55:15.000884 ip-10-0-136-183 kubenswrapper[2574]: I0909 19:55:14.992218    2574 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Sep 09 19:55:15.000884 ip-10-0-136-183 kubenswrapper[2574]: I0909 19:55:14.992223    2574 flags.go:64] FLAG: --rotate-certificates="false"
Sep 09 19:55:15.000884 ip-10-0-136-183 kubenswrapper[2574]: I0909 19:55:14.992227    2574 flags.go:64] FLAG: --rotate-server-certificates="false"
Sep 09 19:55:15.000884 ip-10-0-136-183 kubenswrapper[2574]: I0909 19:55:14.992231    2574 flags.go:64] FLAG: --runonce="false"
Sep 09 19:55:15.000884 ip-10-0-136-183 kubenswrapper[2574]: I0909 19:55:14.992236    2574 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Sep 09 19:55:15.000884 ip-10-0-136-183 kubenswrapper[2574]: I0909 19:55:14.992241    2574 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Sep 09 19:55:15.000884 ip-10-0-136-183 kubenswrapper[2574]: I0909 19:55:14.992247    2574 flags.go:64] FLAG: --seccomp-default="false"
Sep 09 19:55:15.001537 ip-10-0-136-183 kubenswrapper[2574]: I0909 19:55:14.992251    2574 flags.go:64] FLAG: --serialize-image-pulls="true"
Sep 09 19:55:15.001537 ip-10-0-136-183 kubenswrapper[2574]: I0909 19:55:14.992256    2574 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Sep 09 19:55:15.001537 ip-10-0-136-183 kubenswrapper[2574]: I0909 19:55:14.992261    2574 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Sep 09 19:55:15.001537 ip-10-0-136-183 kubenswrapper[2574]: I0909 19:55:14.992266    2574 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Sep 09 19:55:15.001537 ip-10-0-136-183 kubenswrapper[2574]: I0909 19:55:14.992271    2574 flags.go:64] FLAG: --storage-driver-password="root"
Sep 09 19:55:15.001537 ip-10-0-136-183 kubenswrapper[2574]: I0909 19:55:14.992276    2574 flags.go:64] FLAG: --storage-driver-secure="false"
Sep 09 19:55:15.001537 ip-10-0-136-183 kubenswrapper[2574]: I0909 19:55:14.992280    2574 flags.go:64] FLAG: --storage-driver-table="stats"
Sep 09 19:55:15.001537 ip-10-0-136-183 kubenswrapper[2574]: I0909 19:55:14.992285    2574 flags.go:64] FLAG: --storage-driver-user="root"
Sep 09 19:55:15.001537 ip-10-0-136-183 kubenswrapper[2574]: I0909 19:55:14.992290    2574 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Sep 09 19:55:15.001537 ip-10-0-136-183 kubenswrapper[2574]: I0909 19:55:14.992295    2574 flags.go:64] FLAG: --sync-frequency="1m0s"
Sep 09 19:55:15.001537 ip-10-0-136-183 kubenswrapper[2574]: I0909 19:55:14.992300    2574 flags.go:64] FLAG: --system-cgroups=""
Sep 09 19:55:15.001537 ip-10-0-136-183 kubenswrapper[2574]: I0909 19:55:14.992304    2574 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Sep 09 19:55:15.001537 ip-10-0-136-183 kubenswrapper[2574]: I0909 19:55:14.992313    2574 flags.go:64] FLAG: --system-reserved-cgroup=""
Sep 09 19:55:15.001537 ip-10-0-136-183 kubenswrapper[2574]: I0909 19:55:14.992317    2574 flags.go:64] FLAG: --tls-cert-file=""
Sep 09 19:55:15.001537 ip-10-0-136-183 kubenswrapper[2574]: I0909 19:55:14.992322    2574 flags.go:64] FLAG: --tls-cipher-suites="[]"
Sep 09 19:55:15.001537 ip-10-0-136-183 kubenswrapper[2574]: I0909 19:55:14.992332    2574 flags.go:64] FLAG: --tls-min-version=""
Sep 09 19:55:15.001537 ip-10-0-136-183 kubenswrapper[2574]: I0909 19:55:14.992336    2574 flags.go:64] FLAG: --tls-private-key-file=""
Sep 09 19:55:15.001537 ip-10-0-136-183 kubenswrapper[2574]: I0909 19:55:14.992341    2574 flags.go:64] FLAG: --topology-manager-policy="none"
Sep 09 19:55:15.001537 ip-10-0-136-183 kubenswrapper[2574]: I0909 19:55:14.992345    2574 flags.go:64] FLAG: --topology-manager-policy-options=""
Sep 09 19:55:15.001537 ip-10-0-136-183 kubenswrapper[2574]: I0909 19:55:14.992350    2574 flags.go:64] FLAG: --topology-manager-scope="container"
Sep 09 19:55:15.001537 ip-10-0-136-183 kubenswrapper[2574]: I0909 19:55:14.992355    2574 flags.go:64] FLAG: --v="2"
Sep 09 19:55:15.001537 ip-10-0-136-183 kubenswrapper[2574]: I0909 19:55:14.992361    2574 flags.go:64] FLAG: --version="false"
Sep 09 19:55:15.001537 ip-10-0-136-183 kubenswrapper[2574]: I0909 19:55:14.992368    2574 flags.go:64] FLAG: --vmodule=""
Sep 09 19:55:15.001537 ip-10-0-136-183 kubenswrapper[2574]: I0909 19:55:14.992375    2574 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Sep 09 19:55:15.001537 ip-10-0-136-183 kubenswrapper[2574]: I0909 19:55:14.992380    2574 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Sep 09 19:55:15.002141 ip-10-0-136-183 kubenswrapper[2574]: W0909 19:55:14.992591    2574 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 09 19:55:15.002141 ip-10-0-136-183 kubenswrapper[2574]: W0909 19:55:14.992598    2574 feature_gate.go:328] unrecognized feature gate: Example2
Sep 09 19:55:15.002141 ip-10-0-136-183 kubenswrapper[2574]: W0909 19:55:14.992602    2574 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 09 19:55:15.002141 ip-10-0-136-183 kubenswrapper[2574]: W0909 19:55:14.992607    2574 feature_gate.go:328] unrecognized feature gate: Example
Sep 09 19:55:15.002141 ip-10-0-136-183 kubenswrapper[2574]: W0909 19:55:14.992611    2574 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 09 19:55:15.002141 ip-10-0-136-183 kubenswrapper[2574]: W0909 19:55:14.992615    2574 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 09 19:55:15.002141 ip-10-0-136-183 kubenswrapper[2574]: W0909 19:55:14.992620    2574 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 09 19:55:15.002141 ip-10-0-136-183 kubenswrapper[2574]: W0909 19:55:14.992626    2574 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 09 19:55:15.002141 ip-10-0-136-183 kubenswrapper[2574]: W0909 19:55:14.992632    2574 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 09 19:55:15.002141 ip-10-0-136-183 kubenswrapper[2574]: W0909 19:55:14.992637    2574 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 09 19:55:15.002141 ip-10-0-136-183 kubenswrapper[2574]: W0909 19:55:14.992641    2574 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 09 19:55:15.002141 ip-10-0-136-183 kubenswrapper[2574]: W0909 19:55:14.992646    2574 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 09 19:55:15.002141 ip-10-0-136-183 kubenswrapper[2574]: W0909 19:55:14.992651    2574 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 09 19:55:15.002141 ip-10-0-136-183 kubenswrapper[2574]: W0909 19:55:14.992655    2574 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 09 19:55:15.002141 ip-10-0-136-183 kubenswrapper[2574]: W0909 19:55:14.992659    2574 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 09 19:55:15.002141 ip-10-0-136-183 kubenswrapper[2574]: W0909 19:55:14.992664    2574 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 09 19:55:15.002141 ip-10-0-136-183 kubenswrapper[2574]: W0909 19:55:14.992670    2574 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 09 19:55:15.002141 ip-10-0-136-183 kubenswrapper[2574]: W0909 19:55:14.992674    2574 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 09 19:55:15.002141 ip-10-0-136-183 kubenswrapper[2574]: W0909 19:55:14.992679    2574 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 09 19:55:15.002141 ip-10-0-136-183 kubenswrapper[2574]: W0909 19:55:14.992683    2574 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 09 19:55:15.002662 ip-10-0-136-183 kubenswrapper[2574]: W0909 19:55:14.992687    2574 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 09 19:55:15.002662 ip-10-0-136-183 kubenswrapper[2574]: W0909 19:55:14.992691    2574 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 09 19:55:15.002662 ip-10-0-136-183 kubenswrapper[2574]: W0909 19:55:14.992695    2574 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 09 19:55:15.002662 ip-10-0-136-183 kubenswrapper[2574]: W0909 19:55:14.992700    2574 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 09 19:55:15.002662 ip-10-0-136-183 kubenswrapper[2574]: W0909 19:55:14.992704    2574 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 09 19:55:15.002662 ip-10-0-136-183 kubenswrapper[2574]: W0909 19:55:14.992709    2574 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 09 19:55:15.002662 ip-10-0-136-183 kubenswrapper[2574]: W0909 19:55:14.992713    2574 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 09 19:55:15.002662 ip-10-0-136-183 kubenswrapper[2574]: W0909 19:55:14.992717    2574 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 09 19:55:15.002662 ip-10-0-136-183 kubenswrapper[2574]: W0909 19:55:14.992721    2574 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 09 19:55:15.002662 ip-10-0-136-183 kubenswrapper[2574]: W0909 19:55:14.992725    2574 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 09 19:55:15.002662 ip-10-0-136-183 kubenswrapper[2574]: W0909 19:55:14.992730    2574 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 09 19:55:15.002662 ip-10-0-136-183 kubenswrapper[2574]: W0909 19:55:14.992735    2574 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 09 19:55:15.002662 ip-10-0-136-183 kubenswrapper[2574]: W0909 19:55:14.992739    2574 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 09 19:55:15.002662 ip-10-0-136-183 kubenswrapper[2574]: W0909 19:55:14.992743    2574 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 09 19:55:15.002662 ip-10-0-136-183 kubenswrapper[2574]: W0909 19:55:14.992747    2574 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 09 19:55:15.002662 ip-10-0-136-183 kubenswrapper[2574]: W0909 19:55:14.992751    2574 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 09 19:55:15.002662 ip-10-0-136-183 kubenswrapper[2574]: W0909 19:55:14.992755    2574 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 09 19:55:15.002662 ip-10-0-136-183 kubenswrapper[2574]: W0909 19:55:14.992762    2574 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 09 19:55:15.002662 ip-10-0-136-183 kubenswrapper[2574]: W0909 19:55:14.992769    2574 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 09 19:55:15.003123 ip-10-0-136-183 kubenswrapper[2574]: W0909 19:55:14.992775    2574 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 09 19:55:15.003123 ip-10-0-136-183 kubenswrapper[2574]: W0909 19:55:14.992779    2574 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 09 19:55:15.003123 ip-10-0-136-183 kubenswrapper[2574]: W0909 19:55:14.992784    2574 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 09 19:55:15.003123 ip-10-0-136-183 kubenswrapper[2574]: W0909 19:55:14.992788    2574 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 09 19:55:15.003123 ip-10-0-136-183 kubenswrapper[2574]: W0909 19:55:14.992793    2574 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 09 19:55:15.003123 ip-10-0-136-183 kubenswrapper[2574]: W0909 19:55:14.992798    2574 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 09 19:55:15.003123 ip-10-0-136-183 kubenswrapper[2574]: W0909 19:55:14.992802    2574 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 09 19:55:15.003123 ip-10-0-136-183 kubenswrapper[2574]: W0909 19:55:14.992806    2574 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 09 19:55:15.003123 ip-10-0-136-183 kubenswrapper[2574]: W0909 19:55:14.992810    2574 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 09 19:55:15.003123 ip-10-0-136-183 kubenswrapper[2574]: W0909 19:55:14.992815    2574 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 09 19:55:15.003123 ip-10-0-136-183 kubenswrapper[2574]: W0909 19:55:14.992819    2574 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 09 19:55:15.003123 ip-10-0-136-183 kubenswrapper[2574]: W0909 19:55:14.992823    2574 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 09 19:55:15.003123 ip-10-0-136-183 kubenswrapper[2574]: W0909 19:55:14.992829    2574 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 09 19:55:15.003123 ip-10-0-136-183 kubenswrapper[2574]: W0909 19:55:14.992834    2574 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 09 19:55:15.003123 ip-10-0-136-183 kubenswrapper[2574]: W0909 19:55:14.992838    2574 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 09 19:55:15.003123 ip-10-0-136-183 kubenswrapper[2574]: W0909 19:55:14.992842    2574 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 09 19:55:15.003123 ip-10-0-136-183 kubenswrapper[2574]: W0909 19:55:14.992846    2574 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 09 19:55:15.003123 ip-10-0-136-183 kubenswrapper[2574]: W0909 19:55:14.992849    2574 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 09 19:55:15.003123 ip-10-0-136-183 kubenswrapper[2574]: W0909 19:55:14.992853    2574 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 09 19:55:15.003610 ip-10-0-136-183 kubenswrapper[2574]: W0909 19:55:14.992857    2574 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 09 19:55:15.003610 ip-10-0-136-183 kubenswrapper[2574]: W0909 19:55:14.992861    2574 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 09 19:55:15.003610 ip-10-0-136-183 kubenswrapper[2574]: W0909 19:55:14.992865    2574 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 09 19:55:15.003610 ip-10-0-136-183 kubenswrapper[2574]: W0909 19:55:14.992869    2574 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 09 19:55:15.003610 ip-10-0-136-183 kubenswrapper[2574]: W0909 19:55:14.992874    2574 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 09 19:55:15.003610 ip-10-0-136-183 kubenswrapper[2574]: W0909 19:55:14.992879    2574 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 09 19:55:15.003610 ip-10-0-136-183 kubenswrapper[2574]: W0909 19:55:14.992884    2574 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 09 19:55:15.003610 ip-10-0-136-183 kubenswrapper[2574]: W0909 19:55:14.992888    2574 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 09 19:55:15.003610 ip-10-0-136-183 kubenswrapper[2574]: W0909 19:55:14.992892    2574 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 09 19:55:15.003610 ip-10-0-136-183 kubenswrapper[2574]: W0909 19:55:14.992896    2574 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 09 19:55:15.003610 ip-10-0-136-183 kubenswrapper[2574]: W0909 19:55:14.992900    2574 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 09 19:55:15.003610 ip-10-0-136-183 kubenswrapper[2574]: W0909 19:55:14.992906    2574 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 09 19:55:15.003610 ip-10-0-136-183 kubenswrapper[2574]: W0909 19:55:14.992910    2574 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 09 19:55:15.003610 ip-10-0-136-183 kubenswrapper[2574]: W0909 19:55:14.992914    2574 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 09 19:55:15.003610 ip-10-0-136-183 kubenswrapper[2574]: W0909 19:55:14.992919    2574 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 09 19:55:15.003610 ip-10-0-136-183 kubenswrapper[2574]: W0909 19:55:14.992923    2574 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 09 19:55:15.003610 ip-10-0-136-183 kubenswrapper[2574]: W0909 19:55:14.992927    2574 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 09 19:55:15.003610 ip-10-0-136-183 kubenswrapper[2574]: W0909 19:55:14.992931    2574 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 09 19:55:15.003610 ip-10-0-136-183 kubenswrapper[2574]: W0909 19:55:14.992935    2574 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 09 19:55:15.003610 ip-10-0-136-183 kubenswrapper[2574]: W0909 19:55:14.992940    2574 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 09 19:55:15.004100 ip-10-0-136-183 kubenswrapper[2574]: W0909 19:55:14.992944    2574 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 09 19:55:15.004100 ip-10-0-136-183 kubenswrapper[2574]: W0909 19:55:14.992948    2574 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 09 19:55:15.004100 ip-10-0-136-183 kubenswrapper[2574]: W0909 19:55:14.992952    2574 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 09 19:55:15.004100 ip-10-0-136-183 kubenswrapper[2574]: W0909 19:55:14.992957    2574 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 09 19:55:15.004100 ip-10-0-136-183 kubenswrapper[2574]: W0909 19:55:14.992961    2574 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 09 19:55:15.004100 ip-10-0-136-183 kubenswrapper[2574]: W0909 19:55:14.992967    2574 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 09 19:55:15.004100 ip-10-0-136-183 kubenswrapper[2574]: W0909 19:55:14.992972    2574 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 09 19:55:15.004100 ip-10-0-136-183 kubenswrapper[2574]: W0909 19:55:14.992976    2574 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 09 19:55:15.004100 ip-10-0-136-183 kubenswrapper[2574]: W0909 19:55:14.992980    2574 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 09 19:55:15.004100 ip-10-0-136-183 kubenswrapper[2574]: W0909 19:55:14.992984    2574 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 09 19:55:15.004100 ip-10-0-136-183 kubenswrapper[2574]: W0909 19:55:14.992989    2574 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 09 19:55:15.004100 ip-10-0-136-183 kubenswrapper[2574]: W0909 19:55:14.992993    2574 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 09 19:55:15.004100 ip-10-0-136-183 kubenswrapper[2574]: W0909 19:55:14.992996    2574 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 09 19:55:15.004100 ip-10-0-136-183 kubenswrapper[2574]: W0909 19:55:14.993000    2574 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 09 19:55:15.004100 ip-10-0-136-183 kubenswrapper[2574]: W0909 19:55:14.993004    2574 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 09 19:55:15.004100 ip-10-0-136-183 kubenswrapper[2574]: W0909 19:55:14.993009    2574 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 09 19:55:15.004100 ip-10-0-136-183 kubenswrapper[2574]: W0909 19:55:14.993013    2574 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 09 19:55:15.004100 ip-10-0-136-183 kubenswrapper[2574]: W0909 19:55:14.993017    2574 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 09 19:55:15.004100 ip-10-0-136-183 kubenswrapper[2574]: W0909 19:55:14.993022    2574 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 09 19:55:15.004100 ip-10-0-136-183 kubenswrapper[2574]: W0909 19:55:14.993027    2574 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 09 19:55:15.004641 ip-10-0-136-183 kubenswrapper[2574]: W0909 19:55:14.993031    2574 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 09 19:55:15.004641 ip-10-0-136-183 kubenswrapper[2574]: I0909 19:55:14.993827    2574 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 09 19:55:15.004641 ip-10-0-136-183 kubenswrapper[2574]: I0909 19:55:15.001913    2574 server.go:529] "Kubelet version" kubeletVersion="v1.34.9"
Sep 09 19:55:15.004641 ip-10-0-136-183 kubenswrapper[2574]: I0909 19:55:15.001930    2574 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Sep 09 19:55:15.004641 ip-10-0-136-183 kubenswrapper[2574]: I0909 19:55:15.001944    2574 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Sep 09 19:55:15.004641 ip-10-0-136-183 kubenswrapper[2574]: I0909 19:55:15.001950    2574 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Sep 09 19:55:15.004641 ip-10-0-136-183 kubenswrapper[2574]: W0909 19:55:15.002020    2574 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 09 19:55:15.004641 ip-10-0-136-183 kubenswrapper[2574]: W0909 19:55:15.002025    2574 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 09 19:55:15.004641 ip-10-0-136-183 kubenswrapper[2574]: W0909 19:55:15.002028    2574 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 09 19:55:15.004641 ip-10-0-136-183 kubenswrapper[2574]: W0909 19:55:15.002031    2574 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 09 19:55:15.004641 ip-10-0-136-183 kubenswrapper[2574]: W0909 19:55:15.002035    2574 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 09 19:55:15.004641 ip-10-0-136-183 kubenswrapper[2574]: W0909 19:55:15.002038    2574 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 09 19:55:15.004641 ip-10-0-136-183 kubenswrapper[2574]: W0909 19:55:15.002041    2574 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 09 19:55:15.004641 ip-10-0-136-183 kubenswrapper[2574]: W0909 19:55:15.002044    2574 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 09 19:55:15.005003 ip-10-0-136-183 kubenswrapper[2574]: W0909 19:55:15.002046    2574 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 09 19:55:15.005003 ip-10-0-136-183 kubenswrapper[2574]: W0909 19:55:15.002049    2574 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 09 19:55:15.005003 ip-10-0-136-183 kubenswrapper[2574]: W0909 19:55:15.002052    2574 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 09 19:55:15.005003 ip-10-0-136-183 kubenswrapper[2574]: W0909 19:55:15.002054    2574 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 09 19:55:15.005003 ip-10-0-136-183 kubenswrapper[2574]: W0909 19:55:15.002057    2574 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 09 19:55:15.005003 ip-10-0-136-183 kubenswrapper[2574]: W0909 19:55:15.002059    2574 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 09 19:55:15.005003 ip-10-0-136-183 kubenswrapper[2574]: W0909 19:55:15.002062    2574 feature_gate.go:328] unrecognized feature gate: Example
Sep 09 19:55:15.005003 ip-10-0-136-183 kubenswrapper[2574]: W0909 19:55:15.002065    2574 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 09 19:55:15.005003 ip-10-0-136-183 kubenswrapper[2574]: W0909 19:55:15.002067    2574 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 09 19:55:15.005003 ip-10-0-136-183 kubenswrapper[2574]: W0909 19:55:15.002070    2574 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 09 19:55:15.005003 ip-10-0-136-183 kubenswrapper[2574]: W0909 19:55:15.002073    2574 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 09 19:55:15.005003 ip-10-0-136-183 kubenswrapper[2574]: W0909 19:55:15.002075    2574 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 09 19:55:15.005003 ip-10-0-136-183 kubenswrapper[2574]: W0909 19:55:15.002078    2574 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 09 19:55:15.005003 ip-10-0-136-183 kubenswrapper[2574]: W0909 19:55:15.002081    2574 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 09 19:55:15.005003 ip-10-0-136-183 kubenswrapper[2574]: W0909 19:55:15.002083    2574 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 09 19:55:15.005003 ip-10-0-136-183 kubenswrapper[2574]: W0909 19:55:15.002086    2574 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 09 19:55:15.005003 ip-10-0-136-183 kubenswrapper[2574]: W0909 19:55:15.002089    2574 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 09 19:55:15.005003 ip-10-0-136-183 kubenswrapper[2574]: W0909 19:55:15.002091    2574 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 09 19:55:15.005003 ip-10-0-136-183 kubenswrapper[2574]: W0909 19:55:15.002094    2574 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 09 19:55:15.005003 ip-10-0-136-183 kubenswrapper[2574]: W0909 19:55:15.002096    2574 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 09 19:55:15.005557 ip-10-0-136-183 kubenswrapper[2574]: W0909 19:55:15.002099    2574 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 09 19:55:15.005557 ip-10-0-136-183 kubenswrapper[2574]: W0909 19:55:15.002103    2574 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 09 19:55:15.005557 ip-10-0-136-183 kubenswrapper[2574]: W0909 19:55:15.002106    2574 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 09 19:55:15.005557 ip-10-0-136-183 kubenswrapper[2574]: W0909 19:55:15.002109    2574 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 09 19:55:15.005557 ip-10-0-136-183 kubenswrapper[2574]: W0909 19:55:15.002111    2574 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 09 19:55:15.005557 ip-10-0-136-183 kubenswrapper[2574]: W0909 19:55:15.002114    2574 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 09 19:55:15.005557 ip-10-0-136-183 kubenswrapper[2574]: W0909 19:55:15.002117    2574 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 09 19:55:15.005557 ip-10-0-136-183 kubenswrapper[2574]: W0909 19:55:15.002120    2574 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 09 19:55:15.005557 ip-10-0-136-183 kubenswrapper[2574]: W0909 19:55:15.002122    2574 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 09 19:55:15.005557 ip-10-0-136-183 kubenswrapper[2574]: W0909 19:55:15.002125    2574 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 09 19:55:15.005557 ip-10-0-136-183 kubenswrapper[2574]: W0909 19:55:15.002127    2574 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 09 19:55:15.005557 ip-10-0-136-183 kubenswrapper[2574]: W0909 19:55:15.002130    2574 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 09 19:55:15.005557 ip-10-0-136-183 kubenswrapper[2574]: W0909 19:55:15.002132    2574 feature_gate.go:328] unrecognized feature gate: Example2
Sep 09 19:55:15.005557 ip-10-0-136-183 kubenswrapper[2574]: W0909 19:55:15.002135    2574 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 09 19:55:15.005557 ip-10-0-136-183 kubenswrapper[2574]: W0909 19:55:15.002137    2574 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 09 19:55:15.005557 ip-10-0-136-183 kubenswrapper[2574]: W0909 19:55:15.002140    2574 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 09 19:55:15.005557 ip-10-0-136-183 kubenswrapper[2574]: W0909 19:55:15.002142    2574 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 09 19:55:15.005557 ip-10-0-136-183 kubenswrapper[2574]: W0909 19:55:15.002145    2574 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 09 19:55:15.005557 ip-10-0-136-183 kubenswrapper[2574]: W0909 19:55:15.002147    2574 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 09 19:55:15.005557 ip-10-0-136-183 kubenswrapper[2574]: W0909 19:55:15.002150    2574 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 09 19:55:15.006043 ip-10-0-136-183 kubenswrapper[2574]: W0909 19:55:15.002153    2574 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 09 19:55:15.006043 ip-10-0-136-183 kubenswrapper[2574]: W0909 19:55:15.002155    2574 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 09 19:55:15.006043 ip-10-0-136-183 kubenswrapper[2574]: W0909 19:55:15.002158    2574 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 09 19:55:15.006043 ip-10-0-136-183 kubenswrapper[2574]: W0909 19:55:15.002161    2574 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 09 19:55:15.006043 ip-10-0-136-183 kubenswrapper[2574]: W0909 19:55:15.002163    2574 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 09 19:55:15.006043 ip-10-0-136-183 kubenswrapper[2574]: W0909 19:55:15.002165    2574 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 09 19:55:15.006043 ip-10-0-136-183 kubenswrapper[2574]: W0909 19:55:15.002168    2574 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 09 19:55:15.006043 ip-10-0-136-183 kubenswrapper[2574]: W0909 19:55:15.002170    2574 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 09 19:55:15.006043 ip-10-0-136-183 kubenswrapper[2574]: W0909 19:55:15.002173    2574 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 09 19:55:15.006043 ip-10-0-136-183 kubenswrapper[2574]: W0909 19:55:15.002186    2574 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 09 19:55:15.006043 ip-10-0-136-183 kubenswrapper[2574]: W0909 19:55:15.002190    2574 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 09 19:55:15.006043 ip-10-0-136-183 kubenswrapper[2574]: W0909 19:55:15.002193    2574 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 09 19:55:15.006043 ip-10-0-136-183 kubenswrapper[2574]: W0909 19:55:15.002196    2574 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 09 19:55:15.006043 ip-10-0-136-183 kubenswrapper[2574]: W0909 19:55:15.002200    2574 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 09 19:55:15.006043 ip-10-0-136-183 kubenswrapper[2574]: W0909 19:55:15.002205    2574 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 09 19:55:15.006043 ip-10-0-136-183 kubenswrapper[2574]: W0909 19:55:15.002208    2574 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 09 19:55:15.006043 ip-10-0-136-183 kubenswrapper[2574]: W0909 19:55:15.002211    2574 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 09 19:55:15.006043 ip-10-0-136-183 kubenswrapper[2574]: W0909 19:55:15.002214    2574 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 09 19:55:15.006043 ip-10-0-136-183 kubenswrapper[2574]: W0909 19:55:15.002217    2574 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 09 19:55:15.006043 ip-10-0-136-183 kubenswrapper[2574]: W0909 19:55:15.002220    2574 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 09 19:55:15.006540 ip-10-0-136-183 kubenswrapper[2574]: W0909 19:55:15.002223    2574 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 09 19:55:15.006540 ip-10-0-136-183 kubenswrapper[2574]: W0909 19:55:15.002225    2574 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 09 19:55:15.006540 ip-10-0-136-183 kubenswrapper[2574]: W0909 19:55:15.002229    2574 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 09 19:55:15.006540 ip-10-0-136-183 kubenswrapper[2574]: W0909 19:55:15.002233    2574 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 09 19:55:15.006540 ip-10-0-136-183 kubenswrapper[2574]: W0909 19:55:15.002236    2574 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 09 19:55:15.006540 ip-10-0-136-183 kubenswrapper[2574]: W0909 19:55:15.002238    2574 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 09 19:55:15.006540 ip-10-0-136-183 kubenswrapper[2574]: W0909 19:55:15.002241    2574 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 09 19:55:15.006540 ip-10-0-136-183 kubenswrapper[2574]: W0909 19:55:15.002243    2574 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 09 19:55:15.006540 ip-10-0-136-183 kubenswrapper[2574]: W0909 19:55:15.002246    2574 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 09 19:55:15.006540 ip-10-0-136-183 kubenswrapper[2574]: W0909 19:55:15.002249    2574 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 09 19:55:15.006540 ip-10-0-136-183 kubenswrapper[2574]: W0909 19:55:15.002251    2574 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 09 19:55:15.006540 ip-10-0-136-183 kubenswrapper[2574]: W0909 19:55:15.002254    2574 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 09 19:55:15.006540 ip-10-0-136-183 kubenswrapper[2574]: W0909 19:55:15.002257    2574 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 09 19:55:15.006540 ip-10-0-136-183 kubenswrapper[2574]: W0909 19:55:15.002261    2574 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 09 19:55:15.006540 ip-10-0-136-183 kubenswrapper[2574]: W0909 19:55:15.002264    2574 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 09 19:55:15.006540 ip-10-0-136-183 kubenswrapper[2574]: W0909 19:55:15.002267    2574 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 09 19:55:15.006540 ip-10-0-136-183 kubenswrapper[2574]: W0909 19:55:15.002270    2574 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 09 19:55:15.006540 ip-10-0-136-183 kubenswrapper[2574]: W0909 19:55:15.002272    2574 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 09 19:55:15.006540 ip-10-0-136-183 kubenswrapper[2574]: W0909 19:55:15.002275    2574 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 09 19:55:15.006996 ip-10-0-136-183 kubenswrapper[2574]: W0909 19:55:15.002278    2574 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 09 19:55:15.006996 ip-10-0-136-183 kubenswrapper[2574]: W0909 19:55:15.002280    2574 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 09 19:55:15.006996 ip-10-0-136-183 kubenswrapper[2574]: W0909 19:55:15.002283    2574 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 09 19:55:15.006996 ip-10-0-136-183 kubenswrapper[2574]: W0909 19:55:15.002286    2574 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 09 19:55:15.006996 ip-10-0-136-183 kubenswrapper[2574]: W0909 19:55:15.002288    2574 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 09 19:55:15.006996 ip-10-0-136-183 kubenswrapper[2574]: W0909 19:55:15.002292    2574 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 09 19:55:15.006996 ip-10-0-136-183 kubenswrapper[2574]: W0909 19:55:15.002296    2574 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 09 19:55:15.006996 ip-10-0-136-183 kubenswrapper[2574]: W0909 19:55:15.002298    2574 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 09 19:55:15.006996 ip-10-0-136-183 kubenswrapper[2574]: W0909 19:55:15.002305    2574 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 09 19:55:15.006996 ip-10-0-136-183 kubenswrapper[2574]: W0909 19:55:15.002309    2574 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 09 19:55:15.006996 ip-10-0-136-183 kubenswrapper[2574]: W0909 19:55:15.002311    2574 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 09 19:55:15.006996 ip-10-0-136-183 kubenswrapper[2574]: W0909 19:55:15.002314    2574 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 09 19:55:15.006996 ip-10-0-136-183 kubenswrapper[2574]: I0909 19:55:15.002319    2574 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 09 19:55:15.006996 ip-10-0-136-183 kubenswrapper[2574]: W0909 19:55:15.002432    2574 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 09 19:55:15.006996 ip-10-0-136-183 kubenswrapper[2574]: W0909 19:55:15.002437    2574 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 09 19:55:15.007353 ip-10-0-136-183 kubenswrapper[2574]: W0909 19:55:15.002441    2574 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 09 19:55:15.007353 ip-10-0-136-183 kubenswrapper[2574]: W0909 19:55:15.002444    2574 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 09 19:55:15.007353 ip-10-0-136-183 kubenswrapper[2574]: W0909 19:55:15.002447    2574 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 09 19:55:15.007353 ip-10-0-136-183 kubenswrapper[2574]: W0909 19:55:15.002449    2574 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 09 19:55:15.007353 ip-10-0-136-183 kubenswrapper[2574]: W0909 19:55:15.002452    2574 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 09 19:55:15.007353 ip-10-0-136-183 kubenswrapper[2574]: W0909 19:55:15.002468    2574 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 09 19:55:15.007353 ip-10-0-136-183 kubenswrapper[2574]: W0909 19:55:15.002472    2574 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 09 19:55:15.007353 ip-10-0-136-183 kubenswrapper[2574]: W0909 19:55:15.002474    2574 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 09 19:55:15.007353 ip-10-0-136-183 kubenswrapper[2574]: W0909 19:55:15.002491    2574 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 09 19:55:15.007353 ip-10-0-136-183 kubenswrapper[2574]: W0909 19:55:15.002496    2574 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 09 19:55:15.007353 ip-10-0-136-183 kubenswrapper[2574]: W0909 19:55:15.002499    2574 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 09 19:55:15.007353 ip-10-0-136-183 kubenswrapper[2574]: W0909 19:55:15.002502    2574 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 09 19:55:15.007353 ip-10-0-136-183 kubenswrapper[2574]: W0909 19:55:15.002505    2574 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 09 19:55:15.007353 ip-10-0-136-183 kubenswrapper[2574]: W0909 19:55:15.002508    2574 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 09 19:55:15.007353 ip-10-0-136-183 kubenswrapper[2574]: W0909 19:55:15.002511    2574 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 09 19:55:15.007353 ip-10-0-136-183 kubenswrapper[2574]: W0909 19:55:15.002514    2574 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 09 19:55:15.007353 ip-10-0-136-183 kubenswrapper[2574]: W0909 19:55:15.002518    2574 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 09 19:55:15.007353 ip-10-0-136-183 kubenswrapper[2574]: W0909 19:55:15.002522    2574 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 09 19:55:15.007353 ip-10-0-136-183 kubenswrapper[2574]: W0909 19:55:15.002525    2574 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 09 19:55:15.007824 ip-10-0-136-183 kubenswrapper[2574]: W0909 19:55:15.002528    2574 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 09 19:55:15.007824 ip-10-0-136-183 kubenswrapper[2574]: W0909 19:55:15.002530    2574 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 09 19:55:15.007824 ip-10-0-136-183 kubenswrapper[2574]: W0909 19:55:15.002533    2574 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 09 19:55:15.007824 ip-10-0-136-183 kubenswrapper[2574]: W0909 19:55:15.002536    2574 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 09 19:55:15.007824 ip-10-0-136-183 kubenswrapper[2574]: W0909 19:55:15.002538    2574 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 09 19:55:15.007824 ip-10-0-136-183 kubenswrapper[2574]: W0909 19:55:15.002541    2574 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 09 19:55:15.007824 ip-10-0-136-183 kubenswrapper[2574]: W0909 19:55:15.002544    2574 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 09 19:55:15.007824 ip-10-0-136-183 kubenswrapper[2574]: W0909 19:55:15.002547    2574 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 09 19:55:15.007824 ip-10-0-136-183 kubenswrapper[2574]: W0909 19:55:15.002550    2574 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 09 19:55:15.007824 ip-10-0-136-183 kubenswrapper[2574]: W0909 19:55:15.002552    2574 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 09 19:55:15.007824 ip-10-0-136-183 kubenswrapper[2574]: W0909 19:55:15.002555    2574 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 09 19:55:15.007824 ip-10-0-136-183 kubenswrapper[2574]: W0909 19:55:15.002558    2574 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 09 19:55:15.007824 ip-10-0-136-183 kubenswrapper[2574]: W0909 19:55:15.002560    2574 feature_gate.go:328] unrecognized feature gate: Example
Sep 09 19:55:15.007824 ip-10-0-136-183 kubenswrapper[2574]: W0909 19:55:15.002563    2574 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 09 19:55:15.007824 ip-10-0-136-183 kubenswrapper[2574]: W0909 19:55:15.002565    2574 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 09 19:55:15.007824 ip-10-0-136-183 kubenswrapper[2574]: W0909 19:55:15.002568    2574 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 09 19:55:15.007824 ip-10-0-136-183 kubenswrapper[2574]: W0909 19:55:15.002571    2574 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 09 19:55:15.007824 ip-10-0-136-183 kubenswrapper[2574]: W0909 19:55:15.002573    2574 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 09 19:55:15.007824 ip-10-0-136-183 kubenswrapper[2574]: W0909 19:55:15.002576    2574 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 09 19:55:15.007824 ip-10-0-136-183 kubenswrapper[2574]: W0909 19:55:15.002578    2574 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 09 19:55:15.008306 ip-10-0-136-183 kubenswrapper[2574]: W0909 19:55:15.002581    2574 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 09 19:55:15.008306 ip-10-0-136-183 kubenswrapper[2574]: W0909 19:55:15.002584    2574 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 09 19:55:15.008306 ip-10-0-136-183 kubenswrapper[2574]: W0909 19:55:15.002586    2574 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 09 19:55:15.008306 ip-10-0-136-183 kubenswrapper[2574]: W0909 19:55:15.002589    2574 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 09 19:55:15.008306 ip-10-0-136-183 kubenswrapper[2574]: W0909 19:55:15.002591    2574 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 09 19:55:15.008306 ip-10-0-136-183 kubenswrapper[2574]: W0909 19:55:15.002594    2574 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 09 19:55:15.008306 ip-10-0-136-183 kubenswrapper[2574]: W0909 19:55:15.002596    2574 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 09 19:55:15.008306 ip-10-0-136-183 kubenswrapper[2574]: W0909 19:55:15.002599    2574 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 09 19:55:15.008306 ip-10-0-136-183 kubenswrapper[2574]: W0909 19:55:15.002602    2574 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 09 19:55:15.008306 ip-10-0-136-183 kubenswrapper[2574]: W0909 19:55:15.002604    2574 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 09 19:55:15.008306 ip-10-0-136-183 kubenswrapper[2574]: W0909 19:55:15.002607    2574 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 09 19:55:15.008306 ip-10-0-136-183 kubenswrapper[2574]: W0909 19:55:15.002609    2574 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 09 19:55:15.008306 ip-10-0-136-183 kubenswrapper[2574]: W0909 19:55:15.002611    2574 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 09 19:55:15.008306 ip-10-0-136-183 kubenswrapper[2574]: W0909 19:55:15.002614    2574 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 09 19:55:15.008306 ip-10-0-136-183 kubenswrapper[2574]: W0909 19:55:15.002616    2574 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 09 19:55:15.008306 ip-10-0-136-183 kubenswrapper[2574]: W0909 19:55:15.002619    2574 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 09 19:55:15.008306 ip-10-0-136-183 kubenswrapper[2574]: W0909 19:55:15.002621    2574 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 09 19:55:15.008306 ip-10-0-136-183 kubenswrapper[2574]: W0909 19:55:15.002624    2574 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 09 19:55:15.008306 ip-10-0-136-183 kubenswrapper[2574]: W0909 19:55:15.002628    2574 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 09 19:55:15.008780 ip-10-0-136-183 kubenswrapper[2574]: W0909 19:55:15.002631    2574 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 09 19:55:15.008780 ip-10-0-136-183 kubenswrapper[2574]: W0909 19:55:15.002635    2574 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 09 19:55:15.008780 ip-10-0-136-183 kubenswrapper[2574]: W0909 19:55:15.002638    2574 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 09 19:55:15.008780 ip-10-0-136-183 kubenswrapper[2574]: W0909 19:55:15.002643    2574 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 09 19:55:15.008780 ip-10-0-136-183 kubenswrapper[2574]: W0909 19:55:15.002646    2574 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 09 19:55:15.008780 ip-10-0-136-183 kubenswrapper[2574]: W0909 19:55:15.002649    2574 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 09 19:55:15.008780 ip-10-0-136-183 kubenswrapper[2574]: W0909 19:55:15.002652    2574 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 09 19:55:15.008780 ip-10-0-136-183 kubenswrapper[2574]: W0909 19:55:15.002654    2574 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 09 19:55:15.008780 ip-10-0-136-183 kubenswrapper[2574]: W0909 19:55:15.002657    2574 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 09 19:55:15.008780 ip-10-0-136-183 kubenswrapper[2574]: W0909 19:55:15.002659    2574 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 09 19:55:15.008780 ip-10-0-136-183 kubenswrapper[2574]: W0909 19:55:15.002662    2574 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 09 19:55:15.008780 ip-10-0-136-183 kubenswrapper[2574]: W0909 19:55:15.002664    2574 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 09 19:55:15.008780 ip-10-0-136-183 kubenswrapper[2574]: W0909 19:55:15.002667    2574 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 09 19:55:15.008780 ip-10-0-136-183 kubenswrapper[2574]: W0909 19:55:15.002670    2574 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 09 19:55:15.008780 ip-10-0-136-183 kubenswrapper[2574]: W0909 19:55:15.002672    2574 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 09 19:55:15.008780 ip-10-0-136-183 kubenswrapper[2574]: W0909 19:55:15.002675    2574 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 09 19:55:15.008780 ip-10-0-136-183 kubenswrapper[2574]: W0909 19:55:15.002678    2574 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 09 19:55:15.008780 ip-10-0-136-183 kubenswrapper[2574]: W0909 19:55:15.002680    2574 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 09 19:55:15.008780 ip-10-0-136-183 kubenswrapper[2574]: W0909 19:55:15.002683    2574 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 09 19:55:15.009244 ip-10-0-136-183 kubenswrapper[2574]: W0909 19:55:15.002685    2574 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 09 19:55:15.009244 ip-10-0-136-183 kubenswrapper[2574]: W0909 19:55:15.002688    2574 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 09 19:55:15.009244 ip-10-0-136-183 kubenswrapper[2574]: W0909 19:55:15.002690    2574 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 09 19:55:15.009244 ip-10-0-136-183 kubenswrapper[2574]: W0909 19:55:15.002692    2574 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 09 19:55:15.009244 ip-10-0-136-183 kubenswrapper[2574]: W0909 19:55:15.002695    2574 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 09 19:55:15.009244 ip-10-0-136-183 kubenswrapper[2574]: W0909 19:55:15.002697    2574 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 09 19:55:15.009244 ip-10-0-136-183 kubenswrapper[2574]: W0909 19:55:15.002700    2574 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 09 19:55:15.009244 ip-10-0-136-183 kubenswrapper[2574]: W0909 19:55:15.002702    2574 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 09 19:55:15.009244 ip-10-0-136-183 kubenswrapper[2574]: W0909 19:55:15.002704    2574 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 09 19:55:15.009244 ip-10-0-136-183 kubenswrapper[2574]: W0909 19:55:15.002707    2574 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 09 19:55:15.009244 ip-10-0-136-183 kubenswrapper[2574]: W0909 19:55:15.002710    2574 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 09 19:55:15.009244 ip-10-0-136-183 kubenswrapper[2574]: W0909 19:55:15.002713    2574 feature_gate.go:328] unrecognized feature gate: Example2
Sep 09 19:55:15.009244 ip-10-0-136-183 kubenswrapper[2574]: W0909 19:55:15.002715    2574 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 09 19:55:15.009244 ip-10-0-136-183 kubenswrapper[2574]: W0909 19:55:15.002718    2574 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 09 19:55:15.009244 ip-10-0-136-183 kubenswrapper[2574]: W0909 19:55:15.002720    2574 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 09 19:55:15.009244 ip-10-0-136-183 kubenswrapper[2574]: W0909 19:55:15.002723    2574 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 09 19:55:15.009244 ip-10-0-136-183 kubenswrapper[2574]: W0909 19:55:15.002726    2574 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 09 19:55:15.009244 ip-10-0-136-183 kubenswrapper[2574]: W0909 19:55:15.002728    2574 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 09 19:55:15.009244 ip-10-0-136-183 kubenswrapper[2574]: W0909 19:55:15.002731    2574 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 09 19:55:15.009244 ip-10-0-136-183 kubenswrapper[2574]: W0909 19:55:15.002733    2574 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 09 19:55:15.009793 ip-10-0-136-183 kubenswrapper[2574]: I0909 19:55:15.002738    2574 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 09 19:55:15.009793 ip-10-0-136-183 kubenswrapper[2574]: I0909 19:55:15.002842    2574 server.go:962] "Client rotation is on, will bootstrap in background"
Sep 09 19:55:15.009793 ip-10-0-136-183 kubenswrapper[2574]: I0909 19:55:15.004988    2574 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir"
Sep 09 19:55:15.009793 ip-10-0-136-183 kubenswrapper[2574]: I0909 19:55:15.006606    2574 server.go:1019] "Starting client certificate rotation"
Sep 09 19:55:15.009793 ip-10-0-136-183 kubenswrapper[2574]: I0909 19:55:15.006718    2574 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Sep 09 19:55:15.009793 ip-10-0-136-183 kubenswrapper[2574]: I0909 19:55:15.006747    2574 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet"
Sep 09 19:55:15.010541 ip-10-0-136-183 kubenswrapper[2574]: I0909 19:55:15.010518    2574 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Sep 09 19:55:15.012498 ip-10-0-136-183 kubenswrapper[2574]: I0909 19:55:15.012478    2574 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Sep 09 19:55:15.022927 ip-10-0-136-183 kubenswrapper[2574]: I0909 19:55:15.022908    2574 log.go:25] "Validated CRI v1 runtime API"
Sep 09 19:55:15.025956 ip-10-0-136-183 kubenswrapper[2574]: I0909 19:55:15.025941    2574 log.go:25] "Validated CRI v1 image API"
Sep 09 19:55:15.026773 ip-10-0-136-183 kubenswrapper[2574]: I0909 19:55:15.026758    2574 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Sep 09 19:55:15.029096 ip-10-0-136-183 kubenswrapper[2574]: I0909 19:55:15.029076    2574 fs.go:135] Filesystem UUIDs: map[12b69031-35b0-4db0-82e5-6afe9bb79390:/dev/nvme0n1p4 4a8a3a1a-32b4-4a1a-ac67-c6c3d1b10a11:/dev/nvme0n1p3 7B77-95E7:/dev/nvme0n1p2]
Sep 09 19:55:15.029156 ip-10-0-136-183 kubenswrapper[2574]: I0909 19:55:15.029094    2574 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}]
Sep 09 19:55:15.035110 ip-10-0-136-183 kubenswrapper[2574]: I0909 19:55:15.034953    2574 manager.go:216] Machine: {Timestamp:2026-09-09 19:55:15.032757951 +0000 UTC m=+0.327351295 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3099868 MemoryCapacity:33164484608 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec246833ba75faf37f4c71d2c4eec2d4 SystemUUID:ec246833-ba75-faf3-7f4c-71d2c4eec2d4 BootID:8bd4ba6e-cca4-4bd6-847f-f2f3f3defe02 Filesystems:[{Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6111232 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16582242304 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16582242304 Type:vfs Inodes:4048399 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6632898560 Type:vfs Inodes:819200 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:1a:d4:27:67:3d Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:1a:d4:27:67:3d Speed:0 Mtu:9001} {Name:ovs-system MacAddress:aa:41:5d:4d:62:92 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164484608 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Sep 09 19:55:15.035110 ip-10-0-136-183 kubenswrapper[2574]: I0909 19:55:15.035106    2574 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Sep 09 19:55:15.035236 ip-10-0-136-183 kubenswrapper[2574]: I0909 19:55:15.035218    2574 manager.go:232] Version: {KernelVersion:5.14.0-570.138.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260901-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Sep 09 19:55:15.036770 ip-10-0-136-183 kubenswrapper[2574]: I0909 19:55:15.036753    2574 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Sep 09 19:55:15.038806 ip-10-0-136-183 kubenswrapper[2574]: I0909 19:55:15.038778    2574 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Sep 09 19:55:15.038943 ip-10-0-136-183 kubenswrapper[2574]: I0909 19:55:15.038809    2574 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-136-183.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Sep 09 19:55:15.039621 ip-10-0-136-183 kubenswrapper[2574]: I0909 19:55:15.039611    2574 topology_manager.go:138] "Creating topology manager with none policy"
Sep 09 19:55:15.039657 ip-10-0-136-183 kubenswrapper[2574]: I0909 19:55:15.039623    2574 container_manager_linux.go:309] "Creating device plugin manager"
Sep 09 19:55:15.039657 ip-10-0-136-183 kubenswrapper[2574]: I0909 19:55:15.039631    2574 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Sep 09 19:55:15.040562 ip-10-0-136-183 kubenswrapper[2574]: I0909 19:55:15.040551    2574 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Sep 09 19:55:15.042339 ip-10-0-136-183 kubenswrapper[2574]: I0909 19:55:15.042328    2574 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager"
Sep 09 19:55:15.045666 ip-10-0-136-183 kubenswrapper[2574]: I0909 19:55:15.045654    2574 state_mem.go:36] "Initialized new in-memory state store"
Sep 09 19:55:15.045774 ip-10-0-136-183 kubenswrapper[2574]: I0909 19:55:15.045765    2574 server.go:1240] "Using root directory" path="/var/lib/kubelet"
Sep 09 19:55:15.045803 ip-10-0-136-183 kubenswrapper[2574]: I0909 19:55:15.045792    2574 kubelet.go:486] "Attempting to sync node with API server"
Sep 09 19:55:15.045833 ip-10-0-136-183 kubenswrapper[2574]: I0909 19:55:15.045820    2574 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests"
Sep 09 19:55:15.045833 ip-10-0-136-183 kubenswrapper[2574]: I0909 19:55:15.045831    2574 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Sep 09 19:55:15.045886 ip-10-0-136-183 kubenswrapper[2574]: I0909 19:55:15.045840    2574 kubelet.go:398] "Adding apiserver pod source"
Sep 09 19:55:15.045886 ip-10-0-136-183 kubenswrapper[2574]: I0909 19:55:15.045850    2574 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Sep 09 19:55:15.046223 ip-10-0-136-183 kubenswrapper[2574]: I0909 19:55:15.046212    2574 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Sep 09 19:55:15.046259 ip-10-0-136-183 kubenswrapper[2574]: I0909 19:55:15.046230    2574 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Sep 09 19:55:15.048656 ip-10-0-136-183 kubenswrapper[2574]: I0909 19:55:15.048639    2574 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.12-5.rhaos4.21.gitd1ec63c.el9" apiVersion="v1"
Sep 09 19:55:15.051630 ip-10-0-136-183 kubenswrapper[2574]: I0909 19:55:15.051617    2574 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Sep 09 19:55:15.051679 ip-10-0-136-183 kubenswrapper[2574]: I0909 19:55:15.051648    2574 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Sep 09 19:55:15.052681 ip-10-0-136-183 kubenswrapper[2574]: I0909 19:55:15.052667    2574 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Sep 09 19:55:15.052738 ip-10-0-136-183 kubenswrapper[2574]: I0909 19:55:15.052685    2574 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Sep 09 19:55:15.052738 ip-10-0-136-183 kubenswrapper[2574]: I0909 19:55:15.052692    2574 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Sep 09 19:55:15.052738 ip-10-0-136-183 kubenswrapper[2574]: I0909 19:55:15.052697    2574 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Sep 09 19:55:15.052738 ip-10-0-136-183 kubenswrapper[2574]: I0909 19:55:15.052706    2574 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Sep 09 19:55:15.052738 ip-10-0-136-183 kubenswrapper[2574]: I0909 19:55:15.052712    2574 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Sep 09 19:55:15.052738 ip-10-0-136-183 kubenswrapper[2574]: I0909 19:55:15.052725    2574 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Sep 09 19:55:15.052738 ip-10-0-136-183 kubenswrapper[2574]: I0909 19:55:15.052731    2574 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Sep 09 19:55:15.052738 ip-10-0-136-183 kubenswrapper[2574]: I0909 19:55:15.052739    2574 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Sep 09 19:55:15.052941 ip-10-0-136-183 kubenswrapper[2574]: I0909 19:55:15.052745    2574 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Sep 09 19:55:15.052941 ip-10-0-136-183 kubenswrapper[2574]: I0909 19:55:15.052759    2574 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Sep 09 19:55:15.052941 ip-10-0-136-183 kubenswrapper[2574]: I0909 19:55:15.052764    2574 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Sep 09 19:55:15.053653 ip-10-0-136-183 kubenswrapper[2574]: I0909 19:55:15.053642    2574 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Sep 09 19:55:15.053653 ip-10-0-136-183 kubenswrapper[2574]: I0909 19:55:15.053652    2574 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Sep 09 19:55:15.058391 ip-10-0-136-183 kubenswrapper[2574]: I0909 19:55:15.058373    2574 server.go:1268] "Started kubelet"
Sep 09 19:55:15.058554 ip-10-0-136-183 kubenswrapper[2574]: I0909 19:55:15.058442    2574 server.go:180] "Starting to listen" address="0.0.0.0" port=10250
Sep 09 19:55:15.058710 ip-10-0-136-183 kubenswrapper[2574]: I0909 19:55:15.058646    2574 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Sep 09 19:55:15.058815 ip-10-0-136-183 kubenswrapper[2574]: I0909 19:55:15.058746    2574 server_v1.go:49] "podresources" method="list" useActivePods=true
Sep 09 19:55:15.059285 ip-10-0-136-183 systemd[1]: Started Kubernetes Kubelet.
Sep 09 19:55:15.060818 ip-10-0-136-183 kubenswrapper[2574]: I0909 19:55:15.060474    2574 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-bjkx9"
Sep 09 19:55:15.060818 ip-10-0-136-183 kubenswrapper[2574]: E0909 19:55:15.060646    2574 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-136-183.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node"
Sep 09 19:55:15.060818 ip-10-0-136-183 kubenswrapper[2574]: E0909 19:55:15.060644    2574 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service"
Sep 09 19:55:15.060818 ip-10-0-136-183 kubenswrapper[2574]: I0909 19:55:15.060674    2574 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-136-183.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Sep 09 19:55:15.061097 ip-10-0-136-183 kubenswrapper[2574]: I0909 19:55:15.060889    2574 server.go:310] "Adding debug handlers to kubelet server"
Sep 09 19:55:15.061261 ip-10-0-136-183 kubenswrapper[2574]: I0909 19:55:15.061247    2574 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Sep 09 19:55:15.064758 ip-10-0-136-183 kubenswrapper[2574]: E0909 19:55:15.064730    2574 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache"
Sep 09 19:55:15.068023 ip-10-0-136-183 kubenswrapper[2574]: I0909 19:55:15.068002    2574 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer"
Sep 09 19:55:15.068121 ip-10-0-136-183 kubenswrapper[2574]: I0909 19:55:15.068057    2574 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Sep 09 19:55:15.068121 ip-10-0-136-183 kubenswrapper[2574]: I0909 19:55:15.068097    2574 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-bjkx9"
Sep 09 19:55:15.068121 ip-10-0-136-183 kubenswrapper[2574]: E0909 19:55:15.068096    2574 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-136-183.ec2.internal\" not found"
Sep 09 19:55:15.069049 ip-10-0-136-183 kubenswrapper[2574]: I0909 19:55:15.069022    2574 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Sep 09 19:55:15.069049 ip-10-0-136-183 kubenswrapper[2574]: I0909 19:55:15.069027    2574 volume_manager.go:311] "The desired_state_of_world populator starts"
Sep 09 19:55:15.069049 ip-10-0-136-183 kubenswrapper[2574]: I0909 19:55:15.069053    2574 volume_manager.go:313] "Starting Kubelet Volume Manager"
Sep 09 19:55:15.069213 ip-10-0-136-183 kubenswrapper[2574]: I0909 19:55:15.069154    2574 reconstruct.go:97] "Volume reconstruction finished"
Sep 09 19:55:15.069213 ip-10-0-136-183 kubenswrapper[2574]: I0909 19:55:15.069160    2574 reconciler.go:29] "Reconciler: start to sync state"
Sep 09 19:55:15.070020 ip-10-0-136-183 kubenswrapper[2574]: I0909 19:55:15.070001    2574 factory.go:55] Registering systemd factory
Sep 09 19:55:15.070020 ip-10-0-136-183 kubenswrapper[2574]: I0909 19:55:15.070017    2574 factory.go:223] Registration of the systemd container factory successfully
Sep 09 19:55:15.070274 ip-10-0-136-183 kubenswrapper[2574]: I0909 19:55:15.070259    2574 factory.go:153] Registering CRI-O factory
Sep 09 19:55:15.070371 ip-10-0-136-183 kubenswrapper[2574]: I0909 19:55:15.070357    2574 factory.go:223] Registration of the crio container factory successfully
Sep 09 19:55:15.070371 ip-10-0-136-183 kubenswrapper[2574]: E0909 19:55:15.070361    2574 reflector.go:205] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver"
Sep 09 19:55:15.070508 ip-10-0-136-183 kubenswrapper[2574]: E0909 19:55:15.070312    2574 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-136-183.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms"
Sep 09 19:55:15.070508 ip-10-0-136-183 kubenswrapper[2574]: I0909 19:55:15.070417    2574 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Sep 09 19:55:15.070508 ip-10-0-136-183 kubenswrapper[2574]: I0909 19:55:15.070436    2574 factory.go:103] Registering Raw factory
Sep 09 19:55:15.070508 ip-10-0-136-183 kubenswrapper[2574]: I0909 19:55:15.070451    2574 manager.go:1196] Started watching for new ooms in manager
Sep 09 19:55:15.071344 ip-10-0-136-183 kubenswrapper[2574]: E0909 19:55:15.070287    2574 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-136-183.ec2.internal.18d3bf35dd02a999  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-136-183.ec2.internal,UID:ip-10-0-136-183.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-136-183.ec2.internal,},FirstTimestamp:2026-09-09 19:55:15.058330009 +0000 UTC m=+0.352923350,LastTimestamp:2026-09-09 19:55:15.058330009 +0000 UTC m=+0.352923350,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-136-183.ec2.internal,}"
Sep 09 19:55:15.071423 ip-10-0-136-183 kubenswrapper[2574]: I0909 19:55:15.071363    2574 manager.go:349] Starting recovery of all containers
Sep 09 19:55:15.082553 ip-10-0-136-183 kubenswrapper[2574]: I0909 19:55:15.082533    2574 manager.go:354] Recovery completed
Sep 09 19:55:15.085616 ip-10-0-136-183 kubenswrapper[2574]: E0909 19:55:15.085594    2574 watcher.go:152] Failed to watch directory "/sys/fs/cgroup/system.slice/systemd-update-utmp-runlevel.service": inotify_add_watch /sys/fs/cgroup/system.slice/systemd-update-utmp-runlevel.service: no such file or directory
Sep 09 19:55:15.088839 ip-10-0-136-183 kubenswrapper[2574]: I0909 19:55:15.088822    2574 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 09 19:55:15.090813 ip-10-0-136-183 kubenswrapper[2574]: I0909 19:55:15.090793    2574 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-183.ec2.internal" event="NodeHasSufficientMemory"
Sep 09 19:55:15.090902 ip-10-0-136-183 kubenswrapper[2574]: I0909 19:55:15.090827    2574 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-183.ec2.internal" event="NodeHasNoDiskPressure"
Sep 09 19:55:15.090902 ip-10-0-136-183 kubenswrapper[2574]: I0909 19:55:15.090838    2574 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-183.ec2.internal" event="NodeHasSufficientPID"
Sep 09 19:55:15.091438 ip-10-0-136-183 kubenswrapper[2574]: I0909 19:55:15.091405    2574 cpu_manager.go:222] "Starting CPU manager" policy="none"
Sep 09 19:55:15.091438 ip-10-0-136-183 kubenswrapper[2574]: I0909 19:55:15.091418    2574 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s"
Sep 09 19:55:15.091438 ip-10-0-136-183 kubenswrapper[2574]: I0909 19:55:15.091432    2574 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 09 19:55:15.091630 ip-10-0-136-183 kubenswrapper[2574]: I0909 19:55:15.091437    2574 state_mem.go:36] "Initialized new in-memory state store"
Sep 09 19:55:15.092388 ip-10-0-136-183 kubenswrapper[2574]: I0909 19:55:15.092373    2574 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-183.ec2.internal" event="NodeHasSufficientMemory"
Sep 09 19:55:15.092495 ip-10-0-136-183 kubenswrapper[2574]: I0909 19:55:15.092404    2574 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-183.ec2.internal" event="NodeHasNoDiskPressure"
Sep 09 19:55:15.092495 ip-10-0-136-183 kubenswrapper[2574]: I0909 19:55:15.092418    2574 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-183.ec2.internal" event="NodeHasSufficientPID"
Sep 09 19:55:15.092706 ip-10-0-136-183 kubenswrapper[2574]: I0909 19:55:15.092695    2574 policy_none.go:49] "None policy: Start"
Sep 09 19:55:15.092748 ip-10-0-136-183 kubenswrapper[2574]: I0909 19:55:15.092710    2574 memory_manager.go:187] "Starting memorymanager" policy="None"
Sep 09 19:55:15.092748 ip-10-0-136-183 kubenswrapper[2574]: I0909 19:55:15.092720    2574 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Sep 09 19:55:15.093779 ip-10-0-136-183 kubenswrapper[2574]: I0909 19:55:15.093642    2574 policy_none.go:47] "Start"
Sep 09 19:55:15.124124 ip-10-0-136-183 kubenswrapper[2574]: I0909 19:55:15.124101    2574 manager.go:337] "Starting Device Plugin manager"
Sep 09 19:55:15.132231 ip-10-0-136-183 kubenswrapper[2574]: E0909 19:55:15.124150    2574 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Sep 09 19:55:15.132231 ip-10-0-136-183 kubenswrapper[2574]: I0909 19:55:15.124164    2574 server.go:87] "Starting device plugin registration server"
Sep 09 19:55:15.132231 ip-10-0-136-183 kubenswrapper[2574]: I0909 19:55:15.124531    2574 eviction_manager.go:189] "Eviction manager: starting control loop"
Sep 09 19:55:15.132231 ip-10-0-136-183 kubenswrapper[2574]: I0909 19:55:15.124545    2574 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Sep 09 19:55:15.132231 ip-10-0-136-183 kubenswrapper[2574]: I0909 19:55:15.124729    2574 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Sep 09 19:55:15.132231 ip-10-0-136-183 kubenswrapper[2574]: I0909 19:55:15.124799    2574 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts"
Sep 09 19:55:15.132231 ip-10-0-136-183 kubenswrapper[2574]: I0909 19:55:15.124808    2574 plugin_manager.go:118] "Starting Kubelet Plugin Manager"
Sep 09 19:55:15.132231 ip-10-0-136-183 kubenswrapper[2574]: E0909 19:55:15.125525    2574 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Sep 09 19:55:15.132231 ip-10-0-136-183 kubenswrapper[2574]: E0909 19:55:15.125573    2574 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-136-183.ec2.internal\" not found"
Sep 09 19:55:15.225326 ip-10-0-136-183 kubenswrapper[2574]: I0909 19:55:15.225296    2574 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 09 19:55:15.226427 ip-10-0-136-183 kubenswrapper[2574]: I0909 19:55:15.226406    2574 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-183.ec2.internal" event="NodeHasSufficientMemory"
Sep 09 19:55:15.226567 ip-10-0-136-183 kubenswrapper[2574]: I0909 19:55:15.226444    2574 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-183.ec2.internal" event="NodeHasNoDiskPressure"
Sep 09 19:55:15.226567 ip-10-0-136-183 kubenswrapper[2574]: I0909 19:55:15.226474    2574 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-183.ec2.internal" event="NodeHasSufficientPID"
Sep 09 19:55:15.226567 ip-10-0-136-183 kubenswrapper[2574]: I0909 19:55:15.226510    2574 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-136-183.ec2.internal"
Sep 09 19:55:15.235245 ip-10-0-136-183 kubenswrapper[2574]: I0909 19:55:15.235176    2574 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Sep 09 19:55:15.236398 ip-10-0-136-183 kubenswrapper[2574]: I0909 19:55:15.236382    2574 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Sep 09 19:55:15.236476 ip-10-0-136-183 kubenswrapper[2574]: I0909 19:55:15.236435    2574 status_manager.go:244] "Starting to sync pod status with apiserver"
Sep 09 19:55:15.236532 ip-10-0-136-183 kubenswrapper[2574]: I0909 19:55:15.236481    2574 kubelet.go:2448] "Starting kubelet main sync loop"
Sep 09 19:55:15.236625 ip-10-0-136-183 kubenswrapper[2574]: E0909 19:55:15.236611    2574 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful"
Sep 09 19:55:15.238245 ip-10-0-136-183 kubenswrapper[2574]: I0909 19:55:15.238225    2574 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-136-183.ec2.internal"
Sep 09 19:55:15.238329 ip-10-0-136-183 kubenswrapper[2574]: E0909 19:55:15.238254    2574 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-136-183.ec2.internal\": node \"ip-10-0-136-183.ec2.internal\" not found"
Sep 09 19:55:15.239037 ip-10-0-136-183 kubenswrapper[2574]: I0909 19:55:15.239021    2574 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 09 19:55:15.281506 ip-10-0-136-183 kubenswrapper[2574]: E0909 19:55:15.281485    2574 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-136-183.ec2.internal\" not found"
Sep 09 19:55:15.337530 ip-10-0-136-183 kubenswrapper[2574]: I0909 19:55:15.337451    2574 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-183.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-136-183.ec2.internal"]
Sep 09 19:55:15.337670 ip-10-0-136-183 kubenswrapper[2574]: I0909 19:55:15.337590    2574 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 09 19:55:15.338638 ip-10-0-136-183 kubenswrapper[2574]: I0909 19:55:15.338621    2574 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-183.ec2.internal" event="NodeHasSufficientMemory"
Sep 09 19:55:15.338683 ip-10-0-136-183 kubenswrapper[2574]: I0909 19:55:15.338657    2574 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-183.ec2.internal" event="NodeHasNoDiskPressure"
Sep 09 19:55:15.338683 ip-10-0-136-183 kubenswrapper[2574]: I0909 19:55:15.338671    2574 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-183.ec2.internal" event="NodeHasSufficientPID"
Sep 09 19:55:15.339742 ip-10-0-136-183 kubenswrapper[2574]: I0909 19:55:15.339731    2574 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 09 19:55:15.339859 ip-10-0-136-183 kubenswrapper[2574]: I0909 19:55:15.339844    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-183.ec2.internal"
Sep 09 19:55:15.339908 ip-10-0-136-183 kubenswrapper[2574]: I0909 19:55:15.339876    2574 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 09 19:55:15.340497 ip-10-0-136-183 kubenswrapper[2574]: I0909 19:55:15.340483    2574 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-183.ec2.internal" event="NodeHasSufficientMemory"
Sep 09 19:55:15.340568 ip-10-0-136-183 kubenswrapper[2574]: I0909 19:55:15.340511    2574 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-183.ec2.internal" event="NodeHasNoDiskPressure"
Sep 09 19:55:15.340568 ip-10-0-136-183 kubenswrapper[2574]: I0909 19:55:15.340526    2574 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-183.ec2.internal" event="NodeHasSufficientPID"
Sep 09 19:55:15.340971 ip-10-0-136-183 kubenswrapper[2574]: I0909 19:55:15.340953    2574 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-183.ec2.internal" event="NodeHasSufficientMemory"
Sep 09 19:55:15.341061 ip-10-0-136-183 kubenswrapper[2574]: I0909 19:55:15.340996    2574 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-183.ec2.internal" event="NodeHasNoDiskPressure"
Sep 09 19:55:15.341061 ip-10-0-136-183 kubenswrapper[2574]: I0909 19:55:15.341007    2574 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-183.ec2.internal" event="NodeHasSufficientPID"
Sep 09 19:55:15.342075 ip-10-0-136-183 kubenswrapper[2574]: I0909 19:55:15.342062    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-136-183.ec2.internal"
Sep 09 19:55:15.342127 ip-10-0-136-183 kubenswrapper[2574]: I0909 19:55:15.342086    2574 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 09 19:55:15.342829 ip-10-0-136-183 kubenswrapper[2574]: I0909 19:55:15.342808    2574 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-183.ec2.internal" event="NodeHasSufficientMemory"
Sep 09 19:55:15.342914 ip-10-0-136-183 kubenswrapper[2574]: I0909 19:55:15.342836    2574 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-183.ec2.internal" event="NodeHasNoDiskPressure"
Sep 09 19:55:15.342914 ip-10-0-136-183 kubenswrapper[2574]: I0909 19:55:15.342846    2574 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-183.ec2.internal" event="NodeHasSufficientPID"
Sep 09 19:55:15.367240 ip-10-0-136-183 kubenswrapper[2574]: E0909 19:55:15.367211    2574 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-136-183.ec2.internal\" not found" node="ip-10-0-136-183.ec2.internal"
Sep 09 19:55:15.370515 ip-10-0-136-183 kubenswrapper[2574]: I0909 19:55:15.370498    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/9de809fe92e7ce3d095a2762d4b6b77a-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-136-183.ec2.internal\" (UID: \"9de809fe92e7ce3d095a2762d4b6b77a\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-183.ec2.internal"
Sep 09 19:55:15.370611 ip-10-0-136-183 kubenswrapper[2574]: I0909 19:55:15.370526    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/9de809fe92e7ce3d095a2762d4b6b77a-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-136-183.ec2.internal\" (UID: \"9de809fe92e7ce3d095a2762d4b6b77a\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-183.ec2.internal"
Sep 09 19:55:15.370611 ip-10-0-136-183 kubenswrapper[2574]: I0909 19:55:15.370551    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/c82ea50b31881ad827f1ff32ee67e2c3-config\") pod \"kube-apiserver-proxy-ip-10-0-136-183.ec2.internal\" (UID: \"c82ea50b31881ad827f1ff32ee67e2c3\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-136-183.ec2.internal"
Sep 09 19:55:15.371802 ip-10-0-136-183 kubenswrapper[2574]: E0909 19:55:15.371787    2574 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-136-183.ec2.internal\" not found" node="ip-10-0-136-183.ec2.internal"
Sep 09 19:55:15.381830 ip-10-0-136-183 kubenswrapper[2574]: E0909 19:55:15.381813    2574 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-136-183.ec2.internal\" not found"
Sep 09 19:55:15.471181 ip-10-0-136-183 kubenswrapper[2574]: I0909 19:55:15.471151    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/9de809fe92e7ce3d095a2762d4b6b77a-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-136-183.ec2.internal\" (UID: \"9de809fe92e7ce3d095a2762d4b6b77a\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-183.ec2.internal"
Sep 09 19:55:15.471181 ip-10-0-136-183 kubenswrapper[2574]: I0909 19:55:15.471183    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/9de809fe92e7ce3d095a2762d4b6b77a-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-136-183.ec2.internal\" (UID: \"9de809fe92e7ce3d095a2762d4b6b77a\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-183.ec2.internal"
Sep 09 19:55:15.471389 ip-10-0-136-183 kubenswrapper[2574]: I0909 19:55:15.471202    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/c82ea50b31881ad827f1ff32ee67e2c3-config\") pod \"kube-apiserver-proxy-ip-10-0-136-183.ec2.internal\" (UID: \"c82ea50b31881ad827f1ff32ee67e2c3\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-136-183.ec2.internal"
Sep 09 19:55:15.471389 ip-10-0-136-183 kubenswrapper[2574]: I0909 19:55:15.471241    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/c82ea50b31881ad827f1ff32ee67e2c3-config\") pod \"kube-apiserver-proxy-ip-10-0-136-183.ec2.internal\" (UID: \"c82ea50b31881ad827f1ff32ee67e2c3\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-136-183.ec2.internal"
Sep 09 19:55:15.471389 ip-10-0-136-183 kubenswrapper[2574]: I0909 19:55:15.471261    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/9de809fe92e7ce3d095a2762d4b6b77a-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-136-183.ec2.internal\" (UID: \"9de809fe92e7ce3d095a2762d4b6b77a\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-183.ec2.internal"
Sep 09 19:55:15.471389 ip-10-0-136-183 kubenswrapper[2574]: I0909 19:55:15.471241    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/9de809fe92e7ce3d095a2762d4b6b77a-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-136-183.ec2.internal\" (UID: \"9de809fe92e7ce3d095a2762d4b6b77a\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-183.ec2.internal"
Sep 09 19:55:15.482491 ip-10-0-136-183 kubenswrapper[2574]: E0909 19:55:15.482452    2574 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-136-183.ec2.internal\" not found"
Sep 09 19:55:15.583399 ip-10-0-136-183 kubenswrapper[2574]: E0909 19:55:15.583305    2574 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-136-183.ec2.internal\" not found"
Sep 09 19:55:16.918010 ip-10-0-136-183 kubenswrapper[2574]: I0909 19:55:16.917974    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-183.ec2.internal"
Sep 09 19:55:16.923115 ip-10-0-136-183 kubenswrapper[2574]: I0909 19:55:16.923093    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-136-183.ec2.internal"
Sep 09 19:55:16.932658 ip-10-0-136-183 kubenswrapper[2574]: E0909 19:55:16.932640    2574 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-136-183.ec2.internal\" not found"
Sep 09 19:55:17.033261 ip-10-0-136-183 kubenswrapper[2574]: E0909 19:55:17.033224    2574 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-136-183.ec2.internal\" not found"
Sep 09 19:55:17.133755 ip-10-0-136-183 kubenswrapper[2574]: E0909 19:55:17.133722    2574 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-136-183.ec2.internal\" not found"
Sep 09 19:55:17.234320 ip-10-0-136-183 kubenswrapper[2574]: E0909 19:55:17.234290    2574 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-136-183.ec2.internal\" not found"
Sep 09 19:55:17.254456 ip-10-0-136-183 kubenswrapper[2574]: I0909 19:55:17.254435    2574 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials"
Sep 09 19:55:17.254590 ip-10-0-136-183 kubenswrapper[2574]: I0909 19:55:17.254573    2574 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Sep 09 19:55:17.318890 ip-10-0-136-183 kubenswrapper[2574]: I0909 19:55:17.318846    2574 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-09-08 19:50:16 +0000 UTC" deadline="2028-02-26 21:27:23.957967909 +0000 UTC"
Sep 09 19:55:17.318890 ip-10-0-136-183 kubenswrapper[2574]: I0909 19:55:17.318886    2574 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="12841h32m6.639086418s"
Sep 09 19:55:17.334986 ip-10-0-136-183 kubenswrapper[2574]: E0909 19:55:17.334958    2574 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-136-183.ec2.internal\" not found"
Sep 09 19:55:17.421770 ip-10-0-136-183 kubenswrapper[2574]: W0909 19:55:17.421735    2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9de809fe92e7ce3d095a2762d4b6b77a.slice/crio-725d5cd8eb465ddf5fff864bc2c4a0420acd9e55ceba9f8f67224b4e0f5a6871 WatchSource:0}: Error finding container 725d5cd8eb465ddf5fff864bc2c4a0420acd9e55ceba9f8f67224b4e0f5a6871: Status 404 returned error can't find the container with id 725d5cd8eb465ddf5fff864bc2c4a0420acd9e55ceba9f8f67224b4e0f5a6871
Sep 09 19:55:17.425944 ip-10-0-136-183 kubenswrapper[2574]: I0909 19:55:17.425918    2574 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 09 19:55:17.435049 ip-10-0-136-183 kubenswrapper[2574]: E0909 19:55:17.435029    2574 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-136-183.ec2.internal\" not found"
Sep 09 19:55:17.438003 ip-10-0-136-183 kubenswrapper[2574]: I0909 19:55:17.437986    2574 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 09 19:55:17.488018 ip-10-0-136-183 kubenswrapper[2574]: I0909 19:55:17.487973    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-183.ec2.internal" event={"ID":"9de809fe92e7ce3d095a2762d4b6b77a","Type":"ContainerStarted","Data":"725d5cd8eb465ddf5fff864bc2c4a0420acd9e55ceba9f8f67224b4e0f5a6871"}
Sep 09 19:55:17.535510 ip-10-0-136-183 kubenswrapper[2574]: E0909 19:55:17.535471    2574 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-136-183.ec2.internal\" not found"
Sep 09 19:55:17.602527 ip-10-0-136-183 kubenswrapper[2574]: I0909 19:55:17.602500    2574 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 09 19:55:17.636489 ip-10-0-136-183 kubenswrapper[2574]: E0909 19:55:17.636460    2574 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-136-183.ec2.internal\" not found"
Sep 09 19:55:17.654089 ip-10-0-136-183 kubenswrapper[2574]: I0909 19:55:17.653602    2574 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 09 19:55:17.717863 ip-10-0-136-183 kubenswrapper[2574]: I0909 19:55:17.717744    2574 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-183.ec2.internal"
Sep 09 19:55:17.730750 ip-10-0-136-183 kubenswrapper[2574]: I0909 19:55:17.730727    2574 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Sep 09 19:55:17.731584 ip-10-0-136-183 kubenswrapper[2574]: I0909 19:55:17.731570    2574 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-136-183.ec2.internal"
Sep 09 19:55:17.738384 ip-10-0-136-183 kubenswrapper[2574]: I0909 19:55:17.738357    2574 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Sep 09 19:55:18.295755 ip-10-0-136-183 kubenswrapper[2574]: I0909 19:55:18.295688    2574 apiserver.go:52] "Watching apiserver"
Sep 09 19:55:18.302157 ip-10-0-136-183 kubenswrapper[2574]: I0909 19:55:18.302120    2574 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Sep 09 19:55:18.302613 ip-10-0-136-183 kubenswrapper[2574]: I0909 19:55:18.302583    2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-multus/multus-sbx29","openshift-multus/network-metrics-daemon-mdrck","openshift-network-operator/iptables-alerter-gvsgq","openshift-ovn-kubernetes/ovnkube-node-jcr9m","kube-system/global-pull-secret-syncer-gjc9x","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bvxbg","openshift-cluster-node-tuning-operator/tuned-svtdz","openshift-dns/node-resolver-q8krg","openshift-image-registry/node-ca-mgg8m","openshift-multus/multus-additional-cni-plugins-gzqh4","openshift-network-diagnostics/network-check-target-4czxv","kube-system/konnectivity-agent-d9ww4","kube-system/kube-apiserver-proxy-ip-10-0-136-183.ec2.internal","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-183.ec2.internal"]
Sep 09 19:55:18.304184 ip-10-0-136-183 kubenswrapper[2574]: I0909 19:55:18.304155    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-sbx29"
Sep 09 19:55:18.305327 ip-10-0-136-183 kubenswrapper[2574]: I0909 19:55:18.305305    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mdrck"
Sep 09 19:55:18.305425 ip-10-0-136-183 kubenswrapper[2574]: E0909 19:55:18.305374    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-mdrck" podUID="47c2b5b9-456a-4528-8d83-7ce3269cc16e"
Sep 09 19:55:18.306994 ip-10-0-136-183 kubenswrapper[2574]: I0909 19:55:18.306691    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-gvsgq"
Sep 09 19:55:18.307128 ip-10-0-136-183 kubenswrapper[2574]: I0909 19:55:18.307108    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-sfrps\""
Sep 09 19:55:18.307223 ip-10-0-136-183 kubenswrapper[2574]: I0909 19:55:18.307162    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Sep 09 19:55:18.307326 ip-10-0-136-183 kubenswrapper[2574]: I0909 19:55:18.307299    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Sep 09 19:55:18.307565 ip-10-0-136-183 kubenswrapper[2574]: I0909 19:55:18.307424    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Sep 09 19:55:18.308346 ip-10-0-136-183 kubenswrapper[2574]: I0909 19:55:18.308169    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Sep 09 19:55:18.308447 ip-10-0-136-183 kubenswrapper[2574]: I0909 19:55:18.308405    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-jcr9m"
Sep 09 19:55:18.308859 ip-10-0-136-183 kubenswrapper[2574]: I0909 19:55:18.308843    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Sep 09 19:55:18.309297 ip-10-0-136-183 kubenswrapper[2574]: I0909 19:55:18.309275    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Sep 09 19:55:18.309425 ip-10-0-136-183 kubenswrapper[2574]: I0909 19:55:18.309411    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Sep 09 19:55:18.309758 ip-10-0-136-183 kubenswrapper[2574]: I0909 19:55:18.309738    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-gjc9x"
Sep 09 19:55:18.309897 ip-10-0-136-183 kubenswrapper[2574]: E0909 19:55:18.309829    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-gjc9x" podUID="c756f4c9-2360-4fff-a136-b6869b02977e"
Sep 09 19:55:18.309897 ip-10-0-136-183 kubenswrapper[2574]: I0909 19:55:18.309740    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-76dxj\""
Sep 09 19:55:18.310687 ip-10-0-136-183 kubenswrapper[2574]: I0909 19:55:18.310658    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bvxbg"
Sep 09 19:55:18.311068 ip-10-0-136-183 kubenswrapper[2574]: I0909 19:55:18.311048    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Sep 09 19:55:18.311163 ip-10-0-136-183 kubenswrapper[2574]: I0909 19:55:18.311086    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Sep 09 19:55:18.311319 ip-10-0-136-183 kubenswrapper[2574]: I0909 19:55:18.311285    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Sep 09 19:55:18.311319 ip-10-0-136-183 kubenswrapper[2574]: I0909 19:55:18.311312    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Sep 09 19:55:18.311514 ip-10-0-136-183 kubenswrapper[2574]: I0909 19:55:18.311465    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Sep 09 19:55:18.311583 ip-10-0-136-183 kubenswrapper[2574]: I0909 19:55:18.311515    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Sep 09 19:55:18.311641 ip-10-0-136-183 kubenswrapper[2574]: I0909 19:55:18.311628    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-bshc8\""
Sep 09 19:55:18.311948 ip-10-0-136-183 kubenswrapper[2574]: I0909 19:55:18.311881    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-svtdz"
Sep 09 19:55:18.313225 ip-10-0-136-183 kubenswrapper[2574]: I0909 19:55:18.313204    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-q8krg"
Sep 09 19:55:18.315376 ip-10-0-136-183 kubenswrapper[2574]: I0909 19:55:18.315167    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Sep 09 19:55:18.315376 ip-10-0-136-183 kubenswrapper[2574]: I0909 19:55:18.315254    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Sep 09 19:55:18.315951 ip-10-0-136-183 kubenswrapper[2574]: I0909 19:55:18.315908    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Sep 09 19:55:18.316052 ip-10-0-136-183 kubenswrapper[2574]: I0909 19:55:18.315934    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Sep 09 19:55:18.316395 ip-10-0-136-183 kubenswrapper[2574]: I0909 19:55:18.316238    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-mgg8m"
Sep 09 19:55:18.320309 ip-10-0-136-183 kubenswrapper[2574]: I0909 19:55:18.317838    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-r8v26\""
Sep 09 19:55:18.320309 ip-10-0-136-183 kubenswrapper[2574]: I0909 19:55:18.318038    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Sep 09 19:55:18.320309 ip-10-0-136-183 kubenswrapper[2574]: I0909 19:55:18.318241    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-7hjhs\""
Sep 09 19:55:18.322509 ip-10-0-136-183 kubenswrapper[2574]: I0909 19:55:18.320871    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Sep 09 19:55:18.322509 ip-10-0-136-183 kubenswrapper[2574]: I0909 19:55:18.321148    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-mcv4q\""
Sep 09 19:55:18.322509 ip-10-0-136-183 kubenswrapper[2574]: I0909 19:55:18.321530    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Sep 09 19:55:18.322509 ip-10-0-136-183 kubenswrapper[2574]: I0909 19:55:18.321827    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Sep 09 19:55:18.322509 ip-10-0-136-183 kubenswrapper[2574]: I0909 19:55:18.322074    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-gzqh4"
Sep 09 19:55:18.326428 ip-10-0-136-183 kubenswrapper[2574]: I0909 19:55:18.322874    2574 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving"
Sep 09 19:55:18.326428 ip-10-0-136-183 kubenswrapper[2574]: I0909 19:55:18.323880    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-87b2c\""
Sep 09 19:55:18.326428 ip-10-0-136-183 kubenswrapper[2574]: I0909 19:55:18.324167    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-b7ttl\""
Sep 09 19:55:18.326428 ip-10-0-136-183 kubenswrapper[2574]: I0909 19:55:18.324239    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Sep 09 19:55:18.326428 ip-10-0-136-183 kubenswrapper[2574]: I0909 19:55:18.324452    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Sep 09 19:55:18.326428 ip-10-0-136-183 kubenswrapper[2574]: I0909 19:55:18.324521    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Sep 09 19:55:18.326428 ip-10-0-136-183 kubenswrapper[2574]: I0909 19:55:18.324880    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Sep 09 19:55:18.329417 ip-10-0-136-183 kubenswrapper[2574]: I0909 19:55:18.329363    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-4czxv"
Sep 09 19:55:18.331496 ip-10-0-136-183 kubenswrapper[2574]: I0909 19:55:18.331272    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-d9ww4"
Sep 09 19:55:18.331496 ip-10-0-136-183 kubenswrapper[2574]: E0909 19:55:18.331319    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-4czxv" podUID="54458a6d-921b-4637-addc-7c4ee839b592"
Sep 09 19:55:18.331637 ip-10-0-136-183 kubenswrapper[2574]: I0909 19:55:18.331544    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/9c19e4c1-5ffa-4fb8-88f0-93f1b32a591b-system-cni-dir\") pod \"multus-sbx29\" (UID: \"9c19e4c1-5ffa-4fb8-88f0-93f1b32a591b\") " pod="openshift-multus/multus-sbx29"
Sep 09 19:55:18.331637 ip-10-0-136-183 kubenswrapper[2574]: I0909 19:55:18.331583    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/b06375df-d12d-40a3-a5a7-18fa1420f388-etc-sysconfig\") pod \"tuned-svtdz\" (UID: \"b06375df-d12d-40a3-a5a7-18fa1420f388\") " pod="openshift-cluster-node-tuning-operator/tuned-svtdz"
Sep 09 19:55:18.331637 ip-10-0-136-183 kubenswrapper[2574]: I0909 19:55:18.331613    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/b06375df-d12d-40a3-a5a7-18fa1420f388-etc-sysctl-conf\") pod \"tuned-svtdz\" (UID: \"b06375df-d12d-40a3-a5a7-18fa1420f388\") " pod="openshift-cluster-node-tuning-operator/tuned-svtdz"
Sep 09 19:55:18.331784 ip-10-0-136-183 kubenswrapper[2574]: I0909 19:55:18.331638    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p6r7f\" (UniqueName: \"kubernetes.io/projected/39099fe9-527d-4802-b40b-1bf535221165-kube-api-access-p6r7f\") pod \"node-ca-mgg8m\" (UID: \"39099fe9-527d-4802-b40b-1bf535221165\") " pod="openshift-image-registry/node-ca-mgg8m"
Sep 09 19:55:18.331784 ip-10-0-136-183 kubenswrapper[2574]: I0909 19:55:18.331663    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/2eb82961-9118-4b08-9d97-8d5cbac8cac3-log-socket\") pod \"ovnkube-node-jcr9m\" (UID: \"2eb82961-9118-4b08-9d97-8d5cbac8cac3\") " pod="openshift-ovn-kubernetes/ovnkube-node-jcr9m"
Sep 09 19:55:18.331784 ip-10-0-136-183 kubenswrapper[2574]: I0909 19:55:18.331687    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/2eb82961-9118-4b08-9d97-8d5cbac8cac3-host-run-ovn-kubernetes\") pod \"ovnkube-node-jcr9m\" (UID: \"2eb82961-9118-4b08-9d97-8d5cbac8cac3\") " pod="openshift-ovn-kubernetes/ovnkube-node-jcr9m"
Sep 09 19:55:18.331784 ip-10-0-136-183 kubenswrapper[2574]: I0909 19:55:18.331711    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/2eb82961-9118-4b08-9d97-8d5cbac8cac3-ovn-node-metrics-cert\") pod \"ovnkube-node-jcr9m\" (UID: \"2eb82961-9118-4b08-9d97-8d5cbac8cac3\") " pod="openshift-ovn-kubernetes/ovnkube-node-jcr9m"
Sep 09 19:55:18.331784 ip-10-0-136-183 kubenswrapper[2574]: I0909 19:55:18.331747    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/2eb82961-9118-4b08-9d97-8d5cbac8cac3-ovnkube-script-lib\") pod \"ovnkube-node-jcr9m\" (UID: \"2eb82961-9118-4b08-9d97-8d5cbac8cac3\") " pod="openshift-ovn-kubernetes/ovnkube-node-jcr9m"
Sep 09 19:55:18.332039 ip-10-0-136-183 kubenswrapper[2574]: I0909 19:55:18.331831    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/bed9028e-2a22-4c8f-a637-bb5ad9204adc-tuning-conf-dir\") pod \"multus-additional-cni-plugins-gzqh4\" (UID: \"bed9028e-2a22-4c8f-a637-bb5ad9204adc\") " pod="openshift-multus/multus-additional-cni-plugins-gzqh4"
Sep 09 19:55:18.332039 ip-10-0-136-183 kubenswrapper[2574]: I0909 19:55:18.331856    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/71b40e5b-d6b9-489f-bb2e-4b5e1e5915e5-etc-selinux\") pod \"aws-ebs-csi-driver-node-bvxbg\" (UID: \"71b40e5b-d6b9-489f-bb2e-4b5e1e5915e5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bvxbg"
Sep 09 19:55:18.332039 ip-10-0-136-183 kubenswrapper[2574]: I0909 19:55:18.331876    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/b06375df-d12d-40a3-a5a7-18fa1420f388-host\") pod \"tuned-svtdz\" (UID: \"b06375df-d12d-40a3-a5a7-18fa1420f388\") " pod="openshift-cluster-node-tuning-operator/tuned-svtdz"
Sep 09 19:55:18.332039 ip-10-0-136-183 kubenswrapper[2574]: I0909 19:55:18.331900    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/9c19e4c1-5ffa-4fb8-88f0-93f1b32a591b-cnibin\") pod \"multus-sbx29\" (UID: \"9c19e4c1-5ffa-4fb8-88f0-93f1b32a591b\") " pod="openshift-multus/multus-sbx29"
Sep 09 19:55:18.332039 ip-10-0-136-183 kubenswrapper[2574]: I0909 19:55:18.331931    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/9c19e4c1-5ffa-4fb8-88f0-93f1b32a591b-host-run-k8s-cni-cncf-io\") pod \"multus-sbx29\" (UID: \"9c19e4c1-5ffa-4fb8-88f0-93f1b32a591b\") " pod="openshift-multus/multus-sbx29"
Sep 09 19:55:18.332039 ip-10-0-136-183 kubenswrapper[2574]: I0909 19:55:18.331954    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/9c19e4c1-5ffa-4fb8-88f0-93f1b32a591b-host-var-lib-cni-multus\") pod \"multus-sbx29\" (UID: \"9c19e4c1-5ffa-4fb8-88f0-93f1b32a591b\") " pod="openshift-multus/multus-sbx29"
Sep 09 19:55:18.332039 ip-10-0-136-183 kubenswrapper[2574]: I0909 19:55:18.331976    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/9c19e4c1-5ffa-4fb8-88f0-93f1b32a591b-host-var-lib-kubelet\") pod \"multus-sbx29\" (UID: \"9c19e4c1-5ffa-4fb8-88f0-93f1b32a591b\") " pod="openshift-multus/multus-sbx29"
Sep 09 19:55:18.332039 ip-10-0-136-183 kubenswrapper[2574]: I0909 19:55:18.331998    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/9c19e4c1-5ffa-4fb8-88f0-93f1b32a591b-multus-daemon-config\") pod \"multus-sbx29\" (UID: \"9c19e4c1-5ffa-4fb8-88f0-93f1b32a591b\") " pod="openshift-multus/multus-sbx29"
Sep 09 19:55:18.332039 ip-10-0-136-183 kubenswrapper[2574]: I0909 19:55:18.332021    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/9c19e4c1-5ffa-4fb8-88f0-93f1b32a591b-etc-kubernetes\") pod \"multus-sbx29\" (UID: \"9c19e4c1-5ffa-4fb8-88f0-93f1b32a591b\") " pod="openshift-multus/multus-sbx29"
Sep 09 19:55:18.332444 ip-10-0-136-183 kubenswrapper[2574]: I0909 19:55:18.332043    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/71b40e5b-d6b9-489f-bb2e-4b5e1e5915e5-device-dir\") pod \"aws-ebs-csi-driver-node-bvxbg\" (UID: \"71b40e5b-d6b9-489f-bb2e-4b5e1e5915e5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bvxbg"
Sep 09 19:55:18.332444 ip-10-0-136-183 kubenswrapper[2574]: I0909 19:55:18.332068    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vwrk4\" (UniqueName: \"kubernetes.io/projected/71b40e5b-d6b9-489f-bb2e-4b5e1e5915e5-kube-api-access-vwrk4\") pod \"aws-ebs-csi-driver-node-bvxbg\" (UID: \"71b40e5b-d6b9-489f-bb2e-4b5e1e5915e5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bvxbg"
Sep 09 19:55:18.332444 ip-10-0-136-183 kubenswrapper[2574]: I0909 19:55:18.332127    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/2eb82961-9118-4b08-9d97-8d5cbac8cac3-var-lib-openvswitch\") pod \"ovnkube-node-jcr9m\" (UID: \"2eb82961-9118-4b08-9d97-8d5cbac8cac3\") " pod="openshift-ovn-kubernetes/ovnkube-node-jcr9m"
Sep 09 19:55:18.332444 ip-10-0-136-183 kubenswrapper[2574]: I0909 19:55:18.332162    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/2eb82961-9118-4b08-9d97-8d5cbac8cac3-run-ovn\") pod \"ovnkube-node-jcr9m\" (UID: \"2eb82961-9118-4b08-9d97-8d5cbac8cac3\") " pod="openshift-ovn-kubernetes/ovnkube-node-jcr9m"
Sep 09 19:55:18.332444 ip-10-0-136-183 kubenswrapper[2574]: I0909 19:55:18.332188    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/c756f4c9-2360-4fff-a136-b6869b02977e-dbus\") pod \"global-pull-secret-syncer-gjc9x\" (UID: \"c756f4c9-2360-4fff-a136-b6869b02977e\") " pod="kube-system/global-pull-secret-syncer-gjc9x"
Sep 09 19:55:18.332444 ip-10-0-136-183 kubenswrapper[2574]: I0909 19:55:18.332225    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zqd55\" (UniqueName: \"kubernetes.io/projected/54458a6d-921b-4637-addc-7c4ee839b592-kube-api-access-zqd55\") pod \"network-check-target-4czxv\" (UID: \"54458a6d-921b-4637-addc-7c4ee839b592\") " pod="openshift-network-diagnostics/network-check-target-4czxv"
Sep 09 19:55:18.332444 ip-10-0-136-183 kubenswrapper[2574]: I0909 19:55:18.332294    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/9c19e4c1-5ffa-4fb8-88f0-93f1b32a591b-multus-conf-dir\") pod \"multus-sbx29\" (UID: \"9c19e4c1-5ffa-4fb8-88f0-93f1b32a591b\") " pod="openshift-multus/multus-sbx29"
Sep 09 19:55:18.332444 ip-10-0-136-183 kubenswrapper[2574]: I0909 19:55:18.332318    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/9c19e4c1-5ffa-4fb8-88f0-93f1b32a591b-host-run-multus-certs\") pod \"multus-sbx29\" (UID: \"9c19e4c1-5ffa-4fb8-88f0-93f1b32a591b\") " pod="openshift-multus/multus-sbx29"
Sep 09 19:55:18.332444 ip-10-0-136-183 kubenswrapper[2574]: I0909 19:55:18.332340    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/b06375df-d12d-40a3-a5a7-18fa1420f388-sys\") pod \"tuned-svtdz\" (UID: \"b06375df-d12d-40a3-a5a7-18fa1420f388\") " pod="openshift-cluster-node-tuning-operator/tuned-svtdz"
Sep 09 19:55:18.332444 ip-10-0-136-183 kubenswrapper[2574]: I0909 19:55:18.332408    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/2eb82961-9118-4b08-9d97-8d5cbac8cac3-host-kubelet\") pod \"ovnkube-node-jcr9m\" (UID: \"2eb82961-9118-4b08-9d97-8d5cbac8cac3\") " pod="openshift-ovn-kubernetes/ovnkube-node-jcr9m"
Sep 09 19:55:18.332986 ip-10-0-136-183 kubenswrapper[2574]: I0909 19:55:18.332455    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/2eb82961-9118-4b08-9d97-8d5cbac8cac3-host-run-netns\") pod \"ovnkube-node-jcr9m\" (UID: \"2eb82961-9118-4b08-9d97-8d5cbac8cac3\") " pod="openshift-ovn-kubernetes/ovnkube-node-jcr9m"
Sep 09 19:55:18.332986 ip-10-0-136-183 kubenswrapper[2574]: I0909 19:55:18.332522    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/2eb82961-9118-4b08-9d97-8d5cbac8cac3-run-systemd\") pod \"ovnkube-node-jcr9m\" (UID: \"2eb82961-9118-4b08-9d97-8d5cbac8cac3\") " pod="openshift-ovn-kubernetes/ovnkube-node-jcr9m"
Sep 09 19:55:18.332986 ip-10-0-136-183 kubenswrapper[2574]: I0909 19:55:18.332555    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/2eb82961-9118-4b08-9d97-8d5cbac8cac3-run-openvswitch\") pod \"ovnkube-node-jcr9m\" (UID: \"2eb82961-9118-4b08-9d97-8d5cbac8cac3\") " pod="openshift-ovn-kubernetes/ovnkube-node-jcr9m"
Sep 09 19:55:18.332986 ip-10-0-136-183 kubenswrapper[2574]: I0909 19:55:18.332620    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/2eb82961-9118-4b08-9d97-8d5cbac8cac3-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-jcr9m\" (UID: \"2eb82961-9118-4b08-9d97-8d5cbac8cac3\") " pod="openshift-ovn-kubernetes/ovnkube-node-jcr9m"
Sep 09 19:55:18.332986 ip-10-0-136-183 kubenswrapper[2574]: I0909 19:55:18.332644    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/2eb82961-9118-4b08-9d97-8d5cbac8cac3-ovnkube-config\") pod \"ovnkube-node-jcr9m\" (UID: \"2eb82961-9118-4b08-9d97-8d5cbac8cac3\") " pod="openshift-ovn-kubernetes/ovnkube-node-jcr9m"
Sep 09 19:55:18.332986 ip-10-0-136-183 kubenswrapper[2574]: I0909 19:55:18.332709    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/c756f4c9-2360-4fff-a136-b6869b02977e-original-pull-secret\") pod \"global-pull-secret-syncer-gjc9x\" (UID: \"c756f4c9-2360-4fff-a136-b6869b02977e\") " pod="kube-system/global-pull-secret-syncer-gjc9x"
Sep 09 19:55:18.332986 ip-10-0-136-183 kubenswrapper[2574]: I0909 19:55:18.332769    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/71b40e5b-d6b9-489f-bb2e-4b5e1e5915e5-socket-dir\") pod \"aws-ebs-csi-driver-node-bvxbg\" (UID: \"71b40e5b-d6b9-489f-bb2e-4b5e1e5915e5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bvxbg"
Sep 09 19:55:18.332986 ip-10-0-136-183 kubenswrapper[2574]: I0909 19:55:18.332792    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/c756f4c9-2360-4fff-a136-b6869b02977e-kubelet-config\") pod \"global-pull-secret-syncer-gjc9x\" (UID: \"c756f4c9-2360-4fff-a136-b6869b02977e\") " pod="kube-system/global-pull-secret-syncer-gjc9x"
Sep 09 19:55:18.332986 ip-10-0-136-183 kubenswrapper[2574]: I0909 19:55:18.332831    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/bed9028e-2a22-4c8f-a637-bb5ad9204adc-system-cni-dir\") pod \"multus-additional-cni-plugins-gzqh4\" (UID: \"bed9028e-2a22-4c8f-a637-bb5ad9204adc\") " pod="openshift-multus/multus-additional-cni-plugins-gzqh4"
Sep 09 19:55:18.332986 ip-10-0-136-183 kubenswrapper[2574]: I0909 19:55:18.332855    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/bed9028e-2a22-4c8f-a637-bb5ad9204adc-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-gzqh4\" (UID: \"bed9028e-2a22-4c8f-a637-bb5ad9204adc\") " pod="openshift-multus/multus-additional-cni-plugins-gzqh4"
Sep 09 19:55:18.332986 ip-10-0-136-183 kubenswrapper[2574]: I0909 19:55:18.332879    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/71b40e5b-d6b9-489f-bb2e-4b5e1e5915e5-kubelet-dir\") pod \"aws-ebs-csi-driver-node-bvxbg\" (UID: \"71b40e5b-d6b9-489f-bb2e-4b5e1e5915e5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bvxbg"
Sep 09 19:55:18.332986 ip-10-0-136-183 kubenswrapper[2574]: I0909 19:55:18.332917    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/496a3333-1b9c-4eb2-8896-b6586eaf6877-hosts-file\") pod \"node-resolver-q8krg\" (UID: \"496a3333-1b9c-4eb2-8896-b6586eaf6877\") " pod="openshift-dns/node-resolver-q8krg"
Sep 09 19:55:18.332986 ip-10-0-136-183 kubenswrapper[2574]: I0909 19:55:18.332976    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/bed9028e-2a22-4c8f-a637-bb5ad9204adc-cni-binary-copy\") pod \"multus-additional-cni-plugins-gzqh4\" (UID: \"bed9028e-2a22-4c8f-a637-bb5ad9204adc\") " pod="openshift-multus/multus-additional-cni-plugins-gzqh4"
Sep 09 19:55:18.333574 ip-10-0-136-183 kubenswrapper[2574]: I0909 19:55:18.333009    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/9c19e4c1-5ffa-4fb8-88f0-93f1b32a591b-multus-cni-dir\") pod \"multus-sbx29\" (UID: \"9c19e4c1-5ffa-4fb8-88f0-93f1b32a591b\") " pod="openshift-multus/multus-sbx29"
Sep 09 19:55:18.333574 ip-10-0-136-183 kubenswrapper[2574]: I0909 19:55:18.333032    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/1f0e15c0-0c09-4135-baf4-767e800917d5-host-slash\") pod \"iptables-alerter-gvsgq\" (UID: \"1f0e15c0-0c09-4135-baf4-767e800917d5\") " pod="openshift-network-operator/iptables-alerter-gvsgq"
Sep 09 19:55:18.333574 ip-10-0-136-183 kubenswrapper[2574]: I0909 19:55:18.333059    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/39099fe9-527d-4802-b40b-1bf535221165-serviceca\") pod \"node-ca-mgg8m\" (UID: \"39099fe9-527d-4802-b40b-1bf535221165\") " pod="openshift-image-registry/node-ca-mgg8m"
Sep 09 19:55:18.333574 ip-10-0-136-183 kubenswrapper[2574]: I0909 19:55:18.333081    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/2eb82961-9118-4b08-9d97-8d5cbac8cac3-systemd-units\") pod \"ovnkube-node-jcr9m\" (UID: \"2eb82961-9118-4b08-9d97-8d5cbac8cac3\") " pod="openshift-ovn-kubernetes/ovnkube-node-jcr9m"
Sep 09 19:55:18.333574 ip-10-0-136-183 kubenswrapper[2574]: I0909 19:55:18.333109    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/9c19e4c1-5ffa-4fb8-88f0-93f1b32a591b-host-var-lib-cni-bin\") pod \"multus-sbx29\" (UID: \"9c19e4c1-5ffa-4fb8-88f0-93f1b32a591b\") " pod="openshift-multus/multus-sbx29"
Sep 09 19:55:18.333574 ip-10-0-136-183 kubenswrapper[2574]: I0909 19:55:18.333134    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/b06375df-d12d-40a3-a5a7-18fa1420f388-etc-tuned\") pod \"tuned-svtdz\" (UID: \"b06375df-d12d-40a3-a5a7-18fa1420f388\") " pod="openshift-cluster-node-tuning-operator/tuned-svtdz"
Sep 09 19:55:18.333574 ip-10-0-136-183 kubenswrapper[2574]: I0909 19:55:18.333183    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/2eb82961-9118-4b08-9d97-8d5cbac8cac3-host-cni-bin\") pod \"ovnkube-node-jcr9m\" (UID: \"2eb82961-9118-4b08-9d97-8d5cbac8cac3\") " pod="openshift-ovn-kubernetes/ovnkube-node-jcr9m"
Sep 09 19:55:18.333574 ip-10-0-136-183 kubenswrapper[2574]: I0909 19:55:18.333206    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/bed9028e-2a22-4c8f-a637-bb5ad9204adc-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-gzqh4\" (UID: \"bed9028e-2a22-4c8f-a637-bb5ad9204adc\") " pod="openshift-multus/multus-additional-cni-plugins-gzqh4"
Sep 09 19:55:18.333574 ip-10-0-136-183 kubenswrapper[2574]: I0909 19:55:18.333239    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tz7jx\" (UniqueName: \"kubernetes.io/projected/bed9028e-2a22-4c8f-a637-bb5ad9204adc-kube-api-access-tz7jx\") pod \"multus-additional-cni-plugins-gzqh4\" (UID: \"bed9028e-2a22-4c8f-a637-bb5ad9204adc\") " pod="openshift-multus/multus-additional-cni-plugins-gzqh4"
Sep 09 19:55:18.333574 ip-10-0-136-183 kubenswrapper[2574]: I0909 19:55:18.333263    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/9c19e4c1-5ffa-4fb8-88f0-93f1b32a591b-cni-binary-copy\") pod \"multus-sbx29\" (UID: \"9c19e4c1-5ffa-4fb8-88f0-93f1b32a591b\") " pod="openshift-multus/multus-sbx29"
Sep 09 19:55:18.333574 ip-10-0-136-183 kubenswrapper[2574]: I0909 19:55:18.333371    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tf28w\" (UniqueName: \"kubernetes.io/projected/1f0e15c0-0c09-4135-baf4-767e800917d5-kube-api-access-tf28w\") pod \"iptables-alerter-gvsgq\" (UID: \"1f0e15c0-0c09-4135-baf4-767e800917d5\") " pod="openshift-network-operator/iptables-alerter-gvsgq"
Sep 09 19:55:18.333574 ip-10-0-136-183 kubenswrapper[2574]: I0909 19:55:18.333410    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/b06375df-d12d-40a3-a5a7-18fa1420f388-etc-sysctl-d\") pod \"tuned-svtdz\" (UID: \"b06375df-d12d-40a3-a5a7-18fa1420f388\") " pod="openshift-cluster-node-tuning-operator/tuned-svtdz"
Sep 09 19:55:18.333574 ip-10-0-136-183 kubenswrapper[2574]: I0909 19:55:18.333446    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cqtqf\" (UniqueName: \"kubernetes.io/projected/2eb82961-9118-4b08-9d97-8d5cbac8cac3-kube-api-access-cqtqf\") pod \"ovnkube-node-jcr9m\" (UID: \"2eb82961-9118-4b08-9d97-8d5cbac8cac3\") " pod="openshift-ovn-kubernetes/ovnkube-node-jcr9m"
Sep 09 19:55:18.333574 ip-10-0-136-183 kubenswrapper[2574]: I0909 19:55:18.333479    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/9c19e4c1-5ffa-4fb8-88f0-93f1b32a591b-hostroot\") pod \"multus-sbx29\" (UID: \"9c19e4c1-5ffa-4fb8-88f0-93f1b32a591b\") " pod="openshift-multus/multus-sbx29"
Sep 09 19:55:18.333574 ip-10-0-136-183 kubenswrapper[2574]: I0909 19:55:18.333540    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sqkzq\" (UniqueName: \"kubernetes.io/projected/9c19e4c1-5ffa-4fb8-88f0-93f1b32a591b-kube-api-access-sqkzq\") pod \"multus-sbx29\" (UID: \"9c19e4c1-5ffa-4fb8-88f0-93f1b32a591b\") " pod="openshift-multus/multus-sbx29"
Sep 09 19:55:18.333574 ip-10-0-136-183 kubenswrapper[2574]: I0909 19:55:18.333563    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/b06375df-d12d-40a3-a5a7-18fa1420f388-var-lib-kubelet\") pod \"tuned-svtdz\" (UID: \"b06375df-d12d-40a3-a5a7-18fa1420f388\") " pod="openshift-cluster-node-tuning-operator/tuned-svtdz"
Sep 09 19:55:18.334735 ip-10-0-136-183 kubenswrapper[2574]: I0909 19:55:18.333586    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/1f0e15c0-0c09-4135-baf4-767e800917d5-iptables-alerter-script\") pod \"iptables-alerter-gvsgq\" (UID: \"1f0e15c0-0c09-4135-baf4-767e800917d5\") " pod="openshift-network-operator/iptables-alerter-gvsgq"
Sep 09 19:55:18.334735 ip-10-0-136-183 kubenswrapper[2574]: I0909 19:55:18.333611    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/b06375df-d12d-40a3-a5a7-18fa1420f388-run\") pod \"tuned-svtdz\" (UID: \"b06375df-d12d-40a3-a5a7-18fa1420f388\") " pod="openshift-cluster-node-tuning-operator/tuned-svtdz"
Sep 09 19:55:18.334735 ip-10-0-136-183 kubenswrapper[2574]: I0909 19:55:18.333632    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/b06375df-d12d-40a3-a5a7-18fa1420f388-etc-systemd\") pod \"tuned-svtdz\" (UID: \"b06375df-d12d-40a3-a5a7-18fa1420f388\") " pod="openshift-cluster-node-tuning-operator/tuned-svtdz"
Sep 09 19:55:18.334735 ip-10-0-136-183 kubenswrapper[2574]: I0909 19:55:18.333656    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/2eb82961-9118-4b08-9d97-8d5cbac8cac3-node-log\") pod \"ovnkube-node-jcr9m\" (UID: \"2eb82961-9118-4b08-9d97-8d5cbac8cac3\") " pod="openshift-ovn-kubernetes/ovnkube-node-jcr9m"
Sep 09 19:55:18.334735 ip-10-0-136-183 kubenswrapper[2574]: I0909 19:55:18.333699    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/47c2b5b9-456a-4528-8d83-7ce3269cc16e-metrics-certs\") pod \"network-metrics-daemon-mdrck\" (UID: \"47c2b5b9-456a-4528-8d83-7ce3269cc16e\") " pod="openshift-multus/network-metrics-daemon-mdrck"
Sep 09 19:55:18.334735 ip-10-0-136-183 kubenswrapper[2574]: I0909 19:55:18.333742    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/71b40e5b-d6b9-489f-bb2e-4b5e1e5915e5-sys-fs\") pod \"aws-ebs-csi-driver-node-bvxbg\" (UID: \"71b40e5b-d6b9-489f-bb2e-4b5e1e5915e5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bvxbg"
Sep 09 19:55:18.334735 ip-10-0-136-183 kubenswrapper[2574]: I0909 19:55:18.333826    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/b06375df-d12d-40a3-a5a7-18fa1420f388-etc-kubernetes\") pod \"tuned-svtdz\" (UID: \"b06375df-d12d-40a3-a5a7-18fa1420f388\") " pod="openshift-cluster-node-tuning-operator/tuned-svtdz"
Sep 09 19:55:18.334735 ip-10-0-136-183 kubenswrapper[2574]: I0909 19:55:18.333857    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/39099fe9-527d-4802-b40b-1bf535221165-host\") pod \"node-ca-mgg8m\" (UID: \"39099fe9-527d-4802-b40b-1bf535221165\") " pod="openshift-image-registry/node-ca-mgg8m"
Sep 09 19:55:18.334735 ip-10-0-136-183 kubenswrapper[2574]: I0909 19:55:18.333880    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nbnrs\" (UniqueName: \"kubernetes.io/projected/47c2b5b9-456a-4528-8d83-7ce3269cc16e-kube-api-access-nbnrs\") pod \"network-metrics-daemon-mdrck\" (UID: \"47c2b5b9-456a-4528-8d83-7ce3269cc16e\") " pod="openshift-multus/network-metrics-daemon-mdrck"
Sep 09 19:55:18.334735 ip-10-0-136-183 kubenswrapper[2574]: I0909 19:55:18.333942    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/2eb82961-9118-4b08-9d97-8d5cbac8cac3-host-slash\") pod \"ovnkube-node-jcr9m\" (UID: \"2eb82961-9118-4b08-9d97-8d5cbac8cac3\") " pod="openshift-ovn-kubernetes/ovnkube-node-jcr9m"
Sep 09 19:55:18.334735 ip-10-0-136-183 kubenswrapper[2574]: I0909 19:55:18.334014    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/bed9028e-2a22-4c8f-a637-bb5ad9204adc-cnibin\") pod \"multus-additional-cni-plugins-gzqh4\" (UID: \"bed9028e-2a22-4c8f-a637-bb5ad9204adc\") " pod="openshift-multus/multus-additional-cni-plugins-gzqh4"
Sep 09 19:55:18.334735 ip-10-0-136-183 kubenswrapper[2574]: I0909 19:55:18.334036    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/71b40e5b-d6b9-489f-bb2e-4b5e1e5915e5-registration-dir\") pod \"aws-ebs-csi-driver-node-bvxbg\" (UID: \"71b40e5b-d6b9-489f-bb2e-4b5e1e5915e5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bvxbg"
Sep 09 19:55:18.334735 ip-10-0-136-183 kubenswrapper[2574]: I0909 19:55:18.334062    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/b06375df-d12d-40a3-a5a7-18fa1420f388-lib-modules\") pod \"tuned-svtdz\" (UID: \"b06375df-d12d-40a3-a5a7-18fa1420f388\") " pod="openshift-cluster-node-tuning-operator/tuned-svtdz"
Sep 09 19:55:18.334735 ip-10-0-136-183 kubenswrapper[2574]: I0909 19:55:18.334083    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/2eb82961-9118-4b08-9d97-8d5cbac8cac3-etc-openvswitch\") pod \"ovnkube-node-jcr9m\" (UID: \"2eb82961-9118-4b08-9d97-8d5cbac8cac3\") " pod="openshift-ovn-kubernetes/ovnkube-node-jcr9m"
Sep 09 19:55:18.334735 ip-10-0-136-183 kubenswrapper[2574]: I0909 19:55:18.334142    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/2eb82961-9118-4b08-9d97-8d5cbac8cac3-host-cni-netd\") pod \"ovnkube-node-jcr9m\" (UID: \"2eb82961-9118-4b08-9d97-8d5cbac8cac3\") " pod="openshift-ovn-kubernetes/ovnkube-node-jcr9m"
Sep 09 19:55:18.334735 ip-10-0-136-183 kubenswrapper[2574]: I0909 19:55:18.334165    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/9c19e4c1-5ffa-4fb8-88f0-93f1b32a591b-multus-socket-dir-parent\") pod \"multus-sbx29\" (UID: \"9c19e4c1-5ffa-4fb8-88f0-93f1b32a591b\") " pod="openshift-multus/multus-sbx29"
Sep 09 19:55:18.337899 ip-10-0-136-183 kubenswrapper[2574]: I0909 19:55:18.334233    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/9c19e4c1-5ffa-4fb8-88f0-93f1b32a591b-host-run-netns\") pod \"multus-sbx29\" (UID: \"9c19e4c1-5ffa-4fb8-88f0-93f1b32a591b\") " pod="openshift-multus/multus-sbx29"
Sep 09 19:55:18.337899 ip-10-0-136-183 kubenswrapper[2574]: I0909 19:55:18.334293    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7kk79\" (UniqueName: \"kubernetes.io/projected/496a3333-1b9c-4eb2-8896-b6586eaf6877-kube-api-access-7kk79\") pod \"node-resolver-q8krg\" (UID: \"496a3333-1b9c-4eb2-8896-b6586eaf6877\") " pod="openshift-dns/node-resolver-q8krg"
Sep 09 19:55:18.337899 ip-10-0-136-183 kubenswrapper[2574]: I0909 19:55:18.334352    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/b06375df-d12d-40a3-a5a7-18fa1420f388-etc-modprobe-d\") pod \"tuned-svtdz\" (UID: \"b06375df-d12d-40a3-a5a7-18fa1420f388\") " pod="openshift-cluster-node-tuning-operator/tuned-svtdz"
Sep 09 19:55:18.337899 ip-10-0-136-183 kubenswrapper[2574]: I0909 19:55:18.334387    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/b06375df-d12d-40a3-a5a7-18fa1420f388-tmp\") pod \"tuned-svtdz\" (UID: \"b06375df-d12d-40a3-a5a7-18fa1420f388\") " pod="openshift-cluster-node-tuning-operator/tuned-svtdz"
Sep 09 19:55:18.337899 ip-10-0-136-183 kubenswrapper[2574]: I0909 19:55:18.334416    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-djvx9\" (UniqueName: \"kubernetes.io/projected/b06375df-d12d-40a3-a5a7-18fa1420f388-kube-api-access-djvx9\") pod \"tuned-svtdz\" (UID: \"b06375df-d12d-40a3-a5a7-18fa1420f388\") " pod="openshift-cluster-node-tuning-operator/tuned-svtdz"
Sep 09 19:55:18.337899 ip-10-0-136-183 kubenswrapper[2574]: I0909 19:55:18.334443    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/2eb82961-9118-4b08-9d97-8d5cbac8cac3-env-overrides\") pod \"ovnkube-node-jcr9m\" (UID: \"2eb82961-9118-4b08-9d97-8d5cbac8cac3\") " pod="openshift-ovn-kubernetes/ovnkube-node-jcr9m"
Sep 09 19:55:18.337899 ip-10-0-136-183 kubenswrapper[2574]: I0909 19:55:18.334486    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/496a3333-1b9c-4eb2-8896-b6586eaf6877-tmp-dir\") pod \"node-resolver-q8krg\" (UID: \"496a3333-1b9c-4eb2-8896-b6586eaf6877\") " pod="openshift-dns/node-resolver-q8krg"
Sep 09 19:55:18.337899 ip-10-0-136-183 kubenswrapper[2574]: I0909 19:55:18.334496    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-n4hvk\""
Sep 09 19:55:18.337899 ip-10-0-136-183 kubenswrapper[2574]: I0909 19:55:18.334489    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Sep 09 19:55:18.337899 ip-10-0-136-183 kubenswrapper[2574]: I0909 19:55:18.334551    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Sep 09 19:55:18.337899 ip-10-0-136-183 kubenswrapper[2574]: I0909 19:55:18.336893    2574 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Sep 09 19:55:18.353867 ip-10-0-136-183 kubenswrapper[2574]: I0909 19:55:18.353845    2574 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-hpgld"
Sep 09 19:55:18.361905 ip-10-0-136-183 kubenswrapper[2574]: I0909 19:55:18.361883    2574 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-hpgld"
Sep 09 19:55:18.418253 ip-10-0-136-183 kubenswrapper[2574]: I0909 19:55:18.418226    2574 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Sep 09 19:55:18.435560 ip-10-0-136-183 kubenswrapper[2574]: I0909 19:55:18.435525    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/b06375df-d12d-40a3-a5a7-18fa1420f388-run\") pod \"tuned-svtdz\" (UID: \"b06375df-d12d-40a3-a5a7-18fa1420f388\") " pod="openshift-cluster-node-tuning-operator/tuned-svtdz"
Sep 09 19:55:18.435560 ip-10-0-136-183 kubenswrapper[2574]: I0909 19:55:18.435559    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/b06375df-d12d-40a3-a5a7-18fa1420f388-etc-systemd\") pod \"tuned-svtdz\" (UID: \"b06375df-d12d-40a3-a5a7-18fa1420f388\") " pod="openshift-cluster-node-tuning-operator/tuned-svtdz"
Sep 09 19:55:18.435750 ip-10-0-136-183 kubenswrapper[2574]: I0909 19:55:18.435581    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/2eb82961-9118-4b08-9d97-8d5cbac8cac3-node-log\") pod \"ovnkube-node-jcr9m\" (UID: \"2eb82961-9118-4b08-9d97-8d5cbac8cac3\") " pod="openshift-ovn-kubernetes/ovnkube-node-jcr9m"
Sep 09 19:55:18.435750 ip-10-0-136-183 kubenswrapper[2574]: I0909 19:55:18.435606    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/47c2b5b9-456a-4528-8d83-7ce3269cc16e-metrics-certs\") pod \"network-metrics-daemon-mdrck\" (UID: \"47c2b5b9-456a-4528-8d83-7ce3269cc16e\") " pod="openshift-multus/network-metrics-daemon-mdrck"
Sep 09 19:55:18.435750 ip-10-0-136-183 kubenswrapper[2574]: I0909 19:55:18.435632    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/71b40e5b-d6b9-489f-bb2e-4b5e1e5915e5-sys-fs\") pod \"aws-ebs-csi-driver-node-bvxbg\" (UID: \"71b40e5b-d6b9-489f-bb2e-4b5e1e5915e5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bvxbg"
Sep 09 19:55:18.435750 ip-10-0-136-183 kubenswrapper[2574]: I0909 19:55:18.435654    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/b06375df-d12d-40a3-a5a7-18fa1420f388-etc-kubernetes\") pod \"tuned-svtdz\" (UID: \"b06375df-d12d-40a3-a5a7-18fa1420f388\") " pod="openshift-cluster-node-tuning-operator/tuned-svtdz"
Sep 09 19:55:18.435750 ip-10-0-136-183 kubenswrapper[2574]: I0909 19:55:18.435677    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/39099fe9-527d-4802-b40b-1bf535221165-host\") pod \"node-ca-mgg8m\" (UID: \"39099fe9-527d-4802-b40b-1bf535221165\") " pod="openshift-image-registry/node-ca-mgg8m"
Sep 09 19:55:18.435750 ip-10-0-136-183 kubenswrapper[2574]: I0909 19:55:18.435700    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nbnrs\" (UniqueName: \"kubernetes.io/projected/47c2b5b9-456a-4528-8d83-7ce3269cc16e-kube-api-access-nbnrs\") pod \"network-metrics-daemon-mdrck\" (UID: \"47c2b5b9-456a-4528-8d83-7ce3269cc16e\") " pod="openshift-multus/network-metrics-daemon-mdrck"
Sep 09 19:55:18.435750 ip-10-0-136-183 kubenswrapper[2574]: I0909 19:55:18.435750    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/2eb82961-9118-4b08-9d97-8d5cbac8cac3-host-slash\") pod \"ovnkube-node-jcr9m\" (UID: \"2eb82961-9118-4b08-9d97-8d5cbac8cac3\") " pod="openshift-ovn-kubernetes/ovnkube-node-jcr9m"
Sep 09 19:55:18.436108 ip-10-0-136-183 kubenswrapper[2574]: I0909 19:55:18.435776    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/bed9028e-2a22-4c8f-a637-bb5ad9204adc-cnibin\") pod \"multus-additional-cni-plugins-gzqh4\" (UID: \"bed9028e-2a22-4c8f-a637-bb5ad9204adc\") " pod="openshift-multus/multus-additional-cni-plugins-gzqh4"
Sep 09 19:55:18.436108 ip-10-0-136-183 kubenswrapper[2574]: I0909 19:55:18.435809    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/71b40e5b-d6b9-489f-bb2e-4b5e1e5915e5-registration-dir\") pod \"aws-ebs-csi-driver-node-bvxbg\" (UID: \"71b40e5b-d6b9-489f-bb2e-4b5e1e5915e5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bvxbg"
Sep 09 19:55:18.436108 ip-10-0-136-183 kubenswrapper[2574]: I0909 19:55:18.435833    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/b06375df-d12d-40a3-a5a7-18fa1420f388-lib-modules\") pod \"tuned-svtdz\" (UID: \"b06375df-d12d-40a3-a5a7-18fa1420f388\") " pod="openshift-cluster-node-tuning-operator/tuned-svtdz"
Sep 09 19:55:18.436108 ip-10-0-136-183 kubenswrapper[2574]: I0909 19:55:18.435858    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/2eb82961-9118-4b08-9d97-8d5cbac8cac3-etc-openvswitch\") pod \"ovnkube-node-jcr9m\" (UID: \"2eb82961-9118-4b08-9d97-8d5cbac8cac3\") " pod="openshift-ovn-kubernetes/ovnkube-node-jcr9m"
Sep 09 19:55:18.436108 ip-10-0-136-183 kubenswrapper[2574]: I0909 19:55:18.435881    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/2eb82961-9118-4b08-9d97-8d5cbac8cac3-host-cni-netd\") pod \"ovnkube-node-jcr9m\" (UID: \"2eb82961-9118-4b08-9d97-8d5cbac8cac3\") " pod="openshift-ovn-kubernetes/ovnkube-node-jcr9m"
Sep 09 19:55:18.436108 ip-10-0-136-183 kubenswrapper[2574]: I0909 19:55:18.435895    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/2eb82961-9118-4b08-9d97-8d5cbac8cac3-host-slash\") pod \"ovnkube-node-jcr9m\" (UID: \"2eb82961-9118-4b08-9d97-8d5cbac8cac3\") " pod="openshift-ovn-kubernetes/ovnkube-node-jcr9m"
Sep 09 19:55:18.436108 ip-10-0-136-183 kubenswrapper[2574]: I0909 19:55:18.435905    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/9c19e4c1-5ffa-4fb8-88f0-93f1b32a591b-multus-socket-dir-parent\") pod \"multus-sbx29\" (UID: \"9c19e4c1-5ffa-4fb8-88f0-93f1b32a591b\") " pod="openshift-multus/multus-sbx29"
Sep 09 19:55:18.436108 ip-10-0-136-183 kubenswrapper[2574]: I0909 19:55:18.435947    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/9c19e4c1-5ffa-4fb8-88f0-93f1b32a591b-host-run-netns\") pod \"multus-sbx29\" (UID: \"9c19e4c1-5ffa-4fb8-88f0-93f1b32a591b\") " pod="openshift-multus/multus-sbx29"
Sep 09 19:55:18.436108 ip-10-0-136-183 kubenswrapper[2574]: I0909 19:55:18.435971    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/bed9028e-2a22-4c8f-a637-bb5ad9204adc-cnibin\") pod \"multus-additional-cni-plugins-gzqh4\" (UID: \"bed9028e-2a22-4c8f-a637-bb5ad9204adc\") " pod="openshift-multus/multus-additional-cni-plugins-gzqh4"
Sep 09 19:55:18.436108 ip-10-0-136-183 kubenswrapper[2574]: I0909 19:55:18.435971    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7kk79\" (UniqueName: \"kubernetes.io/projected/496a3333-1b9c-4eb2-8896-b6586eaf6877-kube-api-access-7kk79\") pod \"node-resolver-q8krg\" (UID: \"496a3333-1b9c-4eb2-8896-b6586eaf6877\") " pod="openshift-dns/node-resolver-q8krg"
Sep 09 19:55:18.436108 ip-10-0-136-183 kubenswrapper[2574]: I0909 19:55:18.435986    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/39099fe9-527d-4802-b40b-1bf535221165-host\") pod \"node-ca-mgg8m\" (UID: \"39099fe9-527d-4802-b40b-1bf535221165\") " pod="openshift-image-registry/node-ca-mgg8m"
Sep 09 19:55:18.436108 ip-10-0-136-183 kubenswrapper[2574]: I0909 19:55:18.436017    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/b06375df-d12d-40a3-a5a7-18fa1420f388-etc-modprobe-d\") pod \"tuned-svtdz\" (UID: \"b06375df-d12d-40a3-a5a7-18fa1420f388\") " pod="openshift-cluster-node-tuning-operator/tuned-svtdz"
Sep 09 19:55:18.436108 ip-10-0-136-183 kubenswrapper[2574]: I0909 19:55:18.436047    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/9c19e4c1-5ffa-4fb8-88f0-93f1b32a591b-multus-socket-dir-parent\") pod \"multus-sbx29\" (UID: \"9c19e4c1-5ffa-4fb8-88f0-93f1b32a591b\") " pod="openshift-multus/multus-sbx29"
Sep 09 19:55:18.436108 ip-10-0-136-183 kubenswrapper[2574]: I0909 19:55:18.436090    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/2eb82961-9118-4b08-9d97-8d5cbac8cac3-etc-openvswitch\") pod \"ovnkube-node-jcr9m\" (UID: \"2eb82961-9118-4b08-9d97-8d5cbac8cac3\") " pod="openshift-ovn-kubernetes/ovnkube-node-jcr9m"
Sep 09 19:55:18.436108 ip-10-0-136-183 kubenswrapper[2574]: I0909 19:55:18.436104    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/b06375df-d12d-40a3-a5a7-18fa1420f388-etc-systemd\") pod \"tuned-svtdz\" (UID: \"b06375df-d12d-40a3-a5a7-18fa1420f388\") " pod="openshift-cluster-node-tuning-operator/tuned-svtdz"
Sep 09 19:55:18.436708 ip-10-0-136-183 kubenswrapper[2574]: I0909 19:55:18.436125    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/71b40e5b-d6b9-489f-bb2e-4b5e1e5915e5-registration-dir\") pod \"aws-ebs-csi-driver-node-bvxbg\" (UID: \"71b40e5b-d6b9-489f-bb2e-4b5e1e5915e5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bvxbg"
Sep 09 19:55:18.436708 ip-10-0-136-183 kubenswrapper[2574]: I0909 19:55:18.436135    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/2eb82961-9118-4b08-9d97-8d5cbac8cac3-host-cni-netd\") pod \"ovnkube-node-jcr9m\" (UID: \"2eb82961-9118-4b08-9d97-8d5cbac8cac3\") " pod="openshift-ovn-kubernetes/ovnkube-node-jcr9m"
Sep 09 19:55:18.436708 ip-10-0-136-183 kubenswrapper[2574]: I0909 19:55:18.435948    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/b06375df-d12d-40a3-a5a7-18fa1420f388-etc-kubernetes\") pod \"tuned-svtdz\" (UID: \"b06375df-d12d-40a3-a5a7-18fa1420f388\") " pod="openshift-cluster-node-tuning-operator/tuned-svtdz"
Sep 09 19:55:18.436708 ip-10-0-136-183 kubenswrapper[2574]: I0909 19:55:18.436157    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/b06375df-d12d-40a3-a5a7-18fa1420f388-run\") pod \"tuned-svtdz\" (UID: \"b06375df-d12d-40a3-a5a7-18fa1420f388\") " pod="openshift-cluster-node-tuning-operator/tuned-svtdz"
Sep 09 19:55:18.436708 ip-10-0-136-183 kubenswrapper[2574]: I0909 19:55:18.436172    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/9c19e4c1-5ffa-4fb8-88f0-93f1b32a591b-host-run-netns\") pod \"multus-sbx29\" (UID: \"9c19e4c1-5ffa-4fb8-88f0-93f1b32a591b\") " pod="openshift-multus/multus-sbx29"
Sep 09 19:55:18.436708 ip-10-0-136-183 kubenswrapper[2574]: I0909 19:55:18.436187    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/2eb82961-9118-4b08-9d97-8d5cbac8cac3-node-log\") pod \"ovnkube-node-jcr9m\" (UID: \"2eb82961-9118-4b08-9d97-8d5cbac8cac3\") " pod="openshift-ovn-kubernetes/ovnkube-node-jcr9m"
Sep 09 19:55:18.436708 ip-10-0-136-183 kubenswrapper[2574]: E0909 19:55:18.436251    2574 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 09 19:55:18.436708 ip-10-0-136-183 kubenswrapper[2574]: I0909 19:55:18.436271    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/b06375df-d12d-40a3-a5a7-18fa1420f388-lib-modules\") pod \"tuned-svtdz\" (UID: \"b06375df-d12d-40a3-a5a7-18fa1420f388\") " pod="openshift-cluster-node-tuning-operator/tuned-svtdz"
Sep 09 19:55:18.436708 ip-10-0-136-183 kubenswrapper[2574]: I0909 19:55:18.435882    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/71b40e5b-d6b9-489f-bb2e-4b5e1e5915e5-sys-fs\") pod \"aws-ebs-csi-driver-node-bvxbg\" (UID: \"71b40e5b-d6b9-489f-bb2e-4b5e1e5915e5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bvxbg"
Sep 09 19:55:18.436708 ip-10-0-136-183 kubenswrapper[2574]: I0909 19:55:18.436383    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/b06375df-d12d-40a3-a5a7-18fa1420f388-etc-modprobe-d\") pod \"tuned-svtdz\" (UID: \"b06375df-d12d-40a3-a5a7-18fa1420f388\") " pod="openshift-cluster-node-tuning-operator/tuned-svtdz"
Sep 09 19:55:18.436708 ip-10-0-136-183 kubenswrapper[2574]: I0909 19:55:18.436474    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/b06375df-d12d-40a3-a5a7-18fa1420f388-tmp\") pod \"tuned-svtdz\" (UID: \"b06375df-d12d-40a3-a5a7-18fa1420f388\") " pod="openshift-cluster-node-tuning-operator/tuned-svtdz"
Sep 09 19:55:18.436708 ip-10-0-136-183 kubenswrapper[2574]: I0909 19:55:18.436506    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-djvx9\" (UniqueName: \"kubernetes.io/projected/b06375df-d12d-40a3-a5a7-18fa1420f388-kube-api-access-djvx9\") pod \"tuned-svtdz\" (UID: \"b06375df-d12d-40a3-a5a7-18fa1420f388\") " pod="openshift-cluster-node-tuning-operator/tuned-svtdz"
Sep 09 19:55:18.436708 ip-10-0-136-183 kubenswrapper[2574]: I0909 19:55:18.436533    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/2eb82961-9118-4b08-9d97-8d5cbac8cac3-env-overrides\") pod \"ovnkube-node-jcr9m\" (UID: \"2eb82961-9118-4b08-9d97-8d5cbac8cac3\") " pod="openshift-ovn-kubernetes/ovnkube-node-jcr9m"
Sep 09 19:55:18.436708 ip-10-0-136-183 kubenswrapper[2574]: I0909 19:55:18.436557    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/496a3333-1b9c-4eb2-8896-b6586eaf6877-tmp-dir\") pod \"node-resolver-q8krg\" (UID: \"496a3333-1b9c-4eb2-8896-b6586eaf6877\") " pod="openshift-dns/node-resolver-q8krg"
Sep 09 19:55:18.436708 ip-10-0-136-183 kubenswrapper[2574]: I0909 19:55:18.436580    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/9c19e4c1-5ffa-4fb8-88f0-93f1b32a591b-system-cni-dir\") pod \"multus-sbx29\" (UID: \"9c19e4c1-5ffa-4fb8-88f0-93f1b32a591b\") " pod="openshift-multus/multus-sbx29"
Sep 09 19:55:18.436708 ip-10-0-136-183 kubenswrapper[2574]: I0909 19:55:18.436612    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/b06375df-d12d-40a3-a5a7-18fa1420f388-etc-sysconfig\") pod \"tuned-svtdz\" (UID: \"b06375df-d12d-40a3-a5a7-18fa1420f388\") " pod="openshift-cluster-node-tuning-operator/tuned-svtdz"
Sep 09 19:55:18.436708 ip-10-0-136-183 kubenswrapper[2574]: I0909 19:55:18.436629    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/b06375df-d12d-40a3-a5a7-18fa1420f388-etc-sysctl-conf\") pod \"tuned-svtdz\" (UID: \"b06375df-d12d-40a3-a5a7-18fa1420f388\") " pod="openshift-cluster-node-tuning-operator/tuned-svtdz"
Sep 09 19:55:18.436708 ip-10-0-136-183 kubenswrapper[2574]: I0909 19:55:18.436645    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p6r7f\" (UniqueName: \"kubernetes.io/projected/39099fe9-527d-4802-b40b-1bf535221165-kube-api-access-p6r7f\") pod \"node-ca-mgg8m\" (UID: \"39099fe9-527d-4802-b40b-1bf535221165\") " pod="openshift-image-registry/node-ca-mgg8m"
Sep 09 19:55:18.437608 ip-10-0-136-183 kubenswrapper[2574]: I0909 19:55:18.436660    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/2eb82961-9118-4b08-9d97-8d5cbac8cac3-log-socket\") pod \"ovnkube-node-jcr9m\" (UID: \"2eb82961-9118-4b08-9d97-8d5cbac8cac3\") " pod="openshift-ovn-kubernetes/ovnkube-node-jcr9m"
Sep 09 19:55:18.437608 ip-10-0-136-183 kubenswrapper[2574]: I0909 19:55:18.436689    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/2eb82961-9118-4b08-9d97-8d5cbac8cac3-host-run-ovn-kubernetes\") pod \"ovnkube-node-jcr9m\" (UID: \"2eb82961-9118-4b08-9d97-8d5cbac8cac3\") " pod="openshift-ovn-kubernetes/ovnkube-node-jcr9m"
Sep 09 19:55:18.437608 ip-10-0-136-183 kubenswrapper[2574]: I0909 19:55:18.436714    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/2eb82961-9118-4b08-9d97-8d5cbac8cac3-ovn-node-metrics-cert\") pod \"ovnkube-node-jcr9m\" (UID: \"2eb82961-9118-4b08-9d97-8d5cbac8cac3\") " pod="openshift-ovn-kubernetes/ovnkube-node-jcr9m"
Sep 09 19:55:18.437608 ip-10-0-136-183 kubenswrapper[2574]: I0909 19:55:18.436732    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/2eb82961-9118-4b08-9d97-8d5cbac8cac3-ovnkube-script-lib\") pod \"ovnkube-node-jcr9m\" (UID: \"2eb82961-9118-4b08-9d97-8d5cbac8cac3\") " pod="openshift-ovn-kubernetes/ovnkube-node-jcr9m"
Sep 09 19:55:18.437608 ip-10-0-136-183 kubenswrapper[2574]: I0909 19:55:18.436747    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/bed9028e-2a22-4c8f-a637-bb5ad9204adc-tuning-conf-dir\") pod \"multus-additional-cni-plugins-gzqh4\" (UID: \"bed9028e-2a22-4c8f-a637-bb5ad9204adc\") " pod="openshift-multus/multus-additional-cni-plugins-gzqh4"
Sep 09 19:55:18.437608 ip-10-0-136-183 kubenswrapper[2574]: I0909 19:55:18.436770    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/71b40e5b-d6b9-489f-bb2e-4b5e1e5915e5-etc-selinux\") pod \"aws-ebs-csi-driver-node-bvxbg\" (UID: \"71b40e5b-d6b9-489f-bb2e-4b5e1e5915e5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bvxbg"
Sep 09 19:55:18.437608 ip-10-0-136-183 kubenswrapper[2574]: I0909 19:55:18.436793    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/b06375df-d12d-40a3-a5a7-18fa1420f388-host\") pod \"tuned-svtdz\" (UID: \"b06375df-d12d-40a3-a5a7-18fa1420f388\") " pod="openshift-cluster-node-tuning-operator/tuned-svtdz"
Sep 09 19:55:18.437608 ip-10-0-136-183 kubenswrapper[2574]: I0909 19:55:18.436836    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/9c19e4c1-5ffa-4fb8-88f0-93f1b32a591b-cnibin\") pod \"multus-sbx29\" (UID: \"9c19e4c1-5ffa-4fb8-88f0-93f1b32a591b\") " pod="openshift-multus/multus-sbx29"
Sep 09 19:55:18.437608 ip-10-0-136-183 kubenswrapper[2574]: I0909 19:55:18.436858    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/9c19e4c1-5ffa-4fb8-88f0-93f1b32a591b-host-run-k8s-cni-cncf-io\") pod \"multus-sbx29\" (UID: \"9c19e4c1-5ffa-4fb8-88f0-93f1b32a591b\") " pod="openshift-multus/multus-sbx29"
Sep 09 19:55:18.437608 ip-10-0-136-183 kubenswrapper[2574]: I0909 19:55:18.436881    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/9c19e4c1-5ffa-4fb8-88f0-93f1b32a591b-host-var-lib-cni-multus\") pod \"multus-sbx29\" (UID: \"9c19e4c1-5ffa-4fb8-88f0-93f1b32a591b\") " pod="openshift-multus/multus-sbx29"
Sep 09 19:55:18.437608 ip-10-0-136-183 kubenswrapper[2574]: I0909 19:55:18.436911    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/9c19e4c1-5ffa-4fb8-88f0-93f1b32a591b-host-var-lib-kubelet\") pod \"multus-sbx29\" (UID: \"9c19e4c1-5ffa-4fb8-88f0-93f1b32a591b\") " pod="openshift-multus/multus-sbx29"
Sep 09 19:55:18.437608 ip-10-0-136-183 kubenswrapper[2574]: I0909 19:55:18.436960    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/9c19e4c1-5ffa-4fb8-88f0-93f1b32a591b-host-var-lib-kubelet\") pod \"multus-sbx29\" (UID: \"9c19e4c1-5ffa-4fb8-88f0-93f1b32a591b\") " pod="openshift-multus/multus-sbx29"
Sep 09 19:55:18.437608 ip-10-0-136-183 kubenswrapper[2574]: I0909 19:55:18.437033    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/2eb82961-9118-4b08-9d97-8d5cbac8cac3-log-socket\") pod \"ovnkube-node-jcr9m\" (UID: \"2eb82961-9118-4b08-9d97-8d5cbac8cac3\") " pod="openshift-ovn-kubernetes/ovnkube-node-jcr9m"
Sep 09 19:55:18.437608 ip-10-0-136-183 kubenswrapper[2574]: I0909 19:55:18.437080    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/2eb82961-9118-4b08-9d97-8d5cbac8cac3-host-run-ovn-kubernetes\") pod \"ovnkube-node-jcr9m\" (UID: \"2eb82961-9118-4b08-9d97-8d5cbac8cac3\") " pod="openshift-ovn-kubernetes/ovnkube-node-jcr9m"
Sep 09 19:55:18.437608 ip-10-0-136-183 kubenswrapper[2574]: I0909 19:55:18.437435    2574 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Sep 09 19:55:18.438428 ip-10-0-136-183 kubenswrapper[2574]: I0909 19:55:18.437700    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/b06375df-d12d-40a3-a5a7-18fa1420f388-host\") pod \"tuned-svtdz\" (UID: \"b06375df-d12d-40a3-a5a7-18fa1420f388\") " pod="openshift-cluster-node-tuning-operator/tuned-svtdz"
Sep 09 19:55:18.438428 ip-10-0-136-183 kubenswrapper[2574]: I0909 19:55:18.437876    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/2eb82961-9118-4b08-9d97-8d5cbac8cac3-ovnkube-script-lib\") pod \"ovnkube-node-jcr9m\" (UID: \"2eb82961-9118-4b08-9d97-8d5cbac8cac3\") " pod="openshift-ovn-kubernetes/ovnkube-node-jcr9m"
Sep 09 19:55:18.438428 ip-10-0-136-183 kubenswrapper[2574]: I0909 19:55:18.437915    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/bed9028e-2a22-4c8f-a637-bb5ad9204adc-tuning-conf-dir\") pod \"multus-additional-cni-plugins-gzqh4\" (UID: \"bed9028e-2a22-4c8f-a637-bb5ad9204adc\") " pod="openshift-multus/multus-additional-cni-plugins-gzqh4"
Sep 09 19:55:18.438428 ip-10-0-136-183 kubenswrapper[2574]: I0909 19:55:18.437979    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/9c19e4c1-5ffa-4fb8-88f0-93f1b32a591b-cnibin\") pod \"multus-sbx29\" (UID: \"9c19e4c1-5ffa-4fb8-88f0-93f1b32a591b\") " pod="openshift-multus/multus-sbx29"
Sep 09 19:55:18.438428 ip-10-0-136-183 kubenswrapper[2574]: I0909 19:55:18.437984    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/71b40e5b-d6b9-489f-bb2e-4b5e1e5915e5-etc-selinux\") pod \"aws-ebs-csi-driver-node-bvxbg\" (UID: \"71b40e5b-d6b9-489f-bb2e-4b5e1e5915e5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bvxbg"
Sep 09 19:55:18.438428 ip-10-0-136-183 kubenswrapper[2574]: I0909 19:55:18.438006    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/496a3333-1b9c-4eb2-8896-b6586eaf6877-tmp-dir\") pod \"node-resolver-q8krg\" (UID: \"496a3333-1b9c-4eb2-8896-b6586eaf6877\") " pod="openshift-dns/node-resolver-q8krg"
Sep 09 19:55:18.438428 ip-10-0-136-183 kubenswrapper[2574]: I0909 19:55:18.438025    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/9c19e4c1-5ffa-4fb8-88f0-93f1b32a591b-host-run-k8s-cni-cncf-io\") pod \"multus-sbx29\" (UID: \"9c19e4c1-5ffa-4fb8-88f0-93f1b32a591b\") " pod="openshift-multus/multus-sbx29"
Sep 09 19:55:18.438428 ip-10-0-136-183 kubenswrapper[2574]: I0909 19:55:18.438039    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/b06375df-d12d-40a3-a5a7-18fa1420f388-etc-sysconfig\") pod \"tuned-svtdz\" (UID: \"b06375df-d12d-40a3-a5a7-18fa1420f388\") " pod="openshift-cluster-node-tuning-operator/tuned-svtdz"
Sep 09 19:55:18.438428 ip-10-0-136-183 kubenswrapper[2574]: I0909 19:55:18.438072    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/9c19e4c1-5ffa-4fb8-88f0-93f1b32a591b-host-var-lib-cni-multus\") pod \"multus-sbx29\" (UID: \"9c19e4c1-5ffa-4fb8-88f0-93f1b32a591b\") " pod="openshift-multus/multus-sbx29"
Sep 09 19:55:18.438428 ip-10-0-136-183 kubenswrapper[2574]: E0909 19:55:18.438083    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/47c2b5b9-456a-4528-8d83-7ce3269cc16e-metrics-certs podName:47c2b5b9-456a-4528-8d83-7ce3269cc16e nodeName:}" failed. No retries permitted until 2026-09-09 19:55:18.938063204 +0000 UTC m=+2.984316444 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/47c2b5b9-456a-4528-8d83-7ce3269cc16e-metrics-certs") pod "network-metrics-daemon-mdrck" (UID: "47c2b5b9-456a-4528-8d83-7ce3269cc16e") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 09 19:55:18.438428 ip-10-0-136-183 kubenswrapper[2574]: I0909 19:55:18.438095    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/9c19e4c1-5ffa-4fb8-88f0-93f1b32a591b-system-cni-dir\") pod \"multus-sbx29\" (UID: \"9c19e4c1-5ffa-4fb8-88f0-93f1b32a591b\") " pod="openshift-multus/multus-sbx29"
Sep 09 19:55:18.438428 ip-10-0-136-183 kubenswrapper[2574]: I0909 19:55:18.438215    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/b06375df-d12d-40a3-a5a7-18fa1420f388-etc-sysctl-conf\") pod \"tuned-svtdz\" (UID: \"b06375df-d12d-40a3-a5a7-18fa1420f388\") " pod="openshift-cluster-node-tuning-operator/tuned-svtdz"
Sep 09 19:55:18.438428 ip-10-0-136-183 kubenswrapper[2574]: I0909 19:55:18.438266    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/9c19e4c1-5ffa-4fb8-88f0-93f1b32a591b-multus-daemon-config\") pod \"multus-sbx29\" (UID: \"9c19e4c1-5ffa-4fb8-88f0-93f1b32a591b\") " pod="openshift-multus/multus-sbx29"
Sep 09 19:55:18.438428 ip-10-0-136-183 kubenswrapper[2574]: I0909 19:55:18.438296    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/9c19e4c1-5ffa-4fb8-88f0-93f1b32a591b-etc-kubernetes\") pod \"multus-sbx29\" (UID: \"9c19e4c1-5ffa-4fb8-88f0-93f1b32a591b\") " pod="openshift-multus/multus-sbx29"
Sep 09 19:55:18.438428 ip-10-0-136-183 kubenswrapper[2574]: I0909 19:55:18.438320    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/71b40e5b-d6b9-489f-bb2e-4b5e1e5915e5-device-dir\") pod \"aws-ebs-csi-driver-node-bvxbg\" (UID: \"71b40e5b-d6b9-489f-bb2e-4b5e1e5915e5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bvxbg"
Sep 09 19:55:18.438428 ip-10-0-136-183 kubenswrapper[2574]: I0909 19:55:18.438344    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vwrk4\" (UniqueName: \"kubernetes.io/projected/71b40e5b-d6b9-489f-bb2e-4b5e1e5915e5-kube-api-access-vwrk4\") pod \"aws-ebs-csi-driver-node-bvxbg\" (UID: \"71b40e5b-d6b9-489f-bb2e-4b5e1e5915e5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bvxbg"
Sep 09 19:55:18.438428 ip-10-0-136-183 kubenswrapper[2574]: I0909 19:55:18.438368    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/2eb82961-9118-4b08-9d97-8d5cbac8cac3-var-lib-openvswitch\") pod \"ovnkube-node-jcr9m\" (UID: \"2eb82961-9118-4b08-9d97-8d5cbac8cac3\") " pod="openshift-ovn-kubernetes/ovnkube-node-jcr9m"
Sep 09 19:55:18.439380 ip-10-0-136-183 kubenswrapper[2574]: I0909 19:55:18.438393    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/2eb82961-9118-4b08-9d97-8d5cbac8cac3-run-ovn\") pod \"ovnkube-node-jcr9m\" (UID: \"2eb82961-9118-4b08-9d97-8d5cbac8cac3\") " pod="openshift-ovn-kubernetes/ovnkube-node-jcr9m"
Sep 09 19:55:18.439380 ip-10-0-136-183 kubenswrapper[2574]: I0909 19:55:18.438415    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/c756f4c9-2360-4fff-a136-b6869b02977e-dbus\") pod \"global-pull-secret-syncer-gjc9x\" (UID: \"c756f4c9-2360-4fff-a136-b6869b02977e\") " pod="kube-system/global-pull-secret-syncer-gjc9x"
Sep 09 19:55:18.439380 ip-10-0-136-183 kubenswrapper[2574]: I0909 19:55:18.438426    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/2eb82961-9118-4b08-9d97-8d5cbac8cac3-env-overrides\") pod \"ovnkube-node-jcr9m\" (UID: \"2eb82961-9118-4b08-9d97-8d5cbac8cac3\") " pod="openshift-ovn-kubernetes/ovnkube-node-jcr9m"
Sep 09 19:55:18.439380 ip-10-0-136-183 kubenswrapper[2574]: I0909 19:55:18.438438    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zqd55\" (UniqueName: \"kubernetes.io/projected/54458a6d-921b-4637-addc-7c4ee839b592-kube-api-access-zqd55\") pod \"network-check-target-4czxv\" (UID: \"54458a6d-921b-4637-addc-7c4ee839b592\") " pod="openshift-network-diagnostics/network-check-target-4czxv"
Sep 09 19:55:18.439380 ip-10-0-136-183 kubenswrapper[2574]: I0909 19:55:18.438463    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/9c19e4c1-5ffa-4fb8-88f0-93f1b32a591b-multus-conf-dir\") pod \"multus-sbx29\" (UID: \"9c19e4c1-5ffa-4fb8-88f0-93f1b32a591b\") " pod="openshift-multus/multus-sbx29"
Sep 09 19:55:18.439380 ip-10-0-136-183 kubenswrapper[2574]: I0909 19:55:18.438488    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/9c19e4c1-5ffa-4fb8-88f0-93f1b32a591b-host-run-multus-certs\") pod \"multus-sbx29\" (UID: \"9c19e4c1-5ffa-4fb8-88f0-93f1b32a591b\") " pod="openshift-multus/multus-sbx29"
Sep 09 19:55:18.439380 ip-10-0-136-183 kubenswrapper[2574]: I0909 19:55:18.438623    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/9c19e4c1-5ffa-4fb8-88f0-93f1b32a591b-etc-kubernetes\") pod \"multus-sbx29\" (UID: \"9c19e4c1-5ffa-4fb8-88f0-93f1b32a591b\") " pod="openshift-multus/multus-sbx29"
Sep 09 19:55:18.439380 ip-10-0-136-183 kubenswrapper[2574]: I0909 19:55:18.438674    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/71b40e5b-d6b9-489f-bb2e-4b5e1e5915e5-device-dir\") pod \"aws-ebs-csi-driver-node-bvxbg\" (UID: \"71b40e5b-d6b9-489f-bb2e-4b5e1e5915e5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bvxbg"
Sep 09 19:55:18.439380 ip-10-0-136-183 kubenswrapper[2574]: I0909 19:55:18.438721    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/c756f4c9-2360-4fff-a136-b6869b02977e-dbus\") pod \"global-pull-secret-syncer-gjc9x\" (UID: \"c756f4c9-2360-4fff-a136-b6869b02977e\") " pod="kube-system/global-pull-secret-syncer-gjc9x"
Sep 09 19:55:18.439380 ip-10-0-136-183 kubenswrapper[2574]: I0909 19:55:18.438755    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/2eb82961-9118-4b08-9d97-8d5cbac8cac3-var-lib-openvswitch\") pod \"ovnkube-node-jcr9m\" (UID: \"2eb82961-9118-4b08-9d97-8d5cbac8cac3\") " pod="openshift-ovn-kubernetes/ovnkube-node-jcr9m"
Sep 09 19:55:18.439380 ip-10-0-136-183 kubenswrapper[2574]: I0909 19:55:18.438810    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/2eb82961-9118-4b08-9d97-8d5cbac8cac3-run-ovn\") pod \"ovnkube-node-jcr9m\" (UID: \"2eb82961-9118-4b08-9d97-8d5cbac8cac3\") " pod="openshift-ovn-kubernetes/ovnkube-node-jcr9m"
Sep 09 19:55:18.439380 ip-10-0-136-183 kubenswrapper[2574]: I0909 19:55:18.438809    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/9c19e4c1-5ffa-4fb8-88f0-93f1b32a591b-multus-daemon-config\") pod \"multus-sbx29\" (UID: \"9c19e4c1-5ffa-4fb8-88f0-93f1b32a591b\") " pod="openshift-multus/multus-sbx29"
Sep 09 19:55:18.439380 ip-10-0-136-183 kubenswrapper[2574]: I0909 19:55:18.438859    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/9c19e4c1-5ffa-4fb8-88f0-93f1b32a591b-multus-conf-dir\") pod \"multus-sbx29\" (UID: \"9c19e4c1-5ffa-4fb8-88f0-93f1b32a591b\") " pod="openshift-multus/multus-sbx29"
Sep 09 19:55:18.439380 ip-10-0-136-183 kubenswrapper[2574]: I0909 19:55:18.438952    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/9c19e4c1-5ffa-4fb8-88f0-93f1b32a591b-host-run-multus-certs\") pod \"multus-sbx29\" (UID: \"9c19e4c1-5ffa-4fb8-88f0-93f1b32a591b\") " pod="openshift-multus/multus-sbx29"
Sep 09 19:55:18.439380 ip-10-0-136-183 kubenswrapper[2574]: I0909 19:55:18.439001    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/b06375df-d12d-40a3-a5a7-18fa1420f388-sys\") pod \"tuned-svtdz\" (UID: \"b06375df-d12d-40a3-a5a7-18fa1420f388\") " pod="openshift-cluster-node-tuning-operator/tuned-svtdz"
Sep 09 19:55:18.439380 ip-10-0-136-183 kubenswrapper[2574]: I0909 19:55:18.439031    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/2eb82961-9118-4b08-9d97-8d5cbac8cac3-host-kubelet\") pod \"ovnkube-node-jcr9m\" (UID: \"2eb82961-9118-4b08-9d97-8d5cbac8cac3\") " pod="openshift-ovn-kubernetes/ovnkube-node-jcr9m"
Sep 09 19:55:18.439380 ip-10-0-136-183 kubenswrapper[2574]: I0909 19:55:18.439056    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/2eb82961-9118-4b08-9d97-8d5cbac8cac3-host-run-netns\") pod \"ovnkube-node-jcr9m\" (UID: \"2eb82961-9118-4b08-9d97-8d5cbac8cac3\") " pod="openshift-ovn-kubernetes/ovnkube-node-jcr9m"
Sep 09 19:55:18.439380 ip-10-0-136-183 kubenswrapper[2574]: I0909 19:55:18.439088    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/2eb82961-9118-4b08-9d97-8d5cbac8cac3-run-systemd\") pod \"ovnkube-node-jcr9m\" (UID: \"2eb82961-9118-4b08-9d97-8d5cbac8cac3\") " pod="openshift-ovn-kubernetes/ovnkube-node-jcr9m"
Sep 09 19:55:18.440252 ip-10-0-136-183 kubenswrapper[2574]: I0909 19:55:18.439111    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/2eb82961-9118-4b08-9d97-8d5cbac8cac3-run-openvswitch\") pod \"ovnkube-node-jcr9m\" (UID: \"2eb82961-9118-4b08-9d97-8d5cbac8cac3\") " pod="openshift-ovn-kubernetes/ovnkube-node-jcr9m"
Sep 09 19:55:18.440252 ip-10-0-136-183 kubenswrapper[2574]: I0909 19:55:18.439116    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/2eb82961-9118-4b08-9d97-8d5cbac8cac3-host-kubelet\") pod \"ovnkube-node-jcr9m\" (UID: \"2eb82961-9118-4b08-9d97-8d5cbac8cac3\") " pod="openshift-ovn-kubernetes/ovnkube-node-jcr9m"
Sep 09 19:55:18.440252 ip-10-0-136-183 kubenswrapper[2574]: I0909 19:55:18.439137    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/2eb82961-9118-4b08-9d97-8d5cbac8cac3-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-jcr9m\" (UID: \"2eb82961-9118-4b08-9d97-8d5cbac8cac3\") " pod="openshift-ovn-kubernetes/ovnkube-node-jcr9m"
Sep 09 19:55:18.440252 ip-10-0-136-183 kubenswrapper[2574]: I0909 19:55:18.439145    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/b06375df-d12d-40a3-a5a7-18fa1420f388-sys\") pod \"tuned-svtdz\" (UID: \"b06375df-d12d-40a3-a5a7-18fa1420f388\") " pod="openshift-cluster-node-tuning-operator/tuned-svtdz"
Sep 09 19:55:18.440252 ip-10-0-136-183 kubenswrapper[2574]: I0909 19:55:18.439167    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/2eb82961-9118-4b08-9d97-8d5cbac8cac3-ovnkube-config\") pod \"ovnkube-node-jcr9m\" (UID: \"2eb82961-9118-4b08-9d97-8d5cbac8cac3\") " pod="openshift-ovn-kubernetes/ovnkube-node-jcr9m"
Sep 09 19:55:18.440252 ip-10-0-136-183 kubenswrapper[2574]: I0909 19:55:18.439192    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/c756f4c9-2360-4fff-a136-b6869b02977e-original-pull-secret\") pod \"global-pull-secret-syncer-gjc9x\" (UID: \"c756f4c9-2360-4fff-a136-b6869b02977e\") " pod="kube-system/global-pull-secret-syncer-gjc9x"
Sep 09 19:55:18.440252 ip-10-0-136-183 kubenswrapper[2574]: I0909 19:55:18.439195    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/2eb82961-9118-4b08-9d97-8d5cbac8cac3-run-openvswitch\") pod \"ovnkube-node-jcr9m\" (UID: \"2eb82961-9118-4b08-9d97-8d5cbac8cac3\") " pod="openshift-ovn-kubernetes/ovnkube-node-jcr9m"
Sep 09 19:55:18.440252 ip-10-0-136-183 kubenswrapper[2574]: I0909 19:55:18.439216    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/71b40e5b-d6b9-489f-bb2e-4b5e1e5915e5-socket-dir\") pod \"aws-ebs-csi-driver-node-bvxbg\" (UID: \"71b40e5b-d6b9-489f-bb2e-4b5e1e5915e5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bvxbg"
Sep 09 19:55:18.440252 ip-10-0-136-183 kubenswrapper[2574]: I0909 19:55:18.439227    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/2eb82961-9118-4b08-9d97-8d5cbac8cac3-run-systemd\") pod \"ovnkube-node-jcr9m\" (UID: \"2eb82961-9118-4b08-9d97-8d5cbac8cac3\") " pod="openshift-ovn-kubernetes/ovnkube-node-jcr9m"
Sep 09 19:55:18.440252 ip-10-0-136-183 kubenswrapper[2574]: I0909 19:55:18.439240    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/c756f4c9-2360-4fff-a136-b6869b02977e-kubelet-config\") pod \"global-pull-secret-syncer-gjc9x\" (UID: \"c756f4c9-2360-4fff-a136-b6869b02977e\") " pod="kube-system/global-pull-secret-syncer-gjc9x"
Sep 09 19:55:18.440252 ip-10-0-136-183 kubenswrapper[2574]: I0909 19:55:18.439266    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/bed9028e-2a22-4c8f-a637-bb5ad9204adc-system-cni-dir\") pod \"multus-additional-cni-plugins-gzqh4\" (UID: \"bed9028e-2a22-4c8f-a637-bb5ad9204adc\") " pod="openshift-multus/multus-additional-cni-plugins-gzqh4"
Sep 09 19:55:18.440252 ip-10-0-136-183 kubenswrapper[2574]: I0909 19:55:18.439275    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/2eb82961-9118-4b08-9d97-8d5cbac8cac3-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-jcr9m\" (UID: \"2eb82961-9118-4b08-9d97-8d5cbac8cac3\") " pod="openshift-ovn-kubernetes/ovnkube-node-jcr9m"
Sep 09 19:55:18.440252 ip-10-0-136-183 kubenswrapper[2574]: I0909 19:55:18.439290    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/bed9028e-2a22-4c8f-a637-bb5ad9204adc-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-gzqh4\" (UID: \"bed9028e-2a22-4c8f-a637-bb5ad9204adc\") " pod="openshift-multus/multus-additional-cni-plugins-gzqh4"
Sep 09 19:55:18.440252 ip-10-0-136-183 kubenswrapper[2574]: I0909 19:55:18.439315    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/71b40e5b-d6b9-489f-bb2e-4b5e1e5915e5-kubelet-dir\") pod \"aws-ebs-csi-driver-node-bvxbg\" (UID: \"71b40e5b-d6b9-489f-bb2e-4b5e1e5915e5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bvxbg"
Sep 09 19:55:18.440252 ip-10-0-136-183 kubenswrapper[2574]: I0909 19:55:18.439342    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/2d0e0ff3-612a-4eb3-b2df-342af115b6c3-konnectivity-ca\") pod \"konnectivity-agent-d9ww4\" (UID: \"2d0e0ff3-612a-4eb3-b2df-342af115b6c3\") " pod="kube-system/konnectivity-agent-d9ww4"
Sep 09 19:55:18.440252 ip-10-0-136-183 kubenswrapper[2574]: I0909 19:55:18.439369    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/496a3333-1b9c-4eb2-8896-b6586eaf6877-hosts-file\") pod \"node-resolver-q8krg\" (UID: \"496a3333-1b9c-4eb2-8896-b6586eaf6877\") " pod="openshift-dns/node-resolver-q8krg"
Sep 09 19:55:18.440252 ip-10-0-136-183 kubenswrapper[2574]: I0909 19:55:18.439391    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/bed9028e-2a22-4c8f-a637-bb5ad9204adc-cni-binary-copy\") pod \"multus-additional-cni-plugins-gzqh4\" (UID: \"bed9028e-2a22-4c8f-a637-bb5ad9204adc\") " pod="openshift-multus/multus-additional-cni-plugins-gzqh4"
Sep 09 19:55:18.441158 ip-10-0-136-183 kubenswrapper[2574]: I0909 19:55:18.439395    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/71b40e5b-d6b9-489f-bb2e-4b5e1e5915e5-socket-dir\") pod \"aws-ebs-csi-driver-node-bvxbg\" (UID: \"71b40e5b-d6b9-489f-bb2e-4b5e1e5915e5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bvxbg"
Sep 09 19:55:18.441158 ip-10-0-136-183 kubenswrapper[2574]: I0909 19:55:18.439414    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/9c19e4c1-5ffa-4fb8-88f0-93f1b32a591b-multus-cni-dir\") pod \"multus-sbx29\" (UID: \"9c19e4c1-5ffa-4fb8-88f0-93f1b32a591b\") " pod="openshift-multus/multus-sbx29"
Sep 09 19:55:18.441158 ip-10-0-136-183 kubenswrapper[2574]: I0909 19:55:18.439193    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/2eb82961-9118-4b08-9d97-8d5cbac8cac3-host-run-netns\") pod \"ovnkube-node-jcr9m\" (UID: \"2eb82961-9118-4b08-9d97-8d5cbac8cac3\") " pod="openshift-ovn-kubernetes/ovnkube-node-jcr9m"
Sep 09 19:55:18.441158 ip-10-0-136-183 kubenswrapper[2574]: I0909 19:55:18.439441    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/1f0e15c0-0c09-4135-baf4-767e800917d5-host-slash\") pod \"iptables-alerter-gvsgq\" (UID: \"1f0e15c0-0c09-4135-baf4-767e800917d5\") " pod="openshift-network-operator/iptables-alerter-gvsgq"
Sep 09 19:55:18.441158 ip-10-0-136-183 kubenswrapper[2574]: I0909 19:55:18.439466    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/2d0e0ff3-612a-4eb3-b2df-342af115b6c3-agent-certs\") pod \"konnectivity-agent-d9ww4\" (UID: \"2d0e0ff3-612a-4eb3-b2df-342af115b6c3\") " pod="kube-system/konnectivity-agent-d9ww4"
Sep 09 19:55:18.441158 ip-10-0-136-183 kubenswrapper[2574]: I0909 19:55:18.439480    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/c756f4c9-2360-4fff-a136-b6869b02977e-kubelet-config\") pod \"global-pull-secret-syncer-gjc9x\" (UID: \"c756f4c9-2360-4fff-a136-b6869b02977e\") " pod="kube-system/global-pull-secret-syncer-gjc9x"
Sep 09 19:55:18.441158 ip-10-0-136-183 kubenswrapper[2574]: I0909 19:55:18.439489    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/39099fe9-527d-4802-b40b-1bf535221165-serviceca\") pod \"node-ca-mgg8m\" (UID: \"39099fe9-527d-4802-b40b-1bf535221165\") " pod="openshift-image-registry/node-ca-mgg8m"
Sep 09 19:55:18.441158 ip-10-0-136-183 kubenswrapper[2574]: I0909 19:55:18.439511    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/2eb82961-9118-4b08-9d97-8d5cbac8cac3-systemd-units\") pod \"ovnkube-node-jcr9m\" (UID: \"2eb82961-9118-4b08-9d97-8d5cbac8cac3\") " pod="openshift-ovn-kubernetes/ovnkube-node-jcr9m"
Sep 09 19:55:18.441158 ip-10-0-136-183 kubenswrapper[2574]: I0909 19:55:18.439507    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/bed9028e-2a22-4c8f-a637-bb5ad9204adc-system-cni-dir\") pod \"multus-additional-cni-plugins-gzqh4\" (UID: \"bed9028e-2a22-4c8f-a637-bb5ad9204adc\") " pod="openshift-multus/multus-additional-cni-plugins-gzqh4"
Sep 09 19:55:18.441158 ip-10-0-136-183 kubenswrapper[2574]: I0909 19:55:18.439537    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/9c19e4c1-5ffa-4fb8-88f0-93f1b32a591b-host-var-lib-cni-bin\") pod \"multus-sbx29\" (UID: \"9c19e4c1-5ffa-4fb8-88f0-93f1b32a591b\") " pod="openshift-multus/multus-sbx29"
Sep 09 19:55:18.441158 ip-10-0-136-183 kubenswrapper[2574]: I0909 19:55:18.439564    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/b06375df-d12d-40a3-a5a7-18fa1420f388-etc-tuned\") pod \"tuned-svtdz\" (UID: \"b06375df-d12d-40a3-a5a7-18fa1420f388\") " pod="openshift-cluster-node-tuning-operator/tuned-svtdz"
Sep 09 19:55:18.441158 ip-10-0-136-183 kubenswrapper[2574]: I0909 19:55:18.439588    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/2eb82961-9118-4b08-9d97-8d5cbac8cac3-host-cni-bin\") pod \"ovnkube-node-jcr9m\" (UID: \"2eb82961-9118-4b08-9d97-8d5cbac8cac3\") " pod="openshift-ovn-kubernetes/ovnkube-node-jcr9m"
Sep 09 19:55:18.441158 ip-10-0-136-183 kubenswrapper[2574]: I0909 19:55:18.439612    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/bed9028e-2a22-4c8f-a637-bb5ad9204adc-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-gzqh4\" (UID: \"bed9028e-2a22-4c8f-a637-bb5ad9204adc\") " pod="openshift-multus/multus-additional-cni-plugins-gzqh4"
Sep 09 19:55:18.441158 ip-10-0-136-183 kubenswrapper[2574]: I0909 19:55:18.439637    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tz7jx\" (UniqueName: \"kubernetes.io/projected/bed9028e-2a22-4c8f-a637-bb5ad9204adc-kube-api-access-tz7jx\") pod \"multus-additional-cni-plugins-gzqh4\" (UID: \"bed9028e-2a22-4c8f-a637-bb5ad9204adc\") " pod="openshift-multus/multus-additional-cni-plugins-gzqh4"
Sep 09 19:55:18.441158 ip-10-0-136-183 kubenswrapper[2574]: I0909 19:55:18.439660    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/9c19e4c1-5ffa-4fb8-88f0-93f1b32a591b-cni-binary-copy\") pod \"multus-sbx29\" (UID: \"9c19e4c1-5ffa-4fb8-88f0-93f1b32a591b\") " pod="openshift-multus/multus-sbx29"
Sep 09 19:55:18.441158 ip-10-0-136-183 kubenswrapper[2574]: I0909 19:55:18.439683    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tf28w\" (UniqueName: \"kubernetes.io/projected/1f0e15c0-0c09-4135-baf4-767e800917d5-kube-api-access-tf28w\") pod \"iptables-alerter-gvsgq\" (UID: \"1f0e15c0-0c09-4135-baf4-767e800917d5\") " pod="openshift-network-operator/iptables-alerter-gvsgq"
Sep 09 19:55:18.441158 ip-10-0-136-183 kubenswrapper[2574]: I0909 19:55:18.439705    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/71b40e5b-d6b9-489f-bb2e-4b5e1e5915e5-kubelet-dir\") pod \"aws-ebs-csi-driver-node-bvxbg\" (UID: \"71b40e5b-d6b9-489f-bb2e-4b5e1e5915e5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bvxbg"
Sep 09 19:55:18.442025 ip-10-0-136-183 kubenswrapper[2574]: I0909 19:55:18.439707    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/b06375df-d12d-40a3-a5a7-18fa1420f388-etc-sysctl-d\") pod \"tuned-svtdz\" (UID: \"b06375df-d12d-40a3-a5a7-18fa1420f388\") " pod="openshift-cluster-node-tuning-operator/tuned-svtdz"
Sep 09 19:55:18.442025 ip-10-0-136-183 kubenswrapper[2574]: I0909 19:55:18.439745    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cqtqf\" (UniqueName: \"kubernetes.io/projected/2eb82961-9118-4b08-9d97-8d5cbac8cac3-kube-api-access-cqtqf\") pod \"ovnkube-node-jcr9m\" (UID: \"2eb82961-9118-4b08-9d97-8d5cbac8cac3\") " pod="openshift-ovn-kubernetes/ovnkube-node-jcr9m"
Sep 09 19:55:18.442025 ip-10-0-136-183 kubenswrapper[2574]: I0909 19:55:18.439766    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/9c19e4c1-5ffa-4fb8-88f0-93f1b32a591b-hostroot\") pod \"multus-sbx29\" (UID: \"9c19e4c1-5ffa-4fb8-88f0-93f1b32a591b\") " pod="openshift-multus/multus-sbx29"
Sep 09 19:55:18.442025 ip-10-0-136-183 kubenswrapper[2574]: I0909 19:55:18.439786    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sqkzq\" (UniqueName: \"kubernetes.io/projected/9c19e4c1-5ffa-4fb8-88f0-93f1b32a591b-kube-api-access-sqkzq\") pod \"multus-sbx29\" (UID: \"9c19e4c1-5ffa-4fb8-88f0-93f1b32a591b\") " pod="openshift-multus/multus-sbx29"
Sep 09 19:55:18.442025 ip-10-0-136-183 kubenswrapper[2574]: I0909 19:55:18.439821    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/b06375df-d12d-40a3-a5a7-18fa1420f388-etc-sysctl-d\") pod \"tuned-svtdz\" (UID: \"b06375df-d12d-40a3-a5a7-18fa1420f388\") " pod="openshift-cluster-node-tuning-operator/tuned-svtdz"
Sep 09 19:55:18.442025 ip-10-0-136-183 kubenswrapper[2574]: I0909 19:55:18.439876    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/b06375df-d12d-40a3-a5a7-18fa1420f388-var-lib-kubelet\") pod \"tuned-svtdz\" (UID: \"b06375df-d12d-40a3-a5a7-18fa1420f388\") " pod="openshift-cluster-node-tuning-operator/tuned-svtdz"
Sep 09 19:55:18.442025 ip-10-0-136-183 kubenswrapper[2574]: I0909 19:55:18.439929    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/496a3333-1b9c-4eb2-8896-b6586eaf6877-hosts-file\") pod \"node-resolver-q8krg\" (UID: \"496a3333-1b9c-4eb2-8896-b6586eaf6877\") " pod="openshift-dns/node-resolver-q8krg"
Sep 09 19:55:18.442025 ip-10-0-136-183 kubenswrapper[2574]: I0909 19:55:18.440012    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/bed9028e-2a22-4c8f-a637-bb5ad9204adc-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-gzqh4\" (UID: \"bed9028e-2a22-4c8f-a637-bb5ad9204adc\") " pod="openshift-multus/multus-additional-cni-plugins-gzqh4"
Sep 09 19:55:18.442025 ip-10-0-136-183 kubenswrapper[2574]: I0909 19:55:18.440029    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/9c19e4c1-5ffa-4fb8-88f0-93f1b32a591b-hostroot\") pod \"multus-sbx29\" (UID: \"9c19e4c1-5ffa-4fb8-88f0-93f1b32a591b\") " pod="openshift-multus/multus-sbx29"
Sep 09 19:55:18.442025 ip-10-0-136-183 kubenswrapper[2574]: I0909 19:55:18.440088    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/9c19e4c1-5ffa-4fb8-88f0-93f1b32a591b-multus-cni-dir\") pod \"multus-sbx29\" (UID: \"9c19e4c1-5ffa-4fb8-88f0-93f1b32a591b\") " pod="openshift-multus/multus-sbx29"
Sep 09 19:55:18.442025 ip-10-0-136-183 kubenswrapper[2574]: E0909 19:55:18.440141    2574 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 09 19:55:18.442025 ip-10-0-136-183 kubenswrapper[2574]: E0909 19:55:18.440184    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c756f4c9-2360-4fff-a136-b6869b02977e-original-pull-secret podName:c756f4c9-2360-4fff-a136-b6869b02977e nodeName:}" failed. No retries permitted until 2026-09-09 19:55:18.940170401 +0000 UTC m=+2.986423660 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/c756f4c9-2360-4fff-a136-b6869b02977e-original-pull-secret") pod "global-pull-secret-syncer-gjc9x" (UID: "c756f4c9-2360-4fff-a136-b6869b02977e") : object "kube-system"/"original-pull-secret" not registered
Sep 09 19:55:18.442025 ip-10-0-136-183 kubenswrapper[2574]: I0909 19:55:18.440225    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/1f0e15c0-0c09-4135-baf4-767e800917d5-host-slash\") pod \"iptables-alerter-gvsgq\" (UID: \"1f0e15c0-0c09-4135-baf4-767e800917d5\") " pod="openshift-network-operator/iptables-alerter-gvsgq"
Sep 09 19:55:18.442025 ip-10-0-136-183 kubenswrapper[2574]: I0909 19:55:18.439659    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/2eb82961-9118-4b08-9d97-8d5cbac8cac3-ovnkube-config\") pod \"ovnkube-node-jcr9m\" (UID: \"2eb82961-9118-4b08-9d97-8d5cbac8cac3\") " pod="openshift-ovn-kubernetes/ovnkube-node-jcr9m"
Sep 09 19:55:18.442025 ip-10-0-136-183 kubenswrapper[2574]: I0909 19:55:18.440581    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/39099fe9-527d-4802-b40b-1bf535221165-serviceca\") pod \"node-ca-mgg8m\" (UID: \"39099fe9-527d-4802-b40b-1bf535221165\") " pod="openshift-image-registry/node-ca-mgg8m"
Sep 09 19:55:18.442025 ip-10-0-136-183 kubenswrapper[2574]: I0909 19:55:18.440638    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/2eb82961-9118-4b08-9d97-8d5cbac8cac3-systemd-units\") pod \"ovnkube-node-jcr9m\" (UID: \"2eb82961-9118-4b08-9d97-8d5cbac8cac3\") " pod="openshift-ovn-kubernetes/ovnkube-node-jcr9m"
Sep 09 19:55:18.442025 ip-10-0-136-183 kubenswrapper[2574]: I0909 19:55:18.440683    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/9c19e4c1-5ffa-4fb8-88f0-93f1b32a591b-host-var-lib-cni-bin\") pod \"multus-sbx29\" (UID: \"9c19e4c1-5ffa-4fb8-88f0-93f1b32a591b\") " pod="openshift-multus/multus-sbx29"
Sep 09 19:55:18.442850 ip-10-0-136-183 kubenswrapper[2574]: I0909 19:55:18.439824    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/b06375df-d12d-40a3-a5a7-18fa1420f388-var-lib-kubelet\") pod \"tuned-svtdz\" (UID: \"b06375df-d12d-40a3-a5a7-18fa1420f388\") " pod="openshift-cluster-node-tuning-operator/tuned-svtdz"
Sep 09 19:55:18.442850 ip-10-0-136-183 kubenswrapper[2574]: I0909 19:55:18.440730    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/1f0e15c0-0c09-4135-baf4-767e800917d5-iptables-alerter-script\") pod \"iptables-alerter-gvsgq\" (UID: \"1f0e15c0-0c09-4135-baf4-767e800917d5\") " pod="openshift-network-operator/iptables-alerter-gvsgq"
Sep 09 19:55:18.442850 ip-10-0-136-183 kubenswrapper[2574]: I0909 19:55:18.441020    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/2eb82961-9118-4b08-9d97-8d5cbac8cac3-host-cni-bin\") pod \"ovnkube-node-jcr9m\" (UID: \"2eb82961-9118-4b08-9d97-8d5cbac8cac3\") " pod="openshift-ovn-kubernetes/ovnkube-node-jcr9m"
Sep 09 19:55:18.442850 ip-10-0-136-183 kubenswrapper[2574]: I0909 19:55:18.441152    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/9c19e4c1-5ffa-4fb8-88f0-93f1b32a591b-cni-binary-copy\") pod \"multus-sbx29\" (UID: \"9c19e4c1-5ffa-4fb8-88f0-93f1b32a591b\") " pod="openshift-multus/multus-sbx29"
Sep 09 19:55:18.442850 ip-10-0-136-183 kubenswrapper[2574]: I0909 19:55:18.441236    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/1f0e15c0-0c09-4135-baf4-767e800917d5-iptables-alerter-script\") pod \"iptables-alerter-gvsgq\" (UID: \"1f0e15c0-0c09-4135-baf4-767e800917d5\") " pod="openshift-network-operator/iptables-alerter-gvsgq"
Sep 09 19:55:18.442850 ip-10-0-136-183 kubenswrapper[2574]: I0909 19:55:18.441292    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/bed9028e-2a22-4c8f-a637-bb5ad9204adc-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-gzqh4\" (UID: \"bed9028e-2a22-4c8f-a637-bb5ad9204adc\") " pod="openshift-multus/multus-additional-cni-plugins-gzqh4"
Sep 09 19:55:18.442850 ip-10-0-136-183 kubenswrapper[2574]: I0909 19:55:18.441529    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/bed9028e-2a22-4c8f-a637-bb5ad9204adc-cni-binary-copy\") pod \"multus-additional-cni-plugins-gzqh4\" (UID: \"bed9028e-2a22-4c8f-a637-bb5ad9204adc\") " pod="openshift-multus/multus-additional-cni-plugins-gzqh4"
Sep 09 19:55:18.443902 ip-10-0-136-183 kubenswrapper[2574]: I0909 19:55:18.443816    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/b06375df-d12d-40a3-a5a7-18fa1420f388-tmp\") pod \"tuned-svtdz\" (UID: \"b06375df-d12d-40a3-a5a7-18fa1420f388\") " pod="openshift-cluster-node-tuning-operator/tuned-svtdz"
Sep 09 19:55:18.443902 ip-10-0-136-183 kubenswrapper[2574]: I0909 19:55:18.443888    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/b06375df-d12d-40a3-a5a7-18fa1420f388-etc-tuned\") pod \"tuned-svtdz\" (UID: \"b06375df-d12d-40a3-a5a7-18fa1420f388\") " pod="openshift-cluster-node-tuning-operator/tuned-svtdz"
Sep 09 19:55:18.444095 ip-10-0-136-183 kubenswrapper[2574]: I0909 19:55:18.444067    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/2eb82961-9118-4b08-9d97-8d5cbac8cac3-ovn-node-metrics-cert\") pod \"ovnkube-node-jcr9m\" (UID: \"2eb82961-9118-4b08-9d97-8d5cbac8cac3\") " pod="openshift-ovn-kubernetes/ovnkube-node-jcr9m"
Sep 09 19:55:18.451714 ip-10-0-136-183 kubenswrapper[2574]: I0909 19:55:18.451691    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tf28w\" (UniqueName: \"kubernetes.io/projected/1f0e15c0-0c09-4135-baf4-767e800917d5-kube-api-access-tf28w\") pod \"iptables-alerter-gvsgq\" (UID: \"1f0e15c0-0c09-4135-baf4-767e800917d5\") " pod="openshift-network-operator/iptables-alerter-gvsgq"
Sep 09 19:55:18.452660 ip-10-0-136-183 kubenswrapper[2574]: I0909 19:55:18.452638    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vwrk4\" (UniqueName: \"kubernetes.io/projected/71b40e5b-d6b9-489f-bb2e-4b5e1e5915e5-kube-api-access-vwrk4\") pod \"aws-ebs-csi-driver-node-bvxbg\" (UID: \"71b40e5b-d6b9-489f-bb2e-4b5e1e5915e5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bvxbg"
Sep 09 19:55:18.452917 ip-10-0-136-183 kubenswrapper[2574]: I0909 19:55:18.452895    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-p6r7f\" (UniqueName: \"kubernetes.io/projected/39099fe9-527d-4802-b40b-1bf535221165-kube-api-access-p6r7f\") pod \"node-ca-mgg8m\" (UID: \"39099fe9-527d-4802-b40b-1bf535221165\") " pod="openshift-image-registry/node-ca-mgg8m"
Sep 09 19:55:18.453407 ip-10-0-136-183 kubenswrapper[2574]: E0909 19:55:18.453391    2574 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 09 19:55:18.453531 ip-10-0-136-183 kubenswrapper[2574]: E0909 19:55:18.453519    2574 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 09 19:55:18.453621 ip-10-0-136-183 kubenswrapper[2574]: E0909 19:55:18.453612    2574 projected.go:196] Error preparing data for projected volume kube-api-access-zqd55 for pod openshift-network-diagnostics/network-check-target-4czxv: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 09 19:55:18.453760 ip-10-0-136-183 kubenswrapper[2574]: E0909 19:55:18.453747    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/54458a6d-921b-4637-addc-7c4ee839b592-kube-api-access-zqd55 podName:54458a6d-921b-4637-addc-7c4ee839b592 nodeName:}" failed. No retries permitted until 2026-09-09 19:55:18.953729269 +0000 UTC m=+2.999982736 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-zqd55" (UniqueName: "kubernetes.io/projected/54458a6d-921b-4637-addc-7c4ee839b592-kube-api-access-zqd55") pod "network-check-target-4czxv" (UID: "54458a6d-921b-4637-addc-7c4ee839b592") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 09 19:55:18.453912 ip-10-0-136-183 kubenswrapper[2574]: I0909 19:55:18.453890    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tz7jx\" (UniqueName: \"kubernetes.io/projected/bed9028e-2a22-4c8f-a637-bb5ad9204adc-kube-api-access-tz7jx\") pod \"multus-additional-cni-plugins-gzqh4\" (UID: \"bed9028e-2a22-4c8f-a637-bb5ad9204adc\") " pod="openshift-multus/multus-additional-cni-plugins-gzqh4"
Sep 09 19:55:18.453992 ip-10-0-136-183 kubenswrapper[2574]: I0909 19:55:18.453482    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-djvx9\" (UniqueName: \"kubernetes.io/projected/b06375df-d12d-40a3-a5a7-18fa1420f388-kube-api-access-djvx9\") pod \"tuned-svtdz\" (UID: \"b06375df-d12d-40a3-a5a7-18fa1420f388\") " pod="openshift-cluster-node-tuning-operator/tuned-svtdz"
Sep 09 19:55:18.454323 ip-10-0-136-183 kubenswrapper[2574]: I0909 19:55:18.454300    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7kk79\" (UniqueName: \"kubernetes.io/projected/496a3333-1b9c-4eb2-8896-b6586eaf6877-kube-api-access-7kk79\") pod \"node-resolver-q8krg\" (UID: \"496a3333-1b9c-4eb2-8896-b6586eaf6877\") " pod="openshift-dns/node-resolver-q8krg"
Sep 09 19:55:18.454445 ip-10-0-136-183 kubenswrapper[2574]: I0909 19:55:18.454421    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cqtqf\" (UniqueName: \"kubernetes.io/projected/2eb82961-9118-4b08-9d97-8d5cbac8cac3-kube-api-access-cqtqf\") pod \"ovnkube-node-jcr9m\" (UID: \"2eb82961-9118-4b08-9d97-8d5cbac8cac3\") " pod="openshift-ovn-kubernetes/ovnkube-node-jcr9m"
Sep 09 19:55:18.455344 ip-10-0-136-183 kubenswrapper[2574]: I0909 19:55:18.455320    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sqkzq\" (UniqueName: \"kubernetes.io/projected/9c19e4c1-5ffa-4fb8-88f0-93f1b32a591b-kube-api-access-sqkzq\") pod \"multus-sbx29\" (UID: \"9c19e4c1-5ffa-4fb8-88f0-93f1b32a591b\") " pod="openshift-multus/multus-sbx29"
Sep 09 19:55:18.459683 ip-10-0-136-183 kubenswrapper[2574]: I0909 19:55:18.459660    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nbnrs\" (UniqueName: \"kubernetes.io/projected/47c2b5b9-456a-4528-8d83-7ce3269cc16e-kube-api-access-nbnrs\") pod \"network-metrics-daemon-mdrck\" (UID: \"47c2b5b9-456a-4528-8d83-7ce3269cc16e\") " pod="openshift-multus/network-metrics-daemon-mdrck"
Sep 09 19:55:18.490926 ip-10-0-136-183 kubenswrapper[2574]: I0909 19:55:18.490893    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-136-183.ec2.internal" event={"ID":"c82ea50b31881ad827f1ff32ee67e2c3","Type":"ContainerStarted","Data":"9649afeecad74084847b2e26465def795eab5af5559ee57fb31ca46d4e176971"}
Sep 09 19:55:18.542058 ip-10-0-136-183 kubenswrapper[2574]: I0909 19:55:18.542022    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/2d0e0ff3-612a-4eb3-b2df-342af115b6c3-konnectivity-ca\") pod \"konnectivity-agent-d9ww4\" (UID: \"2d0e0ff3-612a-4eb3-b2df-342af115b6c3\") " pod="kube-system/konnectivity-agent-d9ww4"
Sep 09 19:55:18.542236 ip-10-0-136-183 kubenswrapper[2574]: I0909 19:55:18.542073    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/2d0e0ff3-612a-4eb3-b2df-342af115b6c3-agent-certs\") pod \"konnectivity-agent-d9ww4\" (UID: \"2d0e0ff3-612a-4eb3-b2df-342af115b6c3\") " pod="kube-system/konnectivity-agent-d9ww4"
Sep 09 19:55:18.543118 ip-10-0-136-183 kubenswrapper[2574]: I0909 19:55:18.543075    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/2d0e0ff3-612a-4eb3-b2df-342af115b6c3-konnectivity-ca\") pod \"konnectivity-agent-d9ww4\" (UID: \"2d0e0ff3-612a-4eb3-b2df-342af115b6c3\") " pod="kube-system/konnectivity-agent-d9ww4"
Sep 09 19:55:18.545134 ip-10-0-136-183 kubenswrapper[2574]: I0909 19:55:18.545112    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/2d0e0ff3-612a-4eb3-b2df-342af115b6c3-agent-certs\") pod \"konnectivity-agent-d9ww4\" (UID: \"2d0e0ff3-612a-4eb3-b2df-342af115b6c3\") " pod="kube-system/konnectivity-agent-d9ww4"
Sep 09 19:55:18.616371 ip-10-0-136-183 kubenswrapper[2574]: I0909 19:55:18.616345    2574 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 09 19:55:18.620619 ip-10-0-136-183 kubenswrapper[2574]: I0909 19:55:18.620595    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-sbx29"
Sep 09 19:55:18.631538 ip-10-0-136-183 kubenswrapper[2574]: I0909 19:55:18.631511    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-gvsgq"
Sep 09 19:55:18.642245 ip-10-0-136-183 kubenswrapper[2574]: I0909 19:55:18.642220    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-jcr9m"
Sep 09 19:55:18.647997 ip-10-0-136-183 kubenswrapper[2574]: I0909 19:55:18.647977    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bvxbg"
Sep 09 19:55:18.655660 ip-10-0-136-183 kubenswrapper[2574]: I0909 19:55:18.655639    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-svtdz"
Sep 09 19:55:18.664206 ip-10-0-136-183 kubenswrapper[2574]: I0909 19:55:18.664187    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-q8krg"
Sep 09 19:55:18.671623 ip-10-0-136-183 kubenswrapper[2574]: I0909 19:55:18.671588    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-mgg8m"
Sep 09 19:55:18.677068 ip-10-0-136-183 kubenswrapper[2574]: I0909 19:55:18.677046    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-gzqh4"
Sep 09 19:55:18.683667 ip-10-0-136-183 kubenswrapper[2574]: I0909 19:55:18.683648    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-d9ww4"
Sep 09 19:55:18.944787 ip-10-0-136-183 kubenswrapper[2574]: I0909 19:55:18.944696    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/c756f4c9-2360-4fff-a136-b6869b02977e-original-pull-secret\") pod \"global-pull-secret-syncer-gjc9x\" (UID: \"c756f4c9-2360-4fff-a136-b6869b02977e\") " pod="kube-system/global-pull-secret-syncer-gjc9x"
Sep 09 19:55:18.944787 ip-10-0-136-183 kubenswrapper[2574]: I0909 19:55:18.944761    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/47c2b5b9-456a-4528-8d83-7ce3269cc16e-metrics-certs\") pod \"network-metrics-daemon-mdrck\" (UID: \"47c2b5b9-456a-4528-8d83-7ce3269cc16e\") " pod="openshift-multus/network-metrics-daemon-mdrck"
Sep 09 19:55:18.945025 ip-10-0-136-183 kubenswrapper[2574]: E0909 19:55:18.944883    2574 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 09 19:55:18.945025 ip-10-0-136-183 kubenswrapper[2574]: E0909 19:55:18.944886    2574 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 09 19:55:18.945025 ip-10-0-136-183 kubenswrapper[2574]: E0909 19:55:18.944955    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/47c2b5b9-456a-4528-8d83-7ce3269cc16e-metrics-certs podName:47c2b5b9-456a-4528-8d83-7ce3269cc16e nodeName:}" failed. No retries permitted until 2026-09-09 19:55:19.944933742 +0000 UTC m=+3.991186997 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/47c2b5b9-456a-4528-8d83-7ce3269cc16e-metrics-certs") pod "network-metrics-daemon-mdrck" (UID: "47c2b5b9-456a-4528-8d83-7ce3269cc16e") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 09 19:55:18.945025 ip-10-0-136-183 kubenswrapper[2574]: E0909 19:55:18.944975    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c756f4c9-2360-4fff-a136-b6869b02977e-original-pull-secret podName:c756f4c9-2360-4fff-a136-b6869b02977e nodeName:}" failed. No retries permitted until 2026-09-09 19:55:19.944966014 +0000 UTC m=+3.991219256 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/c756f4c9-2360-4fff-a136-b6869b02977e-original-pull-secret") pod "global-pull-secret-syncer-gjc9x" (UID: "c756f4c9-2360-4fff-a136-b6869b02977e") : object "kube-system"/"original-pull-secret" not registered
Sep 09 19:55:19.045312 ip-10-0-136-183 kubenswrapper[2574]: I0909 19:55:19.045279    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zqd55\" (UniqueName: \"kubernetes.io/projected/54458a6d-921b-4637-addc-7c4ee839b592-kube-api-access-zqd55\") pod \"network-check-target-4czxv\" (UID: \"54458a6d-921b-4637-addc-7c4ee839b592\") " pod="openshift-network-diagnostics/network-check-target-4czxv"
Sep 09 19:55:19.045495 ip-10-0-136-183 kubenswrapper[2574]: E0909 19:55:19.045445    2574 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 09 19:55:19.045495 ip-10-0-136-183 kubenswrapper[2574]: E0909 19:55:19.045474    2574 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 09 19:55:19.045495 ip-10-0-136-183 kubenswrapper[2574]: E0909 19:55:19.045486    2574 projected.go:196] Error preparing data for projected volume kube-api-access-zqd55 for pod openshift-network-diagnostics/network-check-target-4czxv: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 09 19:55:19.045606 ip-10-0-136-183 kubenswrapper[2574]: E0909 19:55:19.045540    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/54458a6d-921b-4637-addc-7c4ee839b592-kube-api-access-zqd55 podName:54458a6d-921b-4637-addc-7c4ee839b592 nodeName:}" failed. No retries permitted until 2026-09-09 19:55:20.045526672 +0000 UTC m=+4.091779926 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-zqd55" (UniqueName: "kubernetes.io/projected/54458a6d-921b-4637-addc-7c4ee839b592-kube-api-access-zqd55") pod "network-check-target-4czxv" (UID: "54458a6d-921b-4637-addc-7c4ee839b592") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 09 19:55:19.301578 ip-10-0-136-183 kubenswrapper[2574]: W0909 19:55:19.301547    2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9c19e4c1_5ffa_4fb8_88f0_93f1b32a591b.slice/crio-b9ed0dd9e850514b4a36d87e566418ed3cc063fde3ea805912b386e1b39f1cb6 WatchSource:0}: Error finding container b9ed0dd9e850514b4a36d87e566418ed3cc063fde3ea805912b386e1b39f1cb6: Status 404 returned error can't find the container with id b9ed0dd9e850514b4a36d87e566418ed3cc063fde3ea805912b386e1b39f1cb6
Sep 09 19:55:19.314394 ip-10-0-136-183 kubenswrapper[2574]: W0909 19:55:19.314366    2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1f0e15c0_0c09_4135_baf4_767e800917d5.slice/crio-48675fcd8a3997addd4f3e0bef23c6d67725d3dc1fcd6de951ee0058e3c4b2e6 WatchSource:0}: Error finding container 48675fcd8a3997addd4f3e0bef23c6d67725d3dc1fcd6de951ee0058e3c4b2e6: Status 404 returned error can't find the container with id 48675fcd8a3997addd4f3e0bef23c6d67725d3dc1fcd6de951ee0058e3c4b2e6
Sep 09 19:55:19.317741 ip-10-0-136-183 kubenswrapper[2574]: W0909 19:55:19.317718    2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb06375df_d12d_40a3_a5a7_18fa1420f388.slice/crio-cd79db194721aeabec44b1609fa0b6cd303ccc19d7e1cc9d5e0b3cbfec16ef9a WatchSource:0}: Error finding container cd79db194721aeabec44b1609fa0b6cd303ccc19d7e1cc9d5e0b3cbfec16ef9a: Status 404 returned error can't find the container with id cd79db194721aeabec44b1609fa0b6cd303ccc19d7e1cc9d5e0b3cbfec16ef9a
Sep 09 19:55:19.319049 ip-10-0-136-183 kubenswrapper[2574]: W0909 19:55:19.319008    2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod71b40e5b_d6b9_489f_bb2e_4b5e1e5915e5.slice/crio-734f4c1d11eebb57a6ea2e533d7800d1cb2c068cd9e47699656dccbcca926d7a WatchSource:0}: Error finding container 734f4c1d11eebb57a6ea2e533d7800d1cb2c068cd9e47699656dccbcca926d7a: Status 404 returned error can't find the container with id 734f4c1d11eebb57a6ea2e533d7800d1cb2c068cd9e47699656dccbcca926d7a
Sep 09 19:55:19.319849 ip-10-0-136-183 kubenswrapper[2574]: W0909 19:55:19.319825    2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2eb82961_9118_4b08_9d97_8d5cbac8cac3.slice/crio-dcf6864e49637b1844de501d38fcdaba21c0d92e3c96c048ac03e89049d42c29 WatchSource:0}: Error finding container dcf6864e49637b1844de501d38fcdaba21c0d92e3c96c048ac03e89049d42c29: Status 404 returned error can't find the container with id dcf6864e49637b1844de501d38fcdaba21c0d92e3c96c048ac03e89049d42c29
Sep 09 19:55:19.327421 ip-10-0-136-183 kubenswrapper[2574]: W0909 19:55:19.326390    2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbed9028e_2a22_4c8f_a637_bb5ad9204adc.slice/crio-c8fc08c42194861a7df64d6cba4a9d6349d8209b38b37b599257296c80029ddd WatchSource:0}: Error finding container c8fc08c42194861a7df64d6cba4a9d6349d8209b38b37b599257296c80029ddd: Status 404 returned error can't find the container with id c8fc08c42194861a7df64d6cba4a9d6349d8209b38b37b599257296c80029ddd
Sep 09 19:55:19.363719 ip-10-0-136-183 kubenswrapper[2574]: I0909 19:55:19.363564    2574 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-09-08 19:50:18 +0000 UTC" deadline="2028-03-26 23:52:25.976192381 +0000 UTC"
Sep 09 19:55:19.363719 ip-10-0-136-183 kubenswrapper[2574]: I0909 19:55:19.363716    2574 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13539h57m6.612479317s"
Sep 09 19:55:19.486001 ip-10-0-136-183 kubenswrapper[2574]: I0909 19:55:19.485973    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-4czxv"
Sep 09 19:55:19.486170 ip-10-0-136-183 kubenswrapper[2574]: E0909 19:55:19.486092    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-4czxv" podUID="54458a6d-921b-4637-addc-7c4ee839b592"
Sep 09 19:55:19.486170 ip-10-0-136-183 kubenswrapper[2574]: I0909 19:55:19.486135    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mdrck"
Sep 09 19:55:19.486287 ip-10-0-136-183 kubenswrapper[2574]: E0909 19:55:19.486237    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-mdrck" podUID="47c2b5b9-456a-4528-8d83-7ce3269cc16e"
Sep 09 19:55:19.497911 ip-10-0-136-183 kubenswrapper[2574]: I0909 19:55:19.497874    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-136-183.ec2.internal" event={"ID":"c82ea50b31881ad827f1ff32ee67e2c3","Type":"ContainerStarted","Data":"d92befe3c40bdb0dbebbfe2a7fd5ed485fb6589e2725cd448feeaee8c387499e"}
Sep 09 19:55:19.499677 ip-10-0-136-183 kubenswrapper[2574]: I0909 19:55:19.499647    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-jcr9m" event={"ID":"2eb82961-9118-4b08-9d97-8d5cbac8cac3","Type":"ContainerStarted","Data":"dcf6864e49637b1844de501d38fcdaba21c0d92e3c96c048ac03e89049d42c29"}
Sep 09 19:55:19.501226 ip-10-0-136-183 kubenswrapper[2574]: I0909 19:55:19.501198    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-sbx29" event={"ID":"9c19e4c1-5ffa-4fb8-88f0-93f1b32a591b","Type":"ContainerStarted","Data":"b9ed0dd9e850514b4a36d87e566418ed3cc063fde3ea805912b386e1b39f1cb6"}
Sep 09 19:55:19.502371 ip-10-0-136-183 kubenswrapper[2574]: I0909 19:55:19.502348    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-d9ww4" event={"ID":"2d0e0ff3-612a-4eb3-b2df-342af115b6c3","Type":"ContainerStarted","Data":"4a4ce45832e9ee59f03c50b4b0a97c81848d2598bd597217d9f14cbf95f28464"}
Sep 09 19:55:19.503368 ip-10-0-136-183 kubenswrapper[2574]: I0909 19:55:19.503348    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-q8krg" event={"ID":"496a3333-1b9c-4eb2-8896-b6586eaf6877","Type":"ContainerStarted","Data":"9629f668f319dcbcabac44c54ea0b42e7b01ece7c69ec6d94d253c0ee8fc7043"}
Sep 09 19:55:19.504742 ip-10-0-136-183 kubenswrapper[2574]: I0909 19:55:19.504722    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-gzqh4" event={"ID":"bed9028e-2a22-4c8f-a637-bb5ad9204adc","Type":"ContainerStarted","Data":"c8fc08c42194861a7df64d6cba4a9d6349d8209b38b37b599257296c80029ddd"}
Sep 09 19:55:19.508075 ip-10-0-136-183 kubenswrapper[2574]: I0909 19:55:19.508054    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-mgg8m" event={"ID":"39099fe9-527d-4802-b40b-1bf535221165","Type":"ContainerStarted","Data":"8b4fd9e1030326ed7989d6a24654ab98b985530befb055a4db0a33e4557919aa"}
Sep 09 19:55:19.510724 ip-10-0-136-183 kubenswrapper[2574]: I0909 19:55:19.510707    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bvxbg" event={"ID":"71b40e5b-d6b9-489f-bb2e-4b5e1e5915e5","Type":"ContainerStarted","Data":"734f4c1d11eebb57a6ea2e533d7800d1cb2c068cd9e47699656dccbcca926d7a"}
Sep 09 19:55:19.512307 ip-10-0-136-183 kubenswrapper[2574]: I0909 19:55:19.512289    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-svtdz" event={"ID":"b06375df-d12d-40a3-a5a7-18fa1420f388","Type":"ContainerStarted","Data":"cd79db194721aeabec44b1609fa0b6cd303ccc19d7e1cc9d5e0b3cbfec16ef9a"}
Sep 09 19:55:19.513392 ip-10-0-136-183 kubenswrapper[2574]: I0909 19:55:19.513373    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-gvsgq" event={"ID":"1f0e15c0-0c09-4135-baf4-767e800917d5","Type":"ContainerStarted","Data":"48675fcd8a3997addd4f3e0bef23c6d67725d3dc1fcd6de951ee0058e3c4b2e6"}
Sep 09 19:55:19.953306 ip-10-0-136-183 kubenswrapper[2574]: I0909 19:55:19.953262    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/c756f4c9-2360-4fff-a136-b6869b02977e-original-pull-secret\") pod \"global-pull-secret-syncer-gjc9x\" (UID: \"c756f4c9-2360-4fff-a136-b6869b02977e\") " pod="kube-system/global-pull-secret-syncer-gjc9x"
Sep 09 19:55:19.953471 ip-10-0-136-183 kubenswrapper[2574]: I0909 19:55:19.953335    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/47c2b5b9-456a-4528-8d83-7ce3269cc16e-metrics-certs\") pod \"network-metrics-daemon-mdrck\" (UID: \"47c2b5b9-456a-4528-8d83-7ce3269cc16e\") " pod="openshift-multus/network-metrics-daemon-mdrck"
Sep 09 19:55:19.953536 ip-10-0-136-183 kubenswrapper[2574]: E0909 19:55:19.953485    2574 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 09 19:55:19.953592 ip-10-0-136-183 kubenswrapper[2574]: E0909 19:55:19.953553    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/47c2b5b9-456a-4528-8d83-7ce3269cc16e-metrics-certs podName:47c2b5b9-456a-4528-8d83-7ce3269cc16e nodeName:}" failed. No retries permitted until 2026-09-09 19:55:21.953533305 +0000 UTC m=+5.999786560 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/47c2b5b9-456a-4528-8d83-7ce3269cc16e-metrics-certs") pod "network-metrics-daemon-mdrck" (UID: "47c2b5b9-456a-4528-8d83-7ce3269cc16e") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 09 19:55:19.953892 ip-10-0-136-183 kubenswrapper[2574]: E0909 19:55:19.953868    2574 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 09 19:55:19.953963 ip-10-0-136-183 kubenswrapper[2574]: E0909 19:55:19.953941    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c756f4c9-2360-4fff-a136-b6869b02977e-original-pull-secret podName:c756f4c9-2360-4fff-a136-b6869b02977e nodeName:}" failed. No retries permitted until 2026-09-09 19:55:21.953921356 +0000 UTC m=+6.000174603 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/c756f4c9-2360-4fff-a136-b6869b02977e-original-pull-secret") pod "global-pull-secret-syncer-gjc9x" (UID: "c756f4c9-2360-4fff-a136-b6869b02977e") : object "kube-system"/"original-pull-secret" not registered
Sep 09 19:55:20.054506 ip-10-0-136-183 kubenswrapper[2574]: I0909 19:55:20.054464    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zqd55\" (UniqueName: \"kubernetes.io/projected/54458a6d-921b-4637-addc-7c4ee839b592-kube-api-access-zqd55\") pod \"network-check-target-4czxv\" (UID: \"54458a6d-921b-4637-addc-7c4ee839b592\") " pod="openshift-network-diagnostics/network-check-target-4czxv"
Sep 09 19:55:20.054673 ip-10-0-136-183 kubenswrapper[2574]: E0909 19:55:20.054636    2574 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 09 19:55:20.054673 ip-10-0-136-183 kubenswrapper[2574]: E0909 19:55:20.054657    2574 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 09 19:55:20.054673 ip-10-0-136-183 kubenswrapper[2574]: E0909 19:55:20.054668    2574 projected.go:196] Error preparing data for projected volume kube-api-access-zqd55 for pod openshift-network-diagnostics/network-check-target-4czxv: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 09 19:55:20.054876 ip-10-0-136-183 kubenswrapper[2574]: E0909 19:55:20.054728    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/54458a6d-921b-4637-addc-7c4ee839b592-kube-api-access-zqd55 podName:54458a6d-921b-4637-addc-7c4ee839b592 nodeName:}" failed. No retries permitted until 2026-09-09 19:55:22.054709261 +0000 UTC m=+6.100962522 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-zqd55" (UniqueName: "kubernetes.io/projected/54458a6d-921b-4637-addc-7c4ee839b592-kube-api-access-zqd55") pod "network-check-target-4czxv" (UID: "54458a6d-921b-4637-addc-7c4ee839b592") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 09 19:55:20.364299 ip-10-0-136-183 kubenswrapper[2574]: I0909 19:55:20.364190    2574 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-09-08 19:50:18 +0000 UTC" deadline="2028-05-03 16:48:25.498095537 +0000 UTC"
Sep 09 19:55:20.364299 ip-10-0-136-183 kubenswrapper[2574]: I0909 19:55:20.364252    2574 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14444h53m5.133847684s"
Sep 09 19:55:20.488003 ip-10-0-136-183 kubenswrapper[2574]: I0909 19:55:20.487487    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-gjc9x"
Sep 09 19:55:20.488003 ip-10-0-136-183 kubenswrapper[2574]: E0909 19:55:20.487624    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-gjc9x" podUID="c756f4c9-2360-4fff-a136-b6869b02977e"
Sep 09 19:55:20.520901 ip-10-0-136-183 kubenswrapper[2574]: I0909 19:55:20.520849    2574 generic.go:356] "Generic (PLEG): container finished" podID="9de809fe92e7ce3d095a2762d4b6b77a" containerID="7748b3be12ae2ebe717f60e54f9b37237bf14209b7421fdf795115d3216c7755" exitCode=0
Sep 09 19:55:20.521059 ip-10-0-136-183 kubenswrapper[2574]: I0909 19:55:20.520924    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-183.ec2.internal" event={"ID":"9de809fe92e7ce3d095a2762d4b6b77a","Type":"ContainerDied","Data":"7748b3be12ae2ebe717f60e54f9b37237bf14209b7421fdf795115d3216c7755"}
Sep 09 19:55:20.540318 ip-10-0-136-183 kubenswrapper[2574]: I0909 19:55:20.540260    2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-136-183.ec2.internal" podStartSLOduration=3.54023775 podStartE2EDuration="3.54023775s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-09 19:55:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 19:55:19.512387909 +0000 UTC m=+3.558641172" watchObservedRunningTime="2026-09-09 19:55:20.54023775 +0000 UTC m=+4.586491013"
Sep 09 19:55:21.485929 ip-10-0-136-183 kubenswrapper[2574]: I0909 19:55:21.485842    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-4czxv"
Sep 09 19:55:21.486410 ip-10-0-136-183 kubenswrapper[2574]: E0909 19:55:21.485979    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-4czxv" podUID="54458a6d-921b-4637-addc-7c4ee839b592"
Sep 09 19:55:21.486410 ip-10-0-136-183 kubenswrapper[2574]: I0909 19:55:21.486373    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mdrck"
Sep 09 19:55:21.486526 ip-10-0-136-183 kubenswrapper[2574]: E0909 19:55:21.486480    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-mdrck" podUID="47c2b5b9-456a-4528-8d83-7ce3269cc16e"
Sep 09 19:55:21.968843 ip-10-0-136-183 kubenswrapper[2574]: I0909 19:55:21.968792    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/c756f4c9-2360-4fff-a136-b6869b02977e-original-pull-secret\") pod \"global-pull-secret-syncer-gjc9x\" (UID: \"c756f4c9-2360-4fff-a136-b6869b02977e\") " pod="kube-system/global-pull-secret-syncer-gjc9x"
Sep 09 19:55:21.969018 ip-10-0-136-183 kubenswrapper[2574]: I0909 19:55:21.968881    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/47c2b5b9-456a-4528-8d83-7ce3269cc16e-metrics-certs\") pod \"network-metrics-daemon-mdrck\" (UID: \"47c2b5b9-456a-4528-8d83-7ce3269cc16e\") " pod="openshift-multus/network-metrics-daemon-mdrck"
Sep 09 19:55:21.969018 ip-10-0-136-183 kubenswrapper[2574]: E0909 19:55:21.968929    2574 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 09 19:55:21.969018 ip-10-0-136-183 kubenswrapper[2574]: E0909 19:55:21.969004    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c756f4c9-2360-4fff-a136-b6869b02977e-original-pull-secret podName:c756f4c9-2360-4fff-a136-b6869b02977e nodeName:}" failed. No retries permitted until 2026-09-09 19:55:25.968985145 +0000 UTC m=+10.015238402 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/c756f4c9-2360-4fff-a136-b6869b02977e-original-pull-secret") pod "global-pull-secret-syncer-gjc9x" (UID: "c756f4c9-2360-4fff-a136-b6869b02977e") : object "kube-system"/"original-pull-secret" not registered
Sep 09 19:55:21.969198 ip-10-0-136-183 kubenswrapper[2574]: E0909 19:55:21.969035    2574 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 09 19:55:21.969198 ip-10-0-136-183 kubenswrapper[2574]: E0909 19:55:21.969085    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/47c2b5b9-456a-4528-8d83-7ce3269cc16e-metrics-certs podName:47c2b5b9-456a-4528-8d83-7ce3269cc16e nodeName:}" failed. No retries permitted until 2026-09-09 19:55:25.969069913 +0000 UTC m=+10.015323168 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/47c2b5b9-456a-4528-8d83-7ce3269cc16e-metrics-certs") pod "network-metrics-daemon-mdrck" (UID: "47c2b5b9-456a-4528-8d83-7ce3269cc16e") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 09 19:55:22.070375 ip-10-0-136-183 kubenswrapper[2574]: I0909 19:55:22.069741    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zqd55\" (UniqueName: \"kubernetes.io/projected/54458a6d-921b-4637-addc-7c4ee839b592-kube-api-access-zqd55\") pod \"network-check-target-4czxv\" (UID: \"54458a6d-921b-4637-addc-7c4ee839b592\") " pod="openshift-network-diagnostics/network-check-target-4czxv"
Sep 09 19:55:22.070375 ip-10-0-136-183 kubenswrapper[2574]: E0909 19:55:22.069947    2574 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 09 19:55:22.070375 ip-10-0-136-183 kubenswrapper[2574]: E0909 19:55:22.069965    2574 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 09 19:55:22.070375 ip-10-0-136-183 kubenswrapper[2574]: E0909 19:55:22.069977    2574 projected.go:196] Error preparing data for projected volume kube-api-access-zqd55 for pod openshift-network-diagnostics/network-check-target-4czxv: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 09 19:55:22.070375 ip-10-0-136-183 kubenswrapper[2574]: E0909 19:55:22.070035    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/54458a6d-921b-4637-addc-7c4ee839b592-kube-api-access-zqd55 podName:54458a6d-921b-4637-addc-7c4ee839b592 nodeName:}" failed. No retries permitted until 2026-09-09 19:55:26.070016133 +0000 UTC m=+10.116269389 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-zqd55" (UniqueName: "kubernetes.io/projected/54458a6d-921b-4637-addc-7c4ee839b592-kube-api-access-zqd55") pod "network-check-target-4czxv" (UID: "54458a6d-921b-4637-addc-7c4ee839b592") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 09 19:55:22.488265 ip-10-0-136-183 kubenswrapper[2574]: I0909 19:55:22.487740    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-gjc9x"
Sep 09 19:55:22.488265 ip-10-0-136-183 kubenswrapper[2574]: E0909 19:55:22.487890    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-gjc9x" podUID="c756f4c9-2360-4fff-a136-b6869b02977e"
Sep 09 19:55:23.486110 ip-10-0-136-183 kubenswrapper[2574]: I0909 19:55:23.486077    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-4czxv"
Sep 09 19:55:23.486266 ip-10-0-136-183 kubenswrapper[2574]: E0909 19:55:23.486205    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-4czxv" podUID="54458a6d-921b-4637-addc-7c4ee839b592"
Sep 09 19:55:23.486307 ip-10-0-136-183 kubenswrapper[2574]: I0909 19:55:23.486277    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mdrck"
Sep 09 19:55:23.486378 ip-10-0-136-183 kubenswrapper[2574]: E0909 19:55:23.486358    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-mdrck" podUID="47c2b5b9-456a-4528-8d83-7ce3269cc16e"
Sep 09 19:55:24.486095 ip-10-0-136-183 kubenswrapper[2574]: I0909 19:55:24.486060    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-gjc9x"
Sep 09 19:55:24.486532 ip-10-0-136-183 kubenswrapper[2574]: E0909 19:55:24.486203    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-gjc9x" podUID="c756f4c9-2360-4fff-a136-b6869b02977e"
Sep 09 19:55:25.485830 ip-10-0-136-183 kubenswrapper[2574]: I0909 19:55:25.485772    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-4czxv"
Sep 09 19:55:25.485830 ip-10-0-136-183 kubenswrapper[2574]: I0909 19:55:25.485826    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mdrck"
Sep 09 19:55:25.486057 ip-10-0-136-183 kubenswrapper[2574]: E0909 19:55:25.485924    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-4czxv" podUID="54458a6d-921b-4637-addc-7c4ee839b592"
Sep 09 19:55:25.486335 ip-10-0-136-183 kubenswrapper[2574]: E0909 19:55:25.486273    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-mdrck" podUID="47c2b5b9-456a-4528-8d83-7ce3269cc16e"
Sep 09 19:55:26.005097 ip-10-0-136-183 kubenswrapper[2574]: I0909 19:55:26.004478    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/47c2b5b9-456a-4528-8d83-7ce3269cc16e-metrics-certs\") pod \"network-metrics-daemon-mdrck\" (UID: \"47c2b5b9-456a-4528-8d83-7ce3269cc16e\") " pod="openshift-multus/network-metrics-daemon-mdrck"
Sep 09 19:55:26.005097 ip-10-0-136-183 kubenswrapper[2574]: I0909 19:55:26.004568    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/c756f4c9-2360-4fff-a136-b6869b02977e-original-pull-secret\") pod \"global-pull-secret-syncer-gjc9x\" (UID: \"c756f4c9-2360-4fff-a136-b6869b02977e\") " pod="kube-system/global-pull-secret-syncer-gjc9x"
Sep 09 19:55:26.005097 ip-10-0-136-183 kubenswrapper[2574]: E0909 19:55:26.004688    2574 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 09 19:55:26.005097 ip-10-0-136-183 kubenswrapper[2574]: E0909 19:55:26.004752    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c756f4c9-2360-4fff-a136-b6869b02977e-original-pull-secret podName:c756f4c9-2360-4fff-a136-b6869b02977e nodeName:}" failed. No retries permitted until 2026-09-09 19:55:34.004732814 +0000 UTC m=+18.050986070 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/c756f4c9-2360-4fff-a136-b6869b02977e-original-pull-secret") pod "global-pull-secret-syncer-gjc9x" (UID: "c756f4c9-2360-4fff-a136-b6869b02977e") : object "kube-system"/"original-pull-secret" not registered
Sep 09 19:55:26.005458 ip-10-0-136-183 kubenswrapper[2574]: E0909 19:55:26.005178    2574 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 09 19:55:26.005458 ip-10-0-136-183 kubenswrapper[2574]: E0909 19:55:26.005309    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/47c2b5b9-456a-4528-8d83-7ce3269cc16e-metrics-certs podName:47c2b5b9-456a-4528-8d83-7ce3269cc16e nodeName:}" failed. No retries permitted until 2026-09-09 19:55:34.005230542 +0000 UTC m=+18.051483787 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/47c2b5b9-456a-4528-8d83-7ce3269cc16e-metrics-certs") pod "network-metrics-daemon-mdrck" (UID: "47c2b5b9-456a-4528-8d83-7ce3269cc16e") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 09 19:55:26.105188 ip-10-0-136-183 kubenswrapper[2574]: I0909 19:55:26.105151    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zqd55\" (UniqueName: \"kubernetes.io/projected/54458a6d-921b-4637-addc-7c4ee839b592-kube-api-access-zqd55\") pod \"network-check-target-4czxv\" (UID: \"54458a6d-921b-4637-addc-7c4ee839b592\") " pod="openshift-network-diagnostics/network-check-target-4czxv"
Sep 09 19:55:26.105367 ip-10-0-136-183 kubenswrapper[2574]: E0909 19:55:26.105321    2574 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 09 19:55:26.105367 ip-10-0-136-183 kubenswrapper[2574]: E0909 19:55:26.105339    2574 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 09 19:55:26.105367 ip-10-0-136-183 kubenswrapper[2574]: E0909 19:55:26.105352    2574 projected.go:196] Error preparing data for projected volume kube-api-access-zqd55 for pod openshift-network-diagnostics/network-check-target-4czxv: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 09 19:55:26.105531 ip-10-0-136-183 kubenswrapper[2574]: E0909 19:55:26.105410    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/54458a6d-921b-4637-addc-7c4ee839b592-kube-api-access-zqd55 podName:54458a6d-921b-4637-addc-7c4ee839b592 nodeName:}" failed. No retries permitted until 2026-09-09 19:55:34.105391113 +0000 UTC m=+18.151644376 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-zqd55" (UniqueName: "kubernetes.io/projected/54458a6d-921b-4637-addc-7c4ee839b592-kube-api-access-zqd55") pod "network-check-target-4czxv" (UID: "54458a6d-921b-4637-addc-7c4ee839b592") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 09 19:55:26.486893 ip-10-0-136-183 kubenswrapper[2574]: I0909 19:55:26.486411    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-gjc9x"
Sep 09 19:55:26.486893 ip-10-0-136-183 kubenswrapper[2574]: E0909 19:55:26.486526    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-gjc9x" podUID="c756f4c9-2360-4fff-a136-b6869b02977e"
Sep 09 19:55:27.485645 ip-10-0-136-183 kubenswrapper[2574]: I0909 19:55:27.485604    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-4czxv"
Sep 09 19:55:27.485870 ip-10-0-136-183 kubenswrapper[2574]: E0909 19:55:27.485742    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-4czxv" podUID="54458a6d-921b-4637-addc-7c4ee839b592"
Sep 09 19:55:27.485870 ip-10-0-136-183 kubenswrapper[2574]: I0909 19:55:27.485818    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mdrck"
Sep 09 19:55:27.486006 ip-10-0-136-183 kubenswrapper[2574]: E0909 19:55:27.485942    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-mdrck" podUID="47c2b5b9-456a-4528-8d83-7ce3269cc16e"
Sep 09 19:55:28.485456 ip-10-0-136-183 kubenswrapper[2574]: I0909 19:55:28.485416    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-gjc9x"
Sep 09 19:55:28.485940 ip-10-0-136-183 kubenswrapper[2574]: E0909 19:55:28.485560    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-gjc9x" podUID="c756f4c9-2360-4fff-a136-b6869b02977e"
Sep 09 19:55:29.485177 ip-10-0-136-183 kubenswrapper[2574]: I0909 19:55:29.485129    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-4czxv"
Sep 09 19:55:29.485364 ip-10-0-136-183 kubenswrapper[2574]: I0909 19:55:29.485184    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mdrck"
Sep 09 19:55:29.485364 ip-10-0-136-183 kubenswrapper[2574]: E0909 19:55:29.485258    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-4czxv" podUID="54458a6d-921b-4637-addc-7c4ee839b592"
Sep 09 19:55:29.485364 ip-10-0-136-183 kubenswrapper[2574]: E0909 19:55:29.485305    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-mdrck" podUID="47c2b5b9-456a-4528-8d83-7ce3269cc16e"
Sep 09 19:55:30.485473 ip-10-0-136-183 kubenswrapper[2574]: I0909 19:55:30.485439    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-gjc9x"
Sep 09 19:55:30.485921 ip-10-0-136-183 kubenswrapper[2574]: E0909 19:55:30.485564    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-gjc9x" podUID="c756f4c9-2360-4fff-a136-b6869b02977e"
Sep 09 19:55:31.485365 ip-10-0-136-183 kubenswrapper[2574]: I0909 19:55:31.485325    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-4czxv"
Sep 09 19:55:31.485555 ip-10-0-136-183 kubenswrapper[2574]: I0909 19:55:31.485341    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mdrck"
Sep 09 19:55:31.485555 ip-10-0-136-183 kubenswrapper[2574]: E0909 19:55:31.485446    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-4czxv" podUID="54458a6d-921b-4637-addc-7c4ee839b592"
Sep 09 19:55:31.485555 ip-10-0-136-183 kubenswrapper[2574]: E0909 19:55:31.485515    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-mdrck" podUID="47c2b5b9-456a-4528-8d83-7ce3269cc16e"
Sep 09 19:55:32.485926 ip-10-0-136-183 kubenswrapper[2574]: I0909 19:55:32.485895    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-gjc9x"
Sep 09 19:55:32.486283 ip-10-0-136-183 kubenswrapper[2574]: E0909 19:55:32.486034    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-gjc9x" podUID="c756f4c9-2360-4fff-a136-b6869b02977e"
Sep 09 19:55:33.485738 ip-10-0-136-183 kubenswrapper[2574]: I0909 19:55:33.485706    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-4czxv"
Sep 09 19:55:33.485991 ip-10-0-136-183 kubenswrapper[2574]: E0909 19:55:33.485833    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-4czxv" podUID="54458a6d-921b-4637-addc-7c4ee839b592"
Sep 09 19:55:33.485991 ip-10-0-136-183 kubenswrapper[2574]: I0909 19:55:33.485891    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mdrck"
Sep 09 19:55:33.486396 ip-10-0-136-183 kubenswrapper[2574]: E0909 19:55:33.485995    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-mdrck" podUID="47c2b5b9-456a-4528-8d83-7ce3269cc16e"
Sep 09 19:55:34.066590 ip-10-0-136-183 kubenswrapper[2574]: I0909 19:55:34.066555    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/47c2b5b9-456a-4528-8d83-7ce3269cc16e-metrics-certs\") pod \"network-metrics-daemon-mdrck\" (UID: \"47c2b5b9-456a-4528-8d83-7ce3269cc16e\") " pod="openshift-multus/network-metrics-daemon-mdrck"
Sep 09 19:55:34.066782 ip-10-0-136-183 kubenswrapper[2574]: I0909 19:55:34.066646    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/c756f4c9-2360-4fff-a136-b6869b02977e-original-pull-secret\") pod \"global-pull-secret-syncer-gjc9x\" (UID: \"c756f4c9-2360-4fff-a136-b6869b02977e\") " pod="kube-system/global-pull-secret-syncer-gjc9x"
Sep 09 19:55:34.066782 ip-10-0-136-183 kubenswrapper[2574]: E0909 19:55:34.066721    2574 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 09 19:55:34.066914 ip-10-0-136-183 kubenswrapper[2574]: E0909 19:55:34.066810    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/47c2b5b9-456a-4528-8d83-7ce3269cc16e-metrics-certs podName:47c2b5b9-456a-4528-8d83-7ce3269cc16e nodeName:}" failed. No retries permitted until 2026-09-09 19:55:50.066772761 +0000 UTC m=+34.113026016 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/47c2b5b9-456a-4528-8d83-7ce3269cc16e-metrics-certs") pod "network-metrics-daemon-mdrck" (UID: "47c2b5b9-456a-4528-8d83-7ce3269cc16e") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 09 19:55:34.066914 ip-10-0-136-183 kubenswrapper[2574]: E0909 19:55:34.066721    2574 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 09 19:55:34.066914 ip-10-0-136-183 kubenswrapper[2574]: E0909 19:55:34.066891    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c756f4c9-2360-4fff-a136-b6869b02977e-original-pull-secret podName:c756f4c9-2360-4fff-a136-b6869b02977e nodeName:}" failed. No retries permitted until 2026-09-09 19:55:50.066868397 +0000 UTC m=+34.113121653 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/c756f4c9-2360-4fff-a136-b6869b02977e-original-pull-secret") pod "global-pull-secret-syncer-gjc9x" (UID: "c756f4c9-2360-4fff-a136-b6869b02977e") : object "kube-system"/"original-pull-secret" not registered
Sep 09 19:55:34.167971 ip-10-0-136-183 kubenswrapper[2574]: I0909 19:55:34.167937    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zqd55\" (UniqueName: \"kubernetes.io/projected/54458a6d-921b-4637-addc-7c4ee839b592-kube-api-access-zqd55\") pod \"network-check-target-4czxv\" (UID: \"54458a6d-921b-4637-addc-7c4ee839b592\") " pod="openshift-network-diagnostics/network-check-target-4czxv"
Sep 09 19:55:34.168149 ip-10-0-136-183 kubenswrapper[2574]: E0909 19:55:34.168088    2574 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 09 19:55:34.168149 ip-10-0-136-183 kubenswrapper[2574]: E0909 19:55:34.168110    2574 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 09 19:55:34.168149 ip-10-0-136-183 kubenswrapper[2574]: E0909 19:55:34.168121    2574 projected.go:196] Error preparing data for projected volume kube-api-access-zqd55 for pod openshift-network-diagnostics/network-check-target-4czxv: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 09 19:55:34.168300 ip-10-0-136-183 kubenswrapper[2574]: E0909 19:55:34.168178    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/54458a6d-921b-4637-addc-7c4ee839b592-kube-api-access-zqd55 podName:54458a6d-921b-4637-addc-7c4ee839b592 nodeName:}" failed. No retries permitted until 2026-09-09 19:55:50.168161246 +0000 UTC m=+34.214414503 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-zqd55" (UniqueName: "kubernetes.io/projected/54458a6d-921b-4637-addc-7c4ee839b592-kube-api-access-zqd55") pod "network-check-target-4czxv" (UID: "54458a6d-921b-4637-addc-7c4ee839b592") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 09 19:55:34.485474 ip-10-0-136-183 kubenswrapper[2574]: I0909 19:55:34.485433    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-gjc9x"
Sep 09 19:55:34.485660 ip-10-0-136-183 kubenswrapper[2574]: E0909 19:55:34.485554    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-gjc9x" podUID="c756f4c9-2360-4fff-a136-b6869b02977e"
Sep 09 19:55:35.485218 ip-10-0-136-183 kubenswrapper[2574]: I0909 19:55:35.485188    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mdrck"
Sep 09 19:55:35.485645 ip-10-0-136-183 kubenswrapper[2574]: E0909 19:55:35.485314    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-mdrck" podUID="47c2b5b9-456a-4528-8d83-7ce3269cc16e"
Sep 09 19:55:35.485645 ip-10-0-136-183 kubenswrapper[2574]: I0909 19:55:35.485352    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-4czxv"
Sep 09 19:55:35.485645 ip-10-0-136-183 kubenswrapper[2574]: E0909 19:55:35.485455    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-4czxv" podUID="54458a6d-921b-4637-addc-7c4ee839b592"
Sep 09 19:55:36.486196 ip-10-0-136-183 kubenswrapper[2574]: I0909 19:55:36.485885    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-gjc9x"
Sep 09 19:55:36.486662 ip-10-0-136-183 kubenswrapper[2574]: E0909 19:55:36.486295    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-gjc9x" podUID="c756f4c9-2360-4fff-a136-b6869b02977e"
Sep 09 19:55:36.555188 ip-10-0-136-183 kubenswrapper[2574]: I0909 19:55:36.555031    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-mgg8m" event={"ID":"39099fe9-527d-4802-b40b-1bf535221165","Type":"ContainerStarted","Data":"4fc364550fe4e57c8d09e530da749306453092af42955e7c0ada804ad3f27b76"}
Sep 09 19:55:36.557110 ip-10-0-136-183 kubenswrapper[2574]: I0909 19:55:36.557081    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bvxbg" event={"ID":"71b40e5b-d6b9-489f-bb2e-4b5e1e5915e5","Type":"ContainerStarted","Data":"55c274326e7e176e7a8be2fced79c6ee7a89f680da67a487bc9fda6a38b93a1f"}
Sep 09 19:55:36.558759 ip-10-0-136-183 kubenswrapper[2574]: I0909 19:55:36.558729    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-svtdz" event={"ID":"b06375df-d12d-40a3-a5a7-18fa1420f388","Type":"ContainerStarted","Data":"24d62a23a41302d24c5d98ed529557aa552cbf3818348d8fe0d9ca7f3d1f7b10"}
Sep 09 19:55:36.560696 ip-10-0-136-183 kubenswrapper[2574]: I0909 19:55:36.560673    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-183.ec2.internal" event={"ID":"9de809fe92e7ce3d095a2762d4b6b77a","Type":"ContainerStarted","Data":"c85e8e7bc460b969a65aee25b668d52bc2356848307841cbcff07db4b3bc244e"}
Sep 09 19:55:36.563223 ip-10-0-136-183 kubenswrapper[2574]: I0909 19:55:36.563198    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-jcr9m" event={"ID":"2eb82961-9118-4b08-9d97-8d5cbac8cac3","Type":"ContainerStarted","Data":"16427ade3acef35615496d30a453bde73de4937709726c985c87105abe87e372"}
Sep 09 19:55:36.563322 ip-10-0-136-183 kubenswrapper[2574]: I0909 19:55:36.563287    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-jcr9m" event={"ID":"2eb82961-9118-4b08-9d97-8d5cbac8cac3","Type":"ContainerStarted","Data":"5af9cc07710e9cc21e9ed898c6b491cd46c804c2dd359fc79379c8b50314aac1"}
Sep 09 19:55:36.563379 ip-10-0-136-183 kubenswrapper[2574]: I0909 19:55:36.563350    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-jcr9m" event={"ID":"2eb82961-9118-4b08-9d97-8d5cbac8cac3","Type":"ContainerStarted","Data":"06b5f1120b0443e0a4cf939da15c81c01103c9006c97e33f28160b45e714cb78"}
Sep 09 19:55:36.563444 ip-10-0-136-183 kubenswrapper[2574]: I0909 19:55:36.563381    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-jcr9m" event={"ID":"2eb82961-9118-4b08-9d97-8d5cbac8cac3","Type":"ContainerStarted","Data":"3f68f6c874b6f970b483a2dcd4c4b62ca993af968324e461a576772d3191bb78"}
Sep 09 19:55:36.564895 ip-10-0-136-183 kubenswrapper[2574]: I0909 19:55:36.564867    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-sbx29" event={"ID":"9c19e4c1-5ffa-4fb8-88f0-93f1b32a591b","Type":"ContainerStarted","Data":"1a193d1c5a5539d2cfe8dfc84dcd56773e8db4bf4c1a85f2969a3eed058d8909"}
Sep 09 19:55:36.566309 ip-10-0-136-183 kubenswrapper[2574]: I0909 19:55:36.566283    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-d9ww4" event={"ID":"2d0e0ff3-612a-4eb3-b2df-342af115b6c3","Type":"ContainerStarted","Data":"9559e520896dbc0b7aa52292ffa9c09274a4620e551cc80bc08c0c6d6294ab93"}
Sep 09 19:55:36.567675 ip-10-0-136-183 kubenswrapper[2574]: I0909 19:55:36.567656    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-q8krg" event={"ID":"496a3333-1b9c-4eb2-8896-b6586eaf6877","Type":"ContainerStarted","Data":"874edbddf55b06f9088abea7425506d8b5fd86d2a2e028d92c9cb836951e4846"}
Sep 09 19:55:36.569120 ip-10-0-136-183 kubenswrapper[2574]: I0909 19:55:36.569099    2574 generic.go:356] "Generic (PLEG): container finished" podID="bed9028e-2a22-4c8f-a637-bb5ad9204adc" containerID="8daaa342425fe44d66d8a46e468d602c4ef5ccf1c167cb736a6714962d980b62" exitCode=0
Sep 09 19:55:36.569204 ip-10-0-136-183 kubenswrapper[2574]: I0909 19:55:36.569127    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-gzqh4" event={"ID":"bed9028e-2a22-4c8f-a637-bb5ad9204adc","Type":"ContainerDied","Data":"8daaa342425fe44d66d8a46e468d602c4ef5ccf1c167cb736a6714962d980b62"}
Sep 09 19:55:36.572358 ip-10-0-136-183 kubenswrapper[2574]: I0909 19:55:36.572322    2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-mgg8m" podStartSLOduration=3.819672335 podStartE2EDuration="20.572312289s" totalImagesPullingTime="16.752639954s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-09 19:55:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 19:55:36.571722353 +0000 UTC m=+20.617975615" watchObservedRunningTime="2026-09-09 19:55:36.572312289 +0000 UTC m=+20.618565551"
Sep 09 19:55:36.602106 ip-10-0-136-183 kubenswrapper[2574]: I0909 19:55:36.602070    2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-svtdz" podStartSLOduration=3.843507876 podStartE2EDuration="20.602056231s" totalImagesPullingTime="16.758548355s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-09 19:55:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 19:55:36.587712394 +0000 UTC m=+20.633965657" watchObservedRunningTime="2026-09-09 19:55:36.602056231 +0000 UTC m=+20.648309492"
Sep 09 19:55:36.602194 ip-10-0-136-183 kubenswrapper[2574]: I0909 19:55:36.602154    2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-q8krg" podStartSLOduration=3.851976348 podStartE2EDuration="20.60214879s" totalImagesPullingTime="16.750172442s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-09 19:55:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 19:55:36.601920235 +0000 UTC m=+20.648173498" watchObservedRunningTime="2026-09-09 19:55:36.60214879 +0000 UTC m=+20.648402052"
Sep 09 19:55:36.638400 ip-10-0-136-183 kubenswrapper[2574]: I0909 19:55:36.638357    2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-d9ww4" podStartSLOduration=10.855812137000001 podStartE2EDuration="20.638341607s" totalImagesPullingTime="9.78252947s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 19:55:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 19:55:36.638239295 +0000 UTC m=+20.684492557" watchObservedRunningTime="2026-09-09 19:55:36.638341607 +0000 UTC m=+20.684594868"
Sep 09 19:55:36.659220 ip-10-0-136-183 kubenswrapper[2574]: I0909 19:55:36.659166    2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-183.ec2.internal" podStartSLOduration=19.659152482 podStartE2EDuration="19.659152482s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-09 19:55:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 19:55:36.652486545 +0000 UTC m=+20.698739807" watchObservedRunningTime="2026-09-09 19:55:36.659152482 +0000 UTC m=+20.705405744"
Sep 09 19:55:36.672352 ip-10-0-136-183 kubenswrapper[2574]: I0909 19:55:36.672315    2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-sbx29" podStartSLOduration=3.839227719 podStartE2EDuration="20.672300228s" totalImagesPullingTime="16.833072509s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-09 19:55:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 19:55:36.672181448 +0000 UTC m=+20.718434710" watchObservedRunningTime="2026-09-09 19:55:36.672300228 +0000 UTC m=+20.718553489"
Sep 09 19:55:37.414134 ip-10-0-136-183 kubenswrapper[2574]: I0909 19:55:37.413995    2574 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Sep 09 19:55:37.485233 ip-10-0-136-183 kubenswrapper[2574]: I0909 19:55:37.485200    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-4czxv"
Sep 09 19:55:37.485370 ip-10-0-136-183 kubenswrapper[2574]: I0909 19:55:37.485200    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mdrck"
Sep 09 19:55:37.485370 ip-10-0-136-183 kubenswrapper[2574]: E0909 19:55:37.485310    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-4czxv" podUID="54458a6d-921b-4637-addc-7c4ee839b592"
Sep 09 19:55:37.485456 ip-10-0-136-183 kubenswrapper[2574]: E0909 19:55:37.485381    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-mdrck" podUID="47c2b5b9-456a-4528-8d83-7ce3269cc16e"
Sep 09 19:55:37.574584 ip-10-0-136-183 kubenswrapper[2574]: I0909 19:55:37.574547    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-jcr9m" event={"ID":"2eb82961-9118-4b08-9d97-8d5cbac8cac3","Type":"ContainerStarted","Data":"1b94f2acee93f5d98f09a19b690fdd7bad31ff2e2a7d7b0bf52b53efb80ab487"}
Sep 09 19:55:37.575109 ip-10-0-136-183 kubenswrapper[2574]: I0909 19:55:37.574637    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-jcr9m" event={"ID":"2eb82961-9118-4b08-9d97-8d5cbac8cac3","Type":"ContainerStarted","Data":"c435959009fdc80115e122faa554f2db10a97743e0f512748a3147c38951b60a"}
Sep 09 19:55:37.576294 ip-10-0-136-183 kubenswrapper[2574]: I0909 19:55:37.576273    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bvxbg" event={"ID":"71b40e5b-d6b9-489f-bb2e-4b5e1e5915e5","Type":"ContainerStarted","Data":"459d3f7baaa11cbd62d10e6505f6d7407d55c0d629135b54b691a4c544053614"}
Sep 09 19:55:37.577698 ip-10-0-136-183 kubenswrapper[2574]: I0909 19:55:37.577675    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-gvsgq" event={"ID":"1f0e15c0-0c09-4135-baf4-767e800917d5","Type":"ContainerStarted","Data":"b6a63fa1f5bbbad32b3d974ea210a0e92dd6112e96aa5b6a461d17a1850cf23d"}
Sep 09 19:55:37.595216 ip-10-0-136-183 kubenswrapper[2574]: I0909 19:55:37.595175    2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-gvsgq" podStartSLOduration=4.834176841 podStartE2EDuration="21.595163268s" totalImagesPullingTime="16.760986427s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-09 19:55:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 19:55:37.595011774 +0000 UTC m=+21.641265036" watchObservedRunningTime="2026-09-09 19:55:37.595163268 +0000 UTC m=+21.641416530"
Sep 09 19:55:38.384903 ip-10-0-136-183 kubenswrapper[2574]: I0909 19:55:38.384787    2574 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-09-09T19:55:37.414130513Z","UUID":"2bf828ef-c191-40b7-8db4-d5fbc2ecbd21","Handler":null,"Name":"","Endpoint":""}
Sep 09 19:55:38.386548 ip-10-0-136-183 kubenswrapper[2574]: I0909 19:55:38.386527    2574 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Sep 09 19:55:38.386651 ip-10-0-136-183 kubenswrapper[2574]: I0909 19:55:38.386556    2574 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Sep 09 19:55:38.485681 ip-10-0-136-183 kubenswrapper[2574]: I0909 19:55:38.485591    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-gjc9x"
Sep 09 19:55:38.485839 ip-10-0-136-183 kubenswrapper[2574]: E0909 19:55:38.485704    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-gjc9x" podUID="c756f4c9-2360-4fff-a136-b6869b02977e"
Sep 09 19:55:38.581388 ip-10-0-136-183 kubenswrapper[2574]: I0909 19:55:38.581357    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bvxbg" event={"ID":"71b40e5b-d6b9-489f-bb2e-4b5e1e5915e5","Type":"ContainerStarted","Data":"fc8ce2f9b3fe182e1244ba0b9403782c79e871aa1f7d998220690cffa33a583c"}
Sep 09 19:55:38.598347 ip-10-0-136-183 kubenswrapper[2574]: I0909 19:55:38.598308    2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bvxbg" podStartSLOduration=4.102681215 podStartE2EDuration="22.598289986s" totalImagesPullingTime="18.495608771s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-09 19:55:16 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 19:55:38.598146892 +0000 UTC m=+22.644400155" watchObservedRunningTime="2026-09-09 19:55:38.598289986 +0000 UTC m=+22.644543251"
Sep 09 19:55:39.485039 ip-10-0-136-183 kubenswrapper[2574]: I0909 19:55:39.485000    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mdrck"
Sep 09 19:55:39.485217 ip-10-0-136-183 kubenswrapper[2574]: I0909 19:55:39.485000    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-4czxv"
Sep 09 19:55:39.485217 ip-10-0-136-183 kubenswrapper[2574]: E0909 19:55:39.485145    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-mdrck" podUID="47c2b5b9-456a-4528-8d83-7ce3269cc16e"
Sep 09 19:55:39.485361 ip-10-0-136-183 kubenswrapper[2574]: E0909 19:55:39.485209    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-4czxv" podUID="54458a6d-921b-4637-addc-7c4ee839b592"
Sep 09 19:55:39.568259 ip-10-0-136-183 kubenswrapper[2574]: I0909 19:55:39.568230    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-d9ww4"
Sep 09 19:55:39.586303 ip-10-0-136-183 kubenswrapper[2574]: I0909 19:55:39.586276    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-jcr9m" event={"ID":"2eb82961-9118-4b08-9d97-8d5cbac8cac3","Type":"ContainerStarted","Data":"f81f4359f46ad3a5087c57c6a28427a343606ad1ea648726bf0c36b568fbe93b"}
Sep 09 19:55:40.486072 ip-10-0-136-183 kubenswrapper[2574]: I0909 19:55:40.486015    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-gjc9x"
Sep 09 19:55:40.486300 ip-10-0-136-183 kubenswrapper[2574]: E0909 19:55:40.486143    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-gjc9x" podUID="c756f4c9-2360-4fff-a136-b6869b02977e"
Sep 09 19:55:40.758179 ip-10-0-136-183 kubenswrapper[2574]: I0909 19:55:40.758100    2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-d9ww4"
Sep 09 19:55:40.758650 ip-10-0-136-183 kubenswrapper[2574]: I0909 19:55:40.758632    2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-d9ww4"
Sep 09 19:55:41.485420 ip-10-0-136-183 kubenswrapper[2574]: I0909 19:55:41.485223    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mdrck"
Sep 09 19:55:41.485588 ip-10-0-136-183 kubenswrapper[2574]: I0909 19:55:41.485284    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-4czxv"
Sep 09 19:55:41.485588 ip-10-0-136-183 kubenswrapper[2574]: E0909 19:55:41.485535    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-mdrck" podUID="47c2b5b9-456a-4528-8d83-7ce3269cc16e"
Sep 09 19:55:41.485588 ip-10-0-136-183 kubenswrapper[2574]: E0909 19:55:41.485579    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-4czxv" podUID="54458a6d-921b-4637-addc-7c4ee839b592"
Sep 09 19:55:41.595678 ip-10-0-136-183 kubenswrapper[2574]: I0909 19:55:41.595637    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-jcr9m" event={"ID":"2eb82961-9118-4b08-9d97-8d5cbac8cac3","Type":"ContainerStarted","Data":"b5430c170d15b275d90f613ffaa6b2adf11ecc90a251384c0ccb28791b369f9f"}
Sep 09 19:55:41.596476 ip-10-0-136-183 kubenswrapper[2574]: I0909 19:55:41.596449    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-d9ww4"
Sep 09 19:55:41.623130 ip-10-0-136-183 kubenswrapper[2574]: I0909 19:55:41.623079    2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-jcr9m" podStartSLOduration=8.800573952 podStartE2EDuration="25.623061699s" totalImagesPullingTime="16.822487747s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-09 19:55:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 19:55:41.622787078 +0000 UTC m=+25.669040344" watchObservedRunningTime="2026-09-09 19:55:41.623061699 +0000 UTC m=+25.669314989"
Sep 09 19:55:42.249356 ip-10-0-136-183 kubenswrapper[2574]: I0909 19:55:42.249260    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-jcr9m"
Sep 09 19:55:42.485744 ip-10-0-136-183 kubenswrapper[2574]: I0909 19:55:42.485713    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-gjc9x"
Sep 09 19:55:42.485929 ip-10-0-136-183 kubenswrapper[2574]: E0909 19:55:42.485874    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-gjc9x" podUID="c756f4c9-2360-4fff-a136-b6869b02977e"
Sep 09 19:55:42.598865 ip-10-0-136-183 kubenswrapper[2574]: I0909 19:55:42.598833    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-jcr9m"
Sep 09 19:55:42.599019 ip-10-0-136-183 kubenswrapper[2574]: I0909 19:55:42.598892    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-jcr9m"
Sep 09 19:55:42.617816 ip-10-0-136-183 kubenswrapper[2574]: I0909 19:55:42.617774    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-jcr9m"
Sep 09 19:55:42.620602 ip-10-0-136-183 kubenswrapper[2574]: I0909 19:55:42.620580    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-jcr9m"
Sep 09 19:55:42.729742 ip-10-0-136-183 kubenswrapper[2574]: I0909 19:55:42.729713    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-gjc9x"]
Sep 09 19:55:42.729923 ip-10-0-136-183 kubenswrapper[2574]: I0909 19:55:42.729866    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-gjc9x"
Sep 09 19:55:42.730019 ip-10-0-136-183 kubenswrapper[2574]: E0909 19:55:42.729992    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-gjc9x" podUID="c756f4c9-2360-4fff-a136-b6869b02977e"
Sep 09 19:55:42.738330 ip-10-0-136-183 kubenswrapper[2574]: I0909 19:55:42.738301    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-mdrck"]
Sep 09 19:55:42.738467 ip-10-0-136-183 kubenswrapper[2574]: I0909 19:55:42.738420    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mdrck"
Sep 09 19:55:42.738531 ip-10-0-136-183 kubenswrapper[2574]: E0909 19:55:42.738516    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-mdrck" podUID="47c2b5b9-456a-4528-8d83-7ce3269cc16e"
Sep 09 19:55:42.741728 ip-10-0-136-183 kubenswrapper[2574]: I0909 19:55:42.741704    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-4czxv"]
Sep 09 19:55:42.741871 ip-10-0-136-183 kubenswrapper[2574]: I0909 19:55:42.741848    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-4czxv"
Sep 09 19:55:42.741959 ip-10-0-136-183 kubenswrapper[2574]: E0909 19:55:42.741938    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-4czxv" podUID="54458a6d-921b-4637-addc-7c4ee839b592"
Sep 09 19:55:44.485160 ip-10-0-136-183 kubenswrapper[2574]: I0909 19:55:44.485129    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-gjc9x"
Sep 09 19:55:44.485697 ip-10-0-136-183 kubenswrapper[2574]: I0909 19:55:44.485135    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-4czxv"
Sep 09 19:55:44.485697 ip-10-0-136-183 kubenswrapper[2574]: E0909 19:55:44.485231    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-gjc9x" podUID="c756f4c9-2360-4fff-a136-b6869b02977e"
Sep 09 19:55:44.485697 ip-10-0-136-183 kubenswrapper[2574]: I0909 19:55:44.485137    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mdrck"
Sep 09 19:55:44.485697 ip-10-0-136-183 kubenswrapper[2574]: E0909 19:55:44.485306    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-4czxv" podUID="54458a6d-921b-4637-addc-7c4ee839b592"
Sep 09 19:55:44.485697 ip-10-0-136-183 kubenswrapper[2574]: E0909 19:55:44.485375    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-mdrck" podUID="47c2b5b9-456a-4528-8d83-7ce3269cc16e"
Sep 09 19:55:45.605936 ip-10-0-136-183 kubenswrapper[2574]: I0909 19:55:45.605728    2574 generic.go:356] "Generic (PLEG): container finished" podID="bed9028e-2a22-4c8f-a637-bb5ad9204adc" containerID="81c71b361a9628536c5ff5cb21c74449f61ca3195b0ebdd2546b4a28e578c6e1" exitCode=0
Sep 09 19:55:45.606558 ip-10-0-136-183 kubenswrapper[2574]: I0909 19:55:45.605824    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-gzqh4" event={"ID":"bed9028e-2a22-4c8f-a637-bb5ad9204adc","Type":"ContainerDied","Data":"81c71b361a9628536c5ff5cb21c74449f61ca3195b0ebdd2546b4a28e578c6e1"}
Sep 09 19:55:46.487255 ip-10-0-136-183 kubenswrapper[2574]: I0909 19:55:46.487173    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mdrck"
Sep 09 19:55:46.487415 ip-10-0-136-183 kubenswrapper[2574]: I0909 19:55:46.487280    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-4czxv"
Sep 09 19:55:46.487415 ip-10-0-136-183 kubenswrapper[2574]: I0909 19:55:46.487321    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-gjc9x"
Sep 09 19:55:46.487415 ip-10-0-136-183 kubenswrapper[2574]: E0909 19:55:46.487323    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-mdrck" podUID="47c2b5b9-456a-4528-8d83-7ce3269cc16e"
Sep 09 19:55:46.487415 ip-10-0-136-183 kubenswrapper[2574]: E0909 19:55:46.487399    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-gjc9x" podUID="c756f4c9-2360-4fff-a136-b6869b02977e"
Sep 09 19:55:46.487615 ip-10-0-136-183 kubenswrapper[2574]: E0909 19:55:46.487469    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-4czxv" podUID="54458a6d-921b-4637-addc-7c4ee839b592"
Sep 09 19:55:46.610292 ip-10-0-136-183 kubenswrapper[2574]: I0909 19:55:46.610260    2574 generic.go:356] "Generic (PLEG): container finished" podID="bed9028e-2a22-4c8f-a637-bb5ad9204adc" containerID="2bdcbe670ae52fb9e49bebd41deb0e394599f7d464255d9a833ee4baa63e74cd" exitCode=0
Sep 09 19:55:46.610730 ip-10-0-136-183 kubenswrapper[2574]: I0909 19:55:46.610317    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-gzqh4" event={"ID":"bed9028e-2a22-4c8f-a637-bb5ad9204adc","Type":"ContainerDied","Data":"2bdcbe670ae52fb9e49bebd41deb0e394599f7d464255d9a833ee4baa63e74cd"}
Sep 09 19:55:48.486083 ip-10-0-136-183 kubenswrapper[2574]: I0909 19:55:48.486050    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-gjc9x"
Sep 09 19:55:48.486652 ip-10-0-136-183 kubenswrapper[2574]: I0909 19:55:48.486088    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-4czxv"
Sep 09 19:55:48.486652 ip-10-0-136-183 kubenswrapper[2574]: I0909 19:55:48.486051    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mdrck"
Sep 09 19:55:48.486652 ip-10-0-136-183 kubenswrapper[2574]: E0909 19:55:48.486155    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-gjc9x" podUID="c756f4c9-2360-4fff-a136-b6869b02977e"
Sep 09 19:55:48.486652 ip-10-0-136-183 kubenswrapper[2574]: E0909 19:55:48.486275    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-mdrck" podUID="47c2b5b9-456a-4528-8d83-7ce3269cc16e"
Sep 09 19:55:48.486652 ip-10-0-136-183 kubenswrapper[2574]: E0909 19:55:48.486369    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-4czxv" podUID="54458a6d-921b-4637-addc-7c4ee839b592"
Sep 09 19:55:48.689336 ip-10-0-136-183 kubenswrapper[2574]: I0909 19:55:48.689309    2574 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-183.ec2.internal" event="NodeReady"
Sep 09 19:55:48.689495 ip-10-0-136-183 kubenswrapper[2574]: I0909 19:55:48.689455    2574 kubelet_node_status.go:489] "Fast updating node status as it just became ready"
Sep 09 19:55:48.734263 ip-10-0-136-183 kubenswrapper[2574]: I0909 19:55:48.734228    2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-4blgq"]
Sep 09 19:55:48.737102 ip-10-0-136-183 kubenswrapper[2574]: I0909 19:55:48.737037    2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-57cmp"]
Sep 09 19:55:48.737227 ip-10-0-136-183 kubenswrapper[2574]: I0909 19:55:48.737149    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-4blgq"
Sep 09 19:55:48.739106 ip-10-0-136-183 kubenswrapper[2574]: I0909 19:55:48.739085    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Sep 09 19:55:48.739227 ip-10-0-136-183 kubenswrapper[2574]: I0909 19:55:48.739088    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Sep 09 19:55:48.739227 ip-10-0-136-183 kubenswrapper[2574]: I0909 19:55:48.739095    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-28vq4\""
Sep 09 19:55:48.739648 ip-10-0-136-183 kubenswrapper[2574]: I0909 19:55:48.739632    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-57cmp"
Sep 09 19:55:48.741390 ip-10-0-136-183 kubenswrapper[2574]: I0909 19:55:48.741362    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Sep 09 19:55:48.741507 ip-10-0-136-183 kubenswrapper[2574]: I0909 19:55:48.741495    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Sep 09 19:55:48.741629 ip-10-0-136-183 kubenswrapper[2574]: I0909 19:55:48.741614    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Sep 09 19:55:48.741697 ip-10-0-136-183 kubenswrapper[2574]: I0909 19:55:48.741676    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-vnftv\""
Sep 09 19:55:48.751000 ip-10-0-136-183 kubenswrapper[2574]: I0909 19:55:48.750942    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-4blgq"]
Sep 09 19:55:48.751463 ip-10-0-136-183 kubenswrapper[2574]: I0909 19:55:48.751443    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-57cmp"]
Sep 09 19:55:48.878838 ip-10-0-136-183 kubenswrapper[2574]: I0909 19:55:48.878788    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/3a81bb5a-0683-466e-b32e-960f4ca22ae6-config-volume\") pod \"dns-default-4blgq\" (UID: \"3a81bb5a-0683-466e-b32e-960f4ca22ae6\") " pod="openshift-dns/dns-default-4blgq"
Sep 09 19:55:48.878838 ip-10-0-136-183 kubenswrapper[2574]: I0909 19:55:48.878838    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m2tdm\" (UniqueName: \"kubernetes.io/projected/3a81bb5a-0683-466e-b32e-960f4ca22ae6-kube-api-access-m2tdm\") pod \"dns-default-4blgq\" (UID: \"3a81bb5a-0683-466e-b32e-960f4ca22ae6\") " pod="openshift-dns/dns-default-4blgq"
Sep 09 19:55:48.879026 ip-10-0-136-183 kubenswrapper[2574]: I0909 19:55:48.878857    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/ceff0c1c-cb80-437c-a942-9ebe28442933-cert\") pod \"ingress-canary-57cmp\" (UID: \"ceff0c1c-cb80-437c-a942-9ebe28442933\") " pod="openshift-ingress-canary/ingress-canary-57cmp"
Sep 09 19:55:48.879026 ip-10-0-136-183 kubenswrapper[2574]: I0909 19:55:48.878878    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/3a81bb5a-0683-466e-b32e-960f4ca22ae6-metrics-tls\") pod \"dns-default-4blgq\" (UID: \"3a81bb5a-0683-466e-b32e-960f4ca22ae6\") " pod="openshift-dns/dns-default-4blgq"
Sep 09 19:55:48.879026 ip-10-0-136-183 kubenswrapper[2574]: I0909 19:55:48.878933    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kfvr4\" (UniqueName: \"kubernetes.io/projected/ceff0c1c-cb80-437c-a942-9ebe28442933-kube-api-access-kfvr4\") pod \"ingress-canary-57cmp\" (UID: \"ceff0c1c-cb80-437c-a942-9ebe28442933\") " pod="openshift-ingress-canary/ingress-canary-57cmp"
Sep 09 19:55:48.879026 ip-10-0-136-183 kubenswrapper[2574]: I0909 19:55:48.878954    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3a81bb5a-0683-466e-b32e-960f4ca22ae6-tmp-dir\") pod \"dns-default-4blgq\" (UID: \"3a81bb5a-0683-466e-b32e-960f4ca22ae6\") " pod="openshift-dns/dns-default-4blgq"
Sep 09 19:55:48.980087 ip-10-0-136-183 kubenswrapper[2574]: I0909 19:55:48.980050    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/3a81bb5a-0683-466e-b32e-960f4ca22ae6-metrics-tls\") pod \"dns-default-4blgq\" (UID: \"3a81bb5a-0683-466e-b32e-960f4ca22ae6\") " pod="openshift-dns/dns-default-4blgq"
Sep 09 19:55:48.980255 ip-10-0-136-183 kubenswrapper[2574]: I0909 19:55:48.980094    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kfvr4\" (UniqueName: \"kubernetes.io/projected/ceff0c1c-cb80-437c-a942-9ebe28442933-kube-api-access-kfvr4\") pod \"ingress-canary-57cmp\" (UID: \"ceff0c1c-cb80-437c-a942-9ebe28442933\") " pod="openshift-ingress-canary/ingress-canary-57cmp"
Sep 09 19:55:48.980255 ip-10-0-136-183 kubenswrapper[2574]: I0909 19:55:48.980112    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3a81bb5a-0683-466e-b32e-960f4ca22ae6-tmp-dir\") pod \"dns-default-4blgq\" (UID: \"3a81bb5a-0683-466e-b32e-960f4ca22ae6\") " pod="openshift-dns/dns-default-4blgq"
Sep 09 19:55:48.980255 ip-10-0-136-183 kubenswrapper[2574]: E0909 19:55:48.980199    2574 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 09 19:55:48.980421 ip-10-0-136-183 kubenswrapper[2574]: E0909 19:55:48.980257    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3a81bb5a-0683-466e-b32e-960f4ca22ae6-metrics-tls podName:3a81bb5a-0683-466e-b32e-960f4ca22ae6 nodeName:}" failed. No retries permitted until 2026-09-09 19:55:49.480240329 +0000 UTC m=+33.526493586 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/3a81bb5a-0683-466e-b32e-960f4ca22ae6-metrics-tls") pod "dns-default-4blgq" (UID: "3a81bb5a-0683-466e-b32e-960f4ca22ae6") : secret "dns-default-metrics-tls" not found
Sep 09 19:55:48.980421 ip-10-0-136-183 kubenswrapper[2574]: I0909 19:55:48.980287    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/3a81bb5a-0683-466e-b32e-960f4ca22ae6-config-volume\") pod \"dns-default-4blgq\" (UID: \"3a81bb5a-0683-466e-b32e-960f4ca22ae6\") " pod="openshift-dns/dns-default-4blgq"
Sep 09 19:55:48.980421 ip-10-0-136-183 kubenswrapper[2574]: I0909 19:55:48.980314    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m2tdm\" (UniqueName: \"kubernetes.io/projected/3a81bb5a-0683-466e-b32e-960f4ca22ae6-kube-api-access-m2tdm\") pod \"dns-default-4blgq\" (UID: \"3a81bb5a-0683-466e-b32e-960f4ca22ae6\") " pod="openshift-dns/dns-default-4blgq"
Sep 09 19:55:48.980421 ip-10-0-136-183 kubenswrapper[2574]: I0909 19:55:48.980330    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/ceff0c1c-cb80-437c-a942-9ebe28442933-cert\") pod \"ingress-canary-57cmp\" (UID: \"ceff0c1c-cb80-437c-a942-9ebe28442933\") " pod="openshift-ingress-canary/ingress-canary-57cmp"
Sep 09 19:55:48.980421 ip-10-0-136-183 kubenswrapper[2574]: I0909 19:55:48.980417    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3a81bb5a-0683-466e-b32e-960f4ca22ae6-tmp-dir\") pod \"dns-default-4blgq\" (UID: \"3a81bb5a-0683-466e-b32e-960f4ca22ae6\") " pod="openshift-dns/dns-default-4blgq"
Sep 09 19:55:48.980784 ip-10-0-136-183 kubenswrapper[2574]: I0909 19:55:48.980768    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/3a81bb5a-0683-466e-b32e-960f4ca22ae6-config-volume\") pod \"dns-default-4blgq\" (UID: \"3a81bb5a-0683-466e-b32e-960f4ca22ae6\") " pod="openshift-dns/dns-default-4blgq"
Sep 09 19:55:48.984874 ip-10-0-136-183 kubenswrapper[2574]: I0909 19:55:48.984849    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/ceff0c1c-cb80-437c-a942-9ebe28442933-cert\") pod \"ingress-canary-57cmp\" (UID: \"ceff0c1c-cb80-437c-a942-9ebe28442933\") " pod="openshift-ingress-canary/ingress-canary-57cmp"
Sep 09 19:55:48.989041 ip-10-0-136-183 kubenswrapper[2574]: I0909 19:55:48.988991    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-m2tdm\" (UniqueName: \"kubernetes.io/projected/3a81bb5a-0683-466e-b32e-960f4ca22ae6-kube-api-access-m2tdm\") pod \"dns-default-4blgq\" (UID: \"3a81bb5a-0683-466e-b32e-960f4ca22ae6\") " pod="openshift-dns/dns-default-4blgq"
Sep 09 19:55:48.989041 ip-10-0-136-183 kubenswrapper[2574]: I0909 19:55:48.988992    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kfvr4\" (UniqueName: \"kubernetes.io/projected/ceff0c1c-cb80-437c-a942-9ebe28442933-kube-api-access-kfvr4\") pod \"ingress-canary-57cmp\" (UID: \"ceff0c1c-cb80-437c-a942-9ebe28442933\") " pod="openshift-ingress-canary/ingress-canary-57cmp"
Sep 09 19:55:49.053318 ip-10-0-136-183 kubenswrapper[2574]: I0909 19:55:49.053285    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-57cmp"
Sep 09 19:55:49.330413 ip-10-0-136-183 kubenswrapper[2574]: I0909 19:55:49.330174    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-57cmp"]
Sep 09 19:55:49.334322 ip-10-0-136-183 kubenswrapper[2574]: W0909 19:55:49.334298    2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podceff0c1c_cb80_437c_a942_9ebe28442933.slice/crio-8e4ea3e6267b1cd5a3047ce01239173076e8d7cde72ccb03020cc1df8d2a450f WatchSource:0}: Error finding container 8e4ea3e6267b1cd5a3047ce01239173076e8d7cde72ccb03020cc1df8d2a450f: Status 404 returned error can't find the container with id 8e4ea3e6267b1cd5a3047ce01239173076e8d7cde72ccb03020cc1df8d2a450f
Sep 09 19:55:49.484791 ip-10-0-136-183 kubenswrapper[2574]: I0909 19:55:49.484759    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/3a81bb5a-0683-466e-b32e-960f4ca22ae6-metrics-tls\") pod \"dns-default-4blgq\" (UID: \"3a81bb5a-0683-466e-b32e-960f4ca22ae6\") " pod="openshift-dns/dns-default-4blgq"
Sep 09 19:55:49.484979 ip-10-0-136-183 kubenswrapper[2574]: E0909 19:55:49.484912    2574 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 09 19:55:49.484979 ip-10-0-136-183 kubenswrapper[2574]: E0909 19:55:49.484978    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3a81bb5a-0683-466e-b32e-960f4ca22ae6-metrics-tls podName:3a81bb5a-0683-466e-b32e-960f4ca22ae6 nodeName:}" failed. No retries permitted until 2026-09-09 19:55:50.484962013 +0000 UTC m=+34.531215257 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/3a81bb5a-0683-466e-b32e-960f4ca22ae6-metrics-tls") pod "dns-default-4blgq" (UID: "3a81bb5a-0683-466e-b32e-960f4ca22ae6") : secret "dns-default-metrics-tls" not found
Sep 09 19:55:49.619152 ip-10-0-136-183 kubenswrapper[2574]: I0909 19:55:49.619108    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-57cmp" event={"ID":"ceff0c1c-cb80-437c-a942-9ebe28442933","Type":"ContainerStarted","Data":"8e4ea3e6267b1cd5a3047ce01239173076e8d7cde72ccb03020cc1df8d2a450f"}
Sep 09 19:55:50.089867 ip-10-0-136-183 kubenswrapper[2574]: I0909 19:55:50.089717    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/47c2b5b9-456a-4528-8d83-7ce3269cc16e-metrics-certs\") pod \"network-metrics-daemon-mdrck\" (UID: \"47c2b5b9-456a-4528-8d83-7ce3269cc16e\") " pod="openshift-multus/network-metrics-daemon-mdrck"
Sep 09 19:55:50.089867 ip-10-0-136-183 kubenswrapper[2574]: I0909 19:55:50.089826    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/c756f4c9-2360-4fff-a136-b6869b02977e-original-pull-secret\") pod \"global-pull-secret-syncer-gjc9x\" (UID: \"c756f4c9-2360-4fff-a136-b6869b02977e\") " pod="kube-system/global-pull-secret-syncer-gjc9x"
Sep 09 19:55:50.090153 ip-10-0-136-183 kubenswrapper[2574]: E0909 19:55:50.089904    2574 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 09 19:55:50.090153 ip-10-0-136-183 kubenswrapper[2574]: E0909 19:55:50.089967    2574 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 09 19:55:50.090153 ip-10-0-136-183 kubenswrapper[2574]: E0909 19:55:50.089981    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/47c2b5b9-456a-4528-8d83-7ce3269cc16e-metrics-certs podName:47c2b5b9-456a-4528-8d83-7ce3269cc16e nodeName:}" failed. No retries permitted until 2026-09-09 19:56:22.089965421 +0000 UTC m=+66.136218666 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/47c2b5b9-456a-4528-8d83-7ce3269cc16e-metrics-certs") pod "network-metrics-daemon-mdrck" (UID: "47c2b5b9-456a-4528-8d83-7ce3269cc16e") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 09 19:55:50.090153 ip-10-0-136-183 kubenswrapper[2574]: E0909 19:55:50.090020    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c756f4c9-2360-4fff-a136-b6869b02977e-original-pull-secret podName:c756f4c9-2360-4fff-a136-b6869b02977e nodeName:}" failed. No retries permitted until 2026-09-09 19:56:22.090003702 +0000 UTC m=+66.136256957 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/c756f4c9-2360-4fff-a136-b6869b02977e-original-pull-secret") pod "global-pull-secret-syncer-gjc9x" (UID: "c756f4c9-2360-4fff-a136-b6869b02977e") : object "kube-system"/"original-pull-secret" not registered
Sep 09 19:55:50.190676 ip-10-0-136-183 kubenswrapper[2574]: I0909 19:55:50.190641    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zqd55\" (UniqueName: \"kubernetes.io/projected/54458a6d-921b-4637-addc-7c4ee839b592-kube-api-access-zqd55\") pod \"network-check-target-4czxv\" (UID: \"54458a6d-921b-4637-addc-7c4ee839b592\") " pod="openshift-network-diagnostics/network-check-target-4czxv"
Sep 09 19:55:50.190859 ip-10-0-136-183 kubenswrapper[2574]: E0909 19:55:50.190838    2574 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 09 19:55:50.190910 ip-10-0-136-183 kubenswrapper[2574]: E0909 19:55:50.190869    2574 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 09 19:55:50.190910 ip-10-0-136-183 kubenswrapper[2574]: E0909 19:55:50.190882    2574 projected.go:196] Error preparing data for projected volume kube-api-access-zqd55 for pod openshift-network-diagnostics/network-check-target-4czxv: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 09 19:55:50.191001 ip-10-0-136-183 kubenswrapper[2574]: E0909 19:55:50.190945    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/54458a6d-921b-4637-addc-7c4ee839b592-kube-api-access-zqd55 podName:54458a6d-921b-4637-addc-7c4ee839b592 nodeName:}" failed. No retries permitted until 2026-09-09 19:56:22.190926141 +0000 UTC m=+66.237179396 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-zqd55" (UniqueName: "kubernetes.io/projected/54458a6d-921b-4637-addc-7c4ee839b592-kube-api-access-zqd55") pod "network-check-target-4czxv" (UID: "54458a6d-921b-4637-addc-7c4ee839b592") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 09 19:55:50.485255 ip-10-0-136-183 kubenswrapper[2574]: I0909 19:55:50.485206    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mdrck"
Sep 09 19:55:50.485431 ip-10-0-136-183 kubenswrapper[2574]: I0909 19:55:50.485207    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-gjc9x"
Sep 09 19:55:50.485431 ip-10-0-136-183 kubenswrapper[2574]: I0909 19:55:50.485210    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-4czxv"
Sep 09 19:55:50.488829 ip-10-0-136-183 kubenswrapper[2574]: I0909 19:55:50.488514    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Sep 09 19:55:50.488939 ip-10-0-136-183 kubenswrapper[2574]: I0909 19:55:50.488891    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Sep 09 19:55:50.489219 ip-10-0-136-183 kubenswrapper[2574]: I0909 19:55:50.489189    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Sep 09 19:55:50.489503 ip-10-0-136-183 kubenswrapper[2574]: I0909 19:55:50.489485    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-q8crj\""
Sep 09 19:55:50.489624 ip-10-0-136-183 kubenswrapper[2574]: I0909 19:55:50.489600    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-cvgdp\""
Sep 09 19:55:50.491206 ip-10-0-136-183 kubenswrapper[2574]: I0909 19:55:50.490229    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Sep 09 19:55:50.493365 ip-10-0-136-183 kubenswrapper[2574]: I0909 19:55:50.493314    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/3a81bb5a-0683-466e-b32e-960f4ca22ae6-metrics-tls\") pod \"dns-default-4blgq\" (UID: \"3a81bb5a-0683-466e-b32e-960f4ca22ae6\") " pod="openshift-dns/dns-default-4blgq"
Sep 09 19:55:50.493461 ip-10-0-136-183 kubenswrapper[2574]: E0909 19:55:50.493446    2574 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 09 19:55:50.493522 ip-10-0-136-183 kubenswrapper[2574]: E0909 19:55:50.493515    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3a81bb5a-0683-466e-b32e-960f4ca22ae6-metrics-tls podName:3a81bb5a-0683-466e-b32e-960f4ca22ae6 nodeName:}" failed. No retries permitted until 2026-09-09 19:55:52.493495088 +0000 UTC m=+36.539748345 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/3a81bb5a-0683-466e-b32e-960f4ca22ae6-metrics-tls") pod "dns-default-4blgq" (UID: "3a81bb5a-0683-466e-b32e-960f4ca22ae6") : secret "dns-default-metrics-tls" not found
Sep 09 19:55:50.624793 ip-10-0-136-183 kubenswrapper[2574]: I0909 19:55:50.624749    2574 generic.go:356] "Generic (PLEG): container finished" podID="bed9028e-2a22-4c8f-a637-bb5ad9204adc" containerID="f75dc55738ef6306bfb663c855d50ee54c73020804ffcf27b5076bc3bdf291c3" exitCode=0
Sep 09 19:55:50.625330 ip-10-0-136-183 kubenswrapper[2574]: I0909 19:55:50.624822    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-gzqh4" event={"ID":"bed9028e-2a22-4c8f-a637-bb5ad9204adc","Type":"ContainerDied","Data":"f75dc55738ef6306bfb663c855d50ee54c73020804ffcf27b5076bc3bdf291c3"}
Sep 09 19:55:52.509292 ip-10-0-136-183 kubenswrapper[2574]: I0909 19:55:52.509259    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/3a81bb5a-0683-466e-b32e-960f4ca22ae6-metrics-tls\") pod \"dns-default-4blgq\" (UID: \"3a81bb5a-0683-466e-b32e-960f4ca22ae6\") " pod="openshift-dns/dns-default-4blgq"
Sep 09 19:55:52.509880 ip-10-0-136-183 kubenswrapper[2574]: E0909 19:55:52.509364    2574 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 09 19:55:52.509880 ip-10-0-136-183 kubenswrapper[2574]: E0909 19:55:52.509415    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3a81bb5a-0683-466e-b32e-960f4ca22ae6-metrics-tls podName:3a81bb5a-0683-466e-b32e-960f4ca22ae6 nodeName:}" failed. No retries permitted until 2026-09-09 19:55:56.509400555 +0000 UTC m=+40.555653808 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/3a81bb5a-0683-466e-b32e-960f4ca22ae6-metrics-tls") pod "dns-default-4blgq" (UID: "3a81bb5a-0683-466e-b32e-960f4ca22ae6") : secret "dns-default-metrics-tls" not found
Sep 09 19:55:52.630002 ip-10-0-136-183 kubenswrapper[2574]: I0909 19:55:52.629971    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-57cmp" event={"ID":"ceff0c1c-cb80-437c-a942-9ebe28442933","Type":"ContainerStarted","Data":"4fff15d98552ef3b7821e65d20f5b406f379e1a00cf747b1b502df471cb7ed58"}
Sep 09 19:55:52.645662 ip-10-0-136-183 kubenswrapper[2574]: I0909 19:55:52.645619    2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-57cmp" podStartSLOduration=2.19081335 podStartE2EDuration="4.645601932s" totalImagesPullingTime="2.454788582s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 19:55:48 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 19:55:52.644577627 +0000 UTC m=+36.690830892" watchObservedRunningTime="2026-09-09 19:55:52.645601932 +0000 UTC m=+36.691855198"
Sep 09 19:55:56.537967 ip-10-0-136-183 kubenswrapper[2574]: I0909 19:55:56.537925    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/3a81bb5a-0683-466e-b32e-960f4ca22ae6-metrics-tls\") pod \"dns-default-4blgq\" (UID: \"3a81bb5a-0683-466e-b32e-960f4ca22ae6\") " pod="openshift-dns/dns-default-4blgq"
Sep 09 19:55:56.538446 ip-10-0-136-183 kubenswrapper[2574]: E0909 19:55:56.538059    2574 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 09 19:55:56.538446 ip-10-0-136-183 kubenswrapper[2574]: E0909 19:55:56.538128    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3a81bb5a-0683-466e-b32e-960f4ca22ae6-metrics-tls podName:3a81bb5a-0683-466e-b32e-960f4ca22ae6 nodeName:}" failed. No retries permitted until 2026-09-09 19:56:04.538106849 +0000 UTC m=+48.584360095 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/3a81bb5a-0683-466e-b32e-960f4ca22ae6-metrics-tls") pod "dns-default-4blgq" (UID: "3a81bb5a-0683-466e-b32e-960f4ca22ae6") : secret "dns-default-metrics-tls" not found
Sep 09 19:55:57.645482 ip-10-0-136-183 kubenswrapper[2574]: I0909 19:55:57.645451    2574 generic.go:356] "Generic (PLEG): container finished" podID="bed9028e-2a22-4c8f-a637-bb5ad9204adc" containerID="69ab05c9d219cbc14df3c1f54b0f626e57d8724d46eca82b1fb70f95d33b2f09" exitCode=0
Sep 09 19:55:57.645924 ip-10-0-136-183 kubenswrapper[2574]: I0909 19:55:57.645520    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-gzqh4" event={"ID":"bed9028e-2a22-4c8f-a637-bb5ad9204adc","Type":"ContainerDied","Data":"69ab05c9d219cbc14df3c1f54b0f626e57d8724d46eca82b1fb70f95d33b2f09"}
Sep 09 19:55:58.651451 ip-10-0-136-183 kubenswrapper[2574]: I0909 19:55:58.651413    2574 generic.go:356] "Generic (PLEG): container finished" podID="bed9028e-2a22-4c8f-a637-bb5ad9204adc" containerID="1b733709b33b1a62781e962682c412e606e9c03368319a2279b5d7745304fbcb" exitCode=0
Sep 09 19:55:58.651871 ip-10-0-136-183 kubenswrapper[2574]: I0909 19:55:58.651458    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-gzqh4" event={"ID":"bed9028e-2a22-4c8f-a637-bb5ad9204adc","Type":"ContainerDied","Data":"1b733709b33b1a62781e962682c412e606e9c03368319a2279b5d7745304fbcb"}
Sep 09 19:55:59.657421 ip-10-0-136-183 kubenswrapper[2574]: I0909 19:55:59.657383    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-gzqh4" event={"ID":"bed9028e-2a22-4c8f-a637-bb5ad9204adc","Type":"ContainerStarted","Data":"5f0b6820512d97b314080203edf550b7737dfd33320f816431a7bc0e0b2c503d"}
Sep 09 19:55:59.679042 ip-10-0-136-183 kubenswrapper[2574]: I0909 19:55:59.678991    2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-gzqh4" podStartSLOduration=8.636325145 podStartE2EDuration="43.678972624s" totalImagesPullingTime="35.042647479s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-09 19:55:16 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 19:55:59.677889764 +0000 UTC m=+43.724143026" watchObservedRunningTime="2026-09-09 19:55:59.678972624 +0000 UTC m=+43.725225888"
Sep 09 19:56:04.596999 ip-10-0-136-183 kubenswrapper[2574]: I0909 19:56:04.596955    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/3a81bb5a-0683-466e-b32e-960f4ca22ae6-metrics-tls\") pod \"dns-default-4blgq\" (UID: \"3a81bb5a-0683-466e-b32e-960f4ca22ae6\") " pod="openshift-dns/dns-default-4blgq"
Sep 09 19:56:04.597422 ip-10-0-136-183 kubenswrapper[2574]: E0909 19:56:04.597106    2574 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 09 19:56:04.597422 ip-10-0-136-183 kubenswrapper[2574]: E0909 19:56:04.597176    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3a81bb5a-0683-466e-b32e-960f4ca22ae6-metrics-tls podName:3a81bb5a-0683-466e-b32e-960f4ca22ae6 nodeName:}" failed. No retries permitted until 2026-09-09 19:56:20.597159969 +0000 UTC m=+64.643413213 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/3a81bb5a-0683-466e-b32e-960f4ca22ae6-metrics-tls") pod "dns-default-4blgq" (UID: "3a81bb5a-0683-466e-b32e-960f4ca22ae6") : secret "dns-default-metrics-tls" not found
Sep 09 19:56:13.019512 ip-10-0-136-183 kubenswrapper[2574]: I0909 19:56:13.019477    2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-network-console/networking-console-plugin-79977f9844-4c8qr"]
Sep 09 19:56:13.021526 ip-10-0-136-183 kubenswrapper[2574]: I0909 19:56:13.021508    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-79977f9844-4c8qr"
Sep 09 19:56:13.025771 ip-10-0-136-183 kubenswrapper[2574]: I0909 19:56:13.025752    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"default-dockercfg-9jdgt\""
Sep 09 19:56:13.025925 ip-10-0-136-183 kubenswrapper[2574]: I0909 19:56:13.025753    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-console\"/\"networking-console-plugin\""
Sep 09 19:56:13.026053 ip-10-0-136-183 kubenswrapper[2574]: I0909 19:56:13.026036    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"networking-console-plugin-cert\""
Sep 09 19:56:13.031135 ip-10-0-136-183 kubenswrapper[2574]: I0909 19:56:13.031113    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-79977f9844-4c8qr"]
Sep 09 19:56:13.037521 ip-10-0-136-183 kubenswrapper[2574]: I0909 19:56:13.037502    2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-network-diagnostics/network-check-source-685456655f-cf5xl"]
Sep 09 19:56:13.039221 ip-10-0-136-183 kubenswrapper[2574]: I0909 19:56:13.039205    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-685456655f-cf5xl"
Sep 09 19:56:13.043083 ip-10-0-136-183 kubenswrapper[2574]: I0909 19:56:13.043061    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-diagnostics-dockercfg-75t97\""
Sep 09 19:56:13.049400 ip-10-0-136-183 kubenswrapper[2574]: I0909 19:56:13.049375    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-685456655f-cf5xl"]
Sep 09 19:56:13.052844 ip-10-0-136-183 kubenswrapper[2574]: I0909 19:56:13.052815    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/12ae69a5-0173-433e-977f-1b13cc984906-networking-console-plugin-cert\") pod \"networking-console-plugin-79977f9844-4c8qr\" (UID: \"12ae69a5-0173-433e-977f-1b13cc984906\") " pod="openshift-network-console/networking-console-plugin-79977f9844-4c8qr"
Sep 09 19:56:13.052986 ip-10-0-136-183 kubenswrapper[2574]: I0909 19:56:13.052862    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ldrv4\" (UniqueName: \"kubernetes.io/projected/7c707e1b-d4bd-4a62-9828-07976f8b7b7d-kube-api-access-ldrv4\") pod \"network-check-source-685456655f-cf5xl\" (UID: \"7c707e1b-d4bd-4a62-9828-07976f8b7b7d\") " pod="openshift-network-diagnostics/network-check-source-685456655f-cf5xl"
Sep 09 19:56:13.052986 ip-10-0-136-183 kubenswrapper[2574]: I0909 19:56:13.052892    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/12ae69a5-0173-433e-977f-1b13cc984906-nginx-conf\") pod \"networking-console-plugin-79977f9844-4c8qr\" (UID: \"12ae69a5-0173-433e-977f-1b13cc984906\") " pod="openshift-network-console/networking-console-plugin-79977f9844-4c8qr"
Sep 09 19:56:13.153497 ip-10-0-136-183 kubenswrapper[2574]: I0909 19:56:13.153455    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/12ae69a5-0173-433e-977f-1b13cc984906-networking-console-plugin-cert\") pod \"networking-console-plugin-79977f9844-4c8qr\" (UID: \"12ae69a5-0173-433e-977f-1b13cc984906\") " pod="openshift-network-console/networking-console-plugin-79977f9844-4c8qr"
Sep 09 19:56:13.153497 ip-10-0-136-183 kubenswrapper[2574]: I0909 19:56:13.153498    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ldrv4\" (UniqueName: \"kubernetes.io/projected/7c707e1b-d4bd-4a62-9828-07976f8b7b7d-kube-api-access-ldrv4\") pod \"network-check-source-685456655f-cf5xl\" (UID: \"7c707e1b-d4bd-4a62-9828-07976f8b7b7d\") " pod="openshift-network-diagnostics/network-check-source-685456655f-cf5xl"
Sep 09 19:56:13.153744 ip-10-0-136-183 kubenswrapper[2574]: I0909 19:56:13.153525    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/12ae69a5-0173-433e-977f-1b13cc984906-nginx-conf\") pod \"networking-console-plugin-79977f9844-4c8qr\" (UID: \"12ae69a5-0173-433e-977f-1b13cc984906\") " pod="openshift-network-console/networking-console-plugin-79977f9844-4c8qr"
Sep 09 19:56:13.153744 ip-10-0-136-183 kubenswrapper[2574]: E0909 19:56:13.153619    2574 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Sep 09 19:56:13.153744 ip-10-0-136-183 kubenswrapper[2574]: E0909 19:56:13.153703    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/12ae69a5-0173-433e-977f-1b13cc984906-networking-console-plugin-cert podName:12ae69a5-0173-433e-977f-1b13cc984906 nodeName:}" failed. No retries permitted until 2026-09-09 19:56:13.653682154 +0000 UTC m=+57.699935408 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/12ae69a5-0173-433e-977f-1b13cc984906-networking-console-plugin-cert") pod "networking-console-plugin-79977f9844-4c8qr" (UID: "12ae69a5-0173-433e-977f-1b13cc984906") : secret "networking-console-plugin-cert" not found
Sep 09 19:56:13.154248 ip-10-0-136-183 kubenswrapper[2574]: I0909 19:56:13.154224    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/12ae69a5-0173-433e-977f-1b13cc984906-nginx-conf\") pod \"networking-console-plugin-79977f9844-4c8qr\" (UID: \"12ae69a5-0173-433e-977f-1b13cc984906\") " pod="openshift-network-console/networking-console-plugin-79977f9844-4c8qr"
Sep 09 19:56:13.169348 ip-10-0-136-183 kubenswrapper[2574]: I0909 19:56:13.169321    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ldrv4\" (UniqueName: \"kubernetes.io/projected/7c707e1b-d4bd-4a62-9828-07976f8b7b7d-kube-api-access-ldrv4\") pod \"network-check-source-685456655f-cf5xl\" (UID: \"7c707e1b-d4bd-4a62-9828-07976f8b7b7d\") " pod="openshift-network-diagnostics/network-check-source-685456655f-cf5xl"
Sep 09 19:56:13.239867 ip-10-0-136-183 kubenswrapper[2574]: I0909 19:56:13.239830    2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/router-default-58574c9bd-vdsvv"]
Sep 09 19:56:13.241607 ip-10-0-136-183 kubenswrapper[2574]: I0909 19:56:13.241590    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-58574c9bd-vdsvv"
Sep 09 19:56:13.243867 ip-10-0-136-183 kubenswrapper[2574]: I0909 19:56:13.243848    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\""
Sep 09 19:56:13.243975 ip-10-0-136-183 kubenswrapper[2574]: I0909 19:56:13.243867    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-dockercfg-w8t5x\""
Sep 09 19:56:13.243975 ip-10-0-136-183 kubenswrapper[2574]: I0909 19:56:13.243921    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"service-ca-bundle\""
Sep 09 19:56:13.244635 ip-10-0-136-183 kubenswrapper[2574]: I0909 19:56:13.244612    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"default-ingress-cert\""
Sep 09 19:56:13.244827 ip-10-0-136-183 kubenswrapper[2574]: I0909 19:56:13.244786    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\""
Sep 09 19:56:13.244912 ip-10-0-136-183 kubenswrapper[2574]: I0909 19:56:13.244864    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-metrics-certs-default\""
Sep 09 19:56:13.245082 ip-10-0-136-183 kubenswrapper[2574]: I0909 19:56:13.245063    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-stats-default\""
Sep 09 19:56:13.254842 ip-10-0-136-183 kubenswrapper[2574]: I0909 19:56:13.254822    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/85e55211-a74b-4907-bc5b-887db353bdd7-metrics-certs\") pod \"router-default-58574c9bd-vdsvv\" (UID: \"85e55211-a74b-4907-bc5b-887db353bdd7\") " pod="openshift-ingress/router-default-58574c9bd-vdsvv"
Sep 09 19:56:13.254940 ip-10-0-136-183 kubenswrapper[2574]: I0909 19:56:13.254848    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vdt46\" (UniqueName: \"kubernetes.io/projected/85e55211-a74b-4907-bc5b-887db353bdd7-kube-api-access-vdt46\") pod \"router-default-58574c9bd-vdsvv\" (UID: \"85e55211-a74b-4907-bc5b-887db353bdd7\") " pod="openshift-ingress/router-default-58574c9bd-vdsvv"
Sep 09 19:56:13.254940 ip-10-0-136-183 kubenswrapper[2574]: I0909 19:56:13.254872    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/85e55211-a74b-4907-bc5b-887db353bdd7-service-ca-bundle\") pod \"router-default-58574c9bd-vdsvv\" (UID: \"85e55211-a74b-4907-bc5b-887db353bdd7\") " pod="openshift-ingress/router-default-58574c9bd-vdsvv"
Sep 09 19:56:13.254940 ip-10-0-136-183 kubenswrapper[2574]: I0909 19:56:13.254905    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/85e55211-a74b-4907-bc5b-887db353bdd7-default-certificate\") pod \"router-default-58574c9bd-vdsvv\" (UID: \"85e55211-a74b-4907-bc5b-887db353bdd7\") " pod="openshift-ingress/router-default-58574c9bd-vdsvv"
Sep 09 19:56:13.255035 ip-10-0-136-183 kubenswrapper[2574]: I0909 19:56:13.254960    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/85e55211-a74b-4907-bc5b-887db353bdd7-stats-auth\") pod \"router-default-58574c9bd-vdsvv\" (UID: \"85e55211-a74b-4907-bc5b-887db353bdd7\") " pod="openshift-ingress/router-default-58574c9bd-vdsvv"
Sep 09 19:56:13.257603 ip-10-0-136-183 kubenswrapper[2574]: I0909 19:56:13.257583    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/router-default-58574c9bd-vdsvv"]
Sep 09 19:56:13.348173 ip-10-0-136-183 kubenswrapper[2574]: I0909 19:56:13.348085    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-685456655f-cf5xl"
Sep 09 19:56:13.356165 ip-10-0-136-183 kubenswrapper[2574]: I0909 19:56:13.356138    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/85e55211-a74b-4907-bc5b-887db353bdd7-metrics-certs\") pod \"router-default-58574c9bd-vdsvv\" (UID: \"85e55211-a74b-4907-bc5b-887db353bdd7\") " pod="openshift-ingress/router-default-58574c9bd-vdsvv"
Sep 09 19:56:13.356292 ip-10-0-136-183 kubenswrapper[2574]: I0909 19:56:13.356180    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vdt46\" (UniqueName: \"kubernetes.io/projected/85e55211-a74b-4907-bc5b-887db353bdd7-kube-api-access-vdt46\") pod \"router-default-58574c9bd-vdsvv\" (UID: \"85e55211-a74b-4907-bc5b-887db353bdd7\") " pod="openshift-ingress/router-default-58574c9bd-vdsvv"
Sep 09 19:56:13.356292 ip-10-0-136-183 kubenswrapper[2574]: I0909 19:56:13.356222    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/85e55211-a74b-4907-bc5b-887db353bdd7-service-ca-bundle\") pod \"router-default-58574c9bd-vdsvv\" (UID: \"85e55211-a74b-4907-bc5b-887db353bdd7\") " pod="openshift-ingress/router-default-58574c9bd-vdsvv"
Sep 09 19:56:13.356292 ip-10-0-136-183 kubenswrapper[2574]: I0909 19:56:13.356269    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/85e55211-a74b-4907-bc5b-887db353bdd7-default-certificate\") pod \"router-default-58574c9bd-vdsvv\" (UID: \"85e55211-a74b-4907-bc5b-887db353bdd7\") " pod="openshift-ingress/router-default-58574c9bd-vdsvv"
Sep 09 19:56:13.356420 ip-10-0-136-183 kubenswrapper[2574]: I0909 19:56:13.356295    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/85e55211-a74b-4907-bc5b-887db353bdd7-stats-auth\") pod \"router-default-58574c9bd-vdsvv\" (UID: \"85e55211-a74b-4907-bc5b-887db353bdd7\") " pod="openshift-ingress/router-default-58574c9bd-vdsvv"
Sep 09 19:56:13.356420 ip-10-0-136-183 kubenswrapper[2574]: E0909 19:56:13.356303    2574 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Sep 09 19:56:13.356420 ip-10-0-136-183 kubenswrapper[2574]: E0909 19:56:13.356398    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/85e55211-a74b-4907-bc5b-887db353bdd7-metrics-certs podName:85e55211-a74b-4907-bc5b-887db353bdd7 nodeName:}" failed. No retries permitted until 2026-09-09 19:56:13.856363804 +0000 UTC m=+57.902617064 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/85e55211-a74b-4907-bc5b-887db353bdd7-metrics-certs") pod "router-default-58574c9bd-vdsvv" (UID: "85e55211-a74b-4907-bc5b-887db353bdd7") : secret "router-metrics-certs-default" not found
Sep 09 19:56:13.356420 ip-10-0-136-183 kubenswrapper[2574]: E0909 19:56:13.356419    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/85e55211-a74b-4907-bc5b-887db353bdd7-service-ca-bundle podName:85e55211-a74b-4907-bc5b-887db353bdd7 nodeName:}" failed. No retries permitted until 2026-09-09 19:56:13.856408654 +0000 UTC m=+57.902661898 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/85e55211-a74b-4907-bc5b-887db353bdd7-service-ca-bundle") pod "router-default-58574c9bd-vdsvv" (UID: "85e55211-a74b-4907-bc5b-887db353bdd7") : configmap references non-existent config key: service-ca.crt
Sep 09 19:56:13.359135 ip-10-0-136-183 kubenswrapper[2574]: I0909 19:56:13.359095    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/85e55211-a74b-4907-bc5b-887db353bdd7-stats-auth\") pod \"router-default-58574c9bd-vdsvv\" (UID: \"85e55211-a74b-4907-bc5b-887db353bdd7\") " pod="openshift-ingress/router-default-58574c9bd-vdsvv"
Sep 09 19:56:13.359135 ip-10-0-136-183 kubenswrapper[2574]: I0909 19:56:13.359106    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/85e55211-a74b-4907-bc5b-887db353bdd7-default-certificate\") pod \"router-default-58574c9bd-vdsvv\" (UID: \"85e55211-a74b-4907-bc5b-887db353bdd7\") " pod="openshift-ingress/router-default-58574c9bd-vdsvv"
Sep 09 19:56:13.365952 ip-10-0-136-183 kubenswrapper[2574]: I0909 19:56:13.365922    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vdt46\" (UniqueName: \"kubernetes.io/projected/85e55211-a74b-4907-bc5b-887db353bdd7-kube-api-access-vdt46\") pod \"router-default-58574c9bd-vdsvv\" (UID: \"85e55211-a74b-4907-bc5b-887db353bdd7\") " pod="openshift-ingress/router-default-58574c9bd-vdsvv"
Sep 09 19:56:13.615645 ip-10-0-136-183 kubenswrapper[2574]: I0909 19:56:13.615618    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-685456655f-cf5xl"]
Sep 09 19:56:13.618839 ip-10-0-136-183 kubenswrapper[2574]: W0909 19:56:13.618816    2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7c707e1b_d4bd_4a62_9828_07976f8b7b7d.slice/crio-29322551cd3ecec9dae70d25e3359ad0fc79cc3b0a84dc61bde304138e132bdb WatchSource:0}: Error finding container 29322551cd3ecec9dae70d25e3359ad0fc79cc3b0a84dc61bde304138e132bdb: Status 404 returned error can't find the container with id 29322551cd3ecec9dae70d25e3359ad0fc79cc3b0a84dc61bde304138e132bdb
Sep 09 19:56:13.658930 ip-10-0-136-183 kubenswrapper[2574]: I0909 19:56:13.658900    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/12ae69a5-0173-433e-977f-1b13cc984906-networking-console-plugin-cert\") pod \"networking-console-plugin-79977f9844-4c8qr\" (UID: \"12ae69a5-0173-433e-977f-1b13cc984906\") " pod="openshift-network-console/networking-console-plugin-79977f9844-4c8qr"
Sep 09 19:56:13.659059 ip-10-0-136-183 kubenswrapper[2574]: E0909 19:56:13.659043    2574 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Sep 09 19:56:13.659112 ip-10-0-136-183 kubenswrapper[2574]: E0909 19:56:13.659102    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/12ae69a5-0173-433e-977f-1b13cc984906-networking-console-plugin-cert podName:12ae69a5-0173-433e-977f-1b13cc984906 nodeName:}" failed. No retries permitted until 2026-09-09 19:56:14.65908472 +0000 UTC m=+58.705337977 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/12ae69a5-0173-433e-977f-1b13cc984906-networking-console-plugin-cert") pod "networking-console-plugin-79977f9844-4c8qr" (UID: "12ae69a5-0173-433e-977f-1b13cc984906") : secret "networking-console-plugin-cert" not found
Sep 09 19:56:13.691696 ip-10-0-136-183 kubenswrapper[2574]: I0909 19:56:13.691660    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-685456655f-cf5xl" event={"ID":"7c707e1b-d4bd-4a62-9828-07976f8b7b7d","Type":"ContainerStarted","Data":"29322551cd3ecec9dae70d25e3359ad0fc79cc3b0a84dc61bde304138e132bdb"}
Sep 09 19:56:13.860770 ip-10-0-136-183 kubenswrapper[2574]: I0909 19:56:13.860732    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/85e55211-a74b-4907-bc5b-887db353bdd7-metrics-certs\") pod \"router-default-58574c9bd-vdsvv\" (UID: \"85e55211-a74b-4907-bc5b-887db353bdd7\") " pod="openshift-ingress/router-default-58574c9bd-vdsvv"
Sep 09 19:56:13.860770 ip-10-0-136-183 kubenswrapper[2574]: I0909 19:56:13.860779    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/85e55211-a74b-4907-bc5b-887db353bdd7-service-ca-bundle\") pod \"router-default-58574c9bd-vdsvv\" (UID: \"85e55211-a74b-4907-bc5b-887db353bdd7\") " pod="openshift-ingress/router-default-58574c9bd-vdsvv"
Sep 09 19:56:13.860981 ip-10-0-136-183 kubenswrapper[2574]: E0909 19:56:13.860911    2574 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Sep 09 19:56:13.860981 ip-10-0-136-183 kubenswrapper[2574]: E0909 19:56:13.860964    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/85e55211-a74b-4907-bc5b-887db353bdd7-service-ca-bundle podName:85e55211-a74b-4907-bc5b-887db353bdd7 nodeName:}" failed. No retries permitted until 2026-09-09 19:56:14.860941074 +0000 UTC m=+58.907194338 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/85e55211-a74b-4907-bc5b-887db353bdd7-service-ca-bundle") pod "router-default-58574c9bd-vdsvv" (UID: "85e55211-a74b-4907-bc5b-887db353bdd7") : configmap references non-existent config key: service-ca.crt
Sep 09 19:56:13.860981 ip-10-0-136-183 kubenswrapper[2574]: E0909 19:56:13.860983    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/85e55211-a74b-4907-bc5b-887db353bdd7-metrics-certs podName:85e55211-a74b-4907-bc5b-887db353bdd7 nodeName:}" failed. No retries permitted until 2026-09-09 19:56:14.860975987 +0000 UTC m=+58.907229227 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/85e55211-a74b-4907-bc5b-887db353bdd7-metrics-certs") pod "router-default-58574c9bd-vdsvv" (UID: "85e55211-a74b-4907-bc5b-887db353bdd7") : secret "router-metrics-certs-default" not found
Sep 09 19:56:14.615014 ip-10-0-136-183 kubenswrapper[2574]: I0909 19:56:14.614980    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-jcr9m"
Sep 09 19:56:14.668615 ip-10-0-136-183 kubenswrapper[2574]: I0909 19:56:14.668137    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/12ae69a5-0173-433e-977f-1b13cc984906-networking-console-plugin-cert\") pod \"networking-console-plugin-79977f9844-4c8qr\" (UID: \"12ae69a5-0173-433e-977f-1b13cc984906\") " pod="openshift-network-console/networking-console-plugin-79977f9844-4c8qr"
Sep 09 19:56:14.668615 ip-10-0-136-183 kubenswrapper[2574]: E0909 19:56:14.668322    2574 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Sep 09 19:56:14.668615 ip-10-0-136-183 kubenswrapper[2574]: E0909 19:56:14.668413    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/12ae69a5-0173-433e-977f-1b13cc984906-networking-console-plugin-cert podName:12ae69a5-0173-433e-977f-1b13cc984906 nodeName:}" failed. No retries permitted until 2026-09-09 19:56:16.668390325 +0000 UTC m=+60.714643582 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/12ae69a5-0173-433e-977f-1b13cc984906-networking-console-plugin-cert") pod "networking-console-plugin-79977f9844-4c8qr" (UID: "12ae69a5-0173-433e-977f-1b13cc984906") : secret "networking-console-plugin-cert" not found
Sep 09 19:56:14.870233 ip-10-0-136-183 kubenswrapper[2574]: I0909 19:56:14.870135    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/85e55211-a74b-4907-bc5b-887db353bdd7-metrics-certs\") pod \"router-default-58574c9bd-vdsvv\" (UID: \"85e55211-a74b-4907-bc5b-887db353bdd7\") " pod="openshift-ingress/router-default-58574c9bd-vdsvv"
Sep 09 19:56:14.870233 ip-10-0-136-183 kubenswrapper[2574]: I0909 19:56:14.870195    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/85e55211-a74b-4907-bc5b-887db353bdd7-service-ca-bundle\") pod \"router-default-58574c9bd-vdsvv\" (UID: \"85e55211-a74b-4907-bc5b-887db353bdd7\") " pod="openshift-ingress/router-default-58574c9bd-vdsvv"
Sep 09 19:56:14.870464 ip-10-0-136-183 kubenswrapper[2574]: E0909 19:56:14.870328    2574 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Sep 09 19:56:14.870464 ip-10-0-136-183 kubenswrapper[2574]: E0909 19:56:14.870340    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/85e55211-a74b-4907-bc5b-887db353bdd7-service-ca-bundle podName:85e55211-a74b-4907-bc5b-887db353bdd7 nodeName:}" failed. No retries permitted until 2026-09-09 19:56:16.870325312 +0000 UTC m=+60.916578557 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/85e55211-a74b-4907-bc5b-887db353bdd7-service-ca-bundle") pod "router-default-58574c9bd-vdsvv" (UID: "85e55211-a74b-4907-bc5b-887db353bdd7") : configmap references non-existent config key: service-ca.crt
Sep 09 19:56:14.870464 ip-10-0-136-183 kubenswrapper[2574]: E0909 19:56:14.870420    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/85e55211-a74b-4907-bc5b-887db353bdd7-metrics-certs podName:85e55211-a74b-4907-bc5b-887db353bdd7 nodeName:}" failed. No retries permitted until 2026-09-09 19:56:16.870383701 +0000 UTC m=+60.916636949 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/85e55211-a74b-4907-bc5b-887db353bdd7-metrics-certs") pod "router-default-58574c9bd-vdsvv" (UID: "85e55211-a74b-4907-bc5b-887db353bdd7") : secret "router-metrics-certs-default" not found
Sep 09 19:56:16.684940 ip-10-0-136-183 kubenswrapper[2574]: I0909 19:56:16.684830    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/12ae69a5-0173-433e-977f-1b13cc984906-networking-console-plugin-cert\") pod \"networking-console-plugin-79977f9844-4c8qr\" (UID: \"12ae69a5-0173-433e-977f-1b13cc984906\") " pod="openshift-network-console/networking-console-plugin-79977f9844-4c8qr"
Sep 09 19:56:16.685421 ip-10-0-136-183 kubenswrapper[2574]: E0909 19:56:16.684971    2574 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Sep 09 19:56:16.685421 ip-10-0-136-183 kubenswrapper[2574]: E0909 19:56:16.685043    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/12ae69a5-0173-433e-977f-1b13cc984906-networking-console-plugin-cert podName:12ae69a5-0173-433e-977f-1b13cc984906 nodeName:}" failed. No retries permitted until 2026-09-09 19:56:20.685024975 +0000 UTC m=+64.731278228 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/12ae69a5-0173-433e-977f-1b13cc984906-networking-console-plugin-cert") pod "networking-console-plugin-79977f9844-4c8qr" (UID: "12ae69a5-0173-433e-977f-1b13cc984906") : secret "networking-console-plugin-cert" not found
Sep 09 19:56:16.700247 ip-10-0-136-183 kubenswrapper[2574]: I0909 19:56:16.700206    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-685456655f-cf5xl" event={"ID":"7c707e1b-d4bd-4a62-9828-07976f8b7b7d","Type":"ContainerStarted","Data":"8db345a8d077c0065fcf916769605d598d27e1cea74d70d232c9f947bc110c4e"}
Sep 09 19:56:16.716398 ip-10-0-136-183 kubenswrapper[2574]: I0909 19:56:16.716348    2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-source-685456655f-cf5xl" podStartSLOduration=0.963523826 podStartE2EDuration="3.716328119s" totalImagesPullingTime="2.752804293s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 19:56:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 19:56:16.714667422 +0000 UTC m=+60.760920683" watchObservedRunningTime="2026-09-09 19:56:16.716328119 +0000 UTC m=+60.762581382"
Sep 09 19:56:16.886135 ip-10-0-136-183 kubenswrapper[2574]: I0909 19:56:16.886099    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/85e55211-a74b-4907-bc5b-887db353bdd7-service-ca-bundle\") pod \"router-default-58574c9bd-vdsvv\" (UID: \"85e55211-a74b-4907-bc5b-887db353bdd7\") " pod="openshift-ingress/router-default-58574c9bd-vdsvv"
Sep 09 19:56:16.886338 ip-10-0-136-183 kubenswrapper[2574]: I0909 19:56:16.886200    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/85e55211-a74b-4907-bc5b-887db353bdd7-metrics-certs\") pod \"router-default-58574c9bd-vdsvv\" (UID: \"85e55211-a74b-4907-bc5b-887db353bdd7\") " pod="openshift-ingress/router-default-58574c9bd-vdsvv"
Sep 09 19:56:16.886338 ip-10-0-136-183 kubenswrapper[2574]: E0909 19:56:16.886287    2574 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Sep 09 19:56:16.886338 ip-10-0-136-183 kubenswrapper[2574]: E0909 19:56:16.886296    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/85e55211-a74b-4907-bc5b-887db353bdd7-service-ca-bundle podName:85e55211-a74b-4907-bc5b-887db353bdd7 nodeName:}" failed. No retries permitted until 2026-09-09 19:56:20.886274532 +0000 UTC m=+64.932527785 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/85e55211-a74b-4907-bc5b-887db353bdd7-service-ca-bundle") pod "router-default-58574c9bd-vdsvv" (UID: "85e55211-a74b-4907-bc5b-887db353bdd7") : configmap references non-existent config key: service-ca.crt
Sep 09 19:56:16.886338 ip-10-0-136-183 kubenswrapper[2574]: E0909 19:56:16.886325    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/85e55211-a74b-4907-bc5b-887db353bdd7-metrics-certs podName:85e55211-a74b-4907-bc5b-887db353bdd7 nodeName:}" failed. No retries permitted until 2026-09-09 19:56:20.886314209 +0000 UTC m=+64.932567474 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/85e55211-a74b-4907-bc5b-887db353bdd7-metrics-certs") pod "router-default-58574c9bd-vdsvv" (UID: "85e55211-a74b-4907-bc5b-887db353bdd7") : secret "router-metrics-certs-default" not found
Sep 09 19:56:18.261930 ip-10-0-136-183 kubenswrapper[2574]: I0909 19:56:18.261891    2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-k6ctg"]
Sep 09 19:56:18.265121 ip-10-0-136-183 kubenswrapper[2574]: I0909 19:56:18.265106    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-k6ctg"
Sep 09 19:56:18.267889 ip-10-0-136-183 kubenswrapper[2574]: I0909 19:56:18.267866    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Sep 09 19:56:18.267998 ip-10-0-136-183 kubenswrapper[2574]: I0909 19:56:18.267897    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Sep 09 19:56:18.267998 ip-10-0-136-183 kubenswrapper[2574]: I0909 19:56:18.267959    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Sep 09 19:56:18.268177 ip-10-0-136-183 kubenswrapper[2574]: I0909 19:56:18.268108    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Sep 09 19:56:18.268481 ip-10-0-136-183 kubenswrapper[2574]: I0909 19:56:18.268464    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-5xp9v\""
Sep 09 19:56:18.285038 ip-10-0-136-183 kubenswrapper[2574]: I0909 19:56:18.285005    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-k6ctg"]
Sep 09 19:56:18.297460 ip-10-0-136-183 kubenswrapper[2574]: I0909 19:56:18.297430    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/2eee1bd1-9d73-45d9-acf7-323648f0d6ae-crio-socket\") pod \"insights-runtime-extractor-k6ctg\" (UID: \"2eee1bd1-9d73-45d9-acf7-323648f0d6ae\") " pod="openshift-insights/insights-runtime-extractor-k6ctg"
Sep 09 19:56:18.297591 ip-10-0-136-183 kubenswrapper[2574]: I0909 19:56:18.297485    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/2eee1bd1-9d73-45d9-acf7-323648f0d6ae-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-k6ctg\" (UID: \"2eee1bd1-9d73-45d9-acf7-323648f0d6ae\") " pod="openshift-insights/insights-runtime-extractor-k6ctg"
Sep 09 19:56:18.297664 ip-10-0-136-183 kubenswrapper[2574]: I0909 19:56:18.297623    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/2eee1bd1-9d73-45d9-acf7-323648f0d6ae-data-volume\") pod \"insights-runtime-extractor-k6ctg\" (UID: \"2eee1bd1-9d73-45d9-acf7-323648f0d6ae\") " pod="openshift-insights/insights-runtime-extractor-k6ctg"
Sep 09 19:56:18.297717 ip-10-0-136-183 kubenswrapper[2574]: I0909 19:56:18.297682    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/2eee1bd1-9d73-45d9-acf7-323648f0d6ae-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-k6ctg\" (UID: \"2eee1bd1-9d73-45d9-acf7-323648f0d6ae\") " pod="openshift-insights/insights-runtime-extractor-k6ctg"
Sep 09 19:56:18.297717 ip-10-0-136-183 kubenswrapper[2574]: I0909 19:56:18.297708    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zhd4p\" (UniqueName: \"kubernetes.io/projected/2eee1bd1-9d73-45d9-acf7-323648f0d6ae-kube-api-access-zhd4p\") pod \"insights-runtime-extractor-k6ctg\" (UID: \"2eee1bd1-9d73-45d9-acf7-323648f0d6ae\") " pod="openshift-insights/insights-runtime-extractor-k6ctg"
Sep 09 19:56:18.398297 ip-10-0-136-183 kubenswrapper[2574]: I0909 19:56:18.398253    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/2eee1bd1-9d73-45d9-acf7-323648f0d6ae-data-volume\") pod \"insights-runtime-extractor-k6ctg\" (UID: \"2eee1bd1-9d73-45d9-acf7-323648f0d6ae\") " pod="openshift-insights/insights-runtime-extractor-k6ctg"
Sep 09 19:56:18.398297 ip-10-0-136-183 kubenswrapper[2574]: I0909 19:56:18.398302    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/2eee1bd1-9d73-45d9-acf7-323648f0d6ae-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-k6ctg\" (UID: \"2eee1bd1-9d73-45d9-acf7-323648f0d6ae\") " pod="openshift-insights/insights-runtime-extractor-k6ctg"
Sep 09 19:56:18.398512 ip-10-0-136-183 kubenswrapper[2574]: I0909 19:56:18.398319    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zhd4p\" (UniqueName: \"kubernetes.io/projected/2eee1bd1-9d73-45d9-acf7-323648f0d6ae-kube-api-access-zhd4p\") pod \"insights-runtime-extractor-k6ctg\" (UID: \"2eee1bd1-9d73-45d9-acf7-323648f0d6ae\") " pod="openshift-insights/insights-runtime-extractor-k6ctg"
Sep 09 19:56:18.398512 ip-10-0-136-183 kubenswrapper[2574]: I0909 19:56:18.398358    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/2eee1bd1-9d73-45d9-acf7-323648f0d6ae-crio-socket\") pod \"insights-runtime-extractor-k6ctg\" (UID: \"2eee1bd1-9d73-45d9-acf7-323648f0d6ae\") " pod="openshift-insights/insights-runtime-extractor-k6ctg"
Sep 09 19:56:18.398512 ip-10-0-136-183 kubenswrapper[2574]: I0909 19:56:18.398405    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/2eee1bd1-9d73-45d9-acf7-323648f0d6ae-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-k6ctg\" (UID: \"2eee1bd1-9d73-45d9-acf7-323648f0d6ae\") " pod="openshift-insights/insights-runtime-extractor-k6ctg"
Sep 09 19:56:18.398512 ip-10-0-136-183 kubenswrapper[2574]: E0909 19:56:18.398444    2574 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 09 19:56:18.398512 ip-10-0-136-183 kubenswrapper[2574]: E0909 19:56:18.398478    2574 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 09 19:56:18.398512 ip-10-0-136-183 kubenswrapper[2574]: E0909 19:56:18.398513    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2eee1bd1-9d73-45d9-acf7-323648f0d6ae-insights-runtime-extractor-tls podName:2eee1bd1-9d73-45d9-acf7-323648f0d6ae nodeName:}" failed. No retries permitted until 2026-09-09 19:56:18.898495113 +0000 UTC m=+62.944748377 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/2eee1bd1-9d73-45d9-acf7-323648f0d6ae-insights-runtime-extractor-tls") pod "insights-runtime-extractor-k6ctg" (UID: "2eee1bd1-9d73-45d9-acf7-323648f0d6ae") : secret "insights-runtime-extractor-tls" not found
Sep 09 19:56:18.398720 ip-10-0-136-183 kubenswrapper[2574]: I0909 19:56:18.398513    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/2eee1bd1-9d73-45d9-acf7-323648f0d6ae-crio-socket\") pod \"insights-runtime-extractor-k6ctg\" (UID: \"2eee1bd1-9d73-45d9-acf7-323648f0d6ae\") " pod="openshift-insights/insights-runtime-extractor-k6ctg"
Sep 09 19:56:18.398720 ip-10-0-136-183 kubenswrapper[2574]: E0909 19:56:18.398528    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/2eee1bd1-9d73-45d9-acf7-323648f0d6ae-kube-rbac-proxy-cm podName:2eee1bd1-9d73-45d9-acf7-323648f0d6ae nodeName:}" failed. No retries permitted until 2026-09-09 19:56:18.898521593 +0000 UTC m=+62.944774833 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/2eee1bd1-9d73-45d9-acf7-323648f0d6ae-kube-rbac-proxy-cm") pod "insights-runtime-extractor-k6ctg" (UID: "2eee1bd1-9d73-45d9-acf7-323648f0d6ae") : configmap "kube-rbac-proxy" not found
Sep 09 19:56:18.398720 ip-10-0-136-183 kubenswrapper[2574]: I0909 19:56:18.398640    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/2eee1bd1-9d73-45d9-acf7-323648f0d6ae-data-volume\") pod \"insights-runtime-extractor-k6ctg\" (UID: \"2eee1bd1-9d73-45d9-acf7-323648f0d6ae\") " pod="openshift-insights/insights-runtime-extractor-k6ctg"
Sep 09 19:56:18.409156 ip-10-0-136-183 kubenswrapper[2574]: I0909 19:56:18.409129    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zhd4p\" (UniqueName: \"kubernetes.io/projected/2eee1bd1-9d73-45d9-acf7-323648f0d6ae-kube-api-access-zhd4p\") pod \"insights-runtime-extractor-k6ctg\" (UID: \"2eee1bd1-9d73-45d9-acf7-323648f0d6ae\") " pod="openshift-insights/insights-runtime-extractor-k6ctg"
Sep 09 19:56:18.902845 ip-10-0-136-183 kubenswrapper[2574]: I0909 19:56:18.902787    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/2eee1bd1-9d73-45d9-acf7-323648f0d6ae-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-k6ctg\" (UID: \"2eee1bd1-9d73-45d9-acf7-323648f0d6ae\") " pod="openshift-insights/insights-runtime-extractor-k6ctg"
Sep 09 19:56:18.903095 ip-10-0-136-183 kubenswrapper[2574]: I0909 19:56:18.902881    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/2eee1bd1-9d73-45d9-acf7-323648f0d6ae-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-k6ctg\" (UID: \"2eee1bd1-9d73-45d9-acf7-323648f0d6ae\") " pod="openshift-insights/insights-runtime-extractor-k6ctg"
Sep 09 19:56:18.903095 ip-10-0-136-183 kubenswrapper[2574]: E0909 19:56:18.902958    2574 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 09 19:56:18.903095 ip-10-0-136-183 kubenswrapper[2574]: E0909 19:56:18.902966    2574 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 09 19:56:18.903095 ip-10-0-136-183 kubenswrapper[2574]: E0909 19:56:18.903012    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/2eee1bd1-9d73-45d9-acf7-323648f0d6ae-kube-rbac-proxy-cm podName:2eee1bd1-9d73-45d9-acf7-323648f0d6ae nodeName:}" failed. No retries permitted until 2026-09-09 19:56:19.902998119 +0000 UTC m=+63.949251358 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/2eee1bd1-9d73-45d9-acf7-323648f0d6ae-kube-rbac-proxy-cm") pod "insights-runtime-extractor-k6ctg" (UID: "2eee1bd1-9d73-45d9-acf7-323648f0d6ae") : configmap "kube-rbac-proxy" not found
Sep 09 19:56:18.903095 ip-10-0-136-183 kubenswrapper[2574]: E0909 19:56:18.903025    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2eee1bd1-9d73-45d9-acf7-323648f0d6ae-insights-runtime-extractor-tls podName:2eee1bd1-9d73-45d9-acf7-323648f0d6ae nodeName:}" failed. No retries permitted until 2026-09-09 19:56:19.903019022 +0000 UTC m=+63.949272262 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/2eee1bd1-9d73-45d9-acf7-323648f0d6ae-insights-runtime-extractor-tls") pod "insights-runtime-extractor-k6ctg" (UID: "2eee1bd1-9d73-45d9-acf7-323648f0d6ae") : secret "insights-runtime-extractor-tls" not found
Sep 09 19:56:19.083661 ip-10-0-136-183 kubenswrapper[2574]: I0909 19:56:19.083624    2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator/migrator-fdbb8fdff-94tx6"]
Sep 09 19:56:19.108609 ip-10-0-136-183 kubenswrapper[2574]: I0909 19:56:19.108576    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-fdbb8fdff-94tx6"]
Sep 09 19:56:19.108766 ip-10-0-136-183 kubenswrapper[2574]: I0909 19:56:19.108682    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-fdbb8fdff-94tx6"
Sep 09 19:56:19.110891 ip-10-0-136-183 kubenswrapper[2574]: I0909 19:56:19.110868    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-root-ca.crt\""
Sep 09 19:56:19.111030 ip-10-0-136-183 kubenswrapper[2574]: I0909 19:56:19.110869    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-storage-version-migrator-sa-dockercfg-rmpsc\""
Sep 09 19:56:19.111030 ip-10-0-136-183 kubenswrapper[2574]: I0909 19:56:19.110869    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"openshift-service-ca.crt\""
Sep 09 19:56:19.205872 ip-10-0-136-183 kubenswrapper[2574]: I0909 19:56:19.205757    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bp82m\" (UniqueName: \"kubernetes.io/projected/afb6a238-0f7b-4545-8d8c-434c1c329064-kube-api-access-bp82m\") pod \"migrator-fdbb8fdff-94tx6\" (UID: \"afb6a238-0f7b-4545-8d8c-434c1c329064\") " pod="openshift-kube-storage-version-migrator/migrator-fdbb8fdff-94tx6"
Sep 09 19:56:19.306661 ip-10-0-136-183 kubenswrapper[2574]: I0909 19:56:19.306615    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bp82m\" (UniqueName: \"kubernetes.io/projected/afb6a238-0f7b-4545-8d8c-434c1c329064-kube-api-access-bp82m\") pod \"migrator-fdbb8fdff-94tx6\" (UID: \"afb6a238-0f7b-4545-8d8c-434c1c329064\") " pod="openshift-kube-storage-version-migrator/migrator-fdbb8fdff-94tx6"
Sep 09 19:56:19.315113 ip-10-0-136-183 kubenswrapper[2574]: I0909 19:56:19.315088    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bp82m\" (UniqueName: \"kubernetes.io/projected/afb6a238-0f7b-4545-8d8c-434c1c329064-kube-api-access-bp82m\") pod \"migrator-fdbb8fdff-94tx6\" (UID: \"afb6a238-0f7b-4545-8d8c-434c1c329064\") " pod="openshift-kube-storage-version-migrator/migrator-fdbb8fdff-94tx6"
Sep 09 19:56:19.418052 ip-10-0-136-183 kubenswrapper[2574]: I0909 19:56:19.418020    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-fdbb8fdff-94tx6"
Sep 09 19:56:19.695935 ip-10-0-136-183 kubenswrapper[2574]: I0909 19:56:19.695881    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-fdbb8fdff-94tx6"]
Sep 09 19:56:19.699221 ip-10-0-136-183 kubenswrapper[2574]: W0909 19:56:19.699192    2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podafb6a238_0f7b_4545_8d8c_434c1c329064.slice/crio-d157a6776491d75fc5b368123bd6fb857cf270ab9a6dd29a8224408cb8095802 WatchSource:0}: Error finding container d157a6776491d75fc5b368123bd6fb857cf270ab9a6dd29a8224408cb8095802: Status 404 returned error can't find the container with id d157a6776491d75fc5b368123bd6fb857cf270ab9a6dd29a8224408cb8095802
Sep 09 19:56:19.708111 ip-10-0-136-183 kubenswrapper[2574]: I0909 19:56:19.708084    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-fdbb8fdff-94tx6" event={"ID":"afb6a238-0f7b-4545-8d8c-434c1c329064","Type":"ContainerStarted","Data":"d157a6776491d75fc5b368123bd6fb857cf270ab9a6dd29a8224408cb8095802"}
Sep 09 19:56:19.911448 ip-10-0-136-183 kubenswrapper[2574]: I0909 19:56:19.911412    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/2eee1bd1-9d73-45d9-acf7-323648f0d6ae-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-k6ctg\" (UID: \"2eee1bd1-9d73-45d9-acf7-323648f0d6ae\") " pod="openshift-insights/insights-runtime-extractor-k6ctg"
Sep 09 19:56:19.911611 ip-10-0-136-183 kubenswrapper[2574]: I0909 19:56:19.911524    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/2eee1bd1-9d73-45d9-acf7-323648f0d6ae-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-k6ctg\" (UID: \"2eee1bd1-9d73-45d9-acf7-323648f0d6ae\") " pod="openshift-insights/insights-runtime-extractor-k6ctg"
Sep 09 19:56:19.911611 ip-10-0-136-183 kubenswrapper[2574]: E0909 19:56:19.911548    2574 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 09 19:56:19.911732 ip-10-0-136-183 kubenswrapper[2574]: E0909 19:56:19.911620    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/2eee1bd1-9d73-45d9-acf7-323648f0d6ae-kube-rbac-proxy-cm podName:2eee1bd1-9d73-45d9-acf7-323648f0d6ae nodeName:}" failed. No retries permitted until 2026-09-09 19:56:21.911602384 +0000 UTC m=+65.957855634 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/2eee1bd1-9d73-45d9-acf7-323648f0d6ae-kube-rbac-proxy-cm") pod "insights-runtime-extractor-k6ctg" (UID: "2eee1bd1-9d73-45d9-acf7-323648f0d6ae") : configmap "kube-rbac-proxy" not found
Sep 09 19:56:19.911732 ip-10-0-136-183 kubenswrapper[2574]: E0909 19:56:19.911644    2574 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 09 19:56:19.911732 ip-10-0-136-183 kubenswrapper[2574]: E0909 19:56:19.911688    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2eee1bd1-9d73-45d9-acf7-323648f0d6ae-insights-runtime-extractor-tls podName:2eee1bd1-9d73-45d9-acf7-323648f0d6ae nodeName:}" failed. No retries permitted until 2026-09-09 19:56:21.911677631 +0000 UTC m=+65.957930888 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/2eee1bd1-9d73-45d9-acf7-323648f0d6ae-insights-runtime-extractor-tls") pod "insights-runtime-extractor-k6ctg" (UID: "2eee1bd1-9d73-45d9-acf7-323648f0d6ae") : secret "insights-runtime-extractor-tls" not found
Sep 09 19:56:20.210523 ip-10-0-136-183 kubenswrapper[2574]: I0909 19:56:20.210438    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-q8krg_496a3333-1b9c-4eb2-8896-b6586eaf6877/dns-node-resolver/0.log"
Sep 09 19:56:20.617516 ip-10-0-136-183 kubenswrapper[2574]: I0909 19:56:20.617475    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/3a81bb5a-0683-466e-b32e-960f4ca22ae6-metrics-tls\") pod \"dns-default-4blgq\" (UID: \"3a81bb5a-0683-466e-b32e-960f4ca22ae6\") " pod="openshift-dns/dns-default-4blgq"
Sep 09 19:56:20.617985 ip-10-0-136-183 kubenswrapper[2574]: E0909 19:56:20.617660    2574 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 09 19:56:20.617985 ip-10-0-136-183 kubenswrapper[2574]: E0909 19:56:20.617745    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3a81bb5a-0683-466e-b32e-960f4ca22ae6-metrics-tls podName:3a81bb5a-0683-466e-b32e-960f4ca22ae6 nodeName:}" failed. No retries permitted until 2026-09-09 19:56:52.617721449 +0000 UTC m=+96.663974704 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/3a81bb5a-0683-466e-b32e-960f4ca22ae6-metrics-tls") pod "dns-default-4blgq" (UID: "3a81bb5a-0683-466e-b32e-960f4ca22ae6") : secret "dns-default-metrics-tls" not found
Sep 09 19:56:20.718372 ip-10-0-136-183 kubenswrapper[2574]: I0909 19:56:20.718337    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/12ae69a5-0173-433e-977f-1b13cc984906-networking-console-plugin-cert\") pod \"networking-console-plugin-79977f9844-4c8qr\" (UID: \"12ae69a5-0173-433e-977f-1b13cc984906\") " pod="openshift-network-console/networking-console-plugin-79977f9844-4c8qr"
Sep 09 19:56:20.718535 ip-10-0-136-183 kubenswrapper[2574]: E0909 19:56:20.718507    2574 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Sep 09 19:56:20.718584 ip-10-0-136-183 kubenswrapper[2574]: E0909 19:56:20.718570    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/12ae69a5-0173-433e-977f-1b13cc984906-networking-console-plugin-cert podName:12ae69a5-0173-433e-977f-1b13cc984906 nodeName:}" failed. No retries permitted until 2026-09-09 19:56:28.718554601 +0000 UTC m=+72.764807854 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/12ae69a5-0173-433e-977f-1b13cc984906-networking-console-plugin-cert") pod "networking-console-plugin-79977f9844-4c8qr" (UID: "12ae69a5-0173-433e-977f-1b13cc984906") : secret "networking-console-plugin-cert" not found
Sep 09 19:56:20.919873 ip-10-0-136-183 kubenswrapper[2574]: I0909 19:56:20.919752    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/85e55211-a74b-4907-bc5b-887db353bdd7-metrics-certs\") pod \"router-default-58574c9bd-vdsvv\" (UID: \"85e55211-a74b-4907-bc5b-887db353bdd7\") " pod="openshift-ingress/router-default-58574c9bd-vdsvv"
Sep 09 19:56:20.919873 ip-10-0-136-183 kubenswrapper[2574]: I0909 19:56:20.919845    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/85e55211-a74b-4907-bc5b-887db353bdd7-service-ca-bundle\") pod \"router-default-58574c9bd-vdsvv\" (UID: \"85e55211-a74b-4907-bc5b-887db353bdd7\") " pod="openshift-ingress/router-default-58574c9bd-vdsvv"
Sep 09 19:56:20.920070 ip-10-0-136-183 kubenswrapper[2574]: E0909 19:56:20.919907    2574 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Sep 09 19:56:20.920070 ip-10-0-136-183 kubenswrapper[2574]: E0909 19:56:20.919979    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/85e55211-a74b-4907-bc5b-887db353bdd7-metrics-certs podName:85e55211-a74b-4907-bc5b-887db353bdd7 nodeName:}" failed. No retries permitted until 2026-09-09 19:56:28.919962966 +0000 UTC m=+72.966216225 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/85e55211-a74b-4907-bc5b-887db353bdd7-metrics-certs") pod "router-default-58574c9bd-vdsvv" (UID: "85e55211-a74b-4907-bc5b-887db353bdd7") : secret "router-metrics-certs-default" not found
Sep 09 19:56:20.920070 ip-10-0-136-183 kubenswrapper[2574]: E0909 19:56:20.920001    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/85e55211-a74b-4907-bc5b-887db353bdd7-service-ca-bundle podName:85e55211-a74b-4907-bc5b-887db353bdd7 nodeName:}" failed. No retries permitted until 2026-09-09 19:56:28.919984551 +0000 UTC m=+72.966237812 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/85e55211-a74b-4907-bc5b-887db353bdd7-service-ca-bundle") pod "router-default-58574c9bd-vdsvv" (UID: "85e55211-a74b-4907-bc5b-887db353bdd7") : configmap references non-existent config key: service-ca.crt
Sep 09 19:56:21.210981 ip-10-0-136-183 kubenswrapper[2574]: I0909 19:56:21.210906    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-mgg8m_39099fe9-527d-4802-b40b-1bf535221165/node-ca/0.log"
Sep 09 19:56:21.715016 ip-10-0-136-183 kubenswrapper[2574]: I0909 19:56:21.714981    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-fdbb8fdff-94tx6" event={"ID":"afb6a238-0f7b-4545-8d8c-434c1c329064","Type":"ContainerStarted","Data":"c4fd2e6b62a6dfd7b7c9f8cc3f3b9c7a1b270d30db6c23f920b64dc92a8da261"}
Sep 09 19:56:21.715503 ip-10-0-136-183 kubenswrapper[2574]: I0909 19:56:21.715041    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-fdbb8fdff-94tx6" event={"ID":"afb6a238-0f7b-4545-8d8c-434c1c329064","Type":"ContainerStarted","Data":"dfa8bfd1d61ecc1aad83328764c4e5c42aa74d1dbb16c00d81f0dc0edf2a025e"}
Sep 09 19:56:21.734080 ip-10-0-136-183 kubenswrapper[2574]: I0909 19:56:21.729954    2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator/migrator-fdbb8fdff-94tx6" podStartSLOduration=1.679825399 podStartE2EDuration="2.729940744s" totalImagesPullingTime="1.050115345s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 19:56:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 19:56:21.729661169 +0000 UTC m=+65.775914431" watchObservedRunningTime="2026-09-09 19:56:21.729940744 +0000 UTC m=+65.776194006"
Sep 09 19:56:21.925665 ip-10-0-136-183 kubenswrapper[2574]: I0909 19:56:21.925620    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/2eee1bd1-9d73-45d9-acf7-323648f0d6ae-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-k6ctg\" (UID: \"2eee1bd1-9d73-45d9-acf7-323648f0d6ae\") " pod="openshift-insights/insights-runtime-extractor-k6ctg"
Sep 09 19:56:21.925865 ip-10-0-136-183 kubenswrapper[2574]: I0909 19:56:21.925685    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/2eee1bd1-9d73-45d9-acf7-323648f0d6ae-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-k6ctg\" (UID: \"2eee1bd1-9d73-45d9-acf7-323648f0d6ae\") " pod="openshift-insights/insights-runtime-extractor-k6ctg"
Sep 09 19:56:21.925865 ip-10-0-136-183 kubenswrapper[2574]: E0909 19:56:21.925767    2574 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 09 19:56:21.925865 ip-10-0-136-183 kubenswrapper[2574]: E0909 19:56:21.925773    2574 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 09 19:56:21.925865 ip-10-0-136-183 kubenswrapper[2574]: E0909 19:56:21.925844    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/2eee1bd1-9d73-45d9-acf7-323648f0d6ae-kube-rbac-proxy-cm podName:2eee1bd1-9d73-45d9-acf7-323648f0d6ae nodeName:}" failed. No retries permitted until 2026-09-09 19:56:25.925829591 +0000 UTC m=+69.972082845 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/2eee1bd1-9d73-45d9-acf7-323648f0d6ae-kube-rbac-proxy-cm") pod "insights-runtime-extractor-k6ctg" (UID: "2eee1bd1-9d73-45d9-acf7-323648f0d6ae") : configmap "kube-rbac-proxy" not found
Sep 09 19:56:21.925865 ip-10-0-136-183 kubenswrapper[2574]: E0909 19:56:21.925857    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2eee1bd1-9d73-45d9-acf7-323648f0d6ae-insights-runtime-extractor-tls podName:2eee1bd1-9d73-45d9-acf7-323648f0d6ae nodeName:}" failed. No retries permitted until 2026-09-09 19:56:25.925851622 +0000 UTC m=+69.972104862 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/2eee1bd1-9d73-45d9-acf7-323648f0d6ae-insights-runtime-extractor-tls") pod "insights-runtime-extractor-k6ctg" (UID: "2eee1bd1-9d73-45d9-acf7-323648f0d6ae") : secret "insights-runtime-extractor-tls" not found
Sep 09 19:56:22.011384 ip-10-0-136-183 kubenswrapper[2574]: I0909 19:56:22.011310    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-57cmp_ceff0c1c-cb80-437c-a942-9ebe28442933/serve-healthcheck-canary/0.log"
Sep 09 19:56:22.127202 ip-10-0-136-183 kubenswrapper[2574]: I0909 19:56:22.127155    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/c756f4c9-2360-4fff-a136-b6869b02977e-original-pull-secret\") pod \"global-pull-secret-syncer-gjc9x\" (UID: \"c756f4c9-2360-4fff-a136-b6869b02977e\") " pod="kube-system/global-pull-secret-syncer-gjc9x"
Sep 09 19:56:22.127398 ip-10-0-136-183 kubenswrapper[2574]: I0909 19:56:22.127281    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/47c2b5b9-456a-4528-8d83-7ce3269cc16e-metrics-certs\") pod \"network-metrics-daemon-mdrck\" (UID: \"47c2b5b9-456a-4528-8d83-7ce3269cc16e\") " pod="openshift-multus/network-metrics-daemon-mdrck"
Sep 09 19:56:22.129510 ip-10-0-136-183 kubenswrapper[2574]: I0909 19:56:22.129491    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Sep 09 19:56:22.129557 ip-10-0-136-183 kubenswrapper[2574]: I0909 19:56:22.129500    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Sep 09 19:56:22.138322 ip-10-0-136-183 kubenswrapper[2574]: E0909 19:56:22.138296    2574 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
Sep 09 19:56:22.138374 ip-10-0-136-183 kubenswrapper[2574]: E0909 19:56:22.138363    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/47c2b5b9-456a-4528-8d83-7ce3269cc16e-metrics-certs podName:47c2b5b9-456a-4528-8d83-7ce3269cc16e nodeName:}" failed. No retries permitted until 2026-09-09 19:57:26.138346522 +0000 UTC m=+130.184599763 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/47c2b5b9-456a-4528-8d83-7ce3269cc16e-metrics-certs") pod "network-metrics-daemon-mdrck" (UID: "47c2b5b9-456a-4528-8d83-7ce3269cc16e") : secret "metrics-daemon-secret" not found
Sep 09 19:56:22.140469 ip-10-0-136-183 kubenswrapper[2574]: I0909 19:56:22.140445    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/c756f4c9-2360-4fff-a136-b6869b02977e-original-pull-secret\") pod \"global-pull-secret-syncer-gjc9x\" (UID: \"c756f4c9-2360-4fff-a136-b6869b02977e\") " pod="kube-system/global-pull-secret-syncer-gjc9x"
Sep 09 19:56:22.228310 ip-10-0-136-183 kubenswrapper[2574]: I0909 19:56:22.228272    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zqd55\" (UniqueName: \"kubernetes.io/projected/54458a6d-921b-4637-addc-7c4ee839b592-kube-api-access-zqd55\") pod \"network-check-target-4czxv\" (UID: \"54458a6d-921b-4637-addc-7c4ee839b592\") " pod="openshift-network-diagnostics/network-check-target-4czxv"
Sep 09 19:56:22.230764 ip-10-0-136-183 kubenswrapper[2574]: I0909 19:56:22.230740    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zqd55\" (UniqueName: \"kubernetes.io/projected/54458a6d-921b-4637-addc-7c4ee839b592-kube-api-access-zqd55\") pod \"network-check-target-4czxv\" (UID: \"54458a6d-921b-4637-addc-7c4ee839b592\") " pod="openshift-network-diagnostics/network-check-target-4czxv"
Sep 09 19:56:22.306329 ip-10-0-136-183 kubenswrapper[2574]: I0909 19:56:22.306231    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-gjc9x"
Sep 09 19:56:22.314551 ip-10-0-136-183 kubenswrapper[2574]: I0909 19:56:22.314525    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-q8crj\""
Sep 09 19:56:22.323252 ip-10-0-136-183 kubenswrapper[2574]: I0909 19:56:22.323222    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-4czxv"
Sep 09 19:56:22.576354 ip-10-0-136-183 kubenswrapper[2574]: I0909 19:56:22.576266    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-gjc9x"]
Sep 09 19:56:22.579841 ip-10-0-136-183 kubenswrapper[2574]: W0909 19:56:22.579811    2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc756f4c9_2360_4fff_a136_b6869b02977e.slice/crio-25bb3978cbf869bfb8756a7842146d6da4ac2e78f1e2c7467b15b8059fd9c813 WatchSource:0}: Error finding container 25bb3978cbf869bfb8756a7842146d6da4ac2e78f1e2c7467b15b8059fd9c813: Status 404 returned error can't find the container with id 25bb3978cbf869bfb8756a7842146d6da4ac2e78f1e2c7467b15b8059fd9c813
Sep 09 19:56:22.598354 ip-10-0-136-183 kubenswrapper[2574]: I0909 19:56:22.598333    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-4czxv"]
Sep 09 19:56:22.601303 ip-10-0-136-183 kubenswrapper[2574]: W0909 19:56:22.601279    2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod54458a6d_921b_4637_addc_7c4ee839b592.slice/crio-808eba2922c08c8230817bb1cd0b6abeb1d0cce7cc3375a006462c9013a050fc WatchSource:0}: Error finding container 808eba2922c08c8230817bb1cd0b6abeb1d0cce7cc3375a006462c9013a050fc: Status 404 returned error can't find the container with id 808eba2922c08c8230817bb1cd0b6abeb1d0cce7cc3375a006462c9013a050fc
Sep 09 19:56:22.721973 ip-10-0-136-183 kubenswrapper[2574]: I0909 19:56:22.721940    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-4czxv" event={"ID":"54458a6d-921b-4637-addc-7c4ee839b592","Type":"ContainerStarted","Data":"95fb626aeb70b8cdab775966c160b8b2a80b5d072aa0c4f45df6258db77b1027"}
Sep 09 19:56:22.722407 ip-10-0-136-183 kubenswrapper[2574]: I0909 19:56:22.721994    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-4czxv" event={"ID":"54458a6d-921b-4637-addc-7c4ee839b592","Type":"ContainerStarted","Data":"808eba2922c08c8230817bb1cd0b6abeb1d0cce7cc3375a006462c9013a050fc"}
Sep 09 19:56:22.722407 ip-10-0-136-183 kubenswrapper[2574]: I0909 19:56:22.722064    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-4czxv"
Sep 09 19:56:22.723585 ip-10-0-136-183 kubenswrapper[2574]: I0909 19:56:22.723558    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-gjc9x" event={"ID":"c756f4c9-2360-4fff-a136-b6869b02977e","Type":"ContainerStarted","Data":"25bb3978cbf869bfb8756a7842146d6da4ac2e78f1e2c7467b15b8059fd9c813"}
Sep 09 19:56:22.738198 ip-10-0-136-183 kubenswrapper[2574]: I0909 19:56:22.738151    2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-4czxv" podStartSLOduration=66.738136667 podStartE2EDuration="1m6.738136667s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 19:55:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 19:56:22.737373106 +0000 UTC m=+66.783626369" watchObservedRunningTime="2026-09-09 19:56:22.738136667 +0000 UTC m=+66.784389929"
Sep 09 19:56:25.958928 ip-10-0-136-183 kubenswrapper[2574]: I0909 19:56:25.958889    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/2eee1bd1-9d73-45d9-acf7-323648f0d6ae-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-k6ctg\" (UID: \"2eee1bd1-9d73-45d9-acf7-323648f0d6ae\") " pod="openshift-insights/insights-runtime-extractor-k6ctg"
Sep 09 19:56:25.959414 ip-10-0-136-183 kubenswrapper[2574]: I0909 19:56:25.959002    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/2eee1bd1-9d73-45d9-acf7-323648f0d6ae-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-k6ctg\" (UID: \"2eee1bd1-9d73-45d9-acf7-323648f0d6ae\") " pod="openshift-insights/insights-runtime-extractor-k6ctg"
Sep 09 19:56:25.959414 ip-10-0-136-183 kubenswrapper[2574]: E0909 19:56:25.959031    2574 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 09 19:56:25.959414 ip-10-0-136-183 kubenswrapper[2574]: E0909 19:56:25.959116    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/2eee1bd1-9d73-45d9-acf7-323648f0d6ae-kube-rbac-proxy-cm podName:2eee1bd1-9d73-45d9-acf7-323648f0d6ae nodeName:}" failed. No retries permitted until 2026-09-09 19:56:33.95909263 +0000 UTC m=+78.005345890 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/2eee1bd1-9d73-45d9-acf7-323648f0d6ae-kube-rbac-proxy-cm") pod "insights-runtime-extractor-k6ctg" (UID: "2eee1bd1-9d73-45d9-acf7-323648f0d6ae") : configmap "kube-rbac-proxy" not found
Sep 09 19:56:25.959414 ip-10-0-136-183 kubenswrapper[2574]: E0909 19:56:25.959140    2574 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 09 19:56:25.959414 ip-10-0-136-183 kubenswrapper[2574]: E0909 19:56:25.959188    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2eee1bd1-9d73-45d9-acf7-323648f0d6ae-insights-runtime-extractor-tls podName:2eee1bd1-9d73-45d9-acf7-323648f0d6ae nodeName:}" failed. No retries permitted until 2026-09-09 19:56:33.959176811 +0000 UTC m=+78.005430064 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/2eee1bd1-9d73-45d9-acf7-323648f0d6ae-insights-runtime-extractor-tls") pod "insights-runtime-extractor-k6ctg" (UID: "2eee1bd1-9d73-45d9-acf7-323648f0d6ae") : secret "insights-runtime-extractor-tls" not found
Sep 09 19:56:27.739173 ip-10-0-136-183 kubenswrapper[2574]: I0909 19:56:27.739128    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-gjc9x" event={"ID":"c756f4c9-2360-4fff-a136-b6869b02977e","Type":"ContainerStarted","Data":"70263f50262febcb540a59d7acafb1922eba302c1227e7cbb22bd3d5585c1b5d"}
Sep 09 19:56:27.754448 ip-10-0-136-183 kubenswrapper[2574]: I0909 19:56:27.754393    2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-gjc9x" podStartSLOduration=67.483454986 podStartE2EDuration="1m11.754377871s" totalImagesPullingTime="4.270922885s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-09 19:55:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 19:56:27.754078084 +0000 UTC m=+71.800331348" watchObservedRunningTime="2026-09-09 19:56:27.754377871 +0000 UTC m=+71.800631133"
Sep 09 19:56:28.783517 ip-10-0-136-183 kubenswrapper[2574]: I0909 19:56:28.783474    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/12ae69a5-0173-433e-977f-1b13cc984906-networking-console-plugin-cert\") pod \"networking-console-plugin-79977f9844-4c8qr\" (UID: \"12ae69a5-0173-433e-977f-1b13cc984906\") " pod="openshift-network-console/networking-console-plugin-79977f9844-4c8qr"
Sep 09 19:56:28.783926 ip-10-0-136-183 kubenswrapper[2574]: E0909 19:56:28.783609    2574 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Sep 09 19:56:28.783926 ip-10-0-136-183 kubenswrapper[2574]: E0909 19:56:28.783663    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/12ae69a5-0173-433e-977f-1b13cc984906-networking-console-plugin-cert podName:12ae69a5-0173-433e-977f-1b13cc984906 nodeName:}" failed. No retries permitted until 2026-09-09 19:56:44.783648259 +0000 UTC m=+88.829901500 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/12ae69a5-0173-433e-977f-1b13cc984906-networking-console-plugin-cert") pod "networking-console-plugin-79977f9844-4c8qr" (UID: "12ae69a5-0173-433e-977f-1b13cc984906") : secret "networking-console-plugin-cert" not found
Sep 09 19:56:28.984756 ip-10-0-136-183 kubenswrapper[2574]: I0909 19:56:28.984712    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/85e55211-a74b-4907-bc5b-887db353bdd7-metrics-certs\") pod \"router-default-58574c9bd-vdsvv\" (UID: \"85e55211-a74b-4907-bc5b-887db353bdd7\") " pod="openshift-ingress/router-default-58574c9bd-vdsvv"
Sep 09 19:56:28.984940 ip-10-0-136-183 kubenswrapper[2574]: I0909 19:56:28.984768    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/85e55211-a74b-4907-bc5b-887db353bdd7-service-ca-bundle\") pod \"router-default-58574c9bd-vdsvv\" (UID: \"85e55211-a74b-4907-bc5b-887db353bdd7\") " pod="openshift-ingress/router-default-58574c9bd-vdsvv"
Sep 09 19:56:28.984940 ip-10-0-136-183 kubenswrapper[2574]: E0909 19:56:28.984883    2574 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Sep 09 19:56:28.984940 ip-10-0-136-183 kubenswrapper[2574]: E0909 19:56:28.984900    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/85e55211-a74b-4907-bc5b-887db353bdd7-service-ca-bundle podName:85e55211-a74b-4907-bc5b-887db353bdd7 nodeName:}" failed. No retries permitted until 2026-09-09 19:56:44.98488438 +0000 UTC m=+89.031137624 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/85e55211-a74b-4907-bc5b-887db353bdd7-service-ca-bundle") pod "router-default-58574c9bd-vdsvv" (UID: "85e55211-a74b-4907-bc5b-887db353bdd7") : configmap references non-existent config key: service-ca.crt
Sep 09 19:56:28.984940 ip-10-0-136-183 kubenswrapper[2574]: E0909 19:56:28.984935    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/85e55211-a74b-4907-bc5b-887db353bdd7-metrics-certs podName:85e55211-a74b-4907-bc5b-887db353bdd7 nodeName:}" failed. No retries permitted until 2026-09-09 19:56:44.984922195 +0000 UTC m=+89.031175448 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/85e55211-a74b-4907-bc5b-887db353bdd7-metrics-certs") pod "router-default-58574c9bd-vdsvv" (UID: "85e55211-a74b-4907-bc5b-887db353bdd7") : secret "router-metrics-certs-default" not found
Sep 09 19:56:34.027686 ip-10-0-136-183 kubenswrapper[2574]: I0909 19:56:34.027638    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/2eee1bd1-9d73-45d9-acf7-323648f0d6ae-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-k6ctg\" (UID: \"2eee1bd1-9d73-45d9-acf7-323648f0d6ae\") " pod="openshift-insights/insights-runtime-extractor-k6ctg"
Sep 09 19:56:34.028229 ip-10-0-136-183 kubenswrapper[2574]: I0909 19:56:34.027826    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/2eee1bd1-9d73-45d9-acf7-323648f0d6ae-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-k6ctg\" (UID: \"2eee1bd1-9d73-45d9-acf7-323648f0d6ae\") " pod="openshift-insights/insights-runtime-extractor-k6ctg"
Sep 09 19:56:34.028229 ip-10-0-136-183 kubenswrapper[2574]: E0909 19:56:34.027926    2574 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 09 19:56:34.028229 ip-10-0-136-183 kubenswrapper[2574]: E0909 19:56:34.027994    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2eee1bd1-9d73-45d9-acf7-323648f0d6ae-insights-runtime-extractor-tls podName:2eee1bd1-9d73-45d9-acf7-323648f0d6ae nodeName:}" failed. No retries permitted until 2026-09-09 19:56:50.027977502 +0000 UTC m=+94.074230758 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/2eee1bd1-9d73-45d9-acf7-323648f0d6ae-insights-runtime-extractor-tls") pod "insights-runtime-extractor-k6ctg" (UID: "2eee1bd1-9d73-45d9-acf7-323648f0d6ae") : secret "insights-runtime-extractor-tls" not found
Sep 09 19:56:34.028229 ip-10-0-136-183 kubenswrapper[2574]: I0909 19:56:34.028171    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/2eee1bd1-9d73-45d9-acf7-323648f0d6ae-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-k6ctg\" (UID: \"2eee1bd1-9d73-45d9-acf7-323648f0d6ae\") " pod="openshift-insights/insights-runtime-extractor-k6ctg"
Sep 09 19:56:44.809777 ip-10-0-136-183 kubenswrapper[2574]: I0909 19:56:44.809660    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/12ae69a5-0173-433e-977f-1b13cc984906-networking-console-plugin-cert\") pod \"networking-console-plugin-79977f9844-4c8qr\" (UID: \"12ae69a5-0173-433e-977f-1b13cc984906\") " pod="openshift-network-console/networking-console-plugin-79977f9844-4c8qr"
Sep 09 19:56:44.812014 ip-10-0-136-183 kubenswrapper[2574]: I0909 19:56:44.811995    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/12ae69a5-0173-433e-977f-1b13cc984906-networking-console-plugin-cert\") pod \"networking-console-plugin-79977f9844-4c8qr\" (UID: \"12ae69a5-0173-433e-977f-1b13cc984906\") " pod="openshift-network-console/networking-console-plugin-79977f9844-4c8qr"
Sep 09 19:56:44.832024 ip-10-0-136-183 kubenswrapper[2574]: I0909 19:56:44.831999    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"default-dockercfg-9jdgt\""
Sep 09 19:56:44.840653 ip-10-0-136-183 kubenswrapper[2574]: I0909 19:56:44.840636    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-79977f9844-4c8qr"
Sep 09 19:56:45.011288 ip-10-0-136-183 kubenswrapper[2574]: I0909 19:56:45.011241    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/85e55211-a74b-4907-bc5b-887db353bdd7-metrics-certs\") pod \"router-default-58574c9bd-vdsvv\" (UID: \"85e55211-a74b-4907-bc5b-887db353bdd7\") " pod="openshift-ingress/router-default-58574c9bd-vdsvv"
Sep 09 19:56:45.011460 ip-10-0-136-183 kubenswrapper[2574]: I0909 19:56:45.011303    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/85e55211-a74b-4907-bc5b-887db353bdd7-service-ca-bundle\") pod \"router-default-58574c9bd-vdsvv\" (UID: \"85e55211-a74b-4907-bc5b-887db353bdd7\") " pod="openshift-ingress/router-default-58574c9bd-vdsvv"
Sep 09 19:56:45.011914 ip-10-0-136-183 kubenswrapper[2574]: I0909 19:56:45.011897    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/85e55211-a74b-4907-bc5b-887db353bdd7-service-ca-bundle\") pod \"router-default-58574c9bd-vdsvv\" (UID: \"85e55211-a74b-4907-bc5b-887db353bdd7\") " pod="openshift-ingress/router-default-58574c9bd-vdsvv"
Sep 09 19:56:45.013619 ip-10-0-136-183 kubenswrapper[2574]: I0909 19:56:45.013594    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/85e55211-a74b-4907-bc5b-887db353bdd7-metrics-certs\") pod \"router-default-58574c9bd-vdsvv\" (UID: \"85e55211-a74b-4907-bc5b-887db353bdd7\") " pod="openshift-ingress/router-default-58574c9bd-vdsvv"
Sep 09 19:56:45.055719 ip-10-0-136-183 kubenswrapper[2574]: I0909 19:56:45.055683    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-dockercfg-w8t5x\""
Sep 09 19:56:45.064373 ip-10-0-136-183 kubenswrapper[2574]: I0909 19:56:45.064307    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-58574c9bd-vdsvv"
Sep 09 19:56:45.108954 ip-10-0-136-183 kubenswrapper[2574]: I0909 19:56:45.108774    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-79977f9844-4c8qr"]
Sep 09 19:56:45.112968 ip-10-0-136-183 kubenswrapper[2574]: W0909 19:56:45.112920    2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod12ae69a5_0173_433e_977f_1b13cc984906.slice/crio-4644fd4e33c1ab38cb9b33bb33a4accac163a0a0b392572c0cd3425b8dd78cc7 WatchSource:0}: Error finding container 4644fd4e33c1ab38cb9b33bb33a4accac163a0a0b392572c0cd3425b8dd78cc7: Status 404 returned error can't find the container with id 4644fd4e33c1ab38cb9b33bb33a4accac163a0a0b392572c0cd3425b8dd78cc7
Sep 09 19:56:45.332659 ip-10-0-136-183 kubenswrapper[2574]: I0909 19:56:45.332584    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/router-default-58574c9bd-vdsvv"]
Sep 09 19:56:45.335571 ip-10-0-136-183 kubenswrapper[2574]: W0909 19:56:45.335539    2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod85e55211_a74b_4907_bc5b_887db353bdd7.slice/crio-5ea7e742cc8c255d8e2945d5241dd23cc6e7d8dc443a3022c56f4c63275b749a WatchSource:0}: Error finding container 5ea7e742cc8c255d8e2945d5241dd23cc6e7d8dc443a3022c56f4c63275b749a: Status 404 returned error can't find the container with id 5ea7e742cc8c255d8e2945d5241dd23cc6e7d8dc443a3022c56f4c63275b749a
Sep 09 19:56:45.793587 ip-10-0-136-183 kubenswrapper[2574]: I0909 19:56:45.792944    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-58574c9bd-vdsvv" event={"ID":"85e55211-a74b-4907-bc5b-887db353bdd7","Type":"ContainerStarted","Data":"9684043e395e4dd2807929974ac923f5727106f0f46f474bdb86e5b3647c6bd5"}
Sep 09 19:56:45.793587 ip-10-0-136-183 kubenswrapper[2574]: I0909 19:56:45.793011    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-58574c9bd-vdsvv" event={"ID":"85e55211-a74b-4907-bc5b-887db353bdd7","Type":"ContainerStarted","Data":"5ea7e742cc8c255d8e2945d5241dd23cc6e7d8dc443a3022c56f4c63275b749a"}
Sep 09 19:56:45.796363 ip-10-0-136-183 kubenswrapper[2574]: I0909 19:56:45.796335    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-79977f9844-4c8qr" event={"ID":"12ae69a5-0173-433e-977f-1b13cc984906","Type":"ContainerStarted","Data":"4644fd4e33c1ab38cb9b33bb33a4accac163a0a0b392572c0cd3425b8dd78cc7"}
Sep 09 19:56:45.816192 ip-10-0-136-183 kubenswrapper[2574]: I0909 19:56:45.816142    2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/router-default-58574c9bd-vdsvv" podStartSLOduration=32.816126519 podStartE2EDuration="32.816126519s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 19:56:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 19:56:45.814493294 +0000 UTC m=+89.860746559" watchObservedRunningTime="2026-09-09 19:56:45.816126519 +0000 UTC m=+89.862379781"
Sep 09 19:56:45.882127 ip-10-0-136-183 kubenswrapper[2574]: I0909 19:56:45.882087    2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-857f74bfc8-rtzrs"]
Sep 09 19:56:45.884048 ip-10-0-136-183 kubenswrapper[2574]: I0909 19:56:45.884027    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-857f74bfc8-rtzrs"
Sep 09 19:56:45.886244 ip-10-0-136-183 kubenswrapper[2574]: I0909 19:56:45.886223    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"open-cluster-management-image-pull-credentials\""
Sep 09 19:56:45.886762 ip-10-0-136-183 kubenswrapper[2574]: I0909 19:56:45.886721    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"work-manager-hub-kubeconfig\""
Sep 09 19:56:45.886762 ip-10-0-136-183 kubenswrapper[2574]: I0909 19:56:45.886737    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"kube-root-ca.crt\""
Sep 09 19:56:45.886762 ip-10-0-136-183 kubenswrapper[2574]: I0909 19:56:45.886736    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"openshift-service-ca.crt\""
Sep 09 19:56:45.911874 ip-10-0-136-183 kubenswrapper[2574]: I0909 19:56:45.911848    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-857f74bfc8-rtzrs"]
Sep 09 19:56:45.985985 ip-10-0-136-183 kubenswrapper[2574]: I0909 19:56:45.985948    2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-5f4c5fd4d7-jtgss"]
Sep 09 19:56:45.989449 ip-10-0-136-183 kubenswrapper[2574]: I0909 19:56:45.989420    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-5f4c5fd4d7-jtgss"
Sep 09 19:56:45.992196 ip-10-0-136-183 kubenswrapper[2574]: I0909 19:56:45.992164    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\""
Sep 09 19:56:45.992335 ip-10-0-136-183 kubenswrapper[2574]: I0909 19:56:45.992218    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-vz5cf\""
Sep 09 19:56:45.992335 ip-10-0-136-183 kubenswrapper[2574]: I0909 19:56:45.992251    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\""
Sep 09 19:56:45.992845 ip-10-0-136-183 kubenswrapper[2574]: I0909 19:56:45.992777    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\""
Sep 09 19:56:45.998674 ip-10-0-136-183 kubenswrapper[2574]: I0909 19:56:45.998644    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\""
Sep 09 19:56:46.002307 ip-10-0-136-183 kubenswrapper[2574]: I0909 19:56:46.002274    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-5f4c5fd4d7-jtgss"]
Sep 09 19:56:46.019462 ip-10-0-136-183 kubenswrapper[2574]: I0909 19:56:46.019434    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/0d577464-f025-4892-be7b-936334347012-tmp\") pod \"klusterlet-addon-workmgr-857f74bfc8-rtzrs\" (UID: \"0d577464-f025-4892-be7b-936334347012\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-857f74bfc8-rtzrs"
Sep 09 19:56:46.019593 ip-10-0-136-183 kubenswrapper[2574]: I0909 19:56:46.019467    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mvrcx\" (UniqueName: \"kubernetes.io/projected/0d577464-f025-4892-be7b-936334347012-kube-api-access-mvrcx\") pod \"klusterlet-addon-workmgr-857f74bfc8-rtzrs\" (UID: \"0d577464-f025-4892-be7b-936334347012\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-857f74bfc8-rtzrs"
Sep 09 19:56:46.019593 ip-10-0-136-183 kubenswrapper[2574]: I0909 19:56:46.019560    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/0d577464-f025-4892-be7b-936334347012-klusterlet-config\") pod \"klusterlet-addon-workmgr-857f74bfc8-rtzrs\" (UID: \"0d577464-f025-4892-be7b-936334347012\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-857f74bfc8-rtzrs"
Sep 09 19:56:46.064765 ip-10-0-136-183 kubenswrapper[2574]: I0909 19:56:46.064691    2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/router-default-58574c9bd-vdsvv"
Sep 09 19:56:46.067477 ip-10-0-136-183 kubenswrapper[2574]: I0909 19:56:46.067448    2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/router-default-58574c9bd-vdsvv"
Sep 09 19:56:46.120437 ip-10-0-136-183 kubenswrapper[2574]: I0909 19:56:46.120397    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/d69a5037-dd5e-4065-a877-9350010e5bb9-registry-certificates\") pod \"image-registry-5f4c5fd4d7-jtgss\" (UID: \"d69a5037-dd5e-4065-a877-9350010e5bb9\") " pod="openshift-image-registry/image-registry-5f4c5fd4d7-jtgss"
Sep 09 19:56:46.120616 ip-10-0-136-183 kubenswrapper[2574]: I0909 19:56:46.120448    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/0d577464-f025-4892-be7b-936334347012-tmp\") pod \"klusterlet-addon-workmgr-857f74bfc8-rtzrs\" (UID: \"0d577464-f025-4892-be7b-936334347012\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-857f74bfc8-rtzrs"
Sep 09 19:56:46.120616 ip-10-0-136-183 kubenswrapper[2574]: I0909 19:56:46.120480    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/d69a5037-dd5e-4065-a877-9350010e5bb9-image-registry-private-configuration\") pod \"image-registry-5f4c5fd4d7-jtgss\" (UID: \"d69a5037-dd5e-4065-a877-9350010e5bb9\") " pod="openshift-image-registry/image-registry-5f4c5fd4d7-jtgss"
Sep 09 19:56:46.120616 ip-10-0-136-183 kubenswrapper[2574]: I0909 19:56:46.120558    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/d69a5037-dd5e-4065-a877-9350010e5bb9-trusted-ca\") pod \"image-registry-5f4c5fd4d7-jtgss\" (UID: \"d69a5037-dd5e-4065-a877-9350010e5bb9\") " pod="openshift-image-registry/image-registry-5f4c5fd4d7-jtgss"
Sep 09 19:56:46.120616 ip-10-0-136-183 kubenswrapper[2574]: I0909 19:56:46.120583    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/d69a5037-dd5e-4065-a877-9350010e5bb9-registry-tls\") pod \"image-registry-5f4c5fd4d7-jtgss\" (UID: \"d69a5037-dd5e-4065-a877-9350010e5bb9\") " pod="openshift-image-registry/image-registry-5f4c5fd4d7-jtgss"
Sep 09 19:56:46.120616 ip-10-0-136-183 kubenswrapper[2574]: I0909 19:56:46.120607    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/d69a5037-dd5e-4065-a877-9350010e5bb9-ca-trust-extracted\") pod \"image-registry-5f4c5fd4d7-jtgss\" (UID: \"d69a5037-dd5e-4065-a877-9350010e5bb9\") " pod="openshift-image-registry/image-registry-5f4c5fd4d7-jtgss"
Sep 09 19:56:46.120909 ip-10-0-136-183 kubenswrapper[2574]: I0909 19:56:46.120661    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mvrcx\" (UniqueName: \"kubernetes.io/projected/0d577464-f025-4892-be7b-936334347012-kube-api-access-mvrcx\") pod \"klusterlet-addon-workmgr-857f74bfc8-rtzrs\" (UID: \"0d577464-f025-4892-be7b-936334347012\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-857f74bfc8-rtzrs"
Sep 09 19:56:46.120909 ip-10-0-136-183 kubenswrapper[2574]: I0909 19:56:46.120709    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/d69a5037-dd5e-4065-a877-9350010e5bb9-installation-pull-secrets\") pod \"image-registry-5f4c5fd4d7-jtgss\" (UID: \"d69a5037-dd5e-4065-a877-9350010e5bb9\") " pod="openshift-image-registry/image-registry-5f4c5fd4d7-jtgss"
Sep 09 19:56:46.120909 ip-10-0-136-183 kubenswrapper[2574]: I0909 19:56:46.120847    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5ctbc\" (UniqueName: \"kubernetes.io/projected/d69a5037-dd5e-4065-a877-9350010e5bb9-kube-api-access-5ctbc\") pod \"image-registry-5f4c5fd4d7-jtgss\" (UID: \"d69a5037-dd5e-4065-a877-9350010e5bb9\") " pod="openshift-image-registry/image-registry-5f4c5fd4d7-jtgss"
Sep 09 19:56:46.120909 ip-10-0-136-183 kubenswrapper[2574]: I0909 19:56:46.120865    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/0d577464-f025-4892-be7b-936334347012-tmp\") pod \"klusterlet-addon-workmgr-857f74bfc8-rtzrs\" (UID: \"0d577464-f025-4892-be7b-936334347012\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-857f74bfc8-rtzrs"
Sep 09 19:56:46.121098 ip-10-0-136-183 kubenswrapper[2574]: I0909 19:56:46.120939    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/0d577464-f025-4892-be7b-936334347012-klusterlet-config\") pod \"klusterlet-addon-workmgr-857f74bfc8-rtzrs\" (UID: \"0d577464-f025-4892-be7b-936334347012\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-857f74bfc8-rtzrs"
Sep 09 19:56:46.121098 ip-10-0-136-183 kubenswrapper[2574]: I0909 19:56:46.120970    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/d69a5037-dd5e-4065-a877-9350010e5bb9-bound-sa-token\") pod \"image-registry-5f4c5fd4d7-jtgss\" (UID: \"d69a5037-dd5e-4065-a877-9350010e5bb9\") " pod="openshift-image-registry/image-registry-5f4c5fd4d7-jtgss"
Sep 09 19:56:46.123976 ip-10-0-136-183 kubenswrapper[2574]: I0909 19:56:46.123949    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/0d577464-f025-4892-be7b-936334347012-klusterlet-config\") pod \"klusterlet-addon-workmgr-857f74bfc8-rtzrs\" (UID: \"0d577464-f025-4892-be7b-936334347012\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-857f74bfc8-rtzrs"
Sep 09 19:56:46.131667 ip-10-0-136-183 kubenswrapper[2574]: I0909 19:56:46.131637    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mvrcx\" (UniqueName: \"kubernetes.io/projected/0d577464-f025-4892-be7b-936334347012-kube-api-access-mvrcx\") pod \"klusterlet-addon-workmgr-857f74bfc8-rtzrs\" (UID: \"0d577464-f025-4892-be7b-936334347012\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-857f74bfc8-rtzrs"
Sep 09 19:56:46.195309 ip-10-0-136-183 kubenswrapper[2574]: I0909 19:56:46.195273    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-857f74bfc8-rtzrs"
Sep 09 19:56:46.221979 ip-10-0-136-183 kubenswrapper[2574]: I0909 19:56:46.221941    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/d69a5037-dd5e-4065-a877-9350010e5bb9-trusted-ca\") pod \"image-registry-5f4c5fd4d7-jtgss\" (UID: \"d69a5037-dd5e-4065-a877-9350010e5bb9\") " pod="openshift-image-registry/image-registry-5f4c5fd4d7-jtgss"
Sep 09 19:56:46.222122 ip-10-0-136-183 kubenswrapper[2574]: I0909 19:56:46.221981    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/d69a5037-dd5e-4065-a877-9350010e5bb9-registry-tls\") pod \"image-registry-5f4c5fd4d7-jtgss\" (UID: \"d69a5037-dd5e-4065-a877-9350010e5bb9\") " pod="openshift-image-registry/image-registry-5f4c5fd4d7-jtgss"
Sep 09 19:56:46.222122 ip-10-0-136-183 kubenswrapper[2574]: I0909 19:56:46.222011    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/d69a5037-dd5e-4065-a877-9350010e5bb9-ca-trust-extracted\") pod \"image-registry-5f4c5fd4d7-jtgss\" (UID: \"d69a5037-dd5e-4065-a877-9350010e5bb9\") " pod="openshift-image-registry/image-registry-5f4c5fd4d7-jtgss"
Sep 09 19:56:46.222122 ip-10-0-136-183 kubenswrapper[2574]: I0909 19:56:46.222042    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/d69a5037-dd5e-4065-a877-9350010e5bb9-installation-pull-secrets\") pod \"image-registry-5f4c5fd4d7-jtgss\" (UID: \"d69a5037-dd5e-4065-a877-9350010e5bb9\") " pod="openshift-image-registry/image-registry-5f4c5fd4d7-jtgss"
Sep 09 19:56:46.222122 ip-10-0-136-183 kubenswrapper[2574]: I0909 19:56:46.222068    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5ctbc\" (UniqueName: \"kubernetes.io/projected/d69a5037-dd5e-4065-a877-9350010e5bb9-kube-api-access-5ctbc\") pod \"image-registry-5f4c5fd4d7-jtgss\" (UID: \"d69a5037-dd5e-4065-a877-9350010e5bb9\") " pod="openshift-image-registry/image-registry-5f4c5fd4d7-jtgss"
Sep 09 19:56:46.222122 ip-10-0-136-183 kubenswrapper[2574]: I0909 19:56:46.222121    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/d69a5037-dd5e-4065-a877-9350010e5bb9-bound-sa-token\") pod \"image-registry-5f4c5fd4d7-jtgss\" (UID: \"d69a5037-dd5e-4065-a877-9350010e5bb9\") " pod="openshift-image-registry/image-registry-5f4c5fd4d7-jtgss"
Sep 09 19:56:46.222390 ip-10-0-136-183 kubenswrapper[2574]: I0909 19:56:46.222167    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/d69a5037-dd5e-4065-a877-9350010e5bb9-registry-certificates\") pod \"image-registry-5f4c5fd4d7-jtgss\" (UID: \"d69a5037-dd5e-4065-a877-9350010e5bb9\") " pod="openshift-image-registry/image-registry-5f4c5fd4d7-jtgss"
Sep 09 19:56:46.222390 ip-10-0-136-183 kubenswrapper[2574]: I0909 19:56:46.222194    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/d69a5037-dd5e-4065-a877-9350010e5bb9-image-registry-private-configuration\") pod \"image-registry-5f4c5fd4d7-jtgss\" (UID: \"d69a5037-dd5e-4065-a877-9350010e5bb9\") " pod="openshift-image-registry/image-registry-5f4c5fd4d7-jtgss"
Sep 09 19:56:46.223009 ip-10-0-136-183 kubenswrapper[2574]: I0909 19:56:46.222983    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/d69a5037-dd5e-4065-a877-9350010e5bb9-ca-trust-extracted\") pod \"image-registry-5f4c5fd4d7-jtgss\" (UID: \"d69a5037-dd5e-4065-a877-9350010e5bb9\") " pod="openshift-image-registry/image-registry-5f4c5fd4d7-jtgss"
Sep 09 19:56:46.223116 ip-10-0-136-183 kubenswrapper[2574]: I0909 19:56:46.223019    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/d69a5037-dd5e-4065-a877-9350010e5bb9-trusted-ca\") pod \"image-registry-5f4c5fd4d7-jtgss\" (UID: \"d69a5037-dd5e-4065-a877-9350010e5bb9\") " pod="openshift-image-registry/image-registry-5f4c5fd4d7-jtgss"
Sep 09 19:56:46.223116 ip-10-0-136-183 kubenswrapper[2574]: I0909 19:56:46.223066    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/d69a5037-dd5e-4065-a877-9350010e5bb9-registry-certificates\") pod \"image-registry-5f4c5fd4d7-jtgss\" (UID: \"d69a5037-dd5e-4065-a877-9350010e5bb9\") " pod="openshift-image-registry/image-registry-5f4c5fd4d7-jtgss"
Sep 09 19:56:46.224925 ip-10-0-136-183 kubenswrapper[2574]: I0909 19:56:46.224904    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/d69a5037-dd5e-4065-a877-9350010e5bb9-registry-tls\") pod \"image-registry-5f4c5fd4d7-jtgss\" (UID: \"d69a5037-dd5e-4065-a877-9350010e5bb9\") " pod="openshift-image-registry/image-registry-5f4c5fd4d7-jtgss"
Sep 09 19:56:46.225231 ip-10-0-136-183 kubenswrapper[2574]: I0909 19:56:46.225211    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/d69a5037-dd5e-4065-a877-9350010e5bb9-installation-pull-secrets\") pod \"image-registry-5f4c5fd4d7-jtgss\" (UID: \"d69a5037-dd5e-4065-a877-9350010e5bb9\") " pod="openshift-image-registry/image-registry-5f4c5fd4d7-jtgss"
Sep 09 19:56:46.225376 ip-10-0-136-183 kubenswrapper[2574]: I0909 19:56:46.225359    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/d69a5037-dd5e-4065-a877-9350010e5bb9-image-registry-private-configuration\") pod \"image-registry-5f4c5fd4d7-jtgss\" (UID: \"d69a5037-dd5e-4065-a877-9350010e5bb9\") " pod="openshift-image-registry/image-registry-5f4c5fd4d7-jtgss"
Sep 09 19:56:46.234094 ip-10-0-136-183 kubenswrapper[2574]: I0909 19:56:46.234044    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5ctbc\" (UniqueName: \"kubernetes.io/projected/d69a5037-dd5e-4065-a877-9350010e5bb9-kube-api-access-5ctbc\") pod \"image-registry-5f4c5fd4d7-jtgss\" (UID: \"d69a5037-dd5e-4065-a877-9350010e5bb9\") " pod="openshift-image-registry/image-registry-5f4c5fd4d7-jtgss"
Sep 09 19:56:46.234583 ip-10-0-136-183 kubenswrapper[2574]: I0909 19:56:46.234541    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/d69a5037-dd5e-4065-a877-9350010e5bb9-bound-sa-token\") pod \"image-registry-5f4c5fd4d7-jtgss\" (UID: \"d69a5037-dd5e-4065-a877-9350010e5bb9\") " pod="openshift-image-registry/image-registry-5f4c5fd4d7-jtgss"
Sep 09 19:56:46.301117 ip-10-0-136-183 kubenswrapper[2574]: I0909 19:56:46.301080    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-5f4c5fd4d7-jtgss"
Sep 09 19:56:46.470670 ip-10-0-136-183 kubenswrapper[2574]: I0909 19:56:46.470629    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-857f74bfc8-rtzrs"]
Sep 09 19:56:46.473977 ip-10-0-136-183 kubenswrapper[2574]: W0909 19:56:46.473955    2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0d577464_f025_4892_be7b_936334347012.slice/crio-ee42c7c135aaedef0465d15107a004f17c4092f7d5d8d75f7bcf0ca6ab172b7b WatchSource:0}: Error finding container ee42c7c135aaedef0465d15107a004f17c4092f7d5d8d75f7bcf0ca6ab172b7b: Status 404 returned error can't find the container with id ee42c7c135aaedef0465d15107a004f17c4092f7d5d8d75f7bcf0ca6ab172b7b
Sep 09 19:56:46.570181 ip-10-0-136-183 kubenswrapper[2574]: I0909 19:56:46.570151    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-5f4c5fd4d7-jtgss"]
Sep 09 19:56:46.573235 ip-10-0-136-183 kubenswrapper[2574]: W0909 19:56:46.573212    2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd69a5037_dd5e_4065_a877_9350010e5bb9.slice/crio-465268be4884d3ec074216686eee046d16a5c1e7765cf1ef59618ea384acb30a WatchSource:0}: Error finding container 465268be4884d3ec074216686eee046d16a5c1e7765cf1ef59618ea384acb30a: Status 404 returned error can't find the container with id 465268be4884d3ec074216686eee046d16a5c1e7765cf1ef59618ea384acb30a
Sep 09 19:56:46.800655 ip-10-0-136-183 kubenswrapper[2574]: I0909 19:56:46.800619    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-857f74bfc8-rtzrs" event={"ID":"0d577464-f025-4892-be7b-936334347012","Type":"ContainerStarted","Data":"ee42c7c135aaedef0465d15107a004f17c4092f7d5d8d75f7bcf0ca6ab172b7b"}
Sep 09 19:56:46.802150 ip-10-0-136-183 kubenswrapper[2574]: I0909 19:56:46.802126    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-79977f9844-4c8qr" event={"ID":"12ae69a5-0173-433e-977f-1b13cc984906","Type":"ContainerStarted","Data":"f8793364bb01914ef41061779180b2fe34067913081aed1ae031ba54d7f9d29f"}
Sep 09 19:56:46.803715 ip-10-0-136-183 kubenswrapper[2574]: I0909 19:56:46.803689    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-5f4c5fd4d7-jtgss" event={"ID":"d69a5037-dd5e-4065-a877-9350010e5bb9","Type":"ContainerStarted","Data":"42485a006adc2c0399179364ce62ac259401203484759d29f2c49b4094be1d40"}
Sep 09 19:56:46.803819 ip-10-0-136-183 kubenswrapper[2574]: I0909 19:56:46.803729    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-5f4c5fd4d7-jtgss" event={"ID":"d69a5037-dd5e-4065-a877-9350010e5bb9","Type":"ContainerStarted","Data":"465268be4884d3ec074216686eee046d16a5c1e7765cf1ef59618ea384acb30a"}
Sep 09 19:56:46.804114 ip-10-0-136-183 kubenswrapper[2574]: I0909 19:56:46.804088    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/router-default-58574c9bd-vdsvv"
Sep 09 19:56:46.806622 ip-10-0-136-183 kubenswrapper[2574]: I0909 19:56:46.806599    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/router-default-58574c9bd-vdsvv"
Sep 09 19:56:46.820451 ip-10-0-136-183 kubenswrapper[2574]: I0909 19:56:46.820408    2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-console/networking-console-plugin-79977f9844-4c8qr" podStartSLOduration=32.733081671 podStartE2EDuration="33.820395469s" totalImagesPullingTime="1.087313798s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 19:56:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 19:56:46.8189724 +0000 UTC m=+90.865225661" watchObservedRunningTime="2026-09-09 19:56:46.820395469 +0000 UTC m=+90.866648731"
Sep 09 19:56:46.861779 ip-10-0-136-183 kubenswrapper[2574]: I0909 19:56:46.861680    2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/image-registry-5f4c5fd4d7-jtgss" podStartSLOduration=1.861663236 podStartE2EDuration="1.861663236s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 19:56:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 19:56:46.861481576 +0000 UTC m=+90.907734839" watchObservedRunningTime="2026-09-09 19:56:46.861663236 +0000 UTC m=+90.907916489"
Sep 09 19:56:47.806456 ip-10-0-136-183 kubenswrapper[2574]: I0909 19:56:47.806420    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-5f4c5fd4d7-jtgss"
Sep 09 19:56:50.052551 ip-10-0-136-183 kubenswrapper[2574]: I0909 19:56:50.052511    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/2eee1bd1-9d73-45d9-acf7-323648f0d6ae-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-k6ctg\" (UID: \"2eee1bd1-9d73-45d9-acf7-323648f0d6ae\") " pod="openshift-insights/insights-runtime-extractor-k6ctg"
Sep 09 19:56:50.054718 ip-10-0-136-183 kubenswrapper[2574]: I0909 19:56:50.054691    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/2eee1bd1-9d73-45d9-acf7-323648f0d6ae-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-k6ctg\" (UID: \"2eee1bd1-9d73-45d9-acf7-323648f0d6ae\") " pod="openshift-insights/insights-runtime-extractor-k6ctg"
Sep 09 19:56:50.073612 ip-10-0-136-183 kubenswrapper[2574]: I0909 19:56:50.073581    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-k6ctg"
Sep 09 19:56:52.094931 ip-10-0-136-183 kubenswrapper[2574]: I0909 19:56:52.094840    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-k6ctg"]
Sep 09 19:56:52.098276 ip-10-0-136-183 kubenswrapper[2574]: W0909 19:56:52.098244    2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2eee1bd1_9d73_45d9_acf7_323648f0d6ae.slice/crio-c57e14a5c3a9216af2dde0f72fc29dcca8c51e07f6da09dee7cb814c3bed3e5a WatchSource:0}: Error finding container c57e14a5c3a9216af2dde0f72fc29dcca8c51e07f6da09dee7cb814c3bed3e5a: Status 404 returned error can't find the container with id c57e14a5c3a9216af2dde0f72fc29dcca8c51e07f6da09dee7cb814c3bed3e5a
Sep 09 19:56:52.175752 ip-10-0-136-183 kubenswrapper[2574]: I0909 19:56:52.175722    2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-6f4946db56-ht869"]
Sep 09 19:56:52.178942 ip-10-0-136-183 kubenswrapper[2574]: I0909 19:56:52.178927    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-6f4946db56-ht869"
Sep 09 19:56:52.181113 ip-10-0-136-183 kubenswrapper[2574]: I0909 19:56:52.181093    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-tls\""
Sep 09 19:56:52.181253 ip-10-0-136-183 kubenswrapper[2574]: I0909 19:56:52.181120    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-kube-rbac-proxy-config\""
Sep 09 19:56:52.181323 ip-10-0-136-183 kubenswrapper[2574]: I0909 19:56:52.181192    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\""
Sep 09 19:56:52.181789 ip-10-0-136-183 kubenswrapper[2574]: I0909 19:56:52.181768    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-dockercfg-psg6z\""
Sep 09 19:56:52.181913 ip-10-0-136-183 kubenswrapper[2574]: I0909 19:56:52.181771    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
Sep 09 19:56:52.181913 ip-10-0-136-183 kubenswrapper[2574]: I0909 19:56:52.181812    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
Sep 09 19:56:52.187335 ip-10-0-136-183 kubenswrapper[2574]: I0909 19:56:52.187314    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-6f4946db56-ht869"]
Sep 09 19:56:52.270607 ip-10-0-136-183 kubenswrapper[2574]: I0909 19:56:52.270580    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8g4sm\" (UniqueName: \"kubernetes.io/projected/09d0544d-3823-4cb2-8611-82a74d206a45-kube-api-access-8g4sm\") pod \"prometheus-operator-6f4946db56-ht869\" (UID: \"09d0544d-3823-4cb2-8611-82a74d206a45\") " pod="openshift-monitoring/prometheus-operator-6f4946db56-ht869"
Sep 09 19:56:52.270707 ip-10-0-136-183 kubenswrapper[2574]: I0909 19:56:52.270687    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/09d0544d-3823-4cb2-8611-82a74d206a45-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-6f4946db56-ht869\" (UID: \"09d0544d-3823-4cb2-8611-82a74d206a45\") " pod="openshift-monitoring/prometheus-operator-6f4946db56-ht869"
Sep 09 19:56:52.270745 ip-10-0-136-183 kubenswrapper[2574]: I0909 19:56:52.270708    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/09d0544d-3823-4cb2-8611-82a74d206a45-prometheus-operator-tls\") pod \"prometheus-operator-6f4946db56-ht869\" (UID: \"09d0544d-3823-4cb2-8611-82a74d206a45\") " pod="openshift-monitoring/prometheus-operator-6f4946db56-ht869"
Sep 09 19:56:52.270745 ip-10-0-136-183 kubenswrapper[2574]: I0909 19:56:52.270725    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/09d0544d-3823-4cb2-8611-82a74d206a45-metrics-client-ca\") pod \"prometheus-operator-6f4946db56-ht869\" (UID: \"09d0544d-3823-4cb2-8611-82a74d206a45\") " pod="openshift-monitoring/prometheus-operator-6f4946db56-ht869"
Sep 09 19:56:52.371997 ip-10-0-136-183 kubenswrapper[2574]: I0909 19:56:52.371964    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/09d0544d-3823-4cb2-8611-82a74d206a45-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-6f4946db56-ht869\" (UID: \"09d0544d-3823-4cb2-8611-82a74d206a45\") " pod="openshift-monitoring/prometheus-operator-6f4946db56-ht869"
Sep 09 19:56:52.371997 ip-10-0-136-183 kubenswrapper[2574]: I0909 19:56:52.372000    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/09d0544d-3823-4cb2-8611-82a74d206a45-prometheus-operator-tls\") pod \"prometheus-operator-6f4946db56-ht869\" (UID: \"09d0544d-3823-4cb2-8611-82a74d206a45\") " pod="openshift-monitoring/prometheus-operator-6f4946db56-ht869"
Sep 09 19:56:52.372234 ip-10-0-136-183 kubenswrapper[2574]: I0909 19:56:52.372017    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/09d0544d-3823-4cb2-8611-82a74d206a45-metrics-client-ca\") pod \"prometheus-operator-6f4946db56-ht869\" (UID: \"09d0544d-3823-4cb2-8611-82a74d206a45\") " pod="openshift-monitoring/prometheus-operator-6f4946db56-ht869"
Sep 09 19:56:52.372234 ip-10-0-136-183 kubenswrapper[2574]: I0909 19:56:52.372047    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8g4sm\" (UniqueName: \"kubernetes.io/projected/09d0544d-3823-4cb2-8611-82a74d206a45-kube-api-access-8g4sm\") pod \"prometheus-operator-6f4946db56-ht869\" (UID: \"09d0544d-3823-4cb2-8611-82a74d206a45\") " pod="openshift-monitoring/prometheus-operator-6f4946db56-ht869"
Sep 09 19:56:52.372709 ip-10-0-136-183 kubenswrapper[2574]: I0909 19:56:52.372674    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/09d0544d-3823-4cb2-8611-82a74d206a45-metrics-client-ca\") pod \"prometheus-operator-6f4946db56-ht869\" (UID: \"09d0544d-3823-4cb2-8611-82a74d206a45\") " pod="openshift-monitoring/prometheus-operator-6f4946db56-ht869"
Sep 09 19:56:52.374461 ip-10-0-136-183 kubenswrapper[2574]: I0909 19:56:52.374443    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/09d0544d-3823-4cb2-8611-82a74d206a45-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-6f4946db56-ht869\" (UID: \"09d0544d-3823-4cb2-8611-82a74d206a45\") " pod="openshift-monitoring/prometheus-operator-6f4946db56-ht869"
Sep 09 19:56:52.374594 ip-10-0-136-183 kubenswrapper[2574]: I0909 19:56:52.374578    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/09d0544d-3823-4cb2-8611-82a74d206a45-prometheus-operator-tls\") pod \"prometheus-operator-6f4946db56-ht869\" (UID: \"09d0544d-3823-4cb2-8611-82a74d206a45\") " pod="openshift-monitoring/prometheus-operator-6f4946db56-ht869"
Sep 09 19:56:52.381477 ip-10-0-136-183 kubenswrapper[2574]: I0909 19:56:52.381446    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8g4sm\" (UniqueName: \"kubernetes.io/projected/09d0544d-3823-4cb2-8611-82a74d206a45-kube-api-access-8g4sm\") pod \"prometheus-operator-6f4946db56-ht869\" (UID: \"09d0544d-3823-4cb2-8611-82a74d206a45\") " pod="openshift-monitoring/prometheus-operator-6f4946db56-ht869"
Sep 09 19:56:52.490381 ip-10-0-136-183 kubenswrapper[2574]: I0909 19:56:52.490347    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-6f4946db56-ht869"
Sep 09 19:56:52.674776 ip-10-0-136-183 kubenswrapper[2574]: I0909 19:56:52.674689    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/3a81bb5a-0683-466e-b32e-960f4ca22ae6-metrics-tls\") pod \"dns-default-4blgq\" (UID: \"3a81bb5a-0683-466e-b32e-960f4ca22ae6\") " pod="openshift-dns/dns-default-4blgq"
Sep 09 19:56:52.677217 ip-10-0-136-183 kubenswrapper[2574]: I0909 19:56:52.677189    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/3a81bb5a-0683-466e-b32e-960f4ca22ae6-metrics-tls\") pod \"dns-default-4blgq\" (UID: \"3a81bb5a-0683-466e-b32e-960f4ca22ae6\") " pod="openshift-dns/dns-default-4blgq"
Sep 09 19:56:52.761993 ip-10-0-136-183 kubenswrapper[2574]: I0909 19:56:52.761950    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-6f4946db56-ht869"]
Sep 09 19:56:52.768127 ip-10-0-136-183 kubenswrapper[2574]: W0909 19:56:52.768098    2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod09d0544d_3823_4cb2_8611_82a74d206a45.slice/crio-7bcbe5adc138f788249a6b3365b34adeed6d4f5f563ef73ac28cce9811d22dfc WatchSource:0}: Error finding container 7bcbe5adc138f788249a6b3365b34adeed6d4f5f563ef73ac28cce9811d22dfc: Status 404 returned error can't find the container with id 7bcbe5adc138f788249a6b3365b34adeed6d4f5f563ef73ac28cce9811d22dfc
Sep 09 19:56:52.827694 ip-10-0-136-183 kubenswrapper[2574]: I0909 19:56:52.827643    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-857f74bfc8-rtzrs" event={"ID":"0d577464-f025-4892-be7b-936334347012","Type":"ContainerStarted","Data":"e44630fc972dfe1d62cf326cf08a21bce60e9152ff5ea6c0724ca6a26ff02c73"}
Sep 09 19:56:52.827883 ip-10-0-136-183 kubenswrapper[2574]: I0909 19:56:52.827833    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-857f74bfc8-rtzrs"
Sep 09 19:56:52.829072 ip-10-0-136-183 kubenswrapper[2574]: I0909 19:56:52.829035    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-6f4946db56-ht869" event={"ID":"09d0544d-3823-4cb2-8611-82a74d206a45","Type":"ContainerStarted","Data":"7bcbe5adc138f788249a6b3365b34adeed6d4f5f563ef73ac28cce9811d22dfc"}
Sep 09 19:56:52.830001 ip-10-0-136-183 kubenswrapper[2574]: I0909 19:56:52.829979    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-857f74bfc8-rtzrs"
Sep 09 19:56:52.830634 ip-10-0-136-183 kubenswrapper[2574]: I0909 19:56:52.830611    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-k6ctg" event={"ID":"2eee1bd1-9d73-45d9-acf7-323648f0d6ae","Type":"ContainerStarted","Data":"1916b82e15c9fa865345a815758a7746ed5e3d8fd9bd814530340cbeb3293448"}
Sep 09 19:56:52.830756 ip-10-0-136-183 kubenswrapper[2574]: I0909 19:56:52.830661    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-k6ctg" event={"ID":"2eee1bd1-9d73-45d9-acf7-323648f0d6ae","Type":"ContainerStarted","Data":"c57e14a5c3a9216af2dde0f72fc29dcca8c51e07f6da09dee7cb814c3bed3e5a"}
Sep 09 19:56:52.843074 ip-10-0-136-183 kubenswrapper[2574]: I0909 19:56:52.843025    2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-857f74bfc8-rtzrs" podStartSLOduration=2.414194526 podStartE2EDuration="7.843010729s" totalImagesPullingTime="5.428816203s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 19:56:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 19:56:52.842519652 +0000 UTC m=+96.888772916" watchObservedRunningTime="2026-09-09 19:56:52.843010729 +0000 UTC m=+96.889264012"
Sep 09 19:56:52.949209 ip-10-0-136-183 kubenswrapper[2574]: I0909 19:56:52.949122    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-28vq4\""
Sep 09 19:56:52.957584 ip-10-0-136-183 kubenswrapper[2574]: I0909 19:56:52.957565    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-4blgq"
Sep 09 19:56:53.230555 ip-10-0-136-183 kubenswrapper[2574]: I0909 19:56:53.230484    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-4blgq"]
Sep 09 19:56:53.233740 ip-10-0-136-183 kubenswrapper[2574]: W0909 19:56:53.233711    2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3a81bb5a_0683_466e_b32e_960f4ca22ae6.slice/crio-85dd66ba434df3603837665444d5978e04b85f9742b51087b35b696050d17c82 WatchSource:0}: Error finding container 85dd66ba434df3603837665444d5978e04b85f9742b51087b35b696050d17c82: Status 404 returned error can't find the container with id 85dd66ba434df3603837665444d5978e04b85f9742b51087b35b696050d17c82
Sep 09 19:56:53.729546 ip-10-0-136-183 kubenswrapper[2574]: I0909 19:56:53.729162    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-4czxv"
Sep 09 19:56:53.836121 ip-10-0-136-183 kubenswrapper[2574]: I0909 19:56:53.836023    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-4blgq" event={"ID":"3a81bb5a-0683-466e-b32e-960f4ca22ae6","Type":"ContainerStarted","Data":"85dd66ba434df3603837665444d5978e04b85f9742b51087b35b696050d17c82"}
Sep 09 19:56:54.563921 ip-10-0-136-183 kubenswrapper[2574]: I0909 19:56:54.563886    2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-6dbd947b68-2rr89"]
Sep 09 19:56:54.567468 ip-10-0-136-183 kubenswrapper[2574]: I0909 19:56:54.567443    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6dbd947b68-2rr89"
Sep 09 19:56:54.569475 ip-10-0-136-183 kubenswrapper[2574]: I0909 19:56:54.569450    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\""
Sep 09 19:56:54.570231 ip-10-0-136-183 kubenswrapper[2574]: I0909 19:56:54.570208    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\""
Sep 09 19:56:54.570346 ip-10-0-136-183 kubenswrapper[2574]: I0909 19:56:54.570278    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\""
Sep 09 19:56:54.570346 ip-10-0-136-183 kubenswrapper[2574]: I0909 19:56:54.570288    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\""
Sep 09 19:56:54.570346 ip-10-0-136-183 kubenswrapper[2574]: I0909 19:56:54.570297    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-nvnxj\""
Sep 09 19:56:54.570346 ip-10-0-136-183 kubenswrapper[2574]: I0909 19:56:54.570307    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\""
Sep 09 19:56:54.570346 ip-10-0-136-183 kubenswrapper[2574]: I0909 19:56:54.570340    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\""
Sep 09 19:56:54.570631 ip-10-0-136-183 kubenswrapper[2574]: I0909 19:56:54.570285    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\""
Sep 09 19:56:54.577559 ip-10-0-136-183 kubenswrapper[2574]: I0909 19:56:54.577531    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-6dbd947b68-2rr89"]
Sep 09 19:56:54.690891 ip-10-0-136-183 kubenswrapper[2574]: I0909 19:56:54.690849    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/120141dc-7742-4117-9052-6ab138ced47a-console-serving-cert\") pod \"console-6dbd947b68-2rr89\" (UID: \"120141dc-7742-4117-9052-6ab138ced47a\") " pod="openshift-console/console-6dbd947b68-2rr89"
Sep 09 19:56:54.691077 ip-10-0-136-183 kubenswrapper[2574]: I0909 19:56:54.690903    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/120141dc-7742-4117-9052-6ab138ced47a-console-config\") pod \"console-6dbd947b68-2rr89\" (UID: \"120141dc-7742-4117-9052-6ab138ced47a\") " pod="openshift-console/console-6dbd947b68-2rr89"
Sep 09 19:56:54.691077 ip-10-0-136-183 kubenswrapper[2574]: I0909 19:56:54.690975    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/120141dc-7742-4117-9052-6ab138ced47a-oauth-serving-cert\") pod \"console-6dbd947b68-2rr89\" (UID: \"120141dc-7742-4117-9052-6ab138ced47a\") " pod="openshift-console/console-6dbd947b68-2rr89"
Sep 09 19:56:54.691077 ip-10-0-136-183 kubenswrapper[2574]: I0909 19:56:54.691055    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rksfs\" (UniqueName: \"kubernetes.io/projected/120141dc-7742-4117-9052-6ab138ced47a-kube-api-access-rksfs\") pod \"console-6dbd947b68-2rr89\" (UID: \"120141dc-7742-4117-9052-6ab138ced47a\") " pod="openshift-console/console-6dbd947b68-2rr89"
Sep 09 19:56:54.691241 ip-10-0-136-183 kubenswrapper[2574]: I0909 19:56:54.691092    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/120141dc-7742-4117-9052-6ab138ced47a-console-oauth-config\") pod \"console-6dbd947b68-2rr89\" (UID: \"120141dc-7742-4117-9052-6ab138ced47a\") " pod="openshift-console/console-6dbd947b68-2rr89"
Sep 09 19:56:54.691241 ip-10-0-136-183 kubenswrapper[2574]: I0909 19:56:54.691126    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/120141dc-7742-4117-9052-6ab138ced47a-service-ca\") pod \"console-6dbd947b68-2rr89\" (UID: \"120141dc-7742-4117-9052-6ab138ced47a\") " pod="openshift-console/console-6dbd947b68-2rr89"
Sep 09 19:56:54.792429 ip-10-0-136-183 kubenswrapper[2574]: I0909 19:56:54.792390    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/120141dc-7742-4117-9052-6ab138ced47a-service-ca\") pod \"console-6dbd947b68-2rr89\" (UID: \"120141dc-7742-4117-9052-6ab138ced47a\") " pod="openshift-console/console-6dbd947b68-2rr89"
Sep 09 19:56:54.792429 ip-10-0-136-183 kubenswrapper[2574]: I0909 19:56:54.792431    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/120141dc-7742-4117-9052-6ab138ced47a-console-serving-cert\") pod \"console-6dbd947b68-2rr89\" (UID: \"120141dc-7742-4117-9052-6ab138ced47a\") " pod="openshift-console/console-6dbd947b68-2rr89"
Sep 09 19:56:54.792653 ip-10-0-136-183 kubenswrapper[2574]: I0909 19:56:54.792452    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/120141dc-7742-4117-9052-6ab138ced47a-console-config\") pod \"console-6dbd947b68-2rr89\" (UID: \"120141dc-7742-4117-9052-6ab138ced47a\") " pod="openshift-console/console-6dbd947b68-2rr89"
Sep 09 19:56:54.792653 ip-10-0-136-183 kubenswrapper[2574]: I0909 19:56:54.792595    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/120141dc-7742-4117-9052-6ab138ced47a-oauth-serving-cert\") pod \"console-6dbd947b68-2rr89\" (UID: \"120141dc-7742-4117-9052-6ab138ced47a\") " pod="openshift-console/console-6dbd947b68-2rr89"
Sep 09 19:56:54.792761 ip-10-0-136-183 kubenswrapper[2574]: I0909 19:56:54.792682    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rksfs\" (UniqueName: \"kubernetes.io/projected/120141dc-7742-4117-9052-6ab138ced47a-kube-api-access-rksfs\") pod \"console-6dbd947b68-2rr89\" (UID: \"120141dc-7742-4117-9052-6ab138ced47a\") " pod="openshift-console/console-6dbd947b68-2rr89"
Sep 09 19:56:54.792761 ip-10-0-136-183 kubenswrapper[2574]: I0909 19:56:54.792719    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/120141dc-7742-4117-9052-6ab138ced47a-console-oauth-config\") pod \"console-6dbd947b68-2rr89\" (UID: \"120141dc-7742-4117-9052-6ab138ced47a\") " pod="openshift-console/console-6dbd947b68-2rr89"
Sep 09 19:56:54.793195 ip-10-0-136-183 kubenswrapper[2574]: I0909 19:56:54.793172    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/120141dc-7742-4117-9052-6ab138ced47a-service-ca\") pod \"console-6dbd947b68-2rr89\" (UID: \"120141dc-7742-4117-9052-6ab138ced47a\") " pod="openshift-console/console-6dbd947b68-2rr89"
Sep 09 19:56:54.793315 ip-10-0-136-183 kubenswrapper[2574]: I0909 19:56:54.793230    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/120141dc-7742-4117-9052-6ab138ced47a-console-config\") pod \"console-6dbd947b68-2rr89\" (UID: \"120141dc-7742-4117-9052-6ab138ced47a\") " pod="openshift-console/console-6dbd947b68-2rr89"
Sep 09 19:56:54.793315 ip-10-0-136-183 kubenswrapper[2574]: I0909 19:56:54.793272    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/120141dc-7742-4117-9052-6ab138ced47a-oauth-serving-cert\") pod \"console-6dbd947b68-2rr89\" (UID: \"120141dc-7742-4117-9052-6ab138ced47a\") " pod="openshift-console/console-6dbd947b68-2rr89"
Sep 09 19:56:54.795152 ip-10-0-136-183 kubenswrapper[2574]: I0909 19:56:54.795122    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/120141dc-7742-4117-9052-6ab138ced47a-console-oauth-config\") pod \"console-6dbd947b68-2rr89\" (UID: \"120141dc-7742-4117-9052-6ab138ced47a\") " pod="openshift-console/console-6dbd947b68-2rr89"
Sep 09 19:56:54.795272 ip-10-0-136-183 kubenswrapper[2574]: I0909 19:56:54.795172    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/120141dc-7742-4117-9052-6ab138ced47a-console-serving-cert\") pod \"console-6dbd947b68-2rr89\" (UID: \"120141dc-7742-4117-9052-6ab138ced47a\") " pod="openshift-console/console-6dbd947b68-2rr89"
Sep 09 19:56:54.801184 ip-10-0-136-183 kubenswrapper[2574]: I0909 19:56:54.801158    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rksfs\" (UniqueName: \"kubernetes.io/projected/120141dc-7742-4117-9052-6ab138ced47a-kube-api-access-rksfs\") pod \"console-6dbd947b68-2rr89\" (UID: \"120141dc-7742-4117-9052-6ab138ced47a\") " pod="openshift-console/console-6dbd947b68-2rr89"
Sep 09 19:56:54.842592 ip-10-0-136-183 kubenswrapper[2574]: I0909 19:56:54.842501    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-6f4946db56-ht869" event={"ID":"09d0544d-3823-4cb2-8611-82a74d206a45","Type":"ContainerStarted","Data":"aa36ad3a548f3423f390d5833064263e764e7989ae9ac1ea8b791ea4955678ae"}
Sep 09 19:56:54.842592 ip-10-0-136-183 kubenswrapper[2574]: I0909 19:56:54.842566    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-6f4946db56-ht869" event={"ID":"09d0544d-3823-4cb2-8611-82a74d206a45","Type":"ContainerStarted","Data":"6f73ea7cf23692b97edbf9af46018bdfc7f3de79a468e7d6a229f1828b6c24f7"}
Sep 09 19:56:54.865332 ip-10-0-136-183 kubenswrapper[2574]: I0909 19:56:54.865286    2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-6f4946db56-ht869" podStartSLOduration=1.6776237470000002 podStartE2EDuration="2.865269624s" totalImagesPullingTime="1.187645877s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 19:56:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 19:56:54.86408493 +0000 UTC m=+98.910338191" watchObservedRunningTime="2026-09-09 19:56:54.865269624 +0000 UTC m=+98.911522901"
Sep 09 19:56:54.878599 ip-10-0-136-183 kubenswrapper[2574]: I0909 19:56:54.878493    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6dbd947b68-2rr89"
Sep 09 19:56:55.150779 ip-10-0-136-183 kubenswrapper[2574]: I0909 19:56:55.150746    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-6dbd947b68-2rr89"]
Sep 09 19:56:55.153643 ip-10-0-136-183 kubenswrapper[2574]: W0909 19:56:55.153620    2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod120141dc_7742_4117_9052_6ab138ced47a.slice/crio-2eede6dc73b05c32f4258e780f1eb6bd337f108ce3a4ffe702d1a384dc7422a2 WatchSource:0}: Error finding container 2eede6dc73b05c32f4258e780f1eb6bd337f108ce3a4ffe702d1a384dc7422a2: Status 404 returned error can't find the container with id 2eede6dc73b05c32f4258e780f1eb6bd337f108ce3a4ffe702d1a384dc7422a2
Sep 09 19:56:55.848491 ip-10-0-136-183 kubenswrapper[2574]: I0909 19:56:55.848452    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-4blgq" event={"ID":"3a81bb5a-0683-466e-b32e-960f4ca22ae6","Type":"ContainerStarted","Data":"1b9d5d099fc2b0af6d72be795212d4d1fbd59fa72871036629e19a2cb4e858b0"}
Sep 09 19:56:55.849006 ip-10-0-136-183 kubenswrapper[2574]: I0909 19:56:55.848522    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-4blgq" event={"ID":"3a81bb5a-0683-466e-b32e-960f4ca22ae6","Type":"ContainerStarted","Data":"735d453ebb7bea94d3acfa6eeaf48bbfe7d7ce675f89cf0d63d536b6557b2e3e"}
Sep 09 19:56:55.849006 ip-10-0-136-183 kubenswrapper[2574]: I0909 19:56:55.848655    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-4blgq"
Sep 09 19:56:55.850194 ip-10-0-136-183 kubenswrapper[2574]: I0909 19:56:55.850141    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6dbd947b68-2rr89" event={"ID":"120141dc-7742-4117-9052-6ab138ced47a","Type":"ContainerStarted","Data":"2eede6dc73b05c32f4258e780f1eb6bd337f108ce3a4ffe702d1a384dc7422a2"}
Sep 09 19:56:55.852971 ip-10-0-136-183 kubenswrapper[2574]: I0909 19:56:55.852405    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-k6ctg" event={"ID":"2eee1bd1-9d73-45d9-acf7-323648f0d6ae","Type":"ContainerStarted","Data":"b9f2b51e34fe1ccbdcae8d06df7c15f4a687fa505e9f6095fe1c59368cacea13"}
Sep 09 19:56:55.868565 ip-10-0-136-183 kubenswrapper[2574]: I0909 19:56:55.868504    2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-4blgq" podStartSLOduration=66.223788989 podStartE2EDuration="1m7.868486013s" totalImagesPullingTime="1.644697024s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 19:55:48 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 19:56:55.866420685 +0000 UTC m=+99.912673948" watchObservedRunningTime="2026-09-09 19:56:55.868486013 +0000 UTC m=+99.914739324"
Sep 09 19:56:56.630274 ip-10-0-136-183 kubenswrapper[2574]: I0909 19:56:56.630238    2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-8b862"]
Sep 09 19:56:56.633869 ip-10-0-136-183 kubenswrapper[2574]: I0909 19:56:56.633846    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-8b862"
Sep 09 19:56:56.636108 ip-10-0-136-183 kubenswrapper[2574]: I0909 19:56:56.635989    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\""
Sep 09 19:56:56.636108 ip-10-0-136-183 kubenswrapper[2574]: I0909 19:56:56.636015    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-f5wvt\""
Sep 09 19:56:56.636108 ip-10-0-136-183 kubenswrapper[2574]: I0909 19:56:56.636028    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\""
Sep 09 19:56:56.636108 ip-10-0-136-183 kubenswrapper[2574]: I0909 19:56:56.635991    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\""
Sep 09 19:56:56.808819 ip-10-0-136-183 kubenswrapper[2574]: I0909 19:56:56.808707    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/06886459-8d92-4c11-826a-a879fb96aa71-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-8b862\" (UID: \"06886459-8d92-4c11-826a-a879fb96aa71\") " pod="openshift-monitoring/node-exporter-8b862"
Sep 09 19:56:56.808819 ip-10-0-136-183 kubenswrapper[2574]: I0909 19:56:56.808764    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/06886459-8d92-4c11-826a-a879fb96aa71-node-exporter-tls\") pod \"node-exporter-8b862\" (UID: \"06886459-8d92-4c11-826a-a879fb96aa71\") " pod="openshift-monitoring/node-exporter-8b862"
Sep 09 19:56:56.808819 ip-10-0-136-183 kubenswrapper[2574]: I0909 19:56:56.808792    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/06886459-8d92-4c11-826a-a879fb96aa71-node-exporter-accelerators-collector-config\") pod \"node-exporter-8b862\" (UID: \"06886459-8d92-4c11-826a-a879fb96aa71\") " pod="openshift-monitoring/node-exporter-8b862"
Sep 09 19:56:56.809053 ip-10-0-136-183 kubenswrapper[2574]: I0909 19:56:56.808836    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/06886459-8d92-4c11-826a-a879fb96aa71-root\") pod \"node-exporter-8b862\" (UID: \"06886459-8d92-4c11-826a-a879fb96aa71\") " pod="openshift-monitoring/node-exporter-8b862"
Sep 09 19:56:56.809053 ip-10-0-136-183 kubenswrapper[2574]: I0909 19:56:56.808924    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qrj8w\" (UniqueName: \"kubernetes.io/projected/06886459-8d92-4c11-826a-a879fb96aa71-kube-api-access-qrj8w\") pod \"node-exporter-8b862\" (UID: \"06886459-8d92-4c11-826a-a879fb96aa71\") " pod="openshift-monitoring/node-exporter-8b862"
Sep 09 19:56:56.809053 ip-10-0-136-183 kubenswrapper[2574]: I0909 19:56:56.808972    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/06886459-8d92-4c11-826a-a879fb96aa71-node-exporter-textfile\") pod \"node-exporter-8b862\" (UID: \"06886459-8d92-4c11-826a-a879fb96aa71\") " pod="openshift-monitoring/node-exporter-8b862"
Sep 09 19:56:56.809053 ip-10-0-136-183 kubenswrapper[2574]: I0909 19:56:56.809011    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/06886459-8d92-4c11-826a-a879fb96aa71-metrics-client-ca\") pod \"node-exporter-8b862\" (UID: \"06886459-8d92-4c11-826a-a879fb96aa71\") " pod="openshift-monitoring/node-exporter-8b862"
Sep 09 19:56:56.809249 ip-10-0-136-183 kubenswrapper[2574]: I0909 19:56:56.809075    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/06886459-8d92-4c11-826a-a879fb96aa71-node-exporter-wtmp\") pod \"node-exporter-8b862\" (UID: \"06886459-8d92-4c11-826a-a879fb96aa71\") " pod="openshift-monitoring/node-exporter-8b862"
Sep 09 19:56:56.809249 ip-10-0-136-183 kubenswrapper[2574]: I0909 19:56:56.809104    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/06886459-8d92-4c11-826a-a879fb96aa71-sys\") pod \"node-exporter-8b862\" (UID: \"06886459-8d92-4c11-826a-a879fb96aa71\") " pod="openshift-monitoring/node-exporter-8b862"
Sep 09 19:56:56.858688 ip-10-0-136-183 kubenswrapper[2574]: I0909 19:56:56.858648    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-k6ctg" event={"ID":"2eee1bd1-9d73-45d9-acf7-323648f0d6ae","Type":"ContainerStarted","Data":"a5f228428d3f84d30217b267ddf1ac6b651d90019a456c5eb76b448fcf7589fb"}
Sep 09 19:56:56.876407 ip-10-0-136-183 kubenswrapper[2574]: I0909 19:56:56.876204    2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-k6ctg" podStartSLOduration=34.695846095 podStartE2EDuration="38.87618227s" totalImagesPullingTime="4.180336175s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-09 19:56:18 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 19:56:56.875000506 +0000 UTC m=+100.921253770" watchObservedRunningTime="2026-09-09 19:56:56.87618227 +0000 UTC m=+100.922435534"
Sep 09 19:56:56.909575 ip-10-0-136-183 kubenswrapper[2574]: I0909 19:56:56.909542    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qrj8w\" (UniqueName: \"kubernetes.io/projected/06886459-8d92-4c11-826a-a879fb96aa71-kube-api-access-qrj8w\") pod \"node-exporter-8b862\" (UID: \"06886459-8d92-4c11-826a-a879fb96aa71\") " pod="openshift-monitoring/node-exporter-8b862"
Sep 09 19:56:56.909575 ip-10-0-136-183 kubenswrapper[2574]: I0909 19:56:56.909590    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/06886459-8d92-4c11-826a-a879fb96aa71-node-exporter-textfile\") pod \"node-exporter-8b862\" (UID: \"06886459-8d92-4c11-826a-a879fb96aa71\") " pod="openshift-monitoring/node-exporter-8b862"
Sep 09 19:56:56.909832 ip-10-0-136-183 kubenswrapper[2574]: I0909 19:56:56.909612    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/06886459-8d92-4c11-826a-a879fb96aa71-metrics-client-ca\") pod \"node-exporter-8b862\" (UID: \"06886459-8d92-4c11-826a-a879fb96aa71\") " pod="openshift-monitoring/node-exporter-8b862"
Sep 09 19:56:56.909832 ip-10-0-136-183 kubenswrapper[2574]: I0909 19:56:56.909652    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/06886459-8d92-4c11-826a-a879fb96aa71-node-exporter-wtmp\") pod \"node-exporter-8b862\" (UID: \"06886459-8d92-4c11-826a-a879fb96aa71\") " pod="openshift-monitoring/node-exporter-8b862"
Sep 09 19:56:56.909832 ip-10-0-136-183 kubenswrapper[2574]: I0909 19:56:56.909680    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/06886459-8d92-4c11-826a-a879fb96aa71-sys\") pod \"node-exporter-8b862\" (UID: \"06886459-8d92-4c11-826a-a879fb96aa71\") " pod="openshift-monitoring/node-exporter-8b862"
Sep 09 19:56:56.909832 ip-10-0-136-183 kubenswrapper[2574]: I0909 19:56:56.909712    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/06886459-8d92-4c11-826a-a879fb96aa71-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-8b862\" (UID: \"06886459-8d92-4c11-826a-a879fb96aa71\") " pod="openshift-monitoring/node-exporter-8b862"
Sep 09 19:56:56.909832 ip-10-0-136-183 kubenswrapper[2574]: I0909 19:56:56.909736    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/06886459-8d92-4c11-826a-a879fb96aa71-node-exporter-tls\") pod \"node-exporter-8b862\" (UID: \"06886459-8d92-4c11-826a-a879fb96aa71\") " pod="openshift-monitoring/node-exporter-8b862"
Sep 09 19:56:56.909995 ip-10-0-136-183 kubenswrapper[2574]: E0909 19:56:56.909839    2574 secret.go:189] Couldn't get secret openshift-monitoring/node-exporter-tls: secret "node-exporter-tls" not found
Sep 09 19:56:56.909995 ip-10-0-136-183 kubenswrapper[2574]: I0909 19:56:56.909871    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/06886459-8d92-4c11-826a-a879fb96aa71-node-exporter-wtmp\") pod \"node-exporter-8b862\" (UID: \"06886459-8d92-4c11-826a-a879fb96aa71\") " pod="openshift-monitoring/node-exporter-8b862"
Sep 09 19:56:56.909995 ip-10-0-136-183 kubenswrapper[2574]: E0909 19:56:56.909906    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/06886459-8d92-4c11-826a-a879fb96aa71-node-exporter-tls podName:06886459-8d92-4c11-826a-a879fb96aa71 nodeName:}" failed. No retries permitted until 2026-09-09 19:56:57.409885 +0000 UTC m=+101.456138240 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "node-exporter-tls" (UniqueName: "kubernetes.io/secret/06886459-8d92-4c11-826a-a879fb96aa71-node-exporter-tls") pod "node-exporter-8b862" (UID: "06886459-8d92-4c11-826a-a879fb96aa71") : secret "node-exporter-tls" not found
Sep 09 19:56:56.909995 ip-10-0-136-183 kubenswrapper[2574]: I0909 19:56:56.909947    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/06886459-8d92-4c11-826a-a879fb96aa71-sys\") pod \"node-exporter-8b862\" (UID: \"06886459-8d92-4c11-826a-a879fb96aa71\") " pod="openshift-monitoring/node-exporter-8b862"
Sep 09 19:56:56.909995 ip-10-0-136-183 kubenswrapper[2574]: I0909 19:56:56.909988    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/06886459-8d92-4c11-826a-a879fb96aa71-node-exporter-accelerators-collector-config\") pod \"node-exporter-8b862\" (UID: \"06886459-8d92-4c11-826a-a879fb96aa71\") " pod="openshift-monitoring/node-exporter-8b862"
Sep 09 19:56:56.910212 ip-10-0-136-183 kubenswrapper[2574]: I0909 19:56:56.910019    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/06886459-8d92-4c11-826a-a879fb96aa71-root\") pod \"node-exporter-8b862\" (UID: \"06886459-8d92-4c11-826a-a879fb96aa71\") " pod="openshift-monitoring/node-exporter-8b862"
Sep 09 19:56:56.910212 ip-10-0-136-183 kubenswrapper[2574]: I0909 19:56:56.910082    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/06886459-8d92-4c11-826a-a879fb96aa71-node-exporter-textfile\") pod \"node-exporter-8b862\" (UID: \"06886459-8d92-4c11-826a-a879fb96aa71\") " pod="openshift-monitoring/node-exporter-8b862"
Sep 09 19:56:56.910212 ip-10-0-136-183 kubenswrapper[2574]: I0909 19:56:56.910116    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/06886459-8d92-4c11-826a-a879fb96aa71-root\") pod \"node-exporter-8b862\" (UID: \"06886459-8d92-4c11-826a-a879fb96aa71\") " pod="openshift-monitoring/node-exporter-8b862"
Sep 09 19:56:56.910350 ip-10-0-136-183 kubenswrapper[2574]: I0909 19:56:56.910302    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/06886459-8d92-4c11-826a-a879fb96aa71-metrics-client-ca\") pod \"node-exporter-8b862\" (UID: \"06886459-8d92-4c11-826a-a879fb96aa71\") " pod="openshift-monitoring/node-exporter-8b862"
Sep 09 19:56:56.910535 ip-10-0-136-183 kubenswrapper[2574]: I0909 19:56:56.910514    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/06886459-8d92-4c11-826a-a879fb96aa71-node-exporter-accelerators-collector-config\") pod \"node-exporter-8b862\" (UID: \"06886459-8d92-4c11-826a-a879fb96aa71\") " pod="openshift-monitoring/node-exporter-8b862"
Sep 09 19:56:56.912420 ip-10-0-136-183 kubenswrapper[2574]: I0909 19:56:56.912394    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/06886459-8d92-4c11-826a-a879fb96aa71-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-8b862\" (UID: \"06886459-8d92-4c11-826a-a879fb96aa71\") " pod="openshift-monitoring/node-exporter-8b862"
Sep 09 19:56:56.920240 ip-10-0-136-183 kubenswrapper[2574]: I0909 19:56:56.920217    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qrj8w\" (UniqueName: \"kubernetes.io/projected/06886459-8d92-4c11-826a-a879fb96aa71-kube-api-access-qrj8w\") pod \"node-exporter-8b862\" (UID: \"06886459-8d92-4c11-826a-a879fb96aa71\") " pod="openshift-monitoring/node-exporter-8b862"
Sep 09 19:56:57.415575 ip-10-0-136-183 kubenswrapper[2574]: I0909 19:56:57.415536    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/06886459-8d92-4c11-826a-a879fb96aa71-node-exporter-tls\") pod \"node-exporter-8b862\" (UID: \"06886459-8d92-4c11-826a-a879fb96aa71\") " pod="openshift-monitoring/node-exporter-8b862"
Sep 09 19:56:57.418303 ip-10-0-136-183 kubenswrapper[2574]: I0909 19:56:57.418273    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/06886459-8d92-4c11-826a-a879fb96aa71-node-exporter-tls\") pod \"node-exporter-8b862\" (UID: \"06886459-8d92-4c11-826a-a879fb96aa71\") " pod="openshift-monitoring/node-exporter-8b862"
Sep 09 19:56:57.544940 ip-10-0-136-183 kubenswrapper[2574]: I0909 19:56:57.544903    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-8b862"
Sep 09 19:56:58.138558 ip-10-0-136-183 kubenswrapper[2574]: W0909 19:56:58.138520    2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod06886459_8d92_4c11_826a_a879fb96aa71.slice/crio-665fadfc127e4ed101d0fa3b70028c7a2a4f1f38497904c80e958c894180021b WatchSource:0}: Error finding container 665fadfc127e4ed101d0fa3b70028c7a2a4f1f38497904c80e958c894180021b: Status 404 returned error can't find the container with id 665fadfc127e4ed101d0fa3b70028c7a2a4f1f38497904c80e958c894180021b
Sep 09 19:56:58.531086 ip-10-0-136-183 kubenswrapper[2574]: I0909 19:56:58.531010    2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/thanos-querier-65c468894b-882qv"]
Sep 09 19:56:58.534380 ip-10-0-136-183 kubenswrapper[2574]: I0909 19:56:58.534355    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-65c468894b-882qv"
Sep 09 19:56:58.537454 ip-10-0-136-183 kubenswrapper[2574]: I0909 19:56:58.536903    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-grpc-tls-1me5fuvvrpd2b\""
Sep 09 19:56:58.537454 ip-10-0-136-183 kubenswrapper[2574]: I0909 19:56:58.537123    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy\""
Sep 09 19:56:58.537454 ip-10-0-136-183 kubenswrapper[2574]: I0909 19:56:58.537195    2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-654cd7bc6c-sjn5l"]
Sep 09 19:56:58.537454 ip-10-0-136-183 kubenswrapper[2574]: I0909 19:56:58.537335    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-web\""
Sep 09 19:56:58.538220 ip-10-0-136-183 kubenswrapper[2574]: I0909 19:56:58.537538    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-metrics\""
Sep 09 19:56:58.538220 ip-10-0-136-183 kubenswrapper[2574]: I0909 19:56:58.537546    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-tls\""
Sep 09 19:56:58.538220 ip-10-0-136-183 kubenswrapper[2574]: I0909 19:56:58.537611    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-dockercfg-4t9cp\""
Sep 09 19:56:58.538220 ip-10-0-136-183 kubenswrapper[2574]: I0909 19:56:58.537636    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-rules\""
Sep 09 19:56:58.540589 ip-10-0-136-183 kubenswrapper[2574]: I0909 19:56:58.540572    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-654cd7bc6c-sjn5l"
Sep 09 19:56:58.548598 ip-10-0-136-183 kubenswrapper[2574]: I0909 19:56:58.548577    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\""
Sep 09 19:56:58.552318 ip-10-0-136-183 kubenswrapper[2574]: I0909 19:56:58.552297    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-654cd7bc6c-sjn5l"]
Sep 09 19:56:58.553421 ip-10-0-136-183 kubenswrapper[2574]: I0909 19:56:58.553394    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-65c468894b-882qv"]
Sep 09 19:56:58.727438 ip-10-0-136-183 kubenswrapper[2574]: I0909 19:56:58.727404    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/6dcf380f-2f53-4047-8a59-8e93030d4c24-secret-thanos-querier-tls\") pod \"thanos-querier-65c468894b-882qv\" (UID: \"6dcf380f-2f53-4047-8a59-8e93030d4c24\") " pod="openshift-monitoring/thanos-querier-65c468894b-882qv"
Sep 09 19:56:58.727633 ip-10-0-136-183 kubenswrapper[2574]: I0909 19:56:58.727443    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/c520dfd7-ff87-4430-9036-715bfbe8ba9e-console-serving-cert\") pod \"console-654cd7bc6c-sjn5l\" (UID: \"c520dfd7-ff87-4430-9036-715bfbe8ba9e\") " pod="openshift-console/console-654cd7bc6c-sjn5l"
Sep 09 19:56:58.727633 ip-10-0-136-183 kubenswrapper[2574]: I0909 19:56:58.727474    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qccn6\" (UniqueName: \"kubernetes.io/projected/6dcf380f-2f53-4047-8a59-8e93030d4c24-kube-api-access-qccn6\") pod \"thanos-querier-65c468894b-882qv\" (UID: \"6dcf380f-2f53-4047-8a59-8e93030d4c24\") " pod="openshift-monitoring/thanos-querier-65c468894b-882qv"
Sep 09 19:56:58.727633 ip-10-0-136-183 kubenswrapper[2574]: I0909 19:56:58.727543    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dvqxf\" (UniqueName: \"kubernetes.io/projected/c520dfd7-ff87-4430-9036-715bfbe8ba9e-kube-api-access-dvqxf\") pod \"console-654cd7bc6c-sjn5l\" (UID: \"c520dfd7-ff87-4430-9036-715bfbe8ba9e\") " pod="openshift-console/console-654cd7bc6c-sjn5l"
Sep 09 19:56:58.727633 ip-10-0-136-183 kubenswrapper[2574]: I0909 19:56:58.727566    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/6dcf380f-2f53-4047-8a59-8e93030d4c24-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-65c468894b-882qv\" (UID: \"6dcf380f-2f53-4047-8a59-8e93030d4c24\") " pod="openshift-monitoring/thanos-querier-65c468894b-882qv"
Sep 09 19:56:58.727823 ip-10-0-136-183 kubenswrapper[2574]: I0909 19:56:58.727640    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/6dcf380f-2f53-4047-8a59-8e93030d4c24-metrics-client-ca\") pod \"thanos-querier-65c468894b-882qv\" (UID: \"6dcf380f-2f53-4047-8a59-8e93030d4c24\") " pod="openshift-monitoring/thanos-querier-65c468894b-882qv"
Sep 09 19:56:58.727823 ip-10-0-136-183 kubenswrapper[2574]: I0909 19:56:58.727673    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c520dfd7-ff87-4430-9036-715bfbe8ba9e-trusted-ca-bundle\") pod \"console-654cd7bc6c-sjn5l\" (UID: \"c520dfd7-ff87-4430-9036-715bfbe8ba9e\") " pod="openshift-console/console-654cd7bc6c-sjn5l"
Sep 09 19:56:58.727823 ip-10-0-136-183 kubenswrapper[2574]: I0909 19:56:58.727695    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/6dcf380f-2f53-4047-8a59-8e93030d4c24-secret-grpc-tls\") pod \"thanos-querier-65c468894b-882qv\" (UID: \"6dcf380f-2f53-4047-8a59-8e93030d4c24\") " pod="openshift-monitoring/thanos-querier-65c468894b-882qv"
Sep 09 19:56:58.727823 ip-10-0-136-183 kubenswrapper[2574]: I0909 19:56:58.727723    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/6dcf380f-2f53-4047-8a59-8e93030d4c24-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-65c468894b-882qv\" (UID: \"6dcf380f-2f53-4047-8a59-8e93030d4c24\") " pod="openshift-monitoring/thanos-querier-65c468894b-882qv"
Sep 09 19:56:58.727823 ip-10-0-136-183 kubenswrapper[2574]: I0909 19:56:58.727760    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/c520dfd7-ff87-4430-9036-715bfbe8ba9e-console-config\") pod \"console-654cd7bc6c-sjn5l\" (UID: \"c520dfd7-ff87-4430-9036-715bfbe8ba9e\") " pod="openshift-console/console-654cd7bc6c-sjn5l"
Sep 09 19:56:58.727823 ip-10-0-136-183 kubenswrapper[2574]: I0909 19:56:58.727774    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/6dcf380f-2f53-4047-8a59-8e93030d4c24-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-65c468894b-882qv\" (UID: \"6dcf380f-2f53-4047-8a59-8e93030d4c24\") " pod="openshift-monitoring/thanos-querier-65c468894b-882qv"
Sep 09 19:56:58.727823 ip-10-0-136-183 kubenswrapper[2574]: I0909 19:56:58.727792    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/c520dfd7-ff87-4430-9036-715bfbe8ba9e-console-oauth-config\") pod \"console-654cd7bc6c-sjn5l\" (UID: \"c520dfd7-ff87-4430-9036-715bfbe8ba9e\") " pod="openshift-console/console-654cd7bc6c-sjn5l"
Sep 09 19:56:58.728069 ip-10-0-136-183 kubenswrapper[2574]: I0909 19:56:58.727855    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/c520dfd7-ff87-4430-9036-715bfbe8ba9e-oauth-serving-cert\") pod \"console-654cd7bc6c-sjn5l\" (UID: \"c520dfd7-ff87-4430-9036-715bfbe8ba9e\") " pod="openshift-console/console-654cd7bc6c-sjn5l"
Sep 09 19:56:58.728069 ip-10-0-136-183 kubenswrapper[2574]: I0909 19:56:58.727884    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/c520dfd7-ff87-4430-9036-715bfbe8ba9e-service-ca\") pod \"console-654cd7bc6c-sjn5l\" (UID: \"c520dfd7-ff87-4430-9036-715bfbe8ba9e\") " pod="openshift-console/console-654cd7bc6c-sjn5l"
Sep 09 19:56:58.728069 ip-10-0-136-183 kubenswrapper[2574]: I0909 19:56:58.727904    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/6dcf380f-2f53-4047-8a59-8e93030d4c24-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-65c468894b-882qv\" (UID: \"6dcf380f-2f53-4047-8a59-8e93030d4c24\") " pod="openshift-monitoring/thanos-querier-65c468894b-882qv"
Sep 09 19:56:58.828792 ip-10-0-136-183 kubenswrapper[2574]: I0909 19:56:58.828705    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/c520dfd7-ff87-4430-9036-715bfbe8ba9e-oauth-serving-cert\") pod \"console-654cd7bc6c-sjn5l\" (UID: \"c520dfd7-ff87-4430-9036-715bfbe8ba9e\") " pod="openshift-console/console-654cd7bc6c-sjn5l"
Sep 09 19:56:58.828792 ip-10-0-136-183 kubenswrapper[2574]: I0909 19:56:58.828756    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/c520dfd7-ff87-4430-9036-715bfbe8ba9e-service-ca\") pod \"console-654cd7bc6c-sjn5l\" (UID: \"c520dfd7-ff87-4430-9036-715bfbe8ba9e\") " pod="openshift-console/console-654cd7bc6c-sjn5l"
Sep 09 19:56:58.828792 ip-10-0-136-183 kubenswrapper[2574]: I0909 19:56:58.828781    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/6dcf380f-2f53-4047-8a59-8e93030d4c24-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-65c468894b-882qv\" (UID: \"6dcf380f-2f53-4047-8a59-8e93030d4c24\") " pod="openshift-monitoring/thanos-querier-65c468894b-882qv"
Sep 09 19:56:58.828998 ip-10-0-136-183 kubenswrapper[2574]: I0909 19:56:58.828828    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/6dcf380f-2f53-4047-8a59-8e93030d4c24-secret-thanos-querier-tls\") pod \"thanos-querier-65c468894b-882qv\" (UID: \"6dcf380f-2f53-4047-8a59-8e93030d4c24\") " pod="openshift-monitoring/thanos-querier-65c468894b-882qv"
Sep 09 19:56:58.828998 ip-10-0-136-183 kubenswrapper[2574]: I0909 19:56:58.828846    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/c520dfd7-ff87-4430-9036-715bfbe8ba9e-console-serving-cert\") pod \"console-654cd7bc6c-sjn5l\" (UID: \"c520dfd7-ff87-4430-9036-715bfbe8ba9e\") " pod="openshift-console/console-654cd7bc6c-sjn5l"
Sep 09 19:56:58.828998 ip-10-0-136-183 kubenswrapper[2574]: I0909 19:56:58.828862    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qccn6\" (UniqueName: \"kubernetes.io/projected/6dcf380f-2f53-4047-8a59-8e93030d4c24-kube-api-access-qccn6\") pod \"thanos-querier-65c468894b-882qv\" (UID: \"6dcf380f-2f53-4047-8a59-8e93030d4c24\") " pod="openshift-monitoring/thanos-querier-65c468894b-882qv"
Sep 09 19:56:58.828998 ip-10-0-136-183 kubenswrapper[2574]: I0909 19:56:58.828885    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dvqxf\" (UniqueName: \"kubernetes.io/projected/c520dfd7-ff87-4430-9036-715bfbe8ba9e-kube-api-access-dvqxf\") pod \"console-654cd7bc6c-sjn5l\" (UID: \"c520dfd7-ff87-4430-9036-715bfbe8ba9e\") " pod="openshift-console/console-654cd7bc6c-sjn5l"
Sep 09 19:56:58.828998 ip-10-0-136-183 kubenswrapper[2574]: I0909 19:56:58.828900    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/6dcf380f-2f53-4047-8a59-8e93030d4c24-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-65c468894b-882qv\" (UID: \"6dcf380f-2f53-4047-8a59-8e93030d4c24\") " pod="openshift-monitoring/thanos-querier-65c468894b-882qv"
Sep 09 19:56:58.828998 ip-10-0-136-183 kubenswrapper[2574]: I0909 19:56:58.828941    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/6dcf380f-2f53-4047-8a59-8e93030d4c24-metrics-client-ca\") pod \"thanos-querier-65c468894b-882qv\" (UID: \"6dcf380f-2f53-4047-8a59-8e93030d4c24\") " pod="openshift-monitoring/thanos-querier-65c468894b-882qv"
Sep 09 19:56:58.829252 ip-10-0-136-183 kubenswrapper[2574]: I0909 19:56:58.829080    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c520dfd7-ff87-4430-9036-715bfbe8ba9e-trusted-ca-bundle\") pod \"console-654cd7bc6c-sjn5l\" (UID: \"c520dfd7-ff87-4430-9036-715bfbe8ba9e\") " pod="openshift-console/console-654cd7bc6c-sjn5l"
Sep 09 19:56:58.829252 ip-10-0-136-183 kubenswrapper[2574]: I0909 19:56:58.829120    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/6dcf380f-2f53-4047-8a59-8e93030d4c24-secret-grpc-tls\") pod \"thanos-querier-65c468894b-882qv\" (UID: \"6dcf380f-2f53-4047-8a59-8e93030d4c24\") " pod="openshift-monitoring/thanos-querier-65c468894b-882qv"
Sep 09 19:56:58.829252 ip-10-0-136-183 kubenswrapper[2574]: I0909 19:56:58.829155    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/6dcf380f-2f53-4047-8a59-8e93030d4c24-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-65c468894b-882qv\" (UID: \"6dcf380f-2f53-4047-8a59-8e93030d4c24\") " pod="openshift-monitoring/thanos-querier-65c468894b-882qv"
Sep 09 19:56:58.829252 ip-10-0-136-183 kubenswrapper[2574]: I0909 19:56:58.829198    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/c520dfd7-ff87-4430-9036-715bfbe8ba9e-console-config\") pod \"console-654cd7bc6c-sjn5l\" (UID: \"c520dfd7-ff87-4430-9036-715bfbe8ba9e\") " pod="openshift-console/console-654cd7bc6c-sjn5l"
Sep 09 19:56:58.829252 ip-10-0-136-183 kubenswrapper[2574]: I0909 19:56:58.829225    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/6dcf380f-2f53-4047-8a59-8e93030d4c24-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-65c468894b-882qv\" (UID: \"6dcf380f-2f53-4047-8a59-8e93030d4c24\") " pod="openshift-monitoring/thanos-querier-65c468894b-882qv"
Sep 09 19:56:58.829458 ip-10-0-136-183 kubenswrapper[2574]: I0909 19:56:58.829257    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/c520dfd7-ff87-4430-9036-715bfbe8ba9e-console-oauth-config\") pod \"console-654cd7bc6c-sjn5l\" (UID: \"c520dfd7-ff87-4430-9036-715bfbe8ba9e\") " pod="openshift-console/console-654cd7bc6c-sjn5l"
Sep 09 19:56:58.829567 ip-10-0-136-183 kubenswrapper[2574]: I0909 19:56:58.829546    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/c520dfd7-ff87-4430-9036-715bfbe8ba9e-service-ca\") pod \"console-654cd7bc6c-sjn5l\" (UID: \"c520dfd7-ff87-4430-9036-715bfbe8ba9e\") " pod="openshift-console/console-654cd7bc6c-sjn5l"
Sep 09 19:56:58.829903 ip-10-0-136-183 kubenswrapper[2574]: I0909 19:56:58.829867    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/c520dfd7-ff87-4430-9036-715bfbe8ba9e-oauth-serving-cert\") pod \"console-654cd7bc6c-sjn5l\" (UID: \"c520dfd7-ff87-4430-9036-715bfbe8ba9e\") " pod="openshift-console/console-654cd7bc6c-sjn5l"
Sep 09 19:56:58.830535 ip-10-0-136-183 kubenswrapper[2574]: I0909 19:56:58.830191    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c520dfd7-ff87-4430-9036-715bfbe8ba9e-trusted-ca-bundle\") pod \"console-654cd7bc6c-sjn5l\" (UID: \"c520dfd7-ff87-4430-9036-715bfbe8ba9e\") " pod="openshift-console/console-654cd7bc6c-sjn5l"
Sep 09 19:56:58.830535 ip-10-0-136-183 kubenswrapper[2574]: I0909 19:56:58.830416    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/6dcf380f-2f53-4047-8a59-8e93030d4c24-metrics-client-ca\") pod \"thanos-querier-65c468894b-882qv\" (UID: \"6dcf380f-2f53-4047-8a59-8e93030d4c24\") " pod="openshift-monitoring/thanos-querier-65c468894b-882qv"
Sep 09 19:56:58.832330 ip-10-0-136-183 kubenswrapper[2574]: I0909 19:56:58.832080    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/6dcf380f-2f53-4047-8a59-8e93030d4c24-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-65c468894b-882qv\" (UID: \"6dcf380f-2f53-4047-8a59-8e93030d4c24\") " pod="openshift-monitoring/thanos-querier-65c468894b-882qv"
Sep 09 19:56:58.832330 ip-10-0-136-183 kubenswrapper[2574]: I0909 19:56:58.832190    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/6dcf380f-2f53-4047-8a59-8e93030d4c24-secret-grpc-tls\") pod \"thanos-querier-65c468894b-882qv\" (UID: \"6dcf380f-2f53-4047-8a59-8e93030d4c24\") " pod="openshift-monitoring/thanos-querier-65c468894b-882qv"
Sep 09 19:56:58.832330 ip-10-0-136-183 kubenswrapper[2574]: I0909 19:56:58.832204    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/6dcf380f-2f53-4047-8a59-8e93030d4c24-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-65c468894b-882qv\" (UID: \"6dcf380f-2f53-4047-8a59-8e93030d4c24\") " pod="openshift-monitoring/thanos-querier-65c468894b-882qv"
Sep 09 19:56:58.832708 ip-10-0-136-183 kubenswrapper[2574]: I0909 19:56:58.832680    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/6dcf380f-2f53-4047-8a59-8e93030d4c24-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-65c468894b-882qv\" (UID: \"6dcf380f-2f53-4047-8a59-8e93030d4c24\") " pod="openshift-monitoring/thanos-querier-65c468894b-882qv"
Sep 09 19:56:58.832762 ip-10-0-136-183 kubenswrapper[2574]: I0909 19:56:58.832687    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/c520dfd7-ff87-4430-9036-715bfbe8ba9e-console-config\") pod \"console-654cd7bc6c-sjn5l\" (UID: \"c520dfd7-ff87-4430-9036-715bfbe8ba9e\") " pod="openshift-console/console-654cd7bc6c-sjn5l"
Sep 09 19:56:58.832762 ip-10-0-136-183 kubenswrapper[2574]: I0909 19:56:58.832693    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/6dcf380f-2f53-4047-8a59-8e93030d4c24-secret-thanos-querier-tls\") pod \"thanos-querier-65c468894b-882qv\" (UID: \"6dcf380f-2f53-4047-8a59-8e93030d4c24\") " pod="openshift-monitoring/thanos-querier-65c468894b-882qv"
Sep 09 19:56:58.832910 ip-10-0-136-183 kubenswrapper[2574]: I0909 19:56:58.832894    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/c520dfd7-ff87-4430-9036-715bfbe8ba9e-console-oauth-config\") pod \"console-654cd7bc6c-sjn5l\" (UID: \"c520dfd7-ff87-4430-9036-715bfbe8ba9e\") " pod="openshift-console/console-654cd7bc6c-sjn5l"
Sep 09 19:56:58.833059 ip-10-0-136-183 kubenswrapper[2574]: I0909 19:56:58.833042    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/c520dfd7-ff87-4430-9036-715bfbe8ba9e-console-serving-cert\") pod \"console-654cd7bc6c-sjn5l\" (UID: \"c520dfd7-ff87-4430-9036-715bfbe8ba9e\") " pod="openshift-console/console-654cd7bc6c-sjn5l"
Sep 09 19:56:58.833259 ip-10-0-136-183 kubenswrapper[2574]: I0909 19:56:58.833238    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/6dcf380f-2f53-4047-8a59-8e93030d4c24-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-65c468894b-882qv\" (UID: \"6dcf380f-2f53-4047-8a59-8e93030d4c24\") " pod="openshift-monitoring/thanos-querier-65c468894b-882qv"
Sep 09 19:56:58.836651 ip-10-0-136-183 kubenswrapper[2574]: I0909 19:56:58.836632    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qccn6\" (UniqueName: \"kubernetes.io/projected/6dcf380f-2f53-4047-8a59-8e93030d4c24-kube-api-access-qccn6\") pod \"thanos-querier-65c468894b-882qv\" (UID: \"6dcf380f-2f53-4047-8a59-8e93030d4c24\") " pod="openshift-monitoring/thanos-querier-65c468894b-882qv"
Sep 09 19:56:58.836967 ip-10-0-136-183 kubenswrapper[2574]: I0909 19:56:58.836949    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dvqxf\" (UniqueName: \"kubernetes.io/projected/c520dfd7-ff87-4430-9036-715bfbe8ba9e-kube-api-access-dvqxf\") pod \"console-654cd7bc6c-sjn5l\" (UID: \"c520dfd7-ff87-4430-9036-715bfbe8ba9e\") " pod="openshift-console/console-654cd7bc6c-sjn5l"
Sep 09 19:56:58.844819 ip-10-0-136-183 kubenswrapper[2574]: I0909 19:56:58.844790    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-65c468894b-882qv"
Sep 09 19:56:58.852959 ip-10-0-136-183 kubenswrapper[2574]: I0909 19:56:58.852826    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-654cd7bc6c-sjn5l"
Sep 09 19:56:58.870175 ip-10-0-136-183 kubenswrapper[2574]: I0909 19:56:58.870145    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6dbd947b68-2rr89" event={"ID":"120141dc-7742-4117-9052-6ab138ced47a","Type":"ContainerStarted","Data":"15f1ae17d062cb4c3befa423a27ab1499ef12ec33945489fa764b5076b728be3"}
Sep 09 19:56:58.871488 ip-10-0-136-183 kubenswrapper[2574]: I0909 19:56:58.871463    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-8b862" event={"ID":"06886459-8d92-4c11-826a-a879fb96aa71","Type":"ContainerStarted","Data":"665fadfc127e4ed101d0fa3b70028c7a2a4f1f38497904c80e958c894180021b"}
Sep 09 19:56:58.887400 ip-10-0-136-183 kubenswrapper[2574]: I0909 19:56:58.887184    2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-6dbd947b68-2rr89" podStartSLOduration=1.8617115279999998 podStartE2EDuration="4.887164172s" totalImagesPullingTime="3.025452644s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 19:56:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 19:56:58.887148409 +0000 UTC m=+102.933401671" watchObservedRunningTime="2026-09-09 19:56:58.887164172 +0000 UTC m=+102.933417434"
Sep 09 19:56:59.123475 ip-10-0-136-183 kubenswrapper[2574]: I0909 19:56:59.123416    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-65c468894b-882qv"]
Sep 09 19:56:59.126076 ip-10-0-136-183 kubenswrapper[2574]: W0909 19:56:59.126041    2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6dcf380f_2f53_4047_8a59_8e93030d4c24.slice/crio-5898a812617c0fbc0c5ff73b0536643b7c02610499f5994f2df1409a04ff7e6b WatchSource:0}: Error finding container 5898a812617c0fbc0c5ff73b0536643b7c02610499f5994f2df1409a04ff7e6b: Status 404 returned error can't find the container with id 5898a812617c0fbc0c5ff73b0536643b7c02610499f5994f2df1409a04ff7e6b
Sep 09 19:56:59.126213 ip-10-0-136-183 kubenswrapper[2574]: I0909 19:56:59.126169    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-654cd7bc6c-sjn5l"]
Sep 09 19:56:59.128769 ip-10-0-136-183 kubenswrapper[2574]: W0909 19:56:59.128733    2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc520dfd7_ff87_4430_9036_715bfbe8ba9e.slice/crio-9e3cb00153f8115227cb35570429e115bf595443544cd1582c4b51dc82392c0e WatchSource:0}: Error finding container 9e3cb00153f8115227cb35570429e115bf595443544cd1582c4b51dc82392c0e: Status 404 returned error can't find the container with id 9e3cb00153f8115227cb35570429e115bf595443544cd1582c4b51dc82392c0e
Sep 09 19:56:59.876298 ip-10-0-136-183 kubenswrapper[2574]: I0909 19:56:59.876258    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-654cd7bc6c-sjn5l" event={"ID":"c520dfd7-ff87-4430-9036-715bfbe8ba9e","Type":"ContainerStarted","Data":"de3e1b14b7f44a671c978a65a81d0585b7aefc87ae520021155e898e96563dd3"}
Sep 09 19:56:59.876735 ip-10-0-136-183 kubenswrapper[2574]: I0909 19:56:59.876321    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-654cd7bc6c-sjn5l" event={"ID":"c520dfd7-ff87-4430-9036-715bfbe8ba9e","Type":"ContainerStarted","Data":"9e3cb00153f8115227cb35570429e115bf595443544cd1582c4b51dc82392c0e"}
Sep 09 19:56:59.877525 ip-10-0-136-183 kubenswrapper[2574]: I0909 19:56:59.877495    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-65c468894b-882qv" event={"ID":"6dcf380f-2f53-4047-8a59-8e93030d4c24","Type":"ContainerStarted","Data":"5898a812617c0fbc0c5ff73b0536643b7c02610499f5994f2df1409a04ff7e6b"}
Sep 09 19:56:59.894121 ip-10-0-136-183 kubenswrapper[2574]: I0909 19:56:59.894077    2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-654cd7bc6c-sjn5l" podStartSLOduration=1.8940617990000002 podStartE2EDuration="1.894061799s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 19:56:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 19:56:59.893502609 +0000 UTC m=+103.939755874" watchObservedRunningTime="2026-09-09 19:56:59.894061799 +0000 UTC m=+103.940315052"
Sep 09 19:57:01.356265 ip-10-0-136-183 kubenswrapper[2574]: I0909 19:57:01.356222    2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-6dbd947b68-2rr89"]
Sep 09 19:57:01.887296 ip-10-0-136-183 kubenswrapper[2574]: I0909 19:57:01.887206    2574 generic.go:356] "Generic (PLEG): container finished" podID="06886459-8d92-4c11-826a-a879fb96aa71" containerID="6e40eb024bc69bdbdf178c1931289985e9149b3dd0ea6eb479944619b9e0f9f6" exitCode=0
Sep 09 19:57:01.887455 ip-10-0-136-183 kubenswrapper[2574]: I0909 19:57:01.887294    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-8b862" event={"ID":"06886459-8d92-4c11-826a-a879fb96aa71","Type":"ContainerDied","Data":"6e40eb024bc69bdbdf178c1931289985e9149b3dd0ea6eb479944619b9e0f9f6"}
Sep 09 19:57:01.889557 ip-10-0-136-183 kubenswrapper[2574]: I0909 19:57:01.889528    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-65c468894b-882qv" event={"ID":"6dcf380f-2f53-4047-8a59-8e93030d4c24","Type":"ContainerStarted","Data":"e1ebe3b18626298eeb01074729bd5e1a833e1946ae1934d6e39a18bbc0ed9f4c"}
Sep 09 19:57:01.889676 ip-10-0-136-183 kubenswrapper[2574]: I0909 19:57:01.889594    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-65c468894b-882qv" event={"ID":"6dcf380f-2f53-4047-8a59-8e93030d4c24","Type":"ContainerStarted","Data":"19af049d92b3ab9e844ab68b0a3f021657634d85f2389e32ac83e6f7b6f2f79a"}
Sep 09 19:57:01.889676 ip-10-0-136-183 kubenswrapper[2574]: I0909 19:57:01.889638    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-65c468894b-882qv" event={"ID":"6dcf380f-2f53-4047-8a59-8e93030d4c24","Type":"ContainerStarted","Data":"dbc8d03bef57b544522a5741ac07ca93fe917d1bb9b274707ce322af13479388"}
Sep 09 19:57:02.896974 ip-10-0-136-183 kubenswrapper[2574]: I0909 19:57:02.896930    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-8b862" event={"ID":"06886459-8d92-4c11-826a-a879fb96aa71","Type":"ContainerStarted","Data":"528afa57dd196d6f725d44fe18d26abd6cdf309898073a07d82878c7ccf79e45"}
Sep 09 19:57:02.897557 ip-10-0-136-183 kubenswrapper[2574]: I0909 19:57:02.897527    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-8b862" event={"ID":"06886459-8d92-4c11-826a-a879fb96aa71","Type":"ContainerStarted","Data":"4567634b070515432c3e12893818ece07e1b215bb39db7f356749dfd5b396457"}
Sep 09 19:57:02.901853 ip-10-0-136-183 kubenswrapper[2574]: I0909 19:57:02.901822    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-65c468894b-882qv" event={"ID":"6dcf380f-2f53-4047-8a59-8e93030d4c24","Type":"ContainerStarted","Data":"e019830ae71dacc0021df6b34eb545d669eecbca817a4576c10ac3cc47b44fc8"}
Sep 09 19:57:02.901993 ip-10-0-136-183 kubenswrapper[2574]: I0909 19:57:02.901891    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-65c468894b-882qv" event={"ID":"6dcf380f-2f53-4047-8a59-8e93030d4c24","Type":"ContainerStarted","Data":"c02f724affe23d7c46ad18b87c35b56f0eb4af550bea776fe53c5fb3180dc220"}
Sep 09 19:57:02.901993 ip-10-0-136-183 kubenswrapper[2574]: I0909 19:57:02.901942    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/thanos-querier-65c468894b-882qv"
Sep 09 19:57:02.901993 ip-10-0-136-183 kubenswrapper[2574]: I0909 19:57:02.901972    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-65c468894b-882qv" event={"ID":"6dcf380f-2f53-4047-8a59-8e93030d4c24","Type":"ContainerStarted","Data":"a193fe14c53892b9b9bb8629a6357f7294ea11c7244c3c6e66e7222f78516b37"}
Sep 09 19:57:02.922544 ip-10-0-136-183 kubenswrapper[2574]: I0909 19:57:02.922488    2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-8b862" podStartSLOduration=3.422457361 podStartE2EDuration="6.922473211s" totalImagesPullingTime="3.42712887s" totalInitContainerRuntime="72.88698ms" isStatefulPod=true podCreationTimestamp="2026-09-09 19:56:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 19:57:02.921875993 +0000 UTC m=+106.968129259" watchObservedRunningTime="2026-09-09 19:57:02.922473211 +0000 UTC m=+106.968726472"
Sep 09 19:57:02.950230 ip-10-0-136-183 kubenswrapper[2574]: I0909 19:57:02.950179    2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/thanos-querier-65c468894b-882qv" podStartSLOduration=1.788069109 podStartE2EDuration="4.950163545s" totalImagesPullingTime="3.162094436s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 19:56:58 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 19:57:02.949693311 +0000 UTC m=+106.995946573" watchObservedRunningTime="2026-09-09 19:57:02.950163545 +0000 UTC m=+106.996416807"
Sep 09 19:57:04.879110 ip-10-0-136-183 kubenswrapper[2574]: I0909 19:57:04.879068    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-6dbd947b68-2rr89"
Sep 09 19:57:05.861136 ip-10-0-136-183 kubenswrapper[2574]: I0909 19:57:05.861105    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-4blgq"
Sep 09 19:57:08.813724 ip-10-0-136-183 kubenswrapper[2574]: I0909 19:57:08.813696    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-5f4c5fd4d7-jtgss"
Sep 09 19:57:08.853910 ip-10-0-136-183 kubenswrapper[2574]: I0909 19:57:08.853873    2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-654cd7bc6c-sjn5l"
Sep 09 19:57:08.854049 ip-10-0-136-183 kubenswrapper[2574]: I0909 19:57:08.853935    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-654cd7bc6c-sjn5l"
Sep 09 19:57:08.859640 ip-10-0-136-183 kubenswrapper[2574]: I0909 19:57:08.859610    2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-654cd7bc6c-sjn5l"
Sep 09 19:57:08.913952 ip-10-0-136-183 kubenswrapper[2574]: I0909 19:57:08.913918    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/thanos-querier-65c468894b-882qv"
Sep 09 19:57:08.930212 ip-10-0-136-183 kubenswrapper[2574]: I0909 19:57:08.930189    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-654cd7bc6c-sjn5l"
Sep 09 19:57:22.108205 ip-10-0-136-183 kubenswrapper[2574]: I0909 19:57:22.108164    2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-654cd7bc6c-sjn5l"]
Sep 09 19:57:26.158713 ip-10-0-136-183 kubenswrapper[2574]: I0909 19:57:26.158669    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/47c2b5b9-456a-4528-8d83-7ce3269cc16e-metrics-certs\") pod \"network-metrics-daemon-mdrck\" (UID: \"47c2b5b9-456a-4528-8d83-7ce3269cc16e\") " pod="openshift-multus/network-metrics-daemon-mdrck"
Sep 09 19:57:26.160985 ip-10-0-136-183 kubenswrapper[2574]: I0909 19:57:26.160957    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/47c2b5b9-456a-4528-8d83-7ce3269cc16e-metrics-certs\") pod \"network-metrics-daemon-mdrck\" (UID: \"47c2b5b9-456a-4528-8d83-7ce3269cc16e\") " pod="openshift-multus/network-metrics-daemon-mdrck"
Sep 09 19:57:26.201316 ip-10-0-136-183 kubenswrapper[2574]: I0909 19:57:26.201288    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-cvgdp\""
Sep 09 19:57:26.209997 ip-10-0-136-183 kubenswrapper[2574]: I0909 19:57:26.209943    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mdrck"
Sep 09 19:57:26.381108 ip-10-0-136-183 kubenswrapper[2574]: I0909 19:57:26.381066    2574 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-6dbd947b68-2rr89" podUID="120141dc-7742-4117-9052-6ab138ced47a" containerName="console" containerID="cri-o://15f1ae17d062cb4c3befa423a27ab1499ef12ec33945489fa764b5076b728be3" gracePeriod=15
Sep 09 19:57:26.484201 ip-10-0-136-183 kubenswrapper[2574]: I0909 19:57:26.484123    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-mdrck"]
Sep 09 19:57:26.487730 ip-10-0-136-183 kubenswrapper[2574]: W0909 19:57:26.487696    2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod47c2b5b9_456a_4528_8d83_7ce3269cc16e.slice/crio-ee7cdfc13c3dca942b1370bd7fd5d007344561fbef6e0089cdc4ea8f3ed5a7f3 WatchSource:0}: Error finding container ee7cdfc13c3dca942b1370bd7fd5d007344561fbef6e0089cdc4ea8f3ed5a7f3: Status 404 returned error can't find the container with id ee7cdfc13c3dca942b1370bd7fd5d007344561fbef6e0089cdc4ea8f3ed5a7f3
Sep 09 19:57:26.608577 ip-10-0-136-183 kubenswrapper[2574]: I0909 19:57:26.608549    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6dbd947b68-2rr89"
Sep 09 19:57:26.662500 ip-10-0-136-183 kubenswrapper[2574]: I0909 19:57:26.662468    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/120141dc-7742-4117-9052-6ab138ced47a-service-ca\") pod \"120141dc-7742-4117-9052-6ab138ced47a\" (UID: \"120141dc-7742-4117-9052-6ab138ced47a\") "
Sep 09 19:57:26.662671 ip-10-0-136-183 kubenswrapper[2574]: I0909 19:57:26.662575    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rksfs\" (UniqueName: \"kubernetes.io/projected/120141dc-7742-4117-9052-6ab138ced47a-kube-api-access-rksfs\") pod \"120141dc-7742-4117-9052-6ab138ced47a\" (UID: \"120141dc-7742-4117-9052-6ab138ced47a\") "
Sep 09 19:57:26.662671 ip-10-0-136-183 kubenswrapper[2574]: I0909 19:57:26.662614    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/120141dc-7742-4117-9052-6ab138ced47a-console-config\") pod \"120141dc-7742-4117-9052-6ab138ced47a\" (UID: \"120141dc-7742-4117-9052-6ab138ced47a\") "
Sep 09 19:57:26.662671 ip-10-0-136-183 kubenswrapper[2574]: I0909 19:57:26.662649    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/120141dc-7742-4117-9052-6ab138ced47a-console-oauth-config\") pod \"120141dc-7742-4117-9052-6ab138ced47a\" (UID: \"120141dc-7742-4117-9052-6ab138ced47a\") "
Sep 09 19:57:26.662858 ip-10-0-136-183 kubenswrapper[2574]: I0909 19:57:26.662697    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/120141dc-7742-4117-9052-6ab138ced47a-console-serving-cert\") pod \"120141dc-7742-4117-9052-6ab138ced47a\" (UID: \"120141dc-7742-4117-9052-6ab138ced47a\") "
Sep 09 19:57:26.662858 ip-10-0-136-183 kubenswrapper[2574]: I0909 19:57:26.662722    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/120141dc-7742-4117-9052-6ab138ced47a-oauth-serving-cert\") pod \"120141dc-7742-4117-9052-6ab138ced47a\" (UID: \"120141dc-7742-4117-9052-6ab138ced47a\") "
Sep 09 19:57:26.662966 ip-10-0-136-183 kubenswrapper[2574]: I0909 19:57:26.662877    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/120141dc-7742-4117-9052-6ab138ced47a-service-ca" (OuterVolumeSpecName: "service-ca") pod "120141dc-7742-4117-9052-6ab138ced47a" (UID: "120141dc-7742-4117-9052-6ab138ced47a"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 19:57:26.663027 ip-10-0-136-183 kubenswrapper[2574]: I0909 19:57:26.662997    2574 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/120141dc-7742-4117-9052-6ab138ced47a-service-ca\") on node \"ip-10-0-136-183.ec2.internal\" DevicePath \"\""
Sep 09 19:57:26.663075 ip-10-0-136-183 kubenswrapper[2574]: I0909 19:57:26.663035    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/120141dc-7742-4117-9052-6ab138ced47a-console-config" (OuterVolumeSpecName: "console-config") pod "120141dc-7742-4117-9052-6ab138ced47a" (UID: "120141dc-7742-4117-9052-6ab138ced47a"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 19:57:26.663121 ip-10-0-136-183 kubenswrapper[2574]: I0909 19:57:26.663088    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/120141dc-7742-4117-9052-6ab138ced47a-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "120141dc-7742-4117-9052-6ab138ced47a" (UID: "120141dc-7742-4117-9052-6ab138ced47a"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 19:57:26.664983 ip-10-0-136-183 kubenswrapper[2574]: I0909 19:57:26.664940    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/120141dc-7742-4117-9052-6ab138ced47a-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "120141dc-7742-4117-9052-6ab138ced47a" (UID: "120141dc-7742-4117-9052-6ab138ced47a"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 19:57:26.665098 ip-10-0-136-183 kubenswrapper[2574]: I0909 19:57:26.665049    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/120141dc-7742-4117-9052-6ab138ced47a-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "120141dc-7742-4117-9052-6ab138ced47a" (UID: "120141dc-7742-4117-9052-6ab138ced47a"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 19:57:26.665098 ip-10-0-136-183 kubenswrapper[2574]: I0909 19:57:26.665050    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/120141dc-7742-4117-9052-6ab138ced47a-kube-api-access-rksfs" (OuterVolumeSpecName: "kube-api-access-rksfs") pod "120141dc-7742-4117-9052-6ab138ced47a" (UID: "120141dc-7742-4117-9052-6ab138ced47a"). InnerVolumeSpecName "kube-api-access-rksfs". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 19:57:26.763732 ip-10-0-136-183 kubenswrapper[2574]: I0909 19:57:26.763632    2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rksfs\" (UniqueName: \"kubernetes.io/projected/120141dc-7742-4117-9052-6ab138ced47a-kube-api-access-rksfs\") on node \"ip-10-0-136-183.ec2.internal\" DevicePath \"\""
Sep 09 19:57:26.763732 ip-10-0-136-183 kubenswrapper[2574]: I0909 19:57:26.763673    2574 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/120141dc-7742-4117-9052-6ab138ced47a-console-config\") on node \"ip-10-0-136-183.ec2.internal\" DevicePath \"\""
Sep 09 19:57:26.763732 ip-10-0-136-183 kubenswrapper[2574]: I0909 19:57:26.763683    2574 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/120141dc-7742-4117-9052-6ab138ced47a-console-oauth-config\") on node \"ip-10-0-136-183.ec2.internal\" DevicePath \"\""
Sep 09 19:57:26.763732 ip-10-0-136-183 kubenswrapper[2574]: I0909 19:57:26.763691    2574 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/120141dc-7742-4117-9052-6ab138ced47a-console-serving-cert\") on node \"ip-10-0-136-183.ec2.internal\" DevicePath \"\""
Sep 09 19:57:26.763732 ip-10-0-136-183 kubenswrapper[2574]: I0909 19:57:26.763699    2574 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/120141dc-7742-4117-9052-6ab138ced47a-oauth-serving-cert\") on node \"ip-10-0-136-183.ec2.internal\" DevicePath \"\""
Sep 09 19:57:26.995061 ip-10-0-136-183 kubenswrapper[2574]: I0909 19:57:26.995028    2574 generic.go:356] "Generic (PLEG): container finished" podID="120141dc-7742-4117-9052-6ab138ced47a" containerID="15f1ae17d062cb4c3befa423a27ab1499ef12ec33945489fa764b5076b728be3" exitCode=0
Sep 09 19:57:26.995267 ip-10-0-136-183 kubenswrapper[2574]: I0909 19:57:26.995079    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6dbd947b68-2rr89"
Sep 09 19:57:26.995267 ip-10-0-136-183 kubenswrapper[2574]: I0909 19:57:26.995111    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6dbd947b68-2rr89" event={"ID":"120141dc-7742-4117-9052-6ab138ced47a","Type":"ContainerDied","Data":"15f1ae17d062cb4c3befa423a27ab1499ef12ec33945489fa764b5076b728be3"}
Sep 09 19:57:26.995267 ip-10-0-136-183 kubenswrapper[2574]: I0909 19:57:26.995181    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6dbd947b68-2rr89" event={"ID":"120141dc-7742-4117-9052-6ab138ced47a","Type":"ContainerDied","Data":"2eede6dc73b05c32f4258e780f1eb6bd337f108ce3a4ffe702d1a384dc7422a2"}
Sep 09 19:57:26.995267 ip-10-0-136-183 kubenswrapper[2574]: I0909 19:57:26.995224    2574 scope.go:117] "RemoveContainer" containerID="15f1ae17d062cb4c3befa423a27ab1499ef12ec33945489fa764b5076b728be3"
Sep 09 19:57:26.996455 ip-10-0-136-183 kubenswrapper[2574]: I0909 19:57:26.996416    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-mdrck" event={"ID":"47c2b5b9-456a-4528-8d83-7ce3269cc16e","Type":"ContainerStarted","Data":"ee7cdfc13c3dca942b1370bd7fd5d007344561fbef6e0089cdc4ea8f3ed5a7f3"}
Sep 09 19:57:27.004085 ip-10-0-136-183 kubenswrapper[2574]: I0909 19:57:27.004070    2574 scope.go:117] "RemoveContainer" containerID="15f1ae17d062cb4c3befa423a27ab1499ef12ec33945489fa764b5076b728be3"
Sep 09 19:57:27.004369 ip-10-0-136-183 kubenswrapper[2574]: E0909 19:57:27.004346    2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"15f1ae17d062cb4c3befa423a27ab1499ef12ec33945489fa764b5076b728be3\": container with ID starting with 15f1ae17d062cb4c3befa423a27ab1499ef12ec33945489fa764b5076b728be3 not found: ID does not exist" containerID="15f1ae17d062cb4c3befa423a27ab1499ef12ec33945489fa764b5076b728be3"
Sep 09 19:57:27.004455 ip-10-0-136-183 kubenswrapper[2574]: I0909 19:57:27.004375    2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"15f1ae17d062cb4c3befa423a27ab1499ef12ec33945489fa764b5076b728be3"} err="failed to get container status \"15f1ae17d062cb4c3befa423a27ab1499ef12ec33945489fa764b5076b728be3\": rpc error: code = NotFound desc = could not find container \"15f1ae17d062cb4c3befa423a27ab1499ef12ec33945489fa764b5076b728be3\": container with ID starting with 15f1ae17d062cb4c3befa423a27ab1499ef12ec33945489fa764b5076b728be3 not found: ID does not exist"
Sep 09 19:57:27.015790 ip-10-0-136-183 kubenswrapper[2574]: I0909 19:57:27.015714    2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-6dbd947b68-2rr89"]
Sep 09 19:57:27.019622 ip-10-0-136-183 kubenswrapper[2574]: I0909 19:57:27.019598    2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-6dbd947b68-2rr89"]
Sep 09 19:57:28.490212 ip-10-0-136-183 kubenswrapper[2574]: I0909 19:57:28.490176    2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="120141dc-7742-4117-9052-6ab138ced47a" path="/var/lib/kubelet/pods/120141dc-7742-4117-9052-6ab138ced47a/volumes"
Sep 09 19:57:29.008690 ip-10-0-136-183 kubenswrapper[2574]: I0909 19:57:29.008599    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-mdrck" event={"ID":"47c2b5b9-456a-4528-8d83-7ce3269cc16e","Type":"ContainerStarted","Data":"38220eb451b3beff33b1abbfb385ec662ea357fe200836b1ac373a299d15ab32"}
Sep 09 19:57:29.008690 ip-10-0-136-183 kubenswrapper[2574]: I0909 19:57:29.008661    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-mdrck" event={"ID":"47c2b5b9-456a-4528-8d83-7ce3269cc16e","Type":"ContainerStarted","Data":"f9d795ad41306b7b5e9b23cea46362b88259f87b5c0f97caa670ac39e00dc63d"}
Sep 09 19:57:29.025849 ip-10-0-136-183 kubenswrapper[2574]: I0909 19:57:29.025567    2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-mdrck" podStartSLOduration=130.942518787 podStartE2EDuration="2m13.025544586s" totalImagesPullingTime="2.083025799s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 19:55:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 19:57:29.024860541 +0000 UTC m=+133.071113804" watchObservedRunningTime="2026-09-09 19:57:29.025544586 +0000 UTC m=+133.071797850"
Sep 09 19:57:47.129568 ip-10-0-136-183 kubenswrapper[2574]: I0909 19:57:47.129504    2574 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-654cd7bc6c-sjn5l" podUID="c520dfd7-ff87-4430-9036-715bfbe8ba9e" containerName="console" containerID="cri-o://de3e1b14b7f44a671c978a65a81d0585b7aefc87ae520021155e898e96563dd3" gracePeriod=15
Sep 09 19:57:47.366444 ip-10-0-136-183 kubenswrapper[2574]: I0909 19:57:47.366416    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-654cd7bc6c-sjn5l"
Sep 09 19:57:47.528581 ip-10-0-136-183 kubenswrapper[2574]: I0909 19:57:47.528481    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c520dfd7-ff87-4430-9036-715bfbe8ba9e-trusted-ca-bundle\") pod \"c520dfd7-ff87-4430-9036-715bfbe8ba9e\" (UID: \"c520dfd7-ff87-4430-9036-715bfbe8ba9e\") "
Sep 09 19:57:47.528581 ip-10-0-136-183 kubenswrapper[2574]: I0909 19:57:47.528533    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/c520dfd7-ff87-4430-9036-715bfbe8ba9e-service-ca\") pod \"c520dfd7-ff87-4430-9036-715bfbe8ba9e\" (UID: \"c520dfd7-ff87-4430-9036-715bfbe8ba9e\") "
Sep 09 19:57:47.528581 ip-10-0-136-183 kubenswrapper[2574]: I0909 19:57:47.528557    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/c520dfd7-ff87-4430-9036-715bfbe8ba9e-console-oauth-config\") pod \"c520dfd7-ff87-4430-9036-715bfbe8ba9e\" (UID: \"c520dfd7-ff87-4430-9036-715bfbe8ba9e\") "
Sep 09 19:57:47.528581 ip-10-0-136-183 kubenswrapper[2574]: I0909 19:57:47.528593    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/c520dfd7-ff87-4430-9036-715bfbe8ba9e-console-serving-cert\") pod \"c520dfd7-ff87-4430-9036-715bfbe8ba9e\" (UID: \"c520dfd7-ff87-4430-9036-715bfbe8ba9e\") "
Sep 09 19:57:47.528937 ip-10-0-136-183 kubenswrapper[2574]: I0909 19:57:47.528610    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/c520dfd7-ff87-4430-9036-715bfbe8ba9e-oauth-serving-cert\") pod \"c520dfd7-ff87-4430-9036-715bfbe8ba9e\" (UID: \"c520dfd7-ff87-4430-9036-715bfbe8ba9e\") "
Sep 09 19:57:47.528937 ip-10-0-136-183 kubenswrapper[2574]: I0909 19:57:47.528631    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-dvqxf\" (UniqueName: \"kubernetes.io/projected/c520dfd7-ff87-4430-9036-715bfbe8ba9e-kube-api-access-dvqxf\") pod \"c520dfd7-ff87-4430-9036-715bfbe8ba9e\" (UID: \"c520dfd7-ff87-4430-9036-715bfbe8ba9e\") "
Sep 09 19:57:47.528937 ip-10-0-136-183 kubenswrapper[2574]: I0909 19:57:47.528670    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/c520dfd7-ff87-4430-9036-715bfbe8ba9e-console-config\") pod \"c520dfd7-ff87-4430-9036-715bfbe8ba9e\" (UID: \"c520dfd7-ff87-4430-9036-715bfbe8ba9e\") "
Sep 09 19:57:47.529096 ip-10-0-136-183 kubenswrapper[2574]: I0909 19:57:47.529063    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c520dfd7-ff87-4430-9036-715bfbe8ba9e-service-ca" (OuterVolumeSpecName: "service-ca") pod "c520dfd7-ff87-4430-9036-715bfbe8ba9e" (UID: "c520dfd7-ff87-4430-9036-715bfbe8ba9e"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 19:57:47.529190 ip-10-0-136-183 kubenswrapper[2574]: I0909 19:57:47.529165    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c520dfd7-ff87-4430-9036-715bfbe8ba9e-console-config" (OuterVolumeSpecName: "console-config") pod "c520dfd7-ff87-4430-9036-715bfbe8ba9e" (UID: "c520dfd7-ff87-4430-9036-715bfbe8ba9e"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 19:57:47.529247 ip-10-0-136-183 kubenswrapper[2574]: I0909 19:57:47.529167    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c520dfd7-ff87-4430-9036-715bfbe8ba9e-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "c520dfd7-ff87-4430-9036-715bfbe8ba9e" (UID: "c520dfd7-ff87-4430-9036-715bfbe8ba9e"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 19:57:47.529247 ip-10-0-136-183 kubenswrapper[2574]: I0909 19:57:47.529208    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c520dfd7-ff87-4430-9036-715bfbe8ba9e-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "c520dfd7-ff87-4430-9036-715bfbe8ba9e" (UID: "c520dfd7-ff87-4430-9036-715bfbe8ba9e"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 19:57:47.530758 ip-10-0-136-183 kubenswrapper[2574]: I0909 19:57:47.530735    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c520dfd7-ff87-4430-9036-715bfbe8ba9e-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "c520dfd7-ff87-4430-9036-715bfbe8ba9e" (UID: "c520dfd7-ff87-4430-9036-715bfbe8ba9e"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 19:57:47.531202 ip-10-0-136-183 kubenswrapper[2574]: I0909 19:57:47.531184    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c520dfd7-ff87-4430-9036-715bfbe8ba9e-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "c520dfd7-ff87-4430-9036-715bfbe8ba9e" (UID: "c520dfd7-ff87-4430-9036-715bfbe8ba9e"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 19:57:47.531323 ip-10-0-136-183 kubenswrapper[2574]: I0909 19:57:47.531306    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c520dfd7-ff87-4430-9036-715bfbe8ba9e-kube-api-access-dvqxf" (OuterVolumeSpecName: "kube-api-access-dvqxf") pod "c520dfd7-ff87-4430-9036-715bfbe8ba9e" (UID: "c520dfd7-ff87-4430-9036-715bfbe8ba9e"). InnerVolumeSpecName "kube-api-access-dvqxf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 19:57:47.629548 ip-10-0-136-183 kubenswrapper[2574]: I0909 19:57:47.629496    2574 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/c520dfd7-ff87-4430-9036-715bfbe8ba9e-console-config\") on node \"ip-10-0-136-183.ec2.internal\" DevicePath \"\""
Sep 09 19:57:47.629548 ip-10-0-136-183 kubenswrapper[2574]: I0909 19:57:47.629539    2574 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c520dfd7-ff87-4430-9036-715bfbe8ba9e-trusted-ca-bundle\") on node \"ip-10-0-136-183.ec2.internal\" DevicePath \"\""
Sep 09 19:57:47.629548 ip-10-0-136-183 kubenswrapper[2574]: I0909 19:57:47.629550    2574 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/c520dfd7-ff87-4430-9036-715bfbe8ba9e-service-ca\") on node \"ip-10-0-136-183.ec2.internal\" DevicePath \"\""
Sep 09 19:57:47.629548 ip-10-0-136-183 kubenswrapper[2574]: I0909 19:57:47.629559    2574 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/c520dfd7-ff87-4430-9036-715bfbe8ba9e-console-oauth-config\") on node \"ip-10-0-136-183.ec2.internal\" DevicePath \"\""
Sep 09 19:57:47.629548 ip-10-0-136-183 kubenswrapper[2574]: I0909 19:57:47.629567    2574 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/c520dfd7-ff87-4430-9036-715bfbe8ba9e-console-serving-cert\") on node \"ip-10-0-136-183.ec2.internal\" DevicePath \"\""
Sep 09 19:57:47.629881 ip-10-0-136-183 kubenswrapper[2574]: I0909 19:57:47.629576    2574 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/c520dfd7-ff87-4430-9036-715bfbe8ba9e-oauth-serving-cert\") on node \"ip-10-0-136-183.ec2.internal\" DevicePath \"\""
Sep 09 19:57:47.629881 ip-10-0-136-183 kubenswrapper[2574]: I0909 19:57:47.629584    2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dvqxf\" (UniqueName: \"kubernetes.io/projected/c520dfd7-ff87-4430-9036-715bfbe8ba9e-kube-api-access-dvqxf\") on node \"ip-10-0-136-183.ec2.internal\" DevicePath \"\""
Sep 09 19:57:48.076162 ip-10-0-136-183 kubenswrapper[2574]: I0909 19:57:48.076127    2574 generic.go:356] "Generic (PLEG): container finished" podID="c520dfd7-ff87-4430-9036-715bfbe8ba9e" containerID="de3e1b14b7f44a671c978a65a81d0585b7aefc87ae520021155e898e96563dd3" exitCode=0
Sep 09 19:57:48.076342 ip-10-0-136-183 kubenswrapper[2574]: I0909 19:57:48.076173    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-654cd7bc6c-sjn5l"
Sep 09 19:57:48.076342 ip-10-0-136-183 kubenswrapper[2574]: I0909 19:57:48.076200    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-654cd7bc6c-sjn5l" event={"ID":"c520dfd7-ff87-4430-9036-715bfbe8ba9e","Type":"ContainerDied","Data":"de3e1b14b7f44a671c978a65a81d0585b7aefc87ae520021155e898e96563dd3"}
Sep 09 19:57:48.076342 ip-10-0-136-183 kubenswrapper[2574]: I0909 19:57:48.076239    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-654cd7bc6c-sjn5l" event={"ID":"c520dfd7-ff87-4430-9036-715bfbe8ba9e","Type":"ContainerDied","Data":"9e3cb00153f8115227cb35570429e115bf595443544cd1582c4b51dc82392c0e"}
Sep 09 19:57:48.076342 ip-10-0-136-183 kubenswrapper[2574]: I0909 19:57:48.076258    2574 scope.go:117] "RemoveContainer" containerID="de3e1b14b7f44a671c978a65a81d0585b7aefc87ae520021155e898e96563dd3"
Sep 09 19:57:48.084714 ip-10-0-136-183 kubenswrapper[2574]: I0909 19:57:48.084693    2574 scope.go:117] "RemoveContainer" containerID="de3e1b14b7f44a671c978a65a81d0585b7aefc87ae520021155e898e96563dd3"
Sep 09 19:57:48.084998 ip-10-0-136-183 kubenswrapper[2574]: E0909 19:57:48.084979    2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"de3e1b14b7f44a671c978a65a81d0585b7aefc87ae520021155e898e96563dd3\": container with ID starting with de3e1b14b7f44a671c978a65a81d0585b7aefc87ae520021155e898e96563dd3 not found: ID does not exist" containerID="de3e1b14b7f44a671c978a65a81d0585b7aefc87ae520021155e898e96563dd3"
Sep 09 19:57:48.085055 ip-10-0-136-183 kubenswrapper[2574]: I0909 19:57:48.085007    2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"de3e1b14b7f44a671c978a65a81d0585b7aefc87ae520021155e898e96563dd3"} err="failed to get container status \"de3e1b14b7f44a671c978a65a81d0585b7aefc87ae520021155e898e96563dd3\": rpc error: code = NotFound desc = could not find container \"de3e1b14b7f44a671c978a65a81d0585b7aefc87ae520021155e898e96563dd3\": container with ID starting with de3e1b14b7f44a671c978a65a81d0585b7aefc87ae520021155e898e96563dd3 not found: ID does not exist"
Sep 09 19:57:48.095149 ip-10-0-136-183 kubenswrapper[2574]: I0909 19:57:48.095123    2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-654cd7bc6c-sjn5l"]
Sep 09 19:57:48.097069 ip-10-0-136-183 kubenswrapper[2574]: I0909 19:57:48.097047    2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-654cd7bc6c-sjn5l"]
Sep 09 19:57:48.489900 ip-10-0-136-183 kubenswrapper[2574]: I0909 19:57:48.489867    2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c520dfd7-ff87-4430-9036-715bfbe8ba9e" path="/var/lib/kubelet/pods/c520dfd7-ff87-4430-9036-715bfbe8ba9e/volumes"
Sep 09 20:00:16.340622 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:00:16.340579    2574 kubelet.go:1648] "Image garbage collection succeeded"
Sep 09 20:01:51.509269 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:01:51.509226    2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1av4h9m"]
Sep 09 20:01:51.509933 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:01:51.509639    2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="120141dc-7742-4117-9052-6ab138ced47a" containerName="console"
Sep 09 20:01:51.509933 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:01:51.509659    2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="120141dc-7742-4117-9052-6ab138ced47a" containerName="console"
Sep 09 20:01:51.509933 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:01:51.509680    2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c520dfd7-ff87-4430-9036-715bfbe8ba9e" containerName="console"
Sep 09 20:01:51.509933 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:01:51.509688    2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="c520dfd7-ff87-4430-9036-715bfbe8ba9e" containerName="console"
Sep 09 20:01:51.509933 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:01:51.509762    2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="120141dc-7742-4117-9052-6ab138ced47a" containerName="console"
Sep 09 20:01:51.509933 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:01:51.509779    2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="c520dfd7-ff87-4430-9036-715bfbe8ba9e" containerName="console"
Sep 09 20:01:51.512933 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:01:51.512912    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1av4h9m"
Sep 09 20:01:51.515058 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:01:51.515036    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\""
Sep 09 20:01:51.515455 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:01:51.515426    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-qzp6l\""
Sep 09 20:01:51.515568 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:01:51.515439    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\""
Sep 09 20:01:51.529440 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:01:51.529413    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1av4h9m"]
Sep 09 20:01:51.617592 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:01:51.617557    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/c687b3a2-38e9-40b1-9ba2-fb7d79f21bd9-util\") pod \"1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1av4h9m\" (UID: \"c687b3a2-38e9-40b1-9ba2-fb7d79f21bd9\") " pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1av4h9m"
Sep 09 20:01:51.617776 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:01:51.617602    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/c687b3a2-38e9-40b1-9ba2-fb7d79f21bd9-bundle\") pod \"1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1av4h9m\" (UID: \"c687b3a2-38e9-40b1-9ba2-fb7d79f21bd9\") " pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1av4h9m"
Sep 09 20:01:51.617776 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:01:51.617630    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-278xb\" (UniqueName: \"kubernetes.io/projected/c687b3a2-38e9-40b1-9ba2-fb7d79f21bd9-kube-api-access-278xb\") pod \"1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1av4h9m\" (UID: \"c687b3a2-38e9-40b1-9ba2-fb7d79f21bd9\") " pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1av4h9m"
Sep 09 20:01:51.718370 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:01:51.718325    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/c687b3a2-38e9-40b1-9ba2-fb7d79f21bd9-util\") pod \"1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1av4h9m\" (UID: \"c687b3a2-38e9-40b1-9ba2-fb7d79f21bd9\") " pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1av4h9m"
Sep 09 20:01:51.718370 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:01:51.718386    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/c687b3a2-38e9-40b1-9ba2-fb7d79f21bd9-bundle\") pod \"1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1av4h9m\" (UID: \"c687b3a2-38e9-40b1-9ba2-fb7d79f21bd9\") " pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1av4h9m"
Sep 09 20:01:51.718603 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:01:51.718414    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-278xb\" (UniqueName: \"kubernetes.io/projected/c687b3a2-38e9-40b1-9ba2-fb7d79f21bd9-kube-api-access-278xb\") pod \"1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1av4h9m\" (UID: \"c687b3a2-38e9-40b1-9ba2-fb7d79f21bd9\") " pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1av4h9m"
Sep 09 20:01:51.718745 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:01:51.718725    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/c687b3a2-38e9-40b1-9ba2-fb7d79f21bd9-util\") pod \"1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1av4h9m\" (UID: \"c687b3a2-38e9-40b1-9ba2-fb7d79f21bd9\") " pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1av4h9m"
Sep 09 20:01:51.718830 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:01:51.718791    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/c687b3a2-38e9-40b1-9ba2-fb7d79f21bd9-bundle\") pod \"1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1av4h9m\" (UID: \"c687b3a2-38e9-40b1-9ba2-fb7d79f21bd9\") " pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1av4h9m"
Sep 09 20:01:51.726923 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:01:51.726904    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-278xb\" (UniqueName: \"kubernetes.io/projected/c687b3a2-38e9-40b1-9ba2-fb7d79f21bd9-kube-api-access-278xb\") pod \"1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1av4h9m\" (UID: \"c687b3a2-38e9-40b1-9ba2-fb7d79f21bd9\") " pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1av4h9m"
Sep 09 20:01:51.821868 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:01:51.821744    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1av4h9m"
Sep 09 20:01:52.092791 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:01:52.092715    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1av4h9m"]
Sep 09 20:01:52.095350 ip-10-0-136-183 kubenswrapper[2574]: W0909 20:01:52.095321    2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc687b3a2_38e9_40b1_9ba2_fb7d79f21bd9.slice/crio-2210ade1915697e09e658d99a4279d6989d7814eab825071124c9753aacc8015 WatchSource:0}: Error finding container 2210ade1915697e09e658d99a4279d6989d7814eab825071124c9753aacc8015: Status 404 returned error can't find the container with id 2210ade1915697e09e658d99a4279d6989d7814eab825071124c9753aacc8015
Sep 09 20:01:52.097355 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:01:52.097337    2574 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 09 20:01:52.928838 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:01:52.928785    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1av4h9m" event={"ID":"c687b3a2-38e9-40b1-9ba2-fb7d79f21bd9","Type":"ContainerStarted","Data":"2210ade1915697e09e658d99a4279d6989d7814eab825071124c9753aacc8015"}
Sep 09 20:01:59.957927 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:01:59.957888    2574 generic.go:356] "Generic (PLEG): container finished" podID="c687b3a2-38e9-40b1-9ba2-fb7d79f21bd9" containerID="d42d18689b6fdc371b1cd85a9449a627b796f56d5870f63b322dbd2a71166497" exitCode=0
Sep 09 20:01:59.958319 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:01:59.957965    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1av4h9m" event={"ID":"c687b3a2-38e9-40b1-9ba2-fb7d79f21bd9","Type":"ContainerDied","Data":"d42d18689b6fdc371b1cd85a9449a627b796f56d5870f63b322dbd2a71166497"}
Sep 09 20:02:02.972069 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:02:02.972034    2574 generic.go:356] "Generic (PLEG): container finished" podID="c687b3a2-38e9-40b1-9ba2-fb7d79f21bd9" containerID="b5eac3f2e4010ed26518d85fdc19e14ef7c53dcdce51a79bfc0ce889b4111f1c" exitCode=0
Sep 09 20:02:02.972466 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:02:02.972087    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1av4h9m" event={"ID":"c687b3a2-38e9-40b1-9ba2-fb7d79f21bd9","Type":"ContainerDied","Data":"b5eac3f2e4010ed26518d85fdc19e14ef7c53dcdce51a79bfc0ce889b4111f1c"}
Sep 09 20:02:10.001334 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:02:10.001292    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1av4h9m" event={"ID":"c687b3a2-38e9-40b1-9ba2-fb7d79f21bd9","Type":"ContainerStarted","Data":"3bf8265b267c1b29e450e7fb2db76f2f920d50174317ae1a1980d6491304fc99"}
Sep 09 20:02:10.029269 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:02:10.029213    2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1av4h9m" podStartSLOduration=2.317728841 podStartE2EDuration="19.029195842s" totalImagesPullingTime="16.711467001s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 20:01:51 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:02:10.028470712 +0000 UTC m=+414.074723973" watchObservedRunningTime="2026-09-09 20:02:10.029195842 +0000 UTC m=+414.075449104"
Sep 09 20:02:11.007374 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:02:11.007335    2574 generic.go:356] "Generic (PLEG): container finished" podID="c687b3a2-38e9-40b1-9ba2-fb7d79f21bd9" containerID="3bf8265b267c1b29e450e7fb2db76f2f920d50174317ae1a1980d6491304fc99" exitCode=0
Sep 09 20:02:11.007755 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:02:11.007405    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1av4h9m" event={"ID":"c687b3a2-38e9-40b1-9ba2-fb7d79f21bd9","Type":"ContainerDied","Data":"3bf8265b267c1b29e450e7fb2db76f2f920d50174317ae1a1980d6491304fc99"}
Sep 09 20:02:12.141538 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:02:12.141514    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1av4h9m"
Sep 09 20:02:12.290436 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:02:12.290346    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-278xb\" (UniqueName: \"kubernetes.io/projected/c687b3a2-38e9-40b1-9ba2-fb7d79f21bd9-kube-api-access-278xb\") pod \"c687b3a2-38e9-40b1-9ba2-fb7d79f21bd9\" (UID: \"c687b3a2-38e9-40b1-9ba2-fb7d79f21bd9\") "
Sep 09 20:02:12.290436 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:02:12.290392    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/c687b3a2-38e9-40b1-9ba2-fb7d79f21bd9-util\") pod \"c687b3a2-38e9-40b1-9ba2-fb7d79f21bd9\" (UID: \"c687b3a2-38e9-40b1-9ba2-fb7d79f21bd9\") "
Sep 09 20:02:12.290436 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:02:12.290429    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/c687b3a2-38e9-40b1-9ba2-fb7d79f21bd9-bundle\") pod \"c687b3a2-38e9-40b1-9ba2-fb7d79f21bd9\" (UID: \"c687b3a2-38e9-40b1-9ba2-fb7d79f21bd9\") "
Sep 09 20:02:12.291105 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:02:12.291078    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c687b3a2-38e9-40b1-9ba2-fb7d79f21bd9-bundle" (OuterVolumeSpecName: "bundle") pod "c687b3a2-38e9-40b1-9ba2-fb7d79f21bd9" (UID: "c687b3a2-38e9-40b1-9ba2-fb7d79f21bd9"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 20:02:12.292553 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:02:12.292523    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c687b3a2-38e9-40b1-9ba2-fb7d79f21bd9-kube-api-access-278xb" (OuterVolumeSpecName: "kube-api-access-278xb") pod "c687b3a2-38e9-40b1-9ba2-fb7d79f21bd9" (UID: "c687b3a2-38e9-40b1-9ba2-fb7d79f21bd9"). InnerVolumeSpecName "kube-api-access-278xb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 20:02:12.294939 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:02:12.294918    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c687b3a2-38e9-40b1-9ba2-fb7d79f21bd9-util" (OuterVolumeSpecName: "util") pod "c687b3a2-38e9-40b1-9ba2-fb7d79f21bd9" (UID: "c687b3a2-38e9-40b1-9ba2-fb7d79f21bd9"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 20:02:12.391216 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:02:12.391177    2574 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/c687b3a2-38e9-40b1-9ba2-fb7d79f21bd9-util\") on node \"ip-10-0-136-183.ec2.internal\" DevicePath \"\""
Sep 09 20:02:12.391216 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:02:12.391210    2574 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/c687b3a2-38e9-40b1-9ba2-fb7d79f21bd9-bundle\") on node \"ip-10-0-136-183.ec2.internal\" DevicePath \"\""
Sep 09 20:02:12.391216 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:02:12.391221    2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-278xb\" (UniqueName: \"kubernetes.io/projected/c687b3a2-38e9-40b1-9ba2-fb7d79f21bd9-kube-api-access-278xb\") on node \"ip-10-0-136-183.ec2.internal\" DevicePath \"\""
Sep 09 20:02:13.016946 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:02:13.016912    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1av4h9m"
Sep 09 20:02:13.017114 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:02:13.016917    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1av4h9m" event={"ID":"c687b3a2-38e9-40b1-9ba2-fb7d79f21bd9","Type":"ContainerDied","Data":"2210ade1915697e09e658d99a4279d6989d7814eab825071124c9753aacc8015"}
Sep 09 20:02:13.017114 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:02:13.017040    2574 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2210ade1915697e09e658d99a4279d6989d7814eab825071124c9753aacc8015"
Sep 09 20:03:45.305897 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:03:45.305789    2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/kserve-controller-manager-7676c798d9-5jm6b"]
Sep 09 20:03:45.306363 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:03:45.306130    2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c687b3a2-38e9-40b1-9ba2-fb7d79f21bd9" containerName="extract"
Sep 09 20:03:45.306363 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:03:45.306140    2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="c687b3a2-38e9-40b1-9ba2-fb7d79f21bd9" containerName="extract"
Sep 09 20:03:45.306363 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:03:45.306154    2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c687b3a2-38e9-40b1-9ba2-fb7d79f21bd9" containerName="pull"
Sep 09 20:03:45.306363 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:03:45.306161    2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="c687b3a2-38e9-40b1-9ba2-fb7d79f21bd9" containerName="pull"
Sep 09 20:03:45.306363 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:03:45.306175    2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c687b3a2-38e9-40b1-9ba2-fb7d79f21bd9" containerName="util"
Sep 09 20:03:45.306363 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:03:45.306180    2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="c687b3a2-38e9-40b1-9ba2-fb7d79f21bd9" containerName="util"
Sep 09 20:03:45.306363 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:03:45.306225    2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="c687b3a2-38e9-40b1-9ba2-fb7d79f21bd9" containerName="extract"
Sep 09 20:03:45.308050 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:03:45.308031    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/kserve-controller-manager-7676c798d9-5jm6b"
Sep 09 20:03:45.310319 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:03:45.310292    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"kserve-controller-manager-dockercfg-x9ppf\""
Sep 09 20:03:45.310438 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:03:45.310327    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve\"/\"openshift-service-ca.crt\""
Sep 09 20:03:45.310438 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:03:45.310329    2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/llmisvc-controller-manager-645f9c8fd6-lg7w2"]
Sep 09 20:03:45.310633 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:03:45.310618    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"kserve-webhook-server-cert\""
Sep 09 20:03:45.310948 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:03:45.310933    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve\"/\"kube-root-ca.crt\""
Sep 09 20:03:45.312408 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:03:45.312392    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/llmisvc-controller-manager-645f9c8fd6-lg7w2"
Sep 09 20:03:45.314291 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:03:45.314272    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"llmisvc-webhook-server-cert\""
Sep 09 20:03:45.314363 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:03:45.314282    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"llmisvc-controller-manager-dockercfg-t64h7\""
Sep 09 20:03:45.320850 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:03:45.320823    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/kserve-controller-manager-7676c798d9-5jm6b"]
Sep 09 20:03:45.325108 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:03:45.325081    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/llmisvc-controller-manager-645f9c8fd6-lg7w2"]
Sep 09 20:03:45.486057 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:03:45.486022    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/fe0462c4-ccd9-41fd-a85e-1fc4f3ba500d-cert\") pod \"llmisvc-controller-manager-645f9c8fd6-lg7w2\" (UID: \"fe0462c4-ccd9-41fd-a85e-1fc4f3ba500d\") " pod="kserve/llmisvc-controller-manager-645f9c8fd6-lg7w2"
Sep 09 20:03:45.486247 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:03:45.486092    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/9fccc9fc-2171-447d-aca4-b9d62054c8b7-cert\") pod \"kserve-controller-manager-7676c798d9-5jm6b\" (UID: \"9fccc9fc-2171-447d-aca4-b9d62054c8b7\") " pod="kserve/kserve-controller-manager-7676c798d9-5jm6b"
Sep 09 20:03:45.486247 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:03:45.486122    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t286n\" (UniqueName: \"kubernetes.io/projected/fe0462c4-ccd9-41fd-a85e-1fc4f3ba500d-kube-api-access-t286n\") pod \"llmisvc-controller-manager-645f9c8fd6-lg7w2\" (UID: \"fe0462c4-ccd9-41fd-a85e-1fc4f3ba500d\") " pod="kserve/llmisvc-controller-manager-645f9c8fd6-lg7w2"
Sep 09 20:03:45.486247 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:03:45.486155    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2m2bx\" (UniqueName: \"kubernetes.io/projected/9fccc9fc-2171-447d-aca4-b9d62054c8b7-kube-api-access-2m2bx\") pod \"kserve-controller-manager-7676c798d9-5jm6b\" (UID: \"9fccc9fc-2171-447d-aca4-b9d62054c8b7\") " pod="kserve/kserve-controller-manager-7676c798d9-5jm6b"
Sep 09 20:03:45.587380 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:03:45.587288    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/fe0462c4-ccd9-41fd-a85e-1fc4f3ba500d-cert\") pod \"llmisvc-controller-manager-645f9c8fd6-lg7w2\" (UID: \"fe0462c4-ccd9-41fd-a85e-1fc4f3ba500d\") " pod="kserve/llmisvc-controller-manager-645f9c8fd6-lg7w2"
Sep 09 20:03:45.587380 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:03:45.587354    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/9fccc9fc-2171-447d-aca4-b9d62054c8b7-cert\") pod \"kserve-controller-manager-7676c798d9-5jm6b\" (UID: \"9fccc9fc-2171-447d-aca4-b9d62054c8b7\") " pod="kserve/kserve-controller-manager-7676c798d9-5jm6b"
Sep 09 20:03:45.587621 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:03:45.587391    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t286n\" (UniqueName: \"kubernetes.io/projected/fe0462c4-ccd9-41fd-a85e-1fc4f3ba500d-kube-api-access-t286n\") pod \"llmisvc-controller-manager-645f9c8fd6-lg7w2\" (UID: \"fe0462c4-ccd9-41fd-a85e-1fc4f3ba500d\") " pod="kserve/llmisvc-controller-manager-645f9c8fd6-lg7w2"
Sep 09 20:03:45.587621 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:03:45.587439    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2m2bx\" (UniqueName: \"kubernetes.io/projected/9fccc9fc-2171-447d-aca4-b9d62054c8b7-kube-api-access-2m2bx\") pod \"kserve-controller-manager-7676c798d9-5jm6b\" (UID: \"9fccc9fc-2171-447d-aca4-b9d62054c8b7\") " pod="kserve/kserve-controller-manager-7676c798d9-5jm6b"
Sep 09 20:03:45.589762 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:03:45.589725    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/9fccc9fc-2171-447d-aca4-b9d62054c8b7-cert\") pod \"kserve-controller-manager-7676c798d9-5jm6b\" (UID: \"9fccc9fc-2171-447d-aca4-b9d62054c8b7\") " pod="kserve/kserve-controller-manager-7676c798d9-5jm6b"
Sep 09 20:03:45.589886 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:03:45.589873    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/fe0462c4-ccd9-41fd-a85e-1fc4f3ba500d-cert\") pod \"llmisvc-controller-manager-645f9c8fd6-lg7w2\" (UID: \"fe0462c4-ccd9-41fd-a85e-1fc4f3ba500d\") " pod="kserve/llmisvc-controller-manager-645f9c8fd6-lg7w2"
Sep 09 20:03:45.596689 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:03:45.596664    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-t286n\" (UniqueName: \"kubernetes.io/projected/fe0462c4-ccd9-41fd-a85e-1fc4f3ba500d-kube-api-access-t286n\") pod \"llmisvc-controller-manager-645f9c8fd6-lg7w2\" (UID: \"fe0462c4-ccd9-41fd-a85e-1fc4f3ba500d\") " pod="kserve/llmisvc-controller-manager-645f9c8fd6-lg7w2"
Sep 09 20:03:45.596963 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:03:45.596946    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2m2bx\" (UniqueName: \"kubernetes.io/projected/9fccc9fc-2171-447d-aca4-b9d62054c8b7-kube-api-access-2m2bx\") pod \"kserve-controller-manager-7676c798d9-5jm6b\" (UID: \"9fccc9fc-2171-447d-aca4-b9d62054c8b7\") " pod="kserve/kserve-controller-manager-7676c798d9-5jm6b"
Sep 09 20:03:45.619939 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:03:45.619903    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/kserve-controller-manager-7676c798d9-5jm6b"
Sep 09 20:03:45.627036 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:03:45.627002    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/llmisvc-controller-manager-645f9c8fd6-lg7w2"
Sep 09 20:03:45.893573 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:03:45.893494    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/kserve-controller-manager-7676c798d9-5jm6b"]
Sep 09 20:03:45.897128 ip-10-0-136-183 kubenswrapper[2574]: W0909 20:03:45.897105    2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9fccc9fc_2171_447d_aca4_b9d62054c8b7.slice/crio-b22709b83acbfff4bd2a306d177649f8c60e066e0c550a769b5517cde7714336 WatchSource:0}: Error finding container b22709b83acbfff4bd2a306d177649f8c60e066e0c550a769b5517cde7714336: Status 404 returned error can't find the container with id b22709b83acbfff4bd2a306d177649f8c60e066e0c550a769b5517cde7714336
Sep 09 20:03:45.901243 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:03:45.901181    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/llmisvc-controller-manager-645f9c8fd6-lg7w2"]
Sep 09 20:03:45.903255 ip-10-0-136-183 kubenswrapper[2574]: W0909 20:03:45.903230    2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-podfe0462c4_ccd9_41fd_a85e_1fc4f3ba500d.slice/crio-6dd0ed9919d66868e82cc9e34d5f63f2945edc22234f8ed39337190031072538 WatchSource:0}: Error finding container 6dd0ed9919d66868e82cc9e34d5f63f2945edc22234f8ed39337190031072538: Status 404 returned error can't find the container with id 6dd0ed9919d66868e82cc9e34d5f63f2945edc22234f8ed39337190031072538
Sep 09 20:03:46.350188 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:03:46.350099    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/kserve-controller-manager-7676c798d9-5jm6b" event={"ID":"9fccc9fc-2171-447d-aca4-b9d62054c8b7","Type":"ContainerStarted","Data":"b22709b83acbfff4bd2a306d177649f8c60e066e0c550a769b5517cde7714336"}
Sep 09 20:03:46.351489 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:03:46.351450    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/llmisvc-controller-manager-645f9c8fd6-lg7w2" event={"ID":"fe0462c4-ccd9-41fd-a85e-1fc4f3ba500d","Type":"ContainerStarted","Data":"6dd0ed9919d66868e82cc9e34d5f63f2945edc22234f8ed39337190031072538"}
Sep 09 20:03:49.367089 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:03:49.367046    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/kserve-controller-manager-7676c798d9-5jm6b" event={"ID":"9fccc9fc-2171-447d-aca4-b9d62054c8b7","Type":"ContainerStarted","Data":"c0a62492fe5ae7dc195baaa16748b59e4986bc4d4879933fddab4a025e5563a2"}
Sep 09 20:03:49.367558 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:03:49.367123    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve/kserve-controller-manager-7676c798d9-5jm6b"
Sep 09 20:03:49.368449 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:03:49.368430    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/llmisvc-controller-manager-645f9c8fd6-lg7w2" event={"ID":"fe0462c4-ccd9-41fd-a85e-1fc4f3ba500d","Type":"ContainerStarted","Data":"11717abfedd1c8bc446f046b4c0c20a9b57fd6e4459614c1e3ec89d1123ffa71"}
Sep 09 20:03:49.368584 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:03:49.368563    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve/llmisvc-controller-manager-645f9c8fd6-lg7w2"
Sep 09 20:03:49.385044 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:03:49.384989    2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/kserve-controller-manager-7676c798d9-5jm6b" podStartSLOduration=1.262771478 podStartE2EDuration="4.384972644s" totalImagesPullingTime="3.122201166s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 20:03:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:03:49.383771586 +0000 UTC m=+513.430024850" watchObservedRunningTime="2026-09-09 20:03:49.384972644 +0000 UTC m=+513.431225905"
Sep 09 20:03:49.399423 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:03:49.399365    2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/llmisvc-controller-manager-645f9c8fd6-lg7w2" podStartSLOduration=1.292511791 podStartE2EDuration="4.399340927s" totalImagesPullingTime="3.106829136s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 20:03:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:03:49.398459199 +0000 UTC m=+513.444712462" watchObservedRunningTime="2026-09-09 20:03:49.399340927 +0000 UTC m=+513.445594189"
Sep 09 20:04:20.374998 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:04:20.374913    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve/llmisvc-controller-manager-645f9c8fd6-lg7w2"
Sep 09 20:04:20.378028 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:04:20.378004    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve/kserve-controller-manager-7676c798d9-5jm6b"
Sep 09 20:04:28.703457 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:04:28.703417    2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve/kserve-controller-manager-7676c798d9-5jm6b"]
Sep 09 20:04:28.703909 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:04:28.703613    2574 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve/kserve-controller-manager-7676c798d9-5jm6b" podUID="9fccc9fc-2171-447d-aca4-b9d62054c8b7" containerName="manager" containerID="cri-o://c0a62492fe5ae7dc195baaa16748b59e4986bc4d4879933fddab4a025e5563a2" gracePeriod=10
Sep 09 20:04:28.732224 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:04:28.732195    2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/kserve-controller-manager-7676c798d9-b9cfb"]
Sep 09 20:04:28.734307 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:04:28.734291    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/kserve-controller-manager-7676c798d9-b9cfb"
Sep 09 20:04:28.747462 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:04:28.747430    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/kserve-controller-manager-7676c798d9-b9cfb"]
Sep 09 20:04:28.813538 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:04:28.813501    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/93002f61-5ef9-4f71-baf0-75011154f51b-cert\") pod \"kserve-controller-manager-7676c798d9-b9cfb\" (UID: \"93002f61-5ef9-4f71-baf0-75011154f51b\") " pod="kserve/kserve-controller-manager-7676c798d9-b9cfb"
Sep 09 20:04:28.813538 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:04:28.813549    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-55zrl\" (UniqueName: \"kubernetes.io/projected/93002f61-5ef9-4f71-baf0-75011154f51b-kube-api-access-55zrl\") pod \"kserve-controller-manager-7676c798d9-b9cfb\" (UID: \"93002f61-5ef9-4f71-baf0-75011154f51b\") " pod="kserve/kserve-controller-manager-7676c798d9-b9cfb"
Sep 09 20:04:28.914575 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:04:28.914547    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/93002f61-5ef9-4f71-baf0-75011154f51b-cert\") pod \"kserve-controller-manager-7676c798d9-b9cfb\" (UID: \"93002f61-5ef9-4f71-baf0-75011154f51b\") " pod="kserve/kserve-controller-manager-7676c798d9-b9cfb"
Sep 09 20:04:28.914758 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:04:28.914580    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-55zrl\" (UniqueName: \"kubernetes.io/projected/93002f61-5ef9-4f71-baf0-75011154f51b-kube-api-access-55zrl\") pod \"kserve-controller-manager-7676c798d9-b9cfb\" (UID: \"93002f61-5ef9-4f71-baf0-75011154f51b\") " pod="kserve/kserve-controller-manager-7676c798d9-b9cfb"
Sep 09 20:04:28.917018 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:04:28.916990    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/93002f61-5ef9-4f71-baf0-75011154f51b-cert\") pod \"kserve-controller-manager-7676c798d9-b9cfb\" (UID: \"93002f61-5ef9-4f71-baf0-75011154f51b\") " pod="kserve/kserve-controller-manager-7676c798d9-b9cfb"
Sep 09 20:04:28.923320 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:04:28.923296    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-55zrl\" (UniqueName: \"kubernetes.io/projected/93002f61-5ef9-4f71-baf0-75011154f51b-kube-api-access-55zrl\") pod \"kserve-controller-manager-7676c798d9-b9cfb\" (UID: \"93002f61-5ef9-4f71-baf0-75011154f51b\") " pod="kserve/kserve-controller-manager-7676c798d9-b9cfb"
Sep 09 20:04:28.938866 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:04:28.938839    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/kserve-controller-manager-7676c798d9-5jm6b"
Sep 09 20:04:29.015859 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:04:29.015733    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/9fccc9fc-2171-447d-aca4-b9d62054c8b7-cert\") pod \"9fccc9fc-2171-447d-aca4-b9d62054c8b7\" (UID: \"9fccc9fc-2171-447d-aca4-b9d62054c8b7\") "
Sep 09 20:04:29.015859 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:04:29.015838    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2m2bx\" (UniqueName: \"kubernetes.io/projected/9fccc9fc-2171-447d-aca4-b9d62054c8b7-kube-api-access-2m2bx\") pod \"9fccc9fc-2171-447d-aca4-b9d62054c8b7\" (UID: \"9fccc9fc-2171-447d-aca4-b9d62054c8b7\") "
Sep 09 20:04:29.018473 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:04:29.018404    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9fccc9fc-2171-447d-aca4-b9d62054c8b7-cert" (OuterVolumeSpecName: "cert") pod "9fccc9fc-2171-447d-aca4-b9d62054c8b7" (UID: "9fccc9fc-2171-447d-aca4-b9d62054c8b7"). InnerVolumeSpecName "cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 20:04:29.018612 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:04:29.018493    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9fccc9fc-2171-447d-aca4-b9d62054c8b7-kube-api-access-2m2bx" (OuterVolumeSpecName: "kube-api-access-2m2bx") pod "9fccc9fc-2171-447d-aca4-b9d62054c8b7" (UID: "9fccc9fc-2171-447d-aca4-b9d62054c8b7"). InnerVolumeSpecName "kube-api-access-2m2bx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 20:04:29.090540 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:04:29.090497    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/kserve-controller-manager-7676c798d9-b9cfb"
Sep 09 20:04:29.116640 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:04:29.116610    2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2m2bx\" (UniqueName: \"kubernetes.io/projected/9fccc9fc-2171-447d-aca4-b9d62054c8b7-kube-api-access-2m2bx\") on node \"ip-10-0-136-183.ec2.internal\" DevicePath \"\""
Sep 09 20:04:29.116640 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:04:29.116638    2574 reconciler_common.go:299] "Volume detached for volume \"cert\" (UniqueName: \"kubernetes.io/secret/9fccc9fc-2171-447d-aca4-b9d62054c8b7-cert\") on node \"ip-10-0-136-183.ec2.internal\" DevicePath \"\""
Sep 09 20:04:29.358546 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:04:29.358513    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/kserve-controller-manager-7676c798d9-b9cfb"]
Sep 09 20:04:29.362080 ip-10-0-136-183 kubenswrapper[2574]: W0909 20:04:29.362053    2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod93002f61_5ef9_4f71_baf0_75011154f51b.slice/crio-03a4f9506d296b8791b7ea9ecf496404231876a58ca0fa2b05d7a65cae2815b2 WatchSource:0}: Error finding container 03a4f9506d296b8791b7ea9ecf496404231876a58ca0fa2b05d7a65cae2815b2: Status 404 returned error can't find the container with id 03a4f9506d296b8791b7ea9ecf496404231876a58ca0fa2b05d7a65cae2815b2
Sep 09 20:04:29.522673 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:04:29.522625    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/kserve-controller-manager-7676c798d9-b9cfb" event={"ID":"93002f61-5ef9-4f71-baf0-75011154f51b","Type":"ContainerStarted","Data":"03a4f9506d296b8791b7ea9ecf496404231876a58ca0fa2b05d7a65cae2815b2"}
Sep 09 20:04:29.523869 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:04:29.523841    2574 generic.go:356] "Generic (PLEG): container finished" podID="9fccc9fc-2171-447d-aca4-b9d62054c8b7" containerID="c0a62492fe5ae7dc195baaa16748b59e4986bc4d4879933fddab4a025e5563a2" exitCode=0
Sep 09 20:04:29.523954 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:04:29.523901    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/kserve-controller-manager-7676c798d9-5jm6b"
Sep 09 20:04:29.523954 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:04:29.523924    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/kserve-controller-manager-7676c798d9-5jm6b" event={"ID":"9fccc9fc-2171-447d-aca4-b9d62054c8b7","Type":"ContainerDied","Data":"c0a62492fe5ae7dc195baaa16748b59e4986bc4d4879933fddab4a025e5563a2"}
Sep 09 20:04:29.524019 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:04:29.523962    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/kserve-controller-manager-7676c798d9-5jm6b" event={"ID":"9fccc9fc-2171-447d-aca4-b9d62054c8b7","Type":"ContainerDied","Data":"b22709b83acbfff4bd2a306d177649f8c60e066e0c550a769b5517cde7714336"}
Sep 09 20:04:29.524019 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:04:29.523988    2574 scope.go:117] "RemoveContainer" containerID="c0a62492fe5ae7dc195baaa16748b59e4986bc4d4879933fddab4a025e5563a2"
Sep 09 20:04:29.532089 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:04:29.532067    2574 scope.go:117] "RemoveContainer" containerID="c0a62492fe5ae7dc195baaa16748b59e4986bc4d4879933fddab4a025e5563a2"
Sep 09 20:04:29.532347 ip-10-0-136-183 kubenswrapper[2574]: E0909 20:04:29.532327    2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c0a62492fe5ae7dc195baaa16748b59e4986bc4d4879933fddab4a025e5563a2\": container with ID starting with c0a62492fe5ae7dc195baaa16748b59e4986bc4d4879933fddab4a025e5563a2 not found: ID does not exist" containerID="c0a62492fe5ae7dc195baaa16748b59e4986bc4d4879933fddab4a025e5563a2"
Sep 09 20:04:29.532408 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:04:29.532360    2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c0a62492fe5ae7dc195baaa16748b59e4986bc4d4879933fddab4a025e5563a2"} err="failed to get container status \"c0a62492fe5ae7dc195baaa16748b59e4986bc4d4879933fddab4a025e5563a2\": rpc error: code = NotFound desc = could not find container \"c0a62492fe5ae7dc195baaa16748b59e4986bc4d4879933fddab4a025e5563a2\": container with ID starting with c0a62492fe5ae7dc195baaa16748b59e4986bc4d4879933fddab4a025e5563a2 not found: ID does not exist"
Sep 09 20:04:29.543181 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:04:29.543157    2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve/kserve-controller-manager-7676c798d9-5jm6b"]
Sep 09 20:04:29.546767 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:04:29.546746    2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve/kserve-controller-manager-7676c798d9-5jm6b"]
Sep 09 20:04:30.490450 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:04:30.490416    2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9fccc9fc-2171-447d-aca4-b9d62054c8b7" path="/var/lib/kubelet/pods/9fccc9fc-2171-447d-aca4-b9d62054c8b7/volumes"
Sep 09 20:04:30.529605 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:04:30.529566    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/kserve-controller-manager-7676c798d9-b9cfb" event={"ID":"93002f61-5ef9-4f71-baf0-75011154f51b","Type":"ContainerStarted","Data":"a79c46f14580ad31884fa3710fa4fa584c14eea8748805a3b8a5c2938fa1cf3d"}
Sep 09 20:04:30.529816 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:04:30.529693    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve/kserve-controller-manager-7676c798d9-b9cfb"
Sep 09 20:04:30.557425 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:04:30.557372    2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/kserve-controller-manager-7676c798d9-b9cfb" podStartSLOduration=2.017100023 podStartE2EDuration="2.557356502s" totalImagesPullingTime="540.256479ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 20:04:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:04:30.557120656 +0000 UTC m=+554.603373917" watchObservedRunningTime="2026-09-09 20:04:30.557356502 +0000 UTC m=+554.603609763"
Sep 09 20:05:01.539166 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:05:01.539134    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve/kserve-controller-manager-7676c798d9-b9cfb"
Sep 09 20:06:52.019978 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:06:52.019944    2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-6946b55967-fjgdf"]
Sep 09 20:06:52.020458 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:06:52.020244    2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9fccc9fc-2171-447d-aca4-b9d62054c8b7" containerName="manager"
Sep 09 20:06:52.020458 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:06:52.020256    2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="9fccc9fc-2171-447d-aca4-b9d62054c8b7" containerName="manager"
Sep 09 20:06:52.020458 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:06:52.020317    2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="9fccc9fc-2171-447d-aca4-b9d62054c8b7" containerName="manager"
Sep 09 20:06:52.022285 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:06:52.022265    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-6946b55967-fjgdf"
Sep 09 20:06:52.024509 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:06:52.024479    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Sep 09 20:06:52.024509 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:06:52.024503    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-fh9bz\""
Sep 09 20:06:52.024707 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:06:52.024480    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Sep 09 20:06:52.024707 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:06:52.024576    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-rollback-kube-rbac-proxy-sar-config\""
Sep 09 20:06:52.025033 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:06:52.025017    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Sep 09 20:06:52.025119 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:06:52.025077    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-rollback-v2-predictor-serving-cert\""
Sep 09 20:06:52.025119 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:06:52.025082    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Sep 09 20:06:52.035108 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:06:52.035087    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-6946b55967-fjgdf"]
Sep 09 20:06:52.099024 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:06:52.098989    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l6n6b\" (UniqueName: \"kubernetes.io/projected/3d037c93-d716-49a7-8078-12c7a5ab5fd5-kube-api-access-l6n6b\") pod \"isvc-canary-rollback-v2-predictor-6946b55967-fjgdf\" (UID: \"3d037c93-d716-49a7-8078-12c7a5ab5fd5\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-6946b55967-fjgdf"
Sep 09 20:06:52.099024 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:06:52.099025    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/3d037c93-d716-49a7-8078-12c7a5ab5fd5-kube-rbac-proxy-sar-config\") pod \"isvc-canary-rollback-v2-predictor-6946b55967-fjgdf\" (UID: \"3d037c93-d716-49a7-8078-12c7a5ab5fd5\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-6946b55967-fjgdf"
Sep 09 20:06:52.099216 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:06:52.099052    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/3d037c93-d716-49a7-8078-12c7a5ab5fd5-kserve-provision-location\") pod \"isvc-canary-rollback-v2-predictor-6946b55967-fjgdf\" (UID: \"3d037c93-d716-49a7-8078-12c7a5ab5fd5\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-6946b55967-fjgdf"
Sep 09 20:06:52.099216 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:06:52.099112    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/3d037c93-d716-49a7-8078-12c7a5ab5fd5-proxy-tls\") pod \"isvc-canary-rollback-v2-predictor-6946b55967-fjgdf\" (UID: \"3d037c93-d716-49a7-8078-12c7a5ab5fd5\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-6946b55967-fjgdf"
Sep 09 20:06:52.099216 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:06:52.099162    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/3d037c93-d716-49a7-8078-12c7a5ab5fd5-cabundle-cert\") pod \"isvc-canary-rollback-v2-predictor-6946b55967-fjgdf\" (UID: \"3d037c93-d716-49a7-8078-12c7a5ab5fd5\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-6946b55967-fjgdf"
Sep 09 20:06:52.111698 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:06:52.111665    2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-predictor-585bd9bcc8-g4dsw"]
Sep 09 20:06:52.114060 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:06:52.114040    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-585bd9bcc8-g4dsw"
Sep 09 20:06:52.116005 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:06:52.115985    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-rollback-predictor-serving-cert\""
Sep 09 20:06:52.124146 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:06:52.124124    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-predictor-585bd9bcc8-g4dsw"]
Sep 09 20:06:52.200318 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:06:52.200283    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7db6k\" (UniqueName: \"kubernetes.io/projected/1a8f6bd9-81ac-4141-a518-35ae22a056f7-kube-api-access-7db6k\") pod \"isvc-canary-rollback-predictor-585bd9bcc8-g4dsw\" (UID: \"1a8f6bd9-81ac-4141-a518-35ae22a056f7\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-585bd9bcc8-g4dsw"
Sep 09 20:06:52.200318 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:06:52.200318    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/1a8f6bd9-81ac-4141-a518-35ae22a056f7-cabundle-cert\") pod \"isvc-canary-rollback-predictor-585bd9bcc8-g4dsw\" (UID: \"1a8f6bd9-81ac-4141-a518-35ae22a056f7\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-585bd9bcc8-g4dsw"
Sep 09 20:06:52.200512 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:06:52.200340    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l6n6b\" (UniqueName: \"kubernetes.io/projected/3d037c93-d716-49a7-8078-12c7a5ab5fd5-kube-api-access-l6n6b\") pod \"isvc-canary-rollback-v2-predictor-6946b55967-fjgdf\" (UID: \"3d037c93-d716-49a7-8078-12c7a5ab5fd5\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-6946b55967-fjgdf"
Sep 09 20:06:52.200512 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:06:52.200356    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/3d037c93-d716-49a7-8078-12c7a5ab5fd5-kube-rbac-proxy-sar-config\") pod \"isvc-canary-rollback-v2-predictor-6946b55967-fjgdf\" (UID: \"3d037c93-d716-49a7-8078-12c7a5ab5fd5\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-6946b55967-fjgdf"
Sep 09 20:06:52.200512 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:06:52.200372    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/3d037c93-d716-49a7-8078-12c7a5ab5fd5-kserve-provision-location\") pod \"isvc-canary-rollback-v2-predictor-6946b55967-fjgdf\" (UID: \"3d037c93-d716-49a7-8078-12c7a5ab5fd5\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-6946b55967-fjgdf"
Sep 09 20:06:52.200512 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:06:52.200401    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/3d037c93-d716-49a7-8078-12c7a5ab5fd5-proxy-tls\") pod \"isvc-canary-rollback-v2-predictor-6946b55967-fjgdf\" (UID: \"3d037c93-d716-49a7-8078-12c7a5ab5fd5\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-6946b55967-fjgdf"
Sep 09 20:06:52.200512 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:06:52.200416    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/1a8f6bd9-81ac-4141-a518-35ae22a056f7-kserve-provision-location\") pod \"isvc-canary-rollback-predictor-585bd9bcc8-g4dsw\" (UID: \"1a8f6bd9-81ac-4141-a518-35ae22a056f7\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-585bd9bcc8-g4dsw"
Sep 09 20:06:52.200512 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:06:52.200448    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/1a8f6bd9-81ac-4141-a518-35ae22a056f7-kube-rbac-proxy-sar-config\") pod \"isvc-canary-rollback-predictor-585bd9bcc8-g4dsw\" (UID: \"1a8f6bd9-81ac-4141-a518-35ae22a056f7\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-585bd9bcc8-g4dsw"
Sep 09 20:06:52.200745 ip-10-0-136-183 kubenswrapper[2574]: E0909 20:06:52.200535    2574 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-serving-cert: secret "isvc-canary-rollback-v2-predictor-serving-cert" not found
Sep 09 20:06:52.200745 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:06:52.200576    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/3d037c93-d716-49a7-8078-12c7a5ab5fd5-cabundle-cert\") pod \"isvc-canary-rollback-v2-predictor-6946b55967-fjgdf\" (UID: \"3d037c93-d716-49a7-8078-12c7a5ab5fd5\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-6946b55967-fjgdf"
Sep 09 20:06:52.200745 ip-10-0-136-183 kubenswrapper[2574]: E0909 20:06:52.200604    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3d037c93-d716-49a7-8078-12c7a5ab5fd5-proxy-tls podName:3d037c93-d716-49a7-8078-12c7a5ab5fd5 nodeName:}" failed. No retries permitted until 2026-09-09 20:06:52.700588094 +0000 UTC m=+696.746841337 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/3d037c93-d716-49a7-8078-12c7a5ab5fd5-proxy-tls") pod "isvc-canary-rollback-v2-predictor-6946b55967-fjgdf" (UID: "3d037c93-d716-49a7-8078-12c7a5ab5fd5") : secret "isvc-canary-rollback-v2-predictor-serving-cert" not found
Sep 09 20:06:52.200745 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:06:52.200669    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/1a8f6bd9-81ac-4141-a518-35ae22a056f7-proxy-tls\") pod \"isvc-canary-rollback-predictor-585bd9bcc8-g4dsw\" (UID: \"1a8f6bd9-81ac-4141-a518-35ae22a056f7\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-585bd9bcc8-g4dsw"
Sep 09 20:06:52.200929 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:06:52.200860    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/3d037c93-d716-49a7-8078-12c7a5ab5fd5-kserve-provision-location\") pod \"isvc-canary-rollback-v2-predictor-6946b55967-fjgdf\" (UID: \"3d037c93-d716-49a7-8078-12c7a5ab5fd5\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-6946b55967-fjgdf"
Sep 09 20:06:52.201158 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:06:52.201138    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/3d037c93-d716-49a7-8078-12c7a5ab5fd5-kube-rbac-proxy-sar-config\") pod \"isvc-canary-rollback-v2-predictor-6946b55967-fjgdf\" (UID: \"3d037c93-d716-49a7-8078-12c7a5ab5fd5\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-6946b55967-fjgdf"
Sep 09 20:06:52.201202 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:06:52.201179    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/3d037c93-d716-49a7-8078-12c7a5ab5fd5-cabundle-cert\") pod \"isvc-canary-rollback-v2-predictor-6946b55967-fjgdf\" (UID: \"3d037c93-d716-49a7-8078-12c7a5ab5fd5\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-6946b55967-fjgdf"
Sep 09 20:06:52.208532 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:06:52.208509    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-l6n6b\" (UniqueName: \"kubernetes.io/projected/3d037c93-d716-49a7-8078-12c7a5ab5fd5-kube-api-access-l6n6b\") pod \"isvc-canary-rollback-v2-predictor-6946b55967-fjgdf\" (UID: \"3d037c93-d716-49a7-8078-12c7a5ab5fd5\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-6946b55967-fjgdf"
Sep 09 20:06:52.301195 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:06:52.301075    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/1a8f6bd9-81ac-4141-a518-35ae22a056f7-proxy-tls\") pod \"isvc-canary-rollback-predictor-585bd9bcc8-g4dsw\" (UID: \"1a8f6bd9-81ac-4141-a518-35ae22a056f7\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-585bd9bcc8-g4dsw"
Sep 09 20:06:52.301195 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:06:52.301135    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7db6k\" (UniqueName: \"kubernetes.io/projected/1a8f6bd9-81ac-4141-a518-35ae22a056f7-kube-api-access-7db6k\") pod \"isvc-canary-rollback-predictor-585bd9bcc8-g4dsw\" (UID: \"1a8f6bd9-81ac-4141-a518-35ae22a056f7\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-585bd9bcc8-g4dsw"
Sep 09 20:06:52.301490 ip-10-0-136-183 kubenswrapper[2574]: E0909 20:06:52.301278    2574 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-canary-rollback-predictor-serving-cert: secret "isvc-canary-rollback-predictor-serving-cert" not found
Sep 09 20:06:52.301490 ip-10-0-136-183 kubenswrapper[2574]: E0909 20:06:52.301360    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1a8f6bd9-81ac-4141-a518-35ae22a056f7-proxy-tls podName:1a8f6bd9-81ac-4141-a518-35ae22a056f7 nodeName:}" failed. No retries permitted until 2026-09-09 20:06:52.801339772 +0000 UTC m=+696.847593040 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/1a8f6bd9-81ac-4141-a518-35ae22a056f7-proxy-tls") pod "isvc-canary-rollback-predictor-585bd9bcc8-g4dsw" (UID: "1a8f6bd9-81ac-4141-a518-35ae22a056f7") : secret "isvc-canary-rollback-predictor-serving-cert" not found
Sep 09 20:06:52.301490 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:06:52.301391    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/1a8f6bd9-81ac-4141-a518-35ae22a056f7-cabundle-cert\") pod \"isvc-canary-rollback-predictor-585bd9bcc8-g4dsw\" (UID: \"1a8f6bd9-81ac-4141-a518-35ae22a056f7\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-585bd9bcc8-g4dsw"
Sep 09 20:06:52.301490 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:06:52.301455    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/1a8f6bd9-81ac-4141-a518-35ae22a056f7-kserve-provision-location\") pod \"isvc-canary-rollback-predictor-585bd9bcc8-g4dsw\" (UID: \"1a8f6bd9-81ac-4141-a518-35ae22a056f7\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-585bd9bcc8-g4dsw"
Sep 09 20:06:52.301737 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:06:52.301496    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/1a8f6bd9-81ac-4141-a518-35ae22a056f7-kube-rbac-proxy-sar-config\") pod \"isvc-canary-rollback-predictor-585bd9bcc8-g4dsw\" (UID: \"1a8f6bd9-81ac-4141-a518-35ae22a056f7\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-585bd9bcc8-g4dsw"
Sep 09 20:06:52.302163 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:06:52.302138    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/1a8f6bd9-81ac-4141-a518-35ae22a056f7-kube-rbac-proxy-sar-config\") pod \"isvc-canary-rollback-predictor-585bd9bcc8-g4dsw\" (UID: \"1a8f6bd9-81ac-4141-a518-35ae22a056f7\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-585bd9bcc8-g4dsw"
Sep 09 20:06:52.302163 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:06:52.302161    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/1a8f6bd9-81ac-4141-a518-35ae22a056f7-cabundle-cert\") pod \"isvc-canary-rollback-predictor-585bd9bcc8-g4dsw\" (UID: \"1a8f6bd9-81ac-4141-a518-35ae22a056f7\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-585bd9bcc8-g4dsw"
Sep 09 20:06:52.302298 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:06:52.302166    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/1a8f6bd9-81ac-4141-a518-35ae22a056f7-kserve-provision-location\") pod \"isvc-canary-rollback-predictor-585bd9bcc8-g4dsw\" (UID: \"1a8f6bd9-81ac-4141-a518-35ae22a056f7\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-585bd9bcc8-g4dsw"
Sep 09 20:06:52.309412 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:06:52.309388    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7db6k\" (UniqueName: \"kubernetes.io/projected/1a8f6bd9-81ac-4141-a518-35ae22a056f7-kube-api-access-7db6k\") pod \"isvc-canary-rollback-predictor-585bd9bcc8-g4dsw\" (UID: \"1a8f6bd9-81ac-4141-a518-35ae22a056f7\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-585bd9bcc8-g4dsw"
Sep 09 20:06:52.705101 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:06:52.705039    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/3d037c93-d716-49a7-8078-12c7a5ab5fd5-proxy-tls\") pod \"isvc-canary-rollback-v2-predictor-6946b55967-fjgdf\" (UID: \"3d037c93-d716-49a7-8078-12c7a5ab5fd5\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-6946b55967-fjgdf"
Sep 09 20:06:52.707639 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:06:52.707600    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/3d037c93-d716-49a7-8078-12c7a5ab5fd5-proxy-tls\") pod \"isvc-canary-rollback-v2-predictor-6946b55967-fjgdf\" (UID: \"3d037c93-d716-49a7-8078-12c7a5ab5fd5\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-6946b55967-fjgdf"
Sep 09 20:06:52.805616 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:06:52.805579    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/1a8f6bd9-81ac-4141-a518-35ae22a056f7-proxy-tls\") pod \"isvc-canary-rollback-predictor-585bd9bcc8-g4dsw\" (UID: \"1a8f6bd9-81ac-4141-a518-35ae22a056f7\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-585bd9bcc8-g4dsw"
Sep 09 20:06:52.808043 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:06:52.808010    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/1a8f6bd9-81ac-4141-a518-35ae22a056f7-proxy-tls\") pod \"isvc-canary-rollback-predictor-585bd9bcc8-g4dsw\" (UID: \"1a8f6bd9-81ac-4141-a518-35ae22a056f7\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-585bd9bcc8-g4dsw"
Sep 09 20:06:52.933861 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:06:52.933816    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-6946b55967-fjgdf"
Sep 09 20:06:53.024784 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:06:53.024756    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-585bd9bcc8-g4dsw"
Sep 09 20:06:53.217775 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:06:53.217732    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-6946b55967-fjgdf"]
Sep 09 20:06:53.220187 ip-10-0-136-183 kubenswrapper[2574]: W0909 20:06:53.220154    2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3d037c93_d716_49a7_8078_12c7a5ab5fd5.slice/crio-24e793082d998cf099cea5c0608083a52b3a115937550a96ad137de720444c39 WatchSource:0}: Error finding container 24e793082d998cf099cea5c0608083a52b3a115937550a96ad137de720444c39: Status 404 returned error can't find the container with id 24e793082d998cf099cea5c0608083a52b3a115937550a96ad137de720444c39
Sep 09 20:06:53.222205 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:06:53.222184    2574 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 09 20:06:53.298729 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:06:53.298658    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-predictor-585bd9bcc8-g4dsw"]
Sep 09 20:06:53.301488 ip-10-0-136-183 kubenswrapper[2574]: W0909 20:06:53.301462    2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1a8f6bd9_81ac_4141_a518_35ae22a056f7.slice/crio-63745e6f76c05740d90d7ccc21a9fa90df47b21a48c184e67fc454486cc6fe77 WatchSource:0}: Error finding container 63745e6f76c05740d90d7ccc21a9fa90df47b21a48c184e67fc454486cc6fe77: Status 404 returned error can't find the container with id 63745e6f76c05740d90d7ccc21a9fa90df47b21a48c184e67fc454486cc6fe77
Sep 09 20:06:54.057749 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:06:54.057704    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-6946b55967-fjgdf" event={"ID":"3d037c93-d716-49a7-8078-12c7a5ab5fd5","Type":"ContainerStarted","Data":"24e793082d998cf099cea5c0608083a52b3a115937550a96ad137de720444c39"}
Sep 09 20:06:54.059370 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:06:54.059331    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-585bd9bcc8-g4dsw" event={"ID":"1a8f6bd9-81ac-4141-a518-35ae22a056f7","Type":"ContainerStarted","Data":"63745e6f76c05740d90d7ccc21a9fa90df47b21a48c184e67fc454486cc6fe77"}
Sep 09 20:06:57.073967 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:06:57.073923    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-6946b55967-fjgdf" event={"ID":"3d037c93-d716-49a7-8078-12c7a5ab5fd5","Type":"ContainerStarted","Data":"a41ca4c29f590b5ce62f919a4a16f7c67c0ec167c263a8c5f3da1ba55cbf3365"}
Sep 09 20:06:57.075661 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:06:57.075631    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-585bd9bcc8-g4dsw" event={"ID":"1a8f6bd9-81ac-4141-a518-35ae22a056f7","Type":"ContainerStarted","Data":"2ee6d0a52f75944e9bf90b6a9cfe38bbcca25aa2ed0080f41e7f17d82765376e"}
Sep 09 20:07:01.092881 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:07:01.092844    2574 generic.go:356] "Generic (PLEG): container finished" podID="1a8f6bd9-81ac-4141-a518-35ae22a056f7" containerID="2ee6d0a52f75944e9bf90b6a9cfe38bbcca25aa2ed0080f41e7f17d82765376e" exitCode=0
Sep 09 20:07:01.093314 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:07:01.092916    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-585bd9bcc8-g4dsw" event={"ID":"1a8f6bd9-81ac-4141-a518-35ae22a056f7","Type":"ContainerDied","Data":"2ee6d0a52f75944e9bf90b6a9cfe38bbcca25aa2ed0080f41e7f17d82765376e"}
Sep 09 20:07:01.094647 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:07:01.094548    2574 generic.go:356] "Generic (PLEG): container finished" podID="3d037c93-d716-49a7-8078-12c7a5ab5fd5" containerID="a41ca4c29f590b5ce62f919a4a16f7c67c0ec167c263a8c5f3da1ba55cbf3365" exitCode=0
Sep 09 20:07:01.094647 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:07:01.094596    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-6946b55967-fjgdf" event={"ID":"3d037c93-d716-49a7-8078-12c7a5ab5fd5","Type":"ContainerDied","Data":"a41ca4c29f590b5ce62f919a4a16f7c67c0ec167c263a8c5f3da1ba55cbf3365"}
Sep 09 20:07:18.196137 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:07:18.196096    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-585bd9bcc8-g4dsw" event={"ID":"1a8f6bd9-81ac-4141-a518-35ae22a056f7","Type":"ContainerStarted","Data":"0e174274d5201c25a761878402271b52fd3b93f8c64802693df14919cbf7dad9"}
Sep 09 20:07:18.201089 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:07:18.201057    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-6946b55967-fjgdf" event={"ID":"3d037c93-d716-49a7-8078-12c7a5ab5fd5","Type":"ContainerStarted","Data":"217667f539c731b6e7cbb415b2bfa9f9f0f5c5cffac2642cfd84c48e39c0f37a"}
Sep 09 20:07:20.210945 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:07:20.210833    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-585bd9bcc8-g4dsw" event={"ID":"1a8f6bd9-81ac-4141-a518-35ae22a056f7","Type":"ContainerStarted","Data":"8aec52d943725b564e0e2b562c8600abfc0849aa1a19d7ff9dc8da3c4725e4eb"}
Sep 09 20:07:20.210945 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:07:20.210912    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-585bd9bcc8-g4dsw"
Sep 09 20:07:20.211440 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:07:20.210952    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-585bd9bcc8-g4dsw"
Sep 09 20:07:20.212655 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:07:20.212616    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-585bd9bcc8-g4dsw" podUID="1a8f6bd9-81ac-4141-a518-35ae22a056f7" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.24:8080: connect: connection refused"
Sep 09 20:07:20.213093 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:07:20.213074    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-6946b55967-fjgdf" event={"ID":"3d037c93-d716-49a7-8078-12c7a5ab5fd5","Type":"ContainerStarted","Data":"9b17e0fc88f282837b7edfde6af2c7d7444f5d382b6c572ef0e5254e54942aa6"}
Sep 09 20:07:20.213240 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:07:20.213228    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-6946b55967-fjgdf"
Sep 09 20:07:20.233401 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:07:20.233354    2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-585bd9bcc8-g4dsw" podStartSLOduration=2.726546051 podStartE2EDuration="28.233340572s" totalImagesPullingTime="21.966599112s" totalInitContainerRuntime="3.540195409s" isStatefulPod=false podCreationTimestamp="2026-09-09 20:06:52 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:07:20.231861665 +0000 UTC m=+724.278114927" watchObservedRunningTime="2026-09-09 20:07:20.233340572 +0000 UTC m=+724.279593833"
Sep 09 20:07:20.251930 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:07:20.251867    2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-6946b55967-fjgdf" podStartSLOduration=2.400842475 podStartE2EDuration="28.251849333s" totalImagesPullingTime="22.034707087s" totalInitContainerRuntime="3.816299771s" isStatefulPod=false podCreationTimestamp="2026-09-09 20:06:52 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:07:20.250492785 +0000 UTC m=+724.296746073" watchObservedRunningTime="2026-09-09 20:07:20.251849333 +0000 UTC m=+724.298102637"
Sep 09 20:07:21.218039 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:07:21.217995    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-585bd9bcc8-g4dsw" podUID="1a8f6bd9-81ac-4141-a518-35ae22a056f7" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.24:8080: connect: connection refused"
Sep 09 20:07:21.218447 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:07:21.218250    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-6946b55967-fjgdf"
Sep 09 20:07:21.219416 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:07:21.219386    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-6946b55967-fjgdf" podUID="3d037c93-d716-49a7-8078-12c7a5ab5fd5" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.23:8080: connect: connection refused"
Sep 09 20:07:22.221395 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:07:22.221358    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-6946b55967-fjgdf" podUID="3d037c93-d716-49a7-8078-12c7a5ab5fd5" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.23:8080: connect: connection refused"
Sep 09 20:07:26.226279 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:07:26.226250    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-585bd9bcc8-g4dsw"
Sep 09 20:07:26.226772 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:07:26.226739    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-585bd9bcc8-g4dsw" podUID="1a8f6bd9-81ac-4141-a518-35ae22a056f7" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.24:8080: connect: connection refused"
Sep 09 20:07:27.226084 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:07:27.226055    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-6946b55967-fjgdf"
Sep 09 20:07:27.226700 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:07:27.226667    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-6946b55967-fjgdf" podUID="3d037c93-d716-49a7-8078-12c7a5ab5fd5" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.23:8080: connect: connection refused"
Sep 09 20:07:36.227592 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:07:36.227552    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-585bd9bcc8-g4dsw" podUID="1a8f6bd9-81ac-4141-a518-35ae22a056f7" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.24:8080: connect: connection refused"
Sep 09 20:07:37.226788 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:07:37.226746    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-6946b55967-fjgdf" podUID="3d037c93-d716-49a7-8078-12c7a5ab5fd5" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.23:8080: connect: connection refused"
Sep 09 20:07:46.227015 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:07:46.226975    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-585bd9bcc8-g4dsw" podUID="1a8f6bd9-81ac-4141-a518-35ae22a056f7" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.24:8080: connect: connection refused"
Sep 09 20:07:47.227085 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:07:47.227048    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-6946b55967-fjgdf" podUID="3d037c93-d716-49a7-8078-12c7a5ab5fd5" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.23:8080: connect: connection refused"
Sep 09 20:07:56.227247 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:07:56.227206    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-585bd9bcc8-g4dsw" podUID="1a8f6bd9-81ac-4141-a518-35ae22a056f7" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.24:8080: connect: connection refused"
Sep 09 20:07:57.226580 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:07:57.226541    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-6946b55967-fjgdf" podUID="3d037c93-d716-49a7-8078-12c7a5ab5fd5" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.23:8080: connect: connection refused"
Sep 09 20:08:06.227538 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:08:06.227498    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-585bd9bcc8-g4dsw" podUID="1a8f6bd9-81ac-4141-a518-35ae22a056f7" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.24:8080: connect: connection refused"
Sep 09 20:08:07.227067 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:08:07.227028    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-6946b55967-fjgdf" podUID="3d037c93-d716-49a7-8078-12c7a5ab5fd5" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.23:8080: connect: connection refused"
Sep 09 20:08:16.226834 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:08:16.226776    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-585bd9bcc8-g4dsw" podUID="1a8f6bd9-81ac-4141-a518-35ae22a056f7" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.24:8080: connect: connection refused"
Sep 09 20:08:17.227504 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:08:17.227465    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-6946b55967-fjgdf" podUID="3d037c93-d716-49a7-8078-12c7a5ab5fd5" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.23:8080: connect: connection refused"
Sep 09 20:08:26.227468 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:08:26.227428    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-585bd9bcc8-g4dsw" podUID="1a8f6bd9-81ac-4141-a518-35ae22a056f7" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.24:8080: connect: connection refused"
Sep 09 20:08:27.227352 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:08:27.227315    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-6946b55967-fjgdf" podUID="3d037c93-d716-49a7-8078-12c7a5ab5fd5" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.23:8080: connect: connection refused"
Sep 09 20:08:36.228012 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:08:36.227976    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-585bd9bcc8-g4dsw"
Sep 09 20:08:37.227466 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:08:37.227430    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-6946b55967-fjgdf"
Sep 09 20:08:37.353098 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:08:37.353051    2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-predictor-585bd9bcc8-g4dsw"]
Sep 09 20:08:37.353571 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:08:37.353447    2574 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-585bd9bcc8-g4dsw" podUID="1a8f6bd9-81ac-4141-a518-35ae22a056f7" containerName="kserve-container" containerID="cri-o://0e174274d5201c25a761878402271b52fd3b93f8c64802693df14919cbf7dad9" gracePeriod=30
Sep 09 20:08:37.353826 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:08:37.353778    2574 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-585bd9bcc8-g4dsw" podUID="1a8f6bd9-81ac-4141-a518-35ae22a056f7" containerName="kube-rbac-proxy" containerID="cri-o://8aec52d943725b564e0e2b562c8600abfc0849aa1a19d7ff9dc8da3c4725e4eb" gracePeriod=30
Sep 09 20:08:37.521332 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:08:37.521253    2574 generic.go:356] "Generic (PLEG): container finished" podID="1a8f6bd9-81ac-4141-a518-35ae22a056f7" containerID="8aec52d943725b564e0e2b562c8600abfc0849aa1a19d7ff9dc8da3c4725e4eb" exitCode=2
Sep 09 20:08:37.521332 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:08:37.521292    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-585bd9bcc8-g4dsw" event={"ID":"1a8f6bd9-81ac-4141-a518-35ae22a056f7","Type":"ContainerDied","Data":"8aec52d943725b564e0e2b562c8600abfc0849aa1a19d7ff9dc8da3c4725e4eb"}
Sep 09 20:08:41.219255 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:08:41.219167    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-585bd9bcc8-g4dsw" podUID="1a8f6bd9-81ac-4141-a518-35ae22a056f7" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.24:8643/healthz\": dial tcp 10.133.0.24:8643: connect: connection refused"
Sep 09 20:08:41.706654 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:08:41.706618    2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-6946b55967-fjgdf"]
Sep 09 20:08:41.707004 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:08:41.706964    2574 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-6946b55967-fjgdf" podUID="3d037c93-d716-49a7-8078-12c7a5ab5fd5" containerName="kube-rbac-proxy" containerID="cri-o://9b17e0fc88f282837b7edfde6af2c7d7444f5d382b6c572ef0e5254e54942aa6" gracePeriod=30
Sep 09 20:08:41.707139 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:08:41.706947    2574 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-6946b55967-fjgdf" podUID="3d037c93-d716-49a7-8078-12c7a5ab5fd5" containerName="kserve-container" containerID="cri-o://217667f539c731b6e7cbb415b2bfa9f9f0f5c5cffac2642cfd84c48e39c0f37a" gracePeriod=30
Sep 09 20:08:42.103208 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:08:42.103181    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-585bd9bcc8-g4dsw"
Sep 09 20:08:42.194431 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:08:42.194398    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/1a8f6bd9-81ac-4141-a518-35ae22a056f7-kube-rbac-proxy-sar-config\") pod \"1a8f6bd9-81ac-4141-a518-35ae22a056f7\" (UID: \"1a8f6bd9-81ac-4141-a518-35ae22a056f7\") "
Sep 09 20:08:42.194431 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:08:42.194435    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7db6k\" (UniqueName: \"kubernetes.io/projected/1a8f6bd9-81ac-4141-a518-35ae22a056f7-kube-api-access-7db6k\") pod \"1a8f6bd9-81ac-4141-a518-35ae22a056f7\" (UID: \"1a8f6bd9-81ac-4141-a518-35ae22a056f7\") "
Sep 09 20:08:42.194764 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:08:42.194492    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/1a8f6bd9-81ac-4141-a518-35ae22a056f7-cabundle-cert\") pod \"1a8f6bd9-81ac-4141-a518-35ae22a056f7\" (UID: \"1a8f6bd9-81ac-4141-a518-35ae22a056f7\") "
Sep 09 20:08:42.194764 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:08:42.194516    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/1a8f6bd9-81ac-4141-a518-35ae22a056f7-proxy-tls\") pod \"1a8f6bd9-81ac-4141-a518-35ae22a056f7\" (UID: \"1a8f6bd9-81ac-4141-a518-35ae22a056f7\") "
Sep 09 20:08:42.194764 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:08:42.194555    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/1a8f6bd9-81ac-4141-a518-35ae22a056f7-kserve-provision-location\") pod \"1a8f6bd9-81ac-4141-a518-35ae22a056f7\" (UID: \"1a8f6bd9-81ac-4141-a518-35ae22a056f7\") "
Sep 09 20:08:42.194960 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:08:42.194910    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1a8f6bd9-81ac-4141-a518-35ae22a056f7-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "1a8f6bd9-81ac-4141-a518-35ae22a056f7" (UID: "1a8f6bd9-81ac-4141-a518-35ae22a056f7"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 20:08:42.194960 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:08:42.194912    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1a8f6bd9-81ac-4141-a518-35ae22a056f7-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "1a8f6bd9-81ac-4141-a518-35ae22a056f7" (UID: "1a8f6bd9-81ac-4141-a518-35ae22a056f7"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 20:08:42.194960 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:08:42.194932    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1a8f6bd9-81ac-4141-a518-35ae22a056f7-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "1a8f6bd9-81ac-4141-a518-35ae22a056f7" (UID: "1a8f6bd9-81ac-4141-a518-35ae22a056f7"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 20:08:42.196853 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:08:42.196828    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1a8f6bd9-81ac-4141-a518-35ae22a056f7-kube-api-access-7db6k" (OuterVolumeSpecName: "kube-api-access-7db6k") pod "1a8f6bd9-81ac-4141-a518-35ae22a056f7" (UID: "1a8f6bd9-81ac-4141-a518-35ae22a056f7"). InnerVolumeSpecName "kube-api-access-7db6k". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 20:08:42.197035 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:08:42.197017    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1a8f6bd9-81ac-4141-a518-35ae22a056f7-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "1a8f6bd9-81ac-4141-a518-35ae22a056f7" (UID: "1a8f6bd9-81ac-4141-a518-35ae22a056f7"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 20:08:42.222660 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:08:42.222590    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-6946b55967-fjgdf" podUID="3d037c93-d716-49a7-8078-12c7a5ab5fd5" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.23:8643/healthz\": dial tcp 10.133.0.23:8643: connect: connection refused"
Sep 09 20:08:42.295138 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:08:42.295097    2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7db6k\" (UniqueName: \"kubernetes.io/projected/1a8f6bd9-81ac-4141-a518-35ae22a056f7-kube-api-access-7db6k\") on node \"ip-10-0-136-183.ec2.internal\" DevicePath \"\""
Sep 09 20:08:42.295138 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:08:42.295126    2574 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/1a8f6bd9-81ac-4141-a518-35ae22a056f7-cabundle-cert\") on node \"ip-10-0-136-183.ec2.internal\" DevicePath \"\""
Sep 09 20:08:42.295138 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:08:42.295136    2574 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/1a8f6bd9-81ac-4141-a518-35ae22a056f7-proxy-tls\") on node \"ip-10-0-136-183.ec2.internal\" DevicePath \"\""
Sep 09 20:08:42.295138 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:08:42.295144    2574 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/1a8f6bd9-81ac-4141-a518-35ae22a056f7-kserve-provision-location\") on node \"ip-10-0-136-183.ec2.internal\" DevicePath \"\""
Sep 09 20:08:42.295138 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:08:42.295153    2574 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/1a8f6bd9-81ac-4141-a518-35ae22a056f7-kube-rbac-proxy-sar-config\") on node \"ip-10-0-136-183.ec2.internal\" DevicePath \"\""
Sep 09 20:08:42.541309 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:08:42.541274    2574 generic.go:356] "Generic (PLEG): container finished" podID="3d037c93-d716-49a7-8078-12c7a5ab5fd5" containerID="9b17e0fc88f282837b7edfde6af2c7d7444f5d382b6c572ef0e5254e54942aa6" exitCode=2
Sep 09 20:08:42.541461 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:08:42.541342    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-6946b55967-fjgdf" event={"ID":"3d037c93-d716-49a7-8078-12c7a5ab5fd5","Type":"ContainerDied","Data":"9b17e0fc88f282837b7edfde6af2c7d7444f5d382b6c572ef0e5254e54942aa6"}
Sep 09 20:08:42.543161 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:08:42.543137    2574 generic.go:356] "Generic (PLEG): container finished" podID="1a8f6bd9-81ac-4141-a518-35ae22a056f7" containerID="0e174274d5201c25a761878402271b52fd3b93f8c64802693df14919cbf7dad9" exitCode=0
Sep 09 20:08:42.543280 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:08:42.543163    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-585bd9bcc8-g4dsw" event={"ID":"1a8f6bd9-81ac-4141-a518-35ae22a056f7","Type":"ContainerDied","Data":"0e174274d5201c25a761878402271b52fd3b93f8c64802693df14919cbf7dad9"}
Sep 09 20:08:42.543280 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:08:42.543194    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-585bd9bcc8-g4dsw"
Sep 09 20:08:42.543280 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:08:42.543211    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-585bd9bcc8-g4dsw" event={"ID":"1a8f6bd9-81ac-4141-a518-35ae22a056f7","Type":"ContainerDied","Data":"63745e6f76c05740d90d7ccc21a9fa90df47b21a48c184e67fc454486cc6fe77"}
Sep 09 20:08:42.543280 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:08:42.543237    2574 scope.go:117] "RemoveContainer" containerID="8aec52d943725b564e0e2b562c8600abfc0849aa1a19d7ff9dc8da3c4725e4eb"
Sep 09 20:08:42.555280 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:08:42.555254    2574 scope.go:117] "RemoveContainer" containerID="0e174274d5201c25a761878402271b52fd3b93f8c64802693df14919cbf7dad9"
Sep 09 20:08:42.562440 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:08:42.562415    2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-predictor-585bd9bcc8-g4dsw"]
Sep 09 20:08:42.563527 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:08:42.563512    2574 scope.go:117] "RemoveContainer" containerID="2ee6d0a52f75944e9bf90b6a9cfe38bbcca25aa2ed0080f41e7f17d82765376e"
Sep 09 20:08:42.565577 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:08:42.565559    2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-predictor-585bd9bcc8-g4dsw"]
Sep 09 20:08:42.570992 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:08:42.570892    2574 scope.go:117] "RemoveContainer" containerID="8aec52d943725b564e0e2b562c8600abfc0849aa1a19d7ff9dc8da3c4725e4eb"
Sep 09 20:08:42.571196 ip-10-0-136-183 kubenswrapper[2574]: E0909 20:08:42.571177    2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8aec52d943725b564e0e2b562c8600abfc0849aa1a19d7ff9dc8da3c4725e4eb\": container with ID starting with 8aec52d943725b564e0e2b562c8600abfc0849aa1a19d7ff9dc8da3c4725e4eb not found: ID does not exist" containerID="8aec52d943725b564e0e2b562c8600abfc0849aa1a19d7ff9dc8da3c4725e4eb"
Sep 09 20:08:42.571307 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:08:42.571204    2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8aec52d943725b564e0e2b562c8600abfc0849aa1a19d7ff9dc8da3c4725e4eb"} err="failed to get container status \"8aec52d943725b564e0e2b562c8600abfc0849aa1a19d7ff9dc8da3c4725e4eb\": rpc error: code = NotFound desc = could not find container \"8aec52d943725b564e0e2b562c8600abfc0849aa1a19d7ff9dc8da3c4725e4eb\": container with ID starting with 8aec52d943725b564e0e2b562c8600abfc0849aa1a19d7ff9dc8da3c4725e4eb not found: ID does not exist"
Sep 09 20:08:42.571307 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:08:42.571221    2574 scope.go:117] "RemoveContainer" containerID="0e174274d5201c25a761878402271b52fd3b93f8c64802693df14919cbf7dad9"
Sep 09 20:08:42.571456 ip-10-0-136-183 kubenswrapper[2574]: E0909 20:08:42.571439    2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0e174274d5201c25a761878402271b52fd3b93f8c64802693df14919cbf7dad9\": container with ID starting with 0e174274d5201c25a761878402271b52fd3b93f8c64802693df14919cbf7dad9 not found: ID does not exist" containerID="0e174274d5201c25a761878402271b52fd3b93f8c64802693df14919cbf7dad9"
Sep 09 20:08:42.571495 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:08:42.571460    2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0e174274d5201c25a761878402271b52fd3b93f8c64802693df14919cbf7dad9"} err="failed to get container status \"0e174274d5201c25a761878402271b52fd3b93f8c64802693df14919cbf7dad9\": rpc error: code = NotFound desc = could not find container \"0e174274d5201c25a761878402271b52fd3b93f8c64802693df14919cbf7dad9\": container with ID starting with 0e174274d5201c25a761878402271b52fd3b93f8c64802693df14919cbf7dad9 not found: ID does not exist"
Sep 09 20:08:42.571495 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:08:42.571473    2574 scope.go:117] "RemoveContainer" containerID="2ee6d0a52f75944e9bf90b6a9cfe38bbcca25aa2ed0080f41e7f17d82765376e"
Sep 09 20:08:42.571716 ip-10-0-136-183 kubenswrapper[2574]: E0909 20:08:42.571698    2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2ee6d0a52f75944e9bf90b6a9cfe38bbcca25aa2ed0080f41e7f17d82765376e\": container with ID starting with 2ee6d0a52f75944e9bf90b6a9cfe38bbcca25aa2ed0080f41e7f17d82765376e not found: ID does not exist" containerID="2ee6d0a52f75944e9bf90b6a9cfe38bbcca25aa2ed0080f41e7f17d82765376e"
Sep 09 20:08:42.571768 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:08:42.571724    2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2ee6d0a52f75944e9bf90b6a9cfe38bbcca25aa2ed0080f41e7f17d82765376e"} err="failed to get container status \"2ee6d0a52f75944e9bf90b6a9cfe38bbcca25aa2ed0080f41e7f17d82765376e\": rpc error: code = NotFound desc = could not find container \"2ee6d0a52f75944e9bf90b6a9cfe38bbcca25aa2ed0080f41e7f17d82765376e\": container with ID starting with 2ee6d0a52f75944e9bf90b6a9cfe38bbcca25aa2ed0080f41e7f17d82765376e not found: ID does not exist"
Sep 09 20:08:44.490844 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:08:44.490781    2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1a8f6bd9-81ac-4141-a518-35ae22a056f7" path="/var/lib/kubelet/pods/1a8f6bd9-81ac-4141-a518-35ae22a056f7/volumes"
Sep 09 20:08:46.556395 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:08:46.556364    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-6946b55967-fjgdf"
Sep 09 20:08:46.562931 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:08:46.562901    2574 generic.go:356] "Generic (PLEG): container finished" podID="3d037c93-d716-49a7-8078-12c7a5ab5fd5" containerID="217667f539c731b6e7cbb415b2bfa9f9f0f5c5cffac2642cfd84c48e39c0f37a" exitCode=0
Sep 09 20:08:46.563066 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:08:46.562981    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-6946b55967-fjgdf"
Sep 09 20:08:46.563066 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:08:46.562979    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-6946b55967-fjgdf" event={"ID":"3d037c93-d716-49a7-8078-12c7a5ab5fd5","Type":"ContainerDied","Data":"217667f539c731b6e7cbb415b2bfa9f9f0f5c5cffac2642cfd84c48e39c0f37a"}
Sep 09 20:08:46.563178 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:08:46.563065    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-6946b55967-fjgdf" event={"ID":"3d037c93-d716-49a7-8078-12c7a5ab5fd5","Type":"ContainerDied","Data":"24e793082d998cf099cea5c0608083a52b3a115937550a96ad137de720444c39"}
Sep 09 20:08:46.563178 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:08:46.563084    2574 scope.go:117] "RemoveContainer" containerID="9b17e0fc88f282837b7edfde6af2c7d7444f5d382b6c572ef0e5254e54942aa6"
Sep 09 20:08:46.570880 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:08:46.570863    2574 scope.go:117] "RemoveContainer" containerID="217667f539c731b6e7cbb415b2bfa9f9f0f5c5cffac2642cfd84c48e39c0f37a"
Sep 09 20:08:46.578626 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:08:46.578604    2574 scope.go:117] "RemoveContainer" containerID="a41ca4c29f590b5ce62f919a4a16f7c67c0ec167c263a8c5f3da1ba55cbf3365"
Sep 09 20:08:46.587438 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:08:46.587419    2574 scope.go:117] "RemoveContainer" containerID="9b17e0fc88f282837b7edfde6af2c7d7444f5d382b6c572ef0e5254e54942aa6"
Sep 09 20:08:46.587723 ip-10-0-136-183 kubenswrapper[2574]: E0909 20:08:46.587702    2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9b17e0fc88f282837b7edfde6af2c7d7444f5d382b6c572ef0e5254e54942aa6\": container with ID starting with 9b17e0fc88f282837b7edfde6af2c7d7444f5d382b6c572ef0e5254e54942aa6 not found: ID does not exist" containerID="9b17e0fc88f282837b7edfde6af2c7d7444f5d382b6c572ef0e5254e54942aa6"
Sep 09 20:08:46.587819 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:08:46.587739    2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9b17e0fc88f282837b7edfde6af2c7d7444f5d382b6c572ef0e5254e54942aa6"} err="failed to get container status \"9b17e0fc88f282837b7edfde6af2c7d7444f5d382b6c572ef0e5254e54942aa6\": rpc error: code = NotFound desc = could not find container \"9b17e0fc88f282837b7edfde6af2c7d7444f5d382b6c572ef0e5254e54942aa6\": container with ID starting with 9b17e0fc88f282837b7edfde6af2c7d7444f5d382b6c572ef0e5254e54942aa6 not found: ID does not exist"
Sep 09 20:08:46.587819 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:08:46.587764    2574 scope.go:117] "RemoveContainer" containerID="217667f539c731b6e7cbb415b2bfa9f9f0f5c5cffac2642cfd84c48e39c0f37a"
Sep 09 20:08:46.588041 ip-10-0-136-183 kubenswrapper[2574]: E0909 20:08:46.588022    2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"217667f539c731b6e7cbb415b2bfa9f9f0f5c5cffac2642cfd84c48e39c0f37a\": container with ID starting with 217667f539c731b6e7cbb415b2bfa9f9f0f5c5cffac2642cfd84c48e39c0f37a not found: ID does not exist" containerID="217667f539c731b6e7cbb415b2bfa9f9f0f5c5cffac2642cfd84c48e39c0f37a"
Sep 09 20:08:46.588081 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:08:46.588048    2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"217667f539c731b6e7cbb415b2bfa9f9f0f5c5cffac2642cfd84c48e39c0f37a"} err="failed to get container status \"217667f539c731b6e7cbb415b2bfa9f9f0f5c5cffac2642cfd84c48e39c0f37a\": rpc error: code = NotFound desc = could not find container \"217667f539c731b6e7cbb415b2bfa9f9f0f5c5cffac2642cfd84c48e39c0f37a\": container with ID starting with 217667f539c731b6e7cbb415b2bfa9f9f0f5c5cffac2642cfd84c48e39c0f37a not found: ID does not exist"
Sep 09 20:08:46.588081 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:08:46.588066    2574 scope.go:117] "RemoveContainer" containerID="a41ca4c29f590b5ce62f919a4a16f7c67c0ec167c263a8c5f3da1ba55cbf3365"
Sep 09 20:08:46.588301 ip-10-0-136-183 kubenswrapper[2574]: E0909 20:08:46.588283    2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a41ca4c29f590b5ce62f919a4a16f7c67c0ec167c263a8c5f3da1ba55cbf3365\": container with ID starting with a41ca4c29f590b5ce62f919a4a16f7c67c0ec167c263a8c5f3da1ba55cbf3365 not found: ID does not exist" containerID="a41ca4c29f590b5ce62f919a4a16f7c67c0ec167c263a8c5f3da1ba55cbf3365"
Sep 09 20:08:46.588359 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:08:46.588306    2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a41ca4c29f590b5ce62f919a4a16f7c67c0ec167c263a8c5f3da1ba55cbf3365"} err="failed to get container status \"a41ca4c29f590b5ce62f919a4a16f7c67c0ec167c263a8c5f3da1ba55cbf3365\": rpc error: code = NotFound desc = could not find container \"a41ca4c29f590b5ce62f919a4a16f7c67c0ec167c263a8c5f3da1ba55cbf3365\": container with ID starting with a41ca4c29f590b5ce62f919a4a16f7c67c0ec167c263a8c5f3da1ba55cbf3365 not found: ID does not exist"
Sep 09 20:08:46.731329 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:08:46.731247    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/3d037c93-d716-49a7-8078-12c7a5ab5fd5-kserve-provision-location\") pod \"3d037c93-d716-49a7-8078-12c7a5ab5fd5\" (UID: \"3d037c93-d716-49a7-8078-12c7a5ab5fd5\") "
Sep 09 20:08:46.731329 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:08:46.731298    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-l6n6b\" (UniqueName: \"kubernetes.io/projected/3d037c93-d716-49a7-8078-12c7a5ab5fd5-kube-api-access-l6n6b\") pod \"3d037c93-d716-49a7-8078-12c7a5ab5fd5\" (UID: \"3d037c93-d716-49a7-8078-12c7a5ab5fd5\") "
Sep 09 20:08:46.731329 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:08:46.731328    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/3d037c93-d716-49a7-8078-12c7a5ab5fd5-kube-rbac-proxy-sar-config\") pod \"3d037c93-d716-49a7-8078-12c7a5ab5fd5\" (UID: \"3d037c93-d716-49a7-8078-12c7a5ab5fd5\") "
Sep 09 20:08:46.731604 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:08:46.731361    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/3d037c93-d716-49a7-8078-12c7a5ab5fd5-proxy-tls\") pod \"3d037c93-d716-49a7-8078-12c7a5ab5fd5\" (UID: \"3d037c93-d716-49a7-8078-12c7a5ab5fd5\") "
Sep 09 20:08:46.731604 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:08:46.731392    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/3d037c93-d716-49a7-8078-12c7a5ab5fd5-cabundle-cert\") pod \"3d037c93-d716-49a7-8078-12c7a5ab5fd5\" (UID: \"3d037c93-d716-49a7-8078-12c7a5ab5fd5\") "
Sep 09 20:08:46.731713 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:08:46.731657    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3d037c93-d716-49a7-8078-12c7a5ab5fd5-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "3d037c93-d716-49a7-8078-12c7a5ab5fd5" (UID: "3d037c93-d716-49a7-8078-12c7a5ab5fd5"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 20:08:46.731769 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:08:46.731746    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3d037c93-d716-49a7-8078-12c7a5ab5fd5-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "3d037c93-d716-49a7-8078-12c7a5ab5fd5" (UID: "3d037c93-d716-49a7-8078-12c7a5ab5fd5"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 20:08:46.731872 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:08:46.731756    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3d037c93-d716-49a7-8078-12c7a5ab5fd5-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "3d037c93-d716-49a7-8078-12c7a5ab5fd5" (UID: "3d037c93-d716-49a7-8078-12c7a5ab5fd5"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 20:08:46.733488 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:08:46.733455    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3d037c93-d716-49a7-8078-12c7a5ab5fd5-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "3d037c93-d716-49a7-8078-12c7a5ab5fd5" (UID: "3d037c93-d716-49a7-8078-12c7a5ab5fd5"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 20:08:46.733583 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:08:46.733493    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3d037c93-d716-49a7-8078-12c7a5ab5fd5-kube-api-access-l6n6b" (OuterVolumeSpecName: "kube-api-access-l6n6b") pod "3d037c93-d716-49a7-8078-12c7a5ab5fd5" (UID: "3d037c93-d716-49a7-8078-12c7a5ab5fd5"). InnerVolumeSpecName "kube-api-access-l6n6b". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 20:08:46.832047 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:08:46.832012    2574 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/3d037c93-d716-49a7-8078-12c7a5ab5fd5-kserve-provision-location\") on node \"ip-10-0-136-183.ec2.internal\" DevicePath \"\""
Sep 09 20:08:46.832047 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:08:46.832039    2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l6n6b\" (UniqueName: \"kubernetes.io/projected/3d037c93-d716-49a7-8078-12c7a5ab5fd5-kube-api-access-l6n6b\") on node \"ip-10-0-136-183.ec2.internal\" DevicePath \"\""
Sep 09 20:08:46.832047 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:08:46.832049    2574 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/3d037c93-d716-49a7-8078-12c7a5ab5fd5-kube-rbac-proxy-sar-config\") on node \"ip-10-0-136-183.ec2.internal\" DevicePath \"\""
Sep 09 20:08:46.832277 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:08:46.832059    2574 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/3d037c93-d716-49a7-8078-12c7a5ab5fd5-proxy-tls\") on node \"ip-10-0-136-183.ec2.internal\" DevicePath \"\""
Sep 09 20:08:46.832277 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:08:46.832068    2574 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/3d037c93-d716-49a7-8078-12c7a5ab5fd5-cabundle-cert\") on node \"ip-10-0-136-183.ec2.internal\" DevicePath \"\""
Sep 09 20:08:46.891135 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:08:46.891097    2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-6946b55967-fjgdf"]
Sep 09 20:08:46.902215 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:08:46.902181    2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-6946b55967-fjgdf"]
Sep 09 20:08:46.975603 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:08:46.975565    2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-raw-61cac-predictor-65ccf7895-mdm8w"]
Sep 09 20:08:46.975884 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:08:46.975871    2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3d037c93-d716-49a7-8078-12c7a5ab5fd5" containerName="storage-initializer"
Sep 09 20:08:46.975944 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:08:46.975886    2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="3d037c93-d716-49a7-8078-12c7a5ab5fd5" containerName="storage-initializer"
Sep 09 20:08:46.975944 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:08:46.975904    2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1a8f6bd9-81ac-4141-a518-35ae22a056f7" containerName="storage-initializer"
Sep 09 20:08:46.975944 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:08:46.975910    2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="1a8f6bd9-81ac-4141-a518-35ae22a056f7" containerName="storage-initializer"
Sep 09 20:08:46.975944 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:08:46.975917    2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3d037c93-d716-49a7-8078-12c7a5ab5fd5" containerName="kube-rbac-proxy"
Sep 09 20:08:46.975944 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:08:46.975924    2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="3d037c93-d716-49a7-8078-12c7a5ab5fd5" containerName="kube-rbac-proxy"
Sep 09 20:08:46.975944 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:08:46.975930    2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1a8f6bd9-81ac-4141-a518-35ae22a056f7" containerName="kube-rbac-proxy"
Sep 09 20:08:46.975944 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:08:46.975935    2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="1a8f6bd9-81ac-4141-a518-35ae22a056f7" containerName="kube-rbac-proxy"
Sep 09 20:08:46.975944 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:08:46.975942    2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3d037c93-d716-49a7-8078-12c7a5ab5fd5" containerName="kserve-container"
Sep 09 20:08:46.975944 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:08:46.975947    2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="3d037c93-d716-49a7-8078-12c7a5ab5fd5" containerName="kserve-container"
Sep 09 20:08:46.976276 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:08:46.975952    2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1a8f6bd9-81ac-4141-a518-35ae22a056f7" containerName="kserve-container"
Sep 09 20:08:46.976276 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:08:46.975958    2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="1a8f6bd9-81ac-4141-a518-35ae22a056f7" containerName="kserve-container"
Sep 09 20:08:46.976276 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:08:46.976001    2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="3d037c93-d716-49a7-8078-12c7a5ab5fd5" containerName="kserve-container"
Sep 09 20:08:46.976276 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:08:46.976008    2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="3d037c93-d716-49a7-8078-12c7a5ab5fd5" containerName="kube-rbac-proxy"
Sep 09 20:08:46.976276 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:08:46.976015    2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="1a8f6bd9-81ac-4141-a518-35ae22a056f7" containerName="kserve-container"
Sep 09 20:08:46.976276 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:08:46.976021    2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="1a8f6bd9-81ac-4141-a518-35ae22a056f7" containerName="kube-rbac-proxy"
Sep 09 20:08:46.980560 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:08:46.980538    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-61cac-predictor-65ccf7895-mdm8w"
Sep 09 20:08:46.982917 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:08:46.982855    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-graph-raw-61cac-predictor-serving-cert\""
Sep 09 20:08:46.982917 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:08:46.982880    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Sep 09 20:08:46.983148 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:08:46.982866    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-fh9bz\""
Sep 09 20:08:46.983346 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:08:46.983327    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-graph-raw-61cac-kube-rbac-proxy-sar-config\""
Sep 09 20:08:46.983435 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:08:46.983412    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Sep 09 20:08:46.983502 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:08:46.983466    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Sep 09 20:08:46.983502 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:08:46.983475    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Sep 09 20:08:46.989739 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:08:46.989712    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-raw-61cac-predictor-65ccf7895-mdm8w"]
Sep 09 20:08:47.106168 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:08:47.106136    2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-graph-raw-61cac-predictor-7d776878c7-nbg56"]
Sep 09 20:08:47.109549 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:08:47.109528    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-61cac-predictor-7d776878c7-nbg56"
Sep 09 20:08:47.111663 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:08:47.111641    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-xgboost-graph-raw-61cac-predictor-serving-cert\""
Sep 09 20:08:47.111771 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:08:47.111680    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-xgboost-graph-raw-61cac-kube-rbac-proxy-sar-config\""
Sep 09 20:08:47.122653 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:08:47.122624    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-graph-raw-61cac-predictor-7d776878c7-nbg56"]
Sep 09 20:08:47.134925 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:08:47.134894    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/63e5f7d1-294c-455c-9653-c0194ff17e5f-kserve-provision-location\") pod \"isvc-sklearn-graph-raw-61cac-predictor-65ccf7895-mdm8w\" (UID: \"63e5f7d1-294c-455c-9653-c0194ff17e5f\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-61cac-predictor-65ccf7895-mdm8w"
Sep 09 20:08:47.135058 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:08:47.134931    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/63e5f7d1-294c-455c-9653-c0194ff17e5f-cabundle-cert\") pod \"isvc-sklearn-graph-raw-61cac-predictor-65ccf7895-mdm8w\" (UID: \"63e5f7d1-294c-455c-9653-c0194ff17e5f\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-61cac-predictor-65ccf7895-mdm8w"
Sep 09 20:08:47.135058 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:08:47.134963    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/63e5f7d1-294c-455c-9653-c0194ff17e5f-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-graph-raw-61cac-predictor-65ccf7895-mdm8w\" (UID: \"63e5f7d1-294c-455c-9653-c0194ff17e5f\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-61cac-predictor-65ccf7895-mdm8w"
Sep 09 20:08:47.135058 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:08:47.135029    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/63e5f7d1-294c-455c-9653-c0194ff17e5f-proxy-tls\") pod \"isvc-sklearn-graph-raw-61cac-predictor-65ccf7895-mdm8w\" (UID: \"63e5f7d1-294c-455c-9653-c0194ff17e5f\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-61cac-predictor-65ccf7895-mdm8w"
Sep 09 20:08:47.135176 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:08:47.135088    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k99gx\" (UniqueName: \"kubernetes.io/projected/63e5f7d1-294c-455c-9653-c0194ff17e5f-kube-api-access-k99gx\") pod \"isvc-sklearn-graph-raw-61cac-predictor-65ccf7895-mdm8w\" (UID: \"63e5f7d1-294c-455c-9653-c0194ff17e5f\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-61cac-predictor-65ccf7895-mdm8w"
Sep 09 20:08:47.236549 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:08:47.236467    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/63e5f7d1-294c-455c-9653-c0194ff17e5f-kserve-provision-location\") pod \"isvc-sklearn-graph-raw-61cac-predictor-65ccf7895-mdm8w\" (UID: \"63e5f7d1-294c-455c-9653-c0194ff17e5f\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-61cac-predictor-65ccf7895-mdm8w"
Sep 09 20:08:47.236549 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:08:47.236504    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/63e5f7d1-294c-455c-9653-c0194ff17e5f-cabundle-cert\") pod \"isvc-sklearn-graph-raw-61cac-predictor-65ccf7895-mdm8w\" (UID: \"63e5f7d1-294c-455c-9653-c0194ff17e5f\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-61cac-predictor-65ccf7895-mdm8w"
Sep 09 20:08:47.236549 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:08:47.236528    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e6310063-1870-4f06-9116-a54299c215b2-proxy-tls\") pod \"isvc-xgboost-graph-raw-61cac-predictor-7d776878c7-nbg56\" (UID: \"e6310063-1870-4f06-9116-a54299c215b2\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-61cac-predictor-7d776878c7-nbg56"
Sep 09 20:08:47.236782 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:08:47.236651    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/63e5f7d1-294c-455c-9653-c0194ff17e5f-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-graph-raw-61cac-predictor-65ccf7895-mdm8w\" (UID: \"63e5f7d1-294c-455c-9653-c0194ff17e5f\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-61cac-predictor-65ccf7895-mdm8w"
Sep 09 20:08:47.236782 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:08:47.236707    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/63e5f7d1-294c-455c-9653-c0194ff17e5f-proxy-tls\") pod \"isvc-sklearn-graph-raw-61cac-predictor-65ccf7895-mdm8w\" (UID: \"63e5f7d1-294c-455c-9653-c0194ff17e5f\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-61cac-predictor-65ccf7895-mdm8w"
Sep 09 20:08:47.236782 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:08:47.236754    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k99gx\" (UniqueName: \"kubernetes.io/projected/63e5f7d1-294c-455c-9653-c0194ff17e5f-kube-api-access-k99gx\") pod \"isvc-sklearn-graph-raw-61cac-predictor-65ccf7895-mdm8w\" (UID: \"63e5f7d1-294c-455c-9653-c0194ff17e5f\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-61cac-predictor-65ccf7895-mdm8w"
Sep 09 20:08:47.236950 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:08:47.236783    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/e6310063-1870-4f06-9116-a54299c215b2-cabundle-cert\") pod \"isvc-xgboost-graph-raw-61cac-predictor-7d776878c7-nbg56\" (UID: \"e6310063-1870-4f06-9116-a54299c215b2\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-61cac-predictor-7d776878c7-nbg56"
Sep 09 20:08:47.236950 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:08:47.236840    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e6310063-1870-4f06-9116-a54299c215b2-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-graph-raw-61cac-predictor-7d776878c7-nbg56\" (UID: \"e6310063-1870-4f06-9116-a54299c215b2\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-61cac-predictor-7d776878c7-nbg56"
Sep 09 20:08:47.236950 ip-10-0-136-183 kubenswrapper[2574]: E0909 20:08:47.236859    2574 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-sklearn-graph-raw-61cac-predictor-serving-cert: secret "isvc-sklearn-graph-raw-61cac-predictor-serving-cert" not found
Sep 09 20:08:47.236950 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:08:47.236873    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-24628\" (UniqueName: \"kubernetes.io/projected/e6310063-1870-4f06-9116-a54299c215b2-kube-api-access-24628\") pod \"isvc-xgboost-graph-raw-61cac-predictor-7d776878c7-nbg56\" (UID: \"e6310063-1870-4f06-9116-a54299c215b2\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-61cac-predictor-7d776878c7-nbg56"
Sep 09 20:08:47.236950 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:08:47.236901    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e6310063-1870-4f06-9116-a54299c215b2-kserve-provision-location\") pod \"isvc-xgboost-graph-raw-61cac-predictor-7d776878c7-nbg56\" (UID: \"e6310063-1870-4f06-9116-a54299c215b2\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-61cac-predictor-7d776878c7-nbg56"
Sep 09 20:08:47.236950 ip-10-0-136-183 kubenswrapper[2574]: E0909 20:08:47.236935    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/63e5f7d1-294c-455c-9653-c0194ff17e5f-proxy-tls podName:63e5f7d1-294c-455c-9653-c0194ff17e5f nodeName:}" failed. No retries permitted until 2026-09-09 20:08:47.736910508 +0000 UTC m=+811.783163750 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/63e5f7d1-294c-455c-9653-c0194ff17e5f-proxy-tls") pod "isvc-sklearn-graph-raw-61cac-predictor-65ccf7895-mdm8w" (UID: "63e5f7d1-294c-455c-9653-c0194ff17e5f") : secret "isvc-sklearn-graph-raw-61cac-predictor-serving-cert" not found
Sep 09 20:08:47.237197 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:08:47.237016    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/63e5f7d1-294c-455c-9653-c0194ff17e5f-kserve-provision-location\") pod \"isvc-sklearn-graph-raw-61cac-predictor-65ccf7895-mdm8w\" (UID: \"63e5f7d1-294c-455c-9653-c0194ff17e5f\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-61cac-predictor-65ccf7895-mdm8w"
Sep 09 20:08:47.237260 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:08:47.237242    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/63e5f7d1-294c-455c-9653-c0194ff17e5f-cabundle-cert\") pod \"isvc-sklearn-graph-raw-61cac-predictor-65ccf7895-mdm8w\" (UID: \"63e5f7d1-294c-455c-9653-c0194ff17e5f\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-61cac-predictor-65ccf7895-mdm8w"
Sep 09 20:08:47.237306 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:08:47.237289    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/63e5f7d1-294c-455c-9653-c0194ff17e5f-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-graph-raw-61cac-predictor-65ccf7895-mdm8w\" (UID: \"63e5f7d1-294c-455c-9653-c0194ff17e5f\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-61cac-predictor-65ccf7895-mdm8w"
Sep 09 20:08:47.252861 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:08:47.252836    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-k99gx\" (UniqueName: \"kubernetes.io/projected/63e5f7d1-294c-455c-9653-c0194ff17e5f-kube-api-access-k99gx\") pod \"isvc-sklearn-graph-raw-61cac-predictor-65ccf7895-mdm8w\" (UID: \"63e5f7d1-294c-455c-9653-c0194ff17e5f\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-61cac-predictor-65ccf7895-mdm8w"
Sep 09 20:08:47.338666 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:08:47.338626    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/e6310063-1870-4f06-9116-a54299c215b2-cabundle-cert\") pod \"isvc-xgboost-graph-raw-61cac-predictor-7d776878c7-nbg56\" (UID: \"e6310063-1870-4f06-9116-a54299c215b2\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-61cac-predictor-7d776878c7-nbg56"
Sep 09 20:08:47.338666 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:08:47.338667    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e6310063-1870-4f06-9116-a54299c215b2-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-graph-raw-61cac-predictor-7d776878c7-nbg56\" (UID: \"e6310063-1870-4f06-9116-a54299c215b2\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-61cac-predictor-7d776878c7-nbg56"
Sep 09 20:08:47.338928 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:08:47.338697    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-24628\" (UniqueName: \"kubernetes.io/projected/e6310063-1870-4f06-9116-a54299c215b2-kube-api-access-24628\") pod \"isvc-xgboost-graph-raw-61cac-predictor-7d776878c7-nbg56\" (UID: \"e6310063-1870-4f06-9116-a54299c215b2\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-61cac-predictor-7d776878c7-nbg56"
Sep 09 20:08:47.338928 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:08:47.338852    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e6310063-1870-4f06-9116-a54299c215b2-kserve-provision-location\") pod \"isvc-xgboost-graph-raw-61cac-predictor-7d776878c7-nbg56\" (UID: \"e6310063-1870-4f06-9116-a54299c215b2\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-61cac-predictor-7d776878c7-nbg56"
Sep 09 20:08:47.339012 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:08:47.338939    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e6310063-1870-4f06-9116-a54299c215b2-proxy-tls\") pod \"isvc-xgboost-graph-raw-61cac-predictor-7d776878c7-nbg56\" (UID: \"e6310063-1870-4f06-9116-a54299c215b2\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-61cac-predictor-7d776878c7-nbg56"
Sep 09 20:08:47.339299 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:08:47.339273    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e6310063-1870-4f06-9116-a54299c215b2-kserve-provision-location\") pod \"isvc-xgboost-graph-raw-61cac-predictor-7d776878c7-nbg56\" (UID: \"e6310063-1870-4f06-9116-a54299c215b2\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-61cac-predictor-7d776878c7-nbg56"
Sep 09 20:08:47.339417 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:08:47.339388    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/e6310063-1870-4f06-9116-a54299c215b2-cabundle-cert\") pod \"isvc-xgboost-graph-raw-61cac-predictor-7d776878c7-nbg56\" (UID: \"e6310063-1870-4f06-9116-a54299c215b2\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-61cac-predictor-7d776878c7-nbg56"
Sep 09 20:08:47.339528 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:08:47.339392    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e6310063-1870-4f06-9116-a54299c215b2-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-graph-raw-61cac-predictor-7d776878c7-nbg56\" (UID: \"e6310063-1870-4f06-9116-a54299c215b2\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-61cac-predictor-7d776878c7-nbg56"
Sep 09 20:08:47.341479 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:08:47.341451    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e6310063-1870-4f06-9116-a54299c215b2-proxy-tls\") pod \"isvc-xgboost-graph-raw-61cac-predictor-7d776878c7-nbg56\" (UID: \"e6310063-1870-4f06-9116-a54299c215b2\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-61cac-predictor-7d776878c7-nbg56"
Sep 09 20:08:47.346904 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:08:47.346882    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-24628\" (UniqueName: \"kubernetes.io/projected/e6310063-1870-4f06-9116-a54299c215b2-kube-api-access-24628\") pod \"isvc-xgboost-graph-raw-61cac-predictor-7d776878c7-nbg56\" (UID: \"e6310063-1870-4f06-9116-a54299c215b2\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-61cac-predictor-7d776878c7-nbg56"
Sep 09 20:08:47.419536 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:08:47.419496    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-61cac-predictor-7d776878c7-nbg56"
Sep 09 20:08:47.696732 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:08:47.696695    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-graph-raw-61cac-predictor-7d776878c7-nbg56"]
Sep 09 20:08:47.699951 ip-10-0-136-183 kubenswrapper[2574]: W0909 20:08:47.699918    2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode6310063_1870_4f06_9116_a54299c215b2.slice/crio-b7bef7fa2e5da8bafceb17ea804b08cdcfb0f6d26247c400fdd5e026af7e10e2 WatchSource:0}: Error finding container b7bef7fa2e5da8bafceb17ea804b08cdcfb0f6d26247c400fdd5e026af7e10e2: Status 404 returned error can't find the container with id b7bef7fa2e5da8bafceb17ea804b08cdcfb0f6d26247c400fdd5e026af7e10e2
Sep 09 20:08:47.742609 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:08:47.742585    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/63e5f7d1-294c-455c-9653-c0194ff17e5f-proxy-tls\") pod \"isvc-sklearn-graph-raw-61cac-predictor-65ccf7895-mdm8w\" (UID: \"63e5f7d1-294c-455c-9653-c0194ff17e5f\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-61cac-predictor-65ccf7895-mdm8w"
Sep 09 20:08:47.744951 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:08:47.744923    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/63e5f7d1-294c-455c-9653-c0194ff17e5f-proxy-tls\") pod \"isvc-sklearn-graph-raw-61cac-predictor-65ccf7895-mdm8w\" (UID: \"63e5f7d1-294c-455c-9653-c0194ff17e5f\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-61cac-predictor-65ccf7895-mdm8w"
Sep 09 20:08:47.891357 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:08:47.891304    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-61cac-predictor-65ccf7895-mdm8w"
Sep 09 20:08:48.164566 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:08:48.164534    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-raw-61cac-predictor-65ccf7895-mdm8w"]
Sep 09 20:08:48.167961 ip-10-0-136-183 kubenswrapper[2574]: W0909 20:08:48.167930    2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod63e5f7d1_294c_455c_9653_c0194ff17e5f.slice/crio-13ffd1e413d84d2423e84eed6243d6e0047aa6dd8aeaa1889dab283675258c55 WatchSource:0}: Error finding container 13ffd1e413d84d2423e84eed6243d6e0047aa6dd8aeaa1889dab283675258c55: Status 404 returned error can't find the container with id 13ffd1e413d84d2423e84eed6243d6e0047aa6dd8aeaa1889dab283675258c55
Sep 09 20:08:48.490243 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:08:48.490164    2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3d037c93-d716-49a7-8078-12c7a5ab5fd5" path="/var/lib/kubelet/pods/3d037c93-d716-49a7-8078-12c7a5ab5fd5/volumes"
Sep 09 20:08:48.572685 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:08:48.572652    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-61cac-predictor-65ccf7895-mdm8w" event={"ID":"63e5f7d1-294c-455c-9653-c0194ff17e5f","Type":"ContainerStarted","Data":"e9e88228c25bd447957ce3a248ec8a5f77df8859979a4cdced6cf3419ace2487"}
Sep 09 20:08:48.572889 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:08:48.572717    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-61cac-predictor-65ccf7895-mdm8w" event={"ID":"63e5f7d1-294c-455c-9653-c0194ff17e5f","Type":"ContainerStarted","Data":"13ffd1e413d84d2423e84eed6243d6e0047aa6dd8aeaa1889dab283675258c55"}
Sep 09 20:08:48.574442 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:08:48.574413    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-61cac-predictor-7d776878c7-nbg56" event={"ID":"e6310063-1870-4f06-9116-a54299c215b2","Type":"ContainerStarted","Data":"f28565b395666ab335b5b4ac9aa1129e1d5eacb481170d2eb2e0a50e21e05d9e"}
Sep 09 20:08:48.574540 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:08:48.574481    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-61cac-predictor-7d776878c7-nbg56" event={"ID":"e6310063-1870-4f06-9116-a54299c215b2","Type":"ContainerStarted","Data":"b7bef7fa2e5da8bafceb17ea804b08cdcfb0f6d26247c400fdd5e026af7e10e2"}
Sep 09 20:08:52.590766 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:08:52.590729    2574 generic.go:356] "Generic (PLEG): container finished" podID="e6310063-1870-4f06-9116-a54299c215b2" containerID="f28565b395666ab335b5b4ac9aa1129e1d5eacb481170d2eb2e0a50e21e05d9e" exitCode=0
Sep 09 20:08:52.591278 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:08:52.590817    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-61cac-predictor-7d776878c7-nbg56" event={"ID":"e6310063-1870-4f06-9116-a54299c215b2","Type":"ContainerDied","Data":"f28565b395666ab335b5b4ac9aa1129e1d5eacb481170d2eb2e0a50e21e05d9e"}
Sep 09 20:08:52.592513 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:08:52.592492    2574 generic.go:356] "Generic (PLEG): container finished" podID="63e5f7d1-294c-455c-9653-c0194ff17e5f" containerID="e9e88228c25bd447957ce3a248ec8a5f77df8859979a4cdced6cf3419ace2487" exitCode=0
Sep 09 20:08:52.592614 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:08:52.592530    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-61cac-predictor-65ccf7895-mdm8w" event={"ID":"63e5f7d1-294c-455c-9653-c0194ff17e5f","Type":"ContainerDied","Data":"e9e88228c25bd447957ce3a248ec8a5f77df8859979a4cdced6cf3419ace2487"}
Sep 09 20:08:53.602692 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:08:53.602646    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-61cac-predictor-65ccf7895-mdm8w" event={"ID":"63e5f7d1-294c-455c-9653-c0194ff17e5f","Type":"ContainerStarted","Data":"3cc1c8e1c4c5de2c1baac57ab498cc57177dafd11565c41d8814729e54e2eb42"}
Sep 09 20:08:53.603273 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:08:53.602767    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-61cac-predictor-65ccf7895-mdm8w" event={"ID":"63e5f7d1-294c-455c-9653-c0194ff17e5f","Type":"ContainerStarted","Data":"e5bdbca55d3ffb49c0a530c73690e6abe2293425cedadc88177bb32fa97b44a1"}
Sep 09 20:08:53.603273 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:08:53.603129    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-61cac-predictor-65ccf7895-mdm8w"
Sep 09 20:08:53.603372 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:08:53.603307    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-61cac-predictor-65ccf7895-mdm8w"
Sep 09 20:08:53.605008 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:08:53.604969    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-61cac-predictor-65ccf7895-mdm8w" podUID="63e5f7d1-294c-455c-9653-c0194ff17e5f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.25:8080: connect: connection refused"
Sep 09 20:08:53.638750 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:08:53.638671    2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-61cac-predictor-65ccf7895-mdm8w" podStartSLOduration=4.065366703 podStartE2EDuration="7.638647687s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.573280984s" isStatefulPod=false podCreationTimestamp="2026-09-09 20:08:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:08:53.635886534 +0000 UTC m=+817.682139797" watchObservedRunningTime="2026-09-09 20:08:53.638647687 +0000 UTC m=+817.684900949"
Sep 09 20:08:54.609689 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:08:54.609649    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-61cac-predictor-65ccf7895-mdm8w" podUID="63e5f7d1-294c-455c-9653-c0194ff17e5f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.25:8080: connect: connection refused"
Sep 09 20:08:59.615614 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:08:59.615582    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-61cac-predictor-65ccf7895-mdm8w"
Sep 09 20:08:59.616107 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:08:59.616027    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-61cac-predictor-65ccf7895-mdm8w" podUID="63e5f7d1-294c-455c-9653-c0194ff17e5f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.25:8080: connect: connection refused"
Sep 09 20:09:09.616494 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:09:09.616445    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-61cac-predictor-65ccf7895-mdm8w" podUID="63e5f7d1-294c-455c-9653-c0194ff17e5f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.25:8080: connect: connection refused"
Sep 09 20:09:11.691252 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:09:11.691217    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-61cac-predictor-7d776878c7-nbg56" event={"ID":"e6310063-1870-4f06-9116-a54299c215b2","Type":"ContainerStarted","Data":"b35f9605ecf0fd06402b0b778b39ccad6e47aa548bfe3ee4a0e70bf942f2b524"}
Sep 09 20:09:11.691684 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:09:11.691281    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-61cac-predictor-7d776878c7-nbg56" event={"ID":"e6310063-1870-4f06-9116-a54299c215b2","Type":"ContainerStarted","Data":"9eaee63837e22d159edd7747112392e2ab372f144a90647bd125d4c924ecbce4"}
Sep 09 20:09:11.691684 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:09:11.691560    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-61cac-predictor-7d776878c7-nbg56"
Sep 09 20:09:11.691817 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:09:11.691765    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-61cac-predictor-7d776878c7-nbg56"
Sep 09 20:09:11.693097 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:09:11.693061    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-61cac-predictor-7d776878c7-nbg56" podUID="e6310063-1870-4f06-9116-a54299c215b2" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.26:8080: connect: connection refused"
Sep 09 20:09:11.712164 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:09:11.712115    2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-61cac-predictor-7d776878c7-nbg56" podStartSLOduration=2.635209777 podStartE2EDuration="24.712100062s" totalImagesPullingTime="18.068892538s" totalInitContainerRuntime="4.007997747s" isStatefulPod=false podCreationTimestamp="2026-09-09 20:08:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:09:11.710533742 +0000 UTC m=+835.756787003" watchObservedRunningTime="2026-09-09 20:09:11.712100062 +0000 UTC m=+835.758353324"
Sep 09 20:09:12.696033 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:09:12.695988    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-61cac-predictor-7d776878c7-nbg56" podUID="e6310063-1870-4f06-9116-a54299c215b2" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.26:8080: connect: connection refused"
Sep 09 20:09:17.700313 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:09:17.700285    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-61cac-predictor-7d776878c7-nbg56"
Sep 09 20:09:17.700865 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:09:17.700834    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-61cac-predictor-7d776878c7-nbg56" podUID="e6310063-1870-4f06-9116-a54299c215b2" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.26:8080: connect: connection refused"
Sep 09 20:09:19.616546 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:09:19.616509    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-61cac-predictor-65ccf7895-mdm8w" podUID="63e5f7d1-294c-455c-9653-c0194ff17e5f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.25:8080: connect: connection refused"
Sep 09 20:09:27.701306 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:09:27.701267    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-61cac-predictor-7d776878c7-nbg56" podUID="e6310063-1870-4f06-9116-a54299c215b2" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.26:8080: connect: connection refused"
Sep 09 20:09:29.616158 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:09:29.616116    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-61cac-predictor-65ccf7895-mdm8w" podUID="63e5f7d1-294c-455c-9653-c0194ff17e5f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.25:8080: connect: connection refused"
Sep 09 20:09:37.701362 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:09:37.701324    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-61cac-predictor-7d776878c7-nbg56" podUID="e6310063-1870-4f06-9116-a54299c215b2" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.26:8080: connect: connection refused"
Sep 09 20:09:39.616907 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:09:39.616862    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-61cac-predictor-65ccf7895-mdm8w" podUID="63e5f7d1-294c-455c-9653-c0194ff17e5f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.25:8080: connect: connection refused"
Sep 09 20:09:47.701595 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:09:47.701552    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-61cac-predictor-7d776878c7-nbg56" podUID="e6310063-1870-4f06-9116-a54299c215b2" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.26:8080: connect: connection refused"
Sep 09 20:09:49.616694 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:09:49.616646    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-61cac-predictor-65ccf7895-mdm8w" podUID="63e5f7d1-294c-455c-9653-c0194ff17e5f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.25:8080: connect: connection refused"
Sep 09 20:09:57.701692 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:09:57.701651    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-61cac-predictor-7d776878c7-nbg56" podUID="e6310063-1870-4f06-9116-a54299c215b2" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.26:8080: connect: connection refused"
Sep 09 20:09:59.616300 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:09:59.616259    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-61cac-predictor-65ccf7895-mdm8w" podUID="63e5f7d1-294c-455c-9653-c0194ff17e5f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.25:8080: connect: connection refused"
Sep 09 20:10:07.700741 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:10:07.700704    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-61cac-predictor-7d776878c7-nbg56" podUID="e6310063-1870-4f06-9116-a54299c215b2" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.26:8080: connect: connection refused"
Sep 09 20:10:09.617122 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:10:09.617029    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-61cac-predictor-65ccf7895-mdm8w" podUID="63e5f7d1-294c-455c-9653-c0194ff17e5f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.25:8080: connect: connection refused"
Sep 09 20:10:14.490991 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:10:14.490964    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-61cac-predictor-65ccf7895-mdm8w"
Sep 09 20:10:17.701961 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:10:17.701928    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-61cac-predictor-7d776878c7-nbg56"
Sep 09 20:10:26.997242 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:10:26.997207    2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/model-chainer-raw-61cac-6887f486b7-z4bk4"]
Sep 09 20:10:27.000485 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:10:27.000467    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/model-chainer-raw-61cac-6887f486b7-z4bk4"
Sep 09 20:10:27.002483 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:10:27.002457    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"model-chainer-raw-61cac-kube-rbac-proxy-sar-config\""
Sep 09 20:10:27.002603 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:10:27.002579    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"model-chainer-raw-61cac-serving-cert\""
Sep 09 20:10:27.008169 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:10:27.008149    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/model-chainer-raw-61cac-6887f486b7-z4bk4"]
Sep 09 20:10:27.028573 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:10:27.028539    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/acc34d61-cf66-488a-8229-faeda1df33b6-proxy-tls\") pod \"model-chainer-raw-61cac-6887f486b7-z4bk4\" (UID: \"acc34d61-cf66-488a-8229-faeda1df33b6\") " pod="kserve-ci-e2e-test/model-chainer-raw-61cac-6887f486b7-z4bk4"
Sep 09 20:10:27.028573 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:10:27.028576    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/acc34d61-cf66-488a-8229-faeda1df33b6-openshift-service-ca-bundle\") pod \"model-chainer-raw-61cac-6887f486b7-z4bk4\" (UID: \"acc34d61-cf66-488a-8229-faeda1df33b6\") " pod="kserve-ci-e2e-test/model-chainer-raw-61cac-6887f486b7-z4bk4"
Sep 09 20:10:27.129072 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:10:27.129043    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/acc34d61-cf66-488a-8229-faeda1df33b6-proxy-tls\") pod \"model-chainer-raw-61cac-6887f486b7-z4bk4\" (UID: \"acc34d61-cf66-488a-8229-faeda1df33b6\") " pod="kserve-ci-e2e-test/model-chainer-raw-61cac-6887f486b7-z4bk4"
Sep 09 20:10:27.129237 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:10:27.129079    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/acc34d61-cf66-488a-8229-faeda1df33b6-openshift-service-ca-bundle\") pod \"model-chainer-raw-61cac-6887f486b7-z4bk4\" (UID: \"acc34d61-cf66-488a-8229-faeda1df33b6\") " pod="kserve-ci-e2e-test/model-chainer-raw-61cac-6887f486b7-z4bk4"
Sep 09 20:10:27.129732 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:10:27.129708    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/acc34d61-cf66-488a-8229-faeda1df33b6-openshift-service-ca-bundle\") pod \"model-chainer-raw-61cac-6887f486b7-z4bk4\" (UID: \"acc34d61-cf66-488a-8229-faeda1df33b6\") " pod="kserve-ci-e2e-test/model-chainer-raw-61cac-6887f486b7-z4bk4"
Sep 09 20:10:27.131475 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:10:27.131450    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/acc34d61-cf66-488a-8229-faeda1df33b6-proxy-tls\") pod \"model-chainer-raw-61cac-6887f486b7-z4bk4\" (UID: \"acc34d61-cf66-488a-8229-faeda1df33b6\") " pod="kserve-ci-e2e-test/model-chainer-raw-61cac-6887f486b7-z4bk4"
Sep 09 20:10:27.311428 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:10:27.311338    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/model-chainer-raw-61cac-6887f486b7-z4bk4"
Sep 09 20:10:27.580521 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:10:27.580393    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/model-chainer-raw-61cac-6887f486b7-z4bk4"]
Sep 09 20:10:27.583329 ip-10-0-136-183 kubenswrapper[2574]: W0909 20:10:27.583298    2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podacc34d61_cf66_488a_8229_faeda1df33b6.slice/crio-c120d1cff7e54a227e33b786ae81795a3e0f83dbb8e9dbc8f2406e0de020ce02 WatchSource:0}: Error finding container c120d1cff7e54a227e33b786ae81795a3e0f83dbb8e9dbc8f2406e0de020ce02: Status 404 returned error can't find the container with id c120d1cff7e54a227e33b786ae81795a3e0f83dbb8e9dbc8f2406e0de020ce02
Sep 09 20:10:27.990668 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:10:27.990632    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/model-chainer-raw-61cac-6887f486b7-z4bk4" event={"ID":"acc34d61-cf66-488a-8229-faeda1df33b6","Type":"ContainerStarted","Data":"c120d1cff7e54a227e33b786ae81795a3e0f83dbb8e9dbc8f2406e0de020ce02"}
Sep 09 20:10:29.999599 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:10:29.999511    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/model-chainer-raw-61cac-6887f486b7-z4bk4" event={"ID":"acc34d61-cf66-488a-8229-faeda1df33b6","Type":"ContainerStarted","Data":"0a64f452b83a0617a30586fc2e9a037f007fd868c5aa0be2f3f03e9603a66789"}
Sep 09 20:10:30.000069 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:10:29.999609    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/model-chainer-raw-61cac-6887f486b7-z4bk4"
Sep 09 20:10:30.016582 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:10:30.016530    2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/model-chainer-raw-61cac-6887f486b7-z4bk4" podStartSLOduration=1.90377455 podStartE2EDuration="4.016514828s" totalImagesPullingTime="2.112740278s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 20:10:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:10:30.015496728 +0000 UTC m=+914.061749990" watchObservedRunningTime="2026-09-09 20:10:30.016514828 +0000 UTC m=+914.062768128"
Sep 09 20:10:36.009292 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:10:36.009264    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/model-chainer-raw-61cac-6887f486b7-z4bk4"
Sep 09 20:10:37.055900 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:10:37.055869    2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/model-chainer-raw-61cac-6887f486b7-z4bk4"]
Sep 09 20:10:37.056272 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:10:37.056108    2574 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/model-chainer-raw-61cac-6887f486b7-z4bk4" podUID="acc34d61-cf66-488a-8229-faeda1df33b6" containerName="model-chainer-raw-61cac" containerID="cri-o://0a64f452b83a0617a30586fc2e9a037f007fd868c5aa0be2f3f03e9603a66789" gracePeriod=30
Sep 09 20:10:37.167508 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:10:37.167476    2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-raw-61cac-predictor-65ccf7895-mdm8w"]
Sep 09 20:10:37.167833 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:10:37.167764    2574 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-61cac-predictor-65ccf7895-mdm8w" podUID="63e5f7d1-294c-455c-9653-c0194ff17e5f" containerName="kserve-container" containerID="cri-o://e5bdbca55d3ffb49c0a530c73690e6abe2293425cedadc88177bb32fa97b44a1" gracePeriod=30
Sep 09 20:10:37.167833 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:10:37.167816    2574 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-61cac-predictor-65ccf7895-mdm8w" podUID="63e5f7d1-294c-455c-9653-c0194ff17e5f" containerName="kube-rbac-proxy" containerID="cri-o://3cc1c8e1c4c5de2c1baac57ab498cc57177dafd11565c41d8814729e54e2eb42" gracePeriod=30
Sep 09 20:10:37.211089 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:10:37.211057    2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-graph-raw-61cac-predictor-7d776878c7-nbg56"]
Sep 09 20:10:37.211327 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:10:37.211294    2574 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-61cac-predictor-7d776878c7-nbg56" podUID="e6310063-1870-4f06-9116-a54299c215b2" containerName="kserve-container" containerID="cri-o://9eaee63837e22d159edd7747112392e2ab372f144a90647bd125d4c924ecbce4" gracePeriod=30
Sep 09 20:10:37.211463 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:10:37.211335    2574 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-61cac-predictor-7d776878c7-nbg56" podUID="e6310063-1870-4f06-9116-a54299c215b2" containerName="kube-rbac-proxy" containerID="cri-o://b35f9605ecf0fd06402b0b778b39ccad6e47aa548bfe3ee4a0e70bf942f2b524" gracePeriod=30
Sep 09 20:10:37.310345 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:10:37.310268    2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-5164f-predictor-6bbf8b7859-zrjtt"]
Sep 09 20:10:37.313665 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:10:37.313645    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-5164f-predictor-6bbf8b7859-zrjtt"
Sep 09 20:10:37.315683 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:10:37.315657    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-graph-raw-hpa-5164f-predictor-serving-cert\""
Sep 09 20:10:37.315938 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:10:37.315922    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-graph-raw-hpa-5164f-kube-rbac-proxy-sar-config\""
Sep 09 20:10:37.323489 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:10:37.323469    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-5164f-predictor-6bbf8b7859-zrjtt"]
Sep 09 20:10:37.414134 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:10:37.414101    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/e13f2ebf-3cdb-4ed4-a94a-d35a1fe1573c-cabundle-cert\") pod \"isvc-sklearn-graph-raw-hpa-5164f-predictor-6bbf8b7859-zrjtt\" (UID: \"e13f2ebf-3cdb-4ed4-a94a-d35a1fe1573c\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-5164f-predictor-6bbf8b7859-zrjtt"
Sep 09 20:10:37.414314 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:10:37.414158    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e13f2ebf-3cdb-4ed4-a94a-d35a1fe1573c-proxy-tls\") pod \"isvc-sklearn-graph-raw-hpa-5164f-predictor-6bbf8b7859-zrjtt\" (UID: \"e13f2ebf-3cdb-4ed4-a94a-d35a1fe1573c\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-5164f-predictor-6bbf8b7859-zrjtt"
Sep 09 20:10:37.414314 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:10:37.414179    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z9tsb\" (UniqueName: \"kubernetes.io/projected/e13f2ebf-3cdb-4ed4-a94a-d35a1fe1573c-kube-api-access-z9tsb\") pod \"isvc-sklearn-graph-raw-hpa-5164f-predictor-6bbf8b7859-zrjtt\" (UID: \"e13f2ebf-3cdb-4ed4-a94a-d35a1fe1573c\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-5164f-predictor-6bbf8b7859-zrjtt"
Sep 09 20:10:37.414314 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:10:37.414196    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e13f2ebf-3cdb-4ed4-a94a-d35a1fe1573c-kserve-provision-location\") pod \"isvc-sklearn-graph-raw-hpa-5164f-predictor-6bbf8b7859-zrjtt\" (UID: \"e13f2ebf-3cdb-4ed4-a94a-d35a1fe1573c\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-5164f-predictor-6bbf8b7859-zrjtt"
Sep 09 20:10:37.414314 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:10:37.414251    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e13f2ebf-3cdb-4ed4-a94a-d35a1fe1573c-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-graph-raw-hpa-5164f-predictor-6bbf8b7859-zrjtt\" (UID: \"e13f2ebf-3cdb-4ed4-a94a-d35a1fe1573c\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-5164f-predictor-6bbf8b7859-zrjtt"
Sep 09 20:10:37.514888 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:10:37.514855    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/e13f2ebf-3cdb-4ed4-a94a-d35a1fe1573c-cabundle-cert\") pod \"isvc-sklearn-graph-raw-hpa-5164f-predictor-6bbf8b7859-zrjtt\" (UID: \"e13f2ebf-3cdb-4ed4-a94a-d35a1fe1573c\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-5164f-predictor-6bbf8b7859-zrjtt"
Sep 09 20:10:37.514888 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:10:37.514891    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e13f2ebf-3cdb-4ed4-a94a-d35a1fe1573c-proxy-tls\") pod \"isvc-sklearn-graph-raw-hpa-5164f-predictor-6bbf8b7859-zrjtt\" (UID: \"e13f2ebf-3cdb-4ed4-a94a-d35a1fe1573c\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-5164f-predictor-6bbf8b7859-zrjtt"
Sep 09 20:10:37.515116 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:10:37.514909    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z9tsb\" (UniqueName: \"kubernetes.io/projected/e13f2ebf-3cdb-4ed4-a94a-d35a1fe1573c-kube-api-access-z9tsb\") pod \"isvc-sklearn-graph-raw-hpa-5164f-predictor-6bbf8b7859-zrjtt\" (UID: \"e13f2ebf-3cdb-4ed4-a94a-d35a1fe1573c\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-5164f-predictor-6bbf8b7859-zrjtt"
Sep 09 20:10:37.515203 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:10:37.515177    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e13f2ebf-3cdb-4ed4-a94a-d35a1fe1573c-kserve-provision-location\") pod \"isvc-sklearn-graph-raw-hpa-5164f-predictor-6bbf8b7859-zrjtt\" (UID: \"e13f2ebf-3cdb-4ed4-a94a-d35a1fe1573c\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-5164f-predictor-6bbf8b7859-zrjtt"
Sep 09 20:10:37.515260 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:10:37.515228    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e13f2ebf-3cdb-4ed4-a94a-d35a1fe1573c-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-graph-raw-hpa-5164f-predictor-6bbf8b7859-zrjtt\" (UID: \"e13f2ebf-3cdb-4ed4-a94a-d35a1fe1573c\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-5164f-predictor-6bbf8b7859-zrjtt"
Sep 09 20:10:37.515521 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:10:37.515496    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e13f2ebf-3cdb-4ed4-a94a-d35a1fe1573c-kserve-provision-location\") pod \"isvc-sklearn-graph-raw-hpa-5164f-predictor-6bbf8b7859-zrjtt\" (UID: \"e13f2ebf-3cdb-4ed4-a94a-d35a1fe1573c\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-5164f-predictor-6bbf8b7859-zrjtt"
Sep 09 20:10:37.515706 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:10:37.515688    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/e13f2ebf-3cdb-4ed4-a94a-d35a1fe1573c-cabundle-cert\") pod \"isvc-sklearn-graph-raw-hpa-5164f-predictor-6bbf8b7859-zrjtt\" (UID: \"e13f2ebf-3cdb-4ed4-a94a-d35a1fe1573c\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-5164f-predictor-6bbf8b7859-zrjtt"
Sep 09 20:10:37.515970 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:10:37.515946    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e13f2ebf-3cdb-4ed4-a94a-d35a1fe1573c-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-graph-raw-hpa-5164f-predictor-6bbf8b7859-zrjtt\" (UID: \"e13f2ebf-3cdb-4ed4-a94a-d35a1fe1573c\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-5164f-predictor-6bbf8b7859-zrjtt"
Sep 09 20:10:37.517594 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:10:37.517571    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e13f2ebf-3cdb-4ed4-a94a-d35a1fe1573c-proxy-tls\") pod \"isvc-sklearn-graph-raw-hpa-5164f-predictor-6bbf8b7859-zrjtt\" (UID: \"e13f2ebf-3cdb-4ed4-a94a-d35a1fe1573c\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-5164f-predictor-6bbf8b7859-zrjtt"
Sep 09 20:10:37.523380 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:10:37.523357    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-z9tsb\" (UniqueName: \"kubernetes.io/projected/e13f2ebf-3cdb-4ed4-a94a-d35a1fe1573c-kube-api-access-z9tsb\") pod \"isvc-sklearn-graph-raw-hpa-5164f-predictor-6bbf8b7859-zrjtt\" (UID: \"e13f2ebf-3cdb-4ed4-a94a-d35a1fe1573c\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-5164f-predictor-6bbf8b7859-zrjtt"
Sep 09 20:10:37.625617 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:10:37.625585    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-5164f-predictor-6bbf8b7859-zrjtt"
Sep 09 20:10:37.696861 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:10:37.696787    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-61cac-predictor-7d776878c7-nbg56" podUID="e6310063-1870-4f06-9116-a54299c215b2" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.26:8643/healthz\": dial tcp 10.133.0.26:8643: connect: connection refused"
Sep 09 20:10:37.701263 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:10:37.701194    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-61cac-predictor-7d776878c7-nbg56" podUID="e6310063-1870-4f06-9116-a54299c215b2" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.26:8080: connect: connection refused"
Sep 09 20:10:37.899319 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:10:37.899235    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-5164f-predictor-6bbf8b7859-zrjtt"]
Sep 09 20:10:37.902458 ip-10-0-136-183 kubenswrapper[2574]: W0909 20:10:37.902428    2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode13f2ebf_3cdb_4ed4_a94a_d35a1fe1573c.slice/crio-312f5e2b5e66b60281d9d9bae0a41133c5d6e6f8e109fe033d529f6fbc1bb71b WatchSource:0}: Error finding container 312f5e2b5e66b60281d9d9bae0a41133c5d6e6f8e109fe033d529f6fbc1bb71b: Status 404 returned error can't find the container with id 312f5e2b5e66b60281d9d9bae0a41133c5d6e6f8e109fe033d529f6fbc1bb71b
Sep 09 20:10:38.032402 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:10:38.032366    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-5164f-predictor-6bbf8b7859-zrjtt" event={"ID":"e13f2ebf-3cdb-4ed4-a94a-d35a1fe1573c","Type":"ContainerStarted","Data":"37ed6c1a64415dd9ef60b37dbb88d1082cb5fa60254c929a5546994edac6b8cc"}
Sep 09 20:10:38.032590 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:10:38.032436    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-5164f-predictor-6bbf8b7859-zrjtt" event={"ID":"e13f2ebf-3cdb-4ed4-a94a-d35a1fe1573c","Type":"ContainerStarted","Data":"312f5e2b5e66b60281d9d9bae0a41133c5d6e6f8e109fe033d529f6fbc1bb71b"}
Sep 09 20:10:38.034750 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:10:38.034723    2574 generic.go:356] "Generic (PLEG): container finished" podID="63e5f7d1-294c-455c-9653-c0194ff17e5f" containerID="3cc1c8e1c4c5de2c1baac57ab498cc57177dafd11565c41d8814729e54e2eb42" exitCode=2
Sep 09 20:10:38.034889 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:10:38.034812    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-61cac-predictor-65ccf7895-mdm8w" event={"ID":"63e5f7d1-294c-455c-9653-c0194ff17e5f","Type":"ContainerDied","Data":"3cc1c8e1c4c5de2c1baac57ab498cc57177dafd11565c41d8814729e54e2eb42"}
Sep 09 20:10:38.037119 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:10:38.037093    2574 generic.go:356] "Generic (PLEG): container finished" podID="e6310063-1870-4f06-9116-a54299c215b2" containerID="b35f9605ecf0fd06402b0b778b39ccad6e47aa548bfe3ee4a0e70bf942f2b524" exitCode=2
Sep 09 20:10:38.037201 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:10:38.037141    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-61cac-predictor-7d776878c7-nbg56" event={"ID":"e6310063-1870-4f06-9116-a54299c215b2","Type":"ContainerDied","Data":"b35f9605ecf0fd06402b0b778b39ccad6e47aa548bfe3ee4a0e70bf942f2b524"}
Sep 09 20:10:39.610284 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:10:39.610242    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-61cac-predictor-65ccf7895-mdm8w" podUID="63e5f7d1-294c-455c-9653-c0194ff17e5f" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.25:8643/healthz\": dial tcp 10.133.0.25:8643: connect: connection refused"
Sep 09 20:10:40.962253 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:10:40.962226    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-61cac-predictor-7d776878c7-nbg56"
Sep 09 20:10:41.008039 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:10:41.007998    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/model-chainer-raw-61cac-6887f486b7-z4bk4" podUID="acc34d61-cf66-488a-8229-faeda1df33b6" containerName="model-chainer-raw-61cac" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 09 20:10:41.043601 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:10:41.043565    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e6310063-1870-4f06-9116-a54299c215b2-proxy-tls\") pod \"e6310063-1870-4f06-9116-a54299c215b2\" (UID: \"e6310063-1870-4f06-9116-a54299c215b2\") "
Sep 09 20:10:41.043768 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:10:41.043630    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/e6310063-1870-4f06-9116-a54299c215b2-cabundle-cert\") pod \"e6310063-1870-4f06-9116-a54299c215b2\" (UID: \"e6310063-1870-4f06-9116-a54299c215b2\") "
Sep 09 20:10:41.043768 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:10:41.043649    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e6310063-1870-4f06-9116-a54299c215b2-kube-rbac-proxy-sar-config\") pod \"e6310063-1870-4f06-9116-a54299c215b2\" (UID: \"e6310063-1870-4f06-9116-a54299c215b2\") "
Sep 09 20:10:41.043768 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:10:41.043695    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-24628\" (UniqueName: \"kubernetes.io/projected/e6310063-1870-4f06-9116-a54299c215b2-kube-api-access-24628\") pod \"e6310063-1870-4f06-9116-a54299c215b2\" (UID: \"e6310063-1870-4f06-9116-a54299c215b2\") "
Sep 09 20:10:41.043768 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:10:41.043717    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e6310063-1870-4f06-9116-a54299c215b2-kserve-provision-location\") pod \"e6310063-1870-4f06-9116-a54299c215b2\" (UID: \"e6310063-1870-4f06-9116-a54299c215b2\") "
Sep 09 20:10:41.044104 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:10:41.044060    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e6310063-1870-4f06-9116-a54299c215b2-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "e6310063-1870-4f06-9116-a54299c215b2" (UID: "e6310063-1870-4f06-9116-a54299c215b2"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 20:10:41.044104 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:10:41.044082    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e6310063-1870-4f06-9116-a54299c215b2-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "e6310063-1870-4f06-9116-a54299c215b2" (UID: "e6310063-1870-4f06-9116-a54299c215b2"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 20:10:41.044266 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:10:41.044163    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e6310063-1870-4f06-9116-a54299c215b2-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "e6310063-1870-4f06-9116-a54299c215b2" (UID: "e6310063-1870-4f06-9116-a54299c215b2"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 20:10:41.045931 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:10:41.045865    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e6310063-1870-4f06-9116-a54299c215b2-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "e6310063-1870-4f06-9116-a54299c215b2" (UID: "e6310063-1870-4f06-9116-a54299c215b2"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 20:10:41.045931 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:10:41.045868    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e6310063-1870-4f06-9116-a54299c215b2-kube-api-access-24628" (OuterVolumeSpecName: "kube-api-access-24628") pod "e6310063-1870-4f06-9116-a54299c215b2" (UID: "e6310063-1870-4f06-9116-a54299c215b2"). InnerVolumeSpecName "kube-api-access-24628". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 20:10:41.053270 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:10:41.053242    2574 generic.go:356] "Generic (PLEG): container finished" podID="e6310063-1870-4f06-9116-a54299c215b2" containerID="9eaee63837e22d159edd7747112392e2ab372f144a90647bd125d4c924ecbce4" exitCode=0
Sep 09 20:10:41.053375 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:10:41.053301    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-61cac-predictor-7d776878c7-nbg56"
Sep 09 20:10:41.053375 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:10:41.053321    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-61cac-predictor-7d776878c7-nbg56" event={"ID":"e6310063-1870-4f06-9116-a54299c215b2","Type":"ContainerDied","Data":"9eaee63837e22d159edd7747112392e2ab372f144a90647bd125d4c924ecbce4"}
Sep 09 20:10:41.053488 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:10:41.053386    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-61cac-predictor-7d776878c7-nbg56" event={"ID":"e6310063-1870-4f06-9116-a54299c215b2","Type":"ContainerDied","Data":"b7bef7fa2e5da8bafceb17ea804b08cdcfb0f6d26247c400fdd5e026af7e10e2"}
Sep 09 20:10:41.053488 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:10:41.053416    2574 scope.go:117] "RemoveContainer" containerID="b35f9605ecf0fd06402b0b778b39ccad6e47aa548bfe3ee4a0e70bf942f2b524"
Sep 09 20:10:41.062465 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:10:41.062445    2574 scope.go:117] "RemoveContainer" containerID="9eaee63837e22d159edd7747112392e2ab372f144a90647bd125d4c924ecbce4"
Sep 09 20:10:41.070409 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:10:41.070391    2574 scope.go:117] "RemoveContainer" containerID="f28565b395666ab335b5b4ac9aa1129e1d5eacb481170d2eb2e0a50e21e05d9e"
Sep 09 20:10:41.077940 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:10:41.077913    2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-graph-raw-61cac-predictor-7d776878c7-nbg56"]
Sep 09 20:10:41.079961 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:10:41.079937    2574 scope.go:117] "RemoveContainer" containerID="b35f9605ecf0fd06402b0b778b39ccad6e47aa548bfe3ee4a0e70bf942f2b524"
Sep 09 20:10:41.080346 ip-10-0-136-183 kubenswrapper[2574]: E0909 20:10:41.080319    2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b35f9605ecf0fd06402b0b778b39ccad6e47aa548bfe3ee4a0e70bf942f2b524\": container with ID starting with b35f9605ecf0fd06402b0b778b39ccad6e47aa548bfe3ee4a0e70bf942f2b524 not found: ID does not exist" containerID="b35f9605ecf0fd06402b0b778b39ccad6e47aa548bfe3ee4a0e70bf942f2b524"
Sep 09 20:10:41.080439 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:10:41.080360    2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b35f9605ecf0fd06402b0b778b39ccad6e47aa548bfe3ee4a0e70bf942f2b524"} err="failed to get container status \"b35f9605ecf0fd06402b0b778b39ccad6e47aa548bfe3ee4a0e70bf942f2b524\": rpc error: code = NotFound desc = could not find container \"b35f9605ecf0fd06402b0b778b39ccad6e47aa548bfe3ee4a0e70bf942f2b524\": container with ID starting with b35f9605ecf0fd06402b0b778b39ccad6e47aa548bfe3ee4a0e70bf942f2b524 not found: ID does not exist"
Sep 09 20:10:41.080439 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:10:41.080380    2574 scope.go:117] "RemoveContainer" containerID="9eaee63837e22d159edd7747112392e2ab372f144a90647bd125d4c924ecbce4"
Sep 09 20:10:41.080704 ip-10-0-136-183 kubenswrapper[2574]: E0909 20:10:41.080679    2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9eaee63837e22d159edd7747112392e2ab372f144a90647bd125d4c924ecbce4\": container with ID starting with 9eaee63837e22d159edd7747112392e2ab372f144a90647bd125d4c924ecbce4 not found: ID does not exist" containerID="9eaee63837e22d159edd7747112392e2ab372f144a90647bd125d4c924ecbce4"
Sep 09 20:10:41.080793 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:10:41.080714    2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9eaee63837e22d159edd7747112392e2ab372f144a90647bd125d4c924ecbce4"} err="failed to get container status \"9eaee63837e22d159edd7747112392e2ab372f144a90647bd125d4c924ecbce4\": rpc error: code = NotFound desc = could not find container \"9eaee63837e22d159edd7747112392e2ab372f144a90647bd125d4c924ecbce4\": container with ID starting with 9eaee63837e22d159edd7747112392e2ab372f144a90647bd125d4c924ecbce4 not found: ID does not exist"
Sep 09 20:10:41.080793 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:10:41.080737    2574 scope.go:117] "RemoveContainer" containerID="f28565b395666ab335b5b4ac9aa1129e1d5eacb481170d2eb2e0a50e21e05d9e"
Sep 09 20:10:41.081054 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:10:41.081034    2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-graph-raw-61cac-predictor-7d776878c7-nbg56"]
Sep 09 20:10:41.081121 ip-10-0-136-183 kubenswrapper[2574]: E0909 20:10:41.081044    2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f28565b395666ab335b5b4ac9aa1129e1d5eacb481170d2eb2e0a50e21e05d9e\": container with ID starting with f28565b395666ab335b5b4ac9aa1129e1d5eacb481170d2eb2e0a50e21e05d9e not found: ID does not exist" containerID="f28565b395666ab335b5b4ac9aa1129e1d5eacb481170d2eb2e0a50e21e05d9e"
Sep 09 20:10:41.081121 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:10:41.081075    2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f28565b395666ab335b5b4ac9aa1129e1d5eacb481170d2eb2e0a50e21e05d9e"} err="failed to get container status \"f28565b395666ab335b5b4ac9aa1129e1d5eacb481170d2eb2e0a50e21e05d9e\": rpc error: code = NotFound desc = could not find container \"f28565b395666ab335b5b4ac9aa1129e1d5eacb481170d2eb2e0a50e21e05d9e\": container with ID starting with f28565b395666ab335b5b4ac9aa1129e1d5eacb481170d2eb2e0a50e21e05d9e not found: ID does not exist"
Sep 09 20:10:41.144543 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:10:41.144514    2574 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/e6310063-1870-4f06-9116-a54299c215b2-cabundle-cert\") on node \"ip-10-0-136-183.ec2.internal\" DevicePath \"\""
Sep 09 20:10:41.144543 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:10:41.144539    2574 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e6310063-1870-4f06-9116-a54299c215b2-kube-rbac-proxy-sar-config\") on node \"ip-10-0-136-183.ec2.internal\" DevicePath \"\""
Sep 09 20:10:41.144543 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:10:41.144549    2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-24628\" (UniqueName: \"kubernetes.io/projected/e6310063-1870-4f06-9116-a54299c215b2-kube-api-access-24628\") on node \"ip-10-0-136-183.ec2.internal\" DevicePath \"\""
Sep 09 20:10:41.144742 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:10:41.144558    2574 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e6310063-1870-4f06-9116-a54299c215b2-kserve-provision-location\") on node \"ip-10-0-136-183.ec2.internal\" DevicePath \"\""
Sep 09 20:10:41.144742 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:10:41.144566    2574 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e6310063-1870-4f06-9116-a54299c215b2-proxy-tls\") on node \"ip-10-0-136-183.ec2.internal\" DevicePath \"\""
Sep 09 20:10:42.060426 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:10:42.060398    2574 generic.go:356] "Generic (PLEG): container finished" podID="63e5f7d1-294c-455c-9653-c0194ff17e5f" containerID="e5bdbca55d3ffb49c0a530c73690e6abe2293425cedadc88177bb32fa97b44a1" exitCode=0
Sep 09 20:10:42.060828 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:10:42.060478    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-61cac-predictor-65ccf7895-mdm8w" event={"ID":"63e5f7d1-294c-455c-9653-c0194ff17e5f","Type":"ContainerDied","Data":"e5bdbca55d3ffb49c0a530c73690e6abe2293425cedadc88177bb32fa97b44a1"}
Sep 09 20:10:42.063099 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:10:42.063032    2574 generic.go:356] "Generic (PLEG): container finished" podID="e13f2ebf-3cdb-4ed4-a94a-d35a1fe1573c" containerID="37ed6c1a64415dd9ef60b37dbb88d1082cb5fa60254c929a5546994edac6b8cc" exitCode=0
Sep 09 20:10:42.063099 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:10:42.063064    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-5164f-predictor-6bbf8b7859-zrjtt" event={"ID":"e13f2ebf-3cdb-4ed4-a94a-d35a1fe1573c","Type":"ContainerDied","Data":"37ed6c1a64415dd9ef60b37dbb88d1082cb5fa60254c929a5546994edac6b8cc"}
Sep 09 20:10:42.128480 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:10:42.128446    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-61cac-predictor-65ccf7895-mdm8w"
Sep 09 20:10:42.254784 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:10:42.254755    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/63e5f7d1-294c-455c-9653-c0194ff17e5f-kserve-provision-location\") pod \"63e5f7d1-294c-455c-9653-c0194ff17e5f\" (UID: \"63e5f7d1-294c-455c-9653-c0194ff17e5f\") "
Sep 09 20:10:42.254924 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:10:42.254794    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/63e5f7d1-294c-455c-9653-c0194ff17e5f-proxy-tls\") pod \"63e5f7d1-294c-455c-9653-c0194ff17e5f\" (UID: \"63e5f7d1-294c-455c-9653-c0194ff17e5f\") "
Sep 09 20:10:42.254924 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:10:42.254842    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/63e5f7d1-294c-455c-9653-c0194ff17e5f-cabundle-cert\") pod \"63e5f7d1-294c-455c-9653-c0194ff17e5f\" (UID: \"63e5f7d1-294c-455c-9653-c0194ff17e5f\") "
Sep 09 20:10:42.254924 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:10:42.254869    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-k99gx\" (UniqueName: \"kubernetes.io/projected/63e5f7d1-294c-455c-9653-c0194ff17e5f-kube-api-access-k99gx\") pod \"63e5f7d1-294c-455c-9653-c0194ff17e5f\" (UID: \"63e5f7d1-294c-455c-9653-c0194ff17e5f\") "
Sep 09 20:10:42.255090 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:10:42.254924    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/63e5f7d1-294c-455c-9653-c0194ff17e5f-kube-rbac-proxy-sar-config\") pod \"63e5f7d1-294c-455c-9653-c0194ff17e5f\" (UID: \"63e5f7d1-294c-455c-9653-c0194ff17e5f\") "
Sep 09 20:10:42.255150 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:10:42.255109    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/63e5f7d1-294c-455c-9653-c0194ff17e5f-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "63e5f7d1-294c-455c-9653-c0194ff17e5f" (UID: "63e5f7d1-294c-455c-9653-c0194ff17e5f"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 20:10:42.255243 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:10:42.255221    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/63e5f7d1-294c-455c-9653-c0194ff17e5f-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "63e5f7d1-294c-455c-9653-c0194ff17e5f" (UID: "63e5f7d1-294c-455c-9653-c0194ff17e5f"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 20:10:42.255344 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:10:42.255323    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/63e5f7d1-294c-455c-9653-c0194ff17e5f-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "63e5f7d1-294c-455c-9653-c0194ff17e5f" (UID: "63e5f7d1-294c-455c-9653-c0194ff17e5f"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 20:10:42.256885 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:10:42.256862    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/63e5f7d1-294c-455c-9653-c0194ff17e5f-kube-api-access-k99gx" (OuterVolumeSpecName: "kube-api-access-k99gx") pod "63e5f7d1-294c-455c-9653-c0194ff17e5f" (UID: "63e5f7d1-294c-455c-9653-c0194ff17e5f"). InnerVolumeSpecName "kube-api-access-k99gx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 20:10:42.256953 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:10:42.256942    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/63e5f7d1-294c-455c-9653-c0194ff17e5f-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "63e5f7d1-294c-455c-9653-c0194ff17e5f" (UID: "63e5f7d1-294c-455c-9653-c0194ff17e5f"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 20:10:42.355663 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:10:42.355628    2574 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/63e5f7d1-294c-455c-9653-c0194ff17e5f-kserve-provision-location\") on node \"ip-10-0-136-183.ec2.internal\" DevicePath \"\""
Sep 09 20:10:42.355663 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:10:42.355663    2574 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/63e5f7d1-294c-455c-9653-c0194ff17e5f-proxy-tls\") on node \"ip-10-0-136-183.ec2.internal\" DevicePath \"\""
Sep 09 20:10:42.355896 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:10:42.355674    2574 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/63e5f7d1-294c-455c-9653-c0194ff17e5f-cabundle-cert\") on node \"ip-10-0-136-183.ec2.internal\" DevicePath \"\""
Sep 09 20:10:42.355896 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:10:42.355682    2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k99gx\" (UniqueName: \"kubernetes.io/projected/63e5f7d1-294c-455c-9653-c0194ff17e5f-kube-api-access-k99gx\") on node \"ip-10-0-136-183.ec2.internal\" DevicePath \"\""
Sep 09 20:10:42.355896 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:10:42.355691    2574 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/63e5f7d1-294c-455c-9653-c0194ff17e5f-kube-rbac-proxy-sar-config\") on node \"ip-10-0-136-183.ec2.internal\" DevicePath \"\""
Sep 09 20:10:42.491218 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:10:42.491182    2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e6310063-1870-4f06-9116-a54299c215b2" path="/var/lib/kubelet/pods/e6310063-1870-4f06-9116-a54299c215b2/volumes"
Sep 09 20:10:43.068985 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:10:43.068953    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-5164f-predictor-6bbf8b7859-zrjtt" event={"ID":"e13f2ebf-3cdb-4ed4-a94a-d35a1fe1573c","Type":"ContainerStarted","Data":"1d69bb651bb4d6b325dcf5dbfe50924cd0b9eb19d2a0d78ccf0bb061f813b70e"}
Sep 09 20:10:43.069415 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:10:43.069008    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-5164f-predictor-6bbf8b7859-zrjtt" event={"ID":"e13f2ebf-3cdb-4ed4-a94a-d35a1fe1573c","Type":"ContainerStarted","Data":"0da28e1b37cb97e78ce8dc562afba93b83c91176d6395f1cf31f5b143a226a3b"}
Sep 09 20:10:43.069415 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:10:43.069194    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-5164f-predictor-6bbf8b7859-zrjtt"
Sep 09 20:10:43.071005 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:10:43.070984    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-61cac-predictor-65ccf7895-mdm8w"
Sep 09 20:10:43.071005 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:10:43.070992    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-61cac-predictor-65ccf7895-mdm8w" event={"ID":"63e5f7d1-294c-455c-9653-c0194ff17e5f","Type":"ContainerDied","Data":"13ffd1e413d84d2423e84eed6243d6e0047aa6dd8aeaa1889dab283675258c55"}
Sep 09 20:10:43.071193 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:10:43.071041    2574 scope.go:117] "RemoveContainer" containerID="3cc1c8e1c4c5de2c1baac57ab498cc57177dafd11565c41d8814729e54e2eb42"
Sep 09 20:10:43.078726 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:10:43.078711    2574 scope.go:117] "RemoveContainer" containerID="e5bdbca55d3ffb49c0a530c73690e6abe2293425cedadc88177bb32fa97b44a1"
Sep 09 20:10:43.087498 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:10:43.087479    2574 scope.go:117] "RemoveContainer" containerID="e9e88228c25bd447957ce3a248ec8a5f77df8859979a4cdced6cf3419ace2487"
Sep 09 20:10:43.092820 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:10:43.092074    2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-5164f-predictor-6bbf8b7859-zrjtt" podStartSLOduration=2.23508229 podStartE2EDuration="6.092055559s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.856973269s" isStatefulPod=false podCreationTimestamp="2026-09-09 20:10:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:10:43.089074993 +0000 UTC m=+927.135328256" watchObservedRunningTime="2026-09-09 20:10:43.092055559 +0000 UTC m=+927.138308825"
Sep 09 20:10:43.102487 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:10:43.102463    2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-raw-61cac-predictor-65ccf7895-mdm8w"]
Sep 09 20:10:43.106283 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:10:43.106263    2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-raw-61cac-predictor-65ccf7895-mdm8w"]
Sep 09 20:10:44.077001 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:10:44.076970    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-5164f-predictor-6bbf8b7859-zrjtt"
Sep 09 20:10:44.078416 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:10:44.078392    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-5164f-predictor-6bbf8b7859-zrjtt" podUID="e13f2ebf-3cdb-4ed4-a94a-d35a1fe1573c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.28:8080: connect: connection refused"
Sep 09 20:10:44.492755 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:10:44.492712    2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="63e5f7d1-294c-455c-9653-c0194ff17e5f" path="/var/lib/kubelet/pods/63e5f7d1-294c-455c-9653-c0194ff17e5f/volumes"
Sep 09 20:10:45.081074 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:10:45.081032    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-5164f-predictor-6bbf8b7859-zrjtt" podUID="e13f2ebf-3cdb-4ed4-a94a-d35a1fe1573c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.28:8080: connect: connection refused"
Sep 09 20:10:46.007607 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:10:46.007554    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/model-chainer-raw-61cac-6887f486b7-z4bk4" podUID="acc34d61-cf66-488a-8229-faeda1df33b6" containerName="model-chainer-raw-61cac" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 09 20:10:50.085646 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:10:50.085616    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-5164f-predictor-6bbf8b7859-zrjtt"
Sep 09 20:10:50.086143 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:10:50.086008    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-5164f-predictor-6bbf8b7859-zrjtt" podUID="e13f2ebf-3cdb-4ed4-a94a-d35a1fe1573c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.28:8080: connect: connection refused"
Sep 09 20:10:51.008129 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:10:51.008090    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/model-chainer-raw-61cac-6887f486b7-z4bk4" podUID="acc34d61-cf66-488a-8229-faeda1df33b6" containerName="model-chainer-raw-61cac" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 09 20:10:51.008285 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:10:51.008196    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/model-chainer-raw-61cac-6887f486b7-z4bk4"
Sep 09 20:10:56.007326 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:10:56.007285    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/model-chainer-raw-61cac-6887f486b7-z4bk4" podUID="acc34d61-cf66-488a-8229-faeda1df33b6" containerName="model-chainer-raw-61cac" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 09 20:11:00.086055 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:11:00.086015    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-5164f-predictor-6bbf8b7859-zrjtt" podUID="e13f2ebf-3cdb-4ed4-a94a-d35a1fe1573c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.28:8080: connect: connection refused"
Sep 09 20:11:01.007766 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:11:01.007724    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/model-chainer-raw-61cac-6887f486b7-z4bk4" podUID="acc34d61-cf66-488a-8229-faeda1df33b6" containerName="model-chainer-raw-61cac" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 09 20:11:06.007612 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:11:06.007572    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/model-chainer-raw-61cac-6887f486b7-z4bk4" podUID="acc34d61-cf66-488a-8229-faeda1df33b6" containerName="model-chainer-raw-61cac" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 09 20:11:07.168779 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:11:07.168734    2574 generic.go:356] "Generic (PLEG): container finished" podID="acc34d61-cf66-488a-8229-faeda1df33b6" containerID="0a64f452b83a0617a30586fc2e9a037f007fd868c5aa0be2f3f03e9603a66789" exitCode=0
Sep 09 20:11:07.169154 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:11:07.168833    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/model-chainer-raw-61cac-6887f486b7-z4bk4" event={"ID":"acc34d61-cf66-488a-8229-faeda1df33b6","Type":"ContainerDied","Data":"0a64f452b83a0617a30586fc2e9a037f007fd868c5aa0be2f3f03e9603a66789"}
Sep 09 20:11:07.701123 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:11:07.701100    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/model-chainer-raw-61cac-6887f486b7-z4bk4"
Sep 09 20:11:07.867659 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:11:07.867631    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/acc34d61-cf66-488a-8229-faeda1df33b6-openshift-service-ca-bundle\") pod \"acc34d61-cf66-488a-8229-faeda1df33b6\" (UID: \"acc34d61-cf66-488a-8229-faeda1df33b6\") "
Sep 09 20:11:07.867846 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:11:07.867674    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/acc34d61-cf66-488a-8229-faeda1df33b6-proxy-tls\") pod \"acc34d61-cf66-488a-8229-faeda1df33b6\" (UID: \"acc34d61-cf66-488a-8229-faeda1df33b6\") "
Sep 09 20:11:07.868004 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:11:07.867976    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/acc34d61-cf66-488a-8229-faeda1df33b6-openshift-service-ca-bundle" (OuterVolumeSpecName: "openshift-service-ca-bundle") pod "acc34d61-cf66-488a-8229-faeda1df33b6" (UID: "acc34d61-cf66-488a-8229-faeda1df33b6"). InnerVolumeSpecName "openshift-service-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 20:11:07.869675 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:11:07.869653    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/acc34d61-cf66-488a-8229-faeda1df33b6-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "acc34d61-cf66-488a-8229-faeda1df33b6" (UID: "acc34d61-cf66-488a-8229-faeda1df33b6"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 20:11:07.969128 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:11:07.969086    2574 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/acc34d61-cf66-488a-8229-faeda1df33b6-openshift-service-ca-bundle\") on node \"ip-10-0-136-183.ec2.internal\" DevicePath \"\""
Sep 09 20:11:07.969128 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:11:07.969121    2574 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/acc34d61-cf66-488a-8229-faeda1df33b6-proxy-tls\") on node \"ip-10-0-136-183.ec2.internal\" DevicePath \"\""
Sep 09 20:11:08.174123 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:11:08.174044    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/model-chainer-raw-61cac-6887f486b7-z4bk4"
Sep 09 20:11:08.174521 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:11:08.174045    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/model-chainer-raw-61cac-6887f486b7-z4bk4" event={"ID":"acc34d61-cf66-488a-8229-faeda1df33b6","Type":"ContainerDied","Data":"c120d1cff7e54a227e33b786ae81795a3e0f83dbb8e9dbc8f2406e0de020ce02"}
Sep 09 20:11:08.174521 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:11:08.174180    2574 scope.go:117] "RemoveContainer" containerID="0a64f452b83a0617a30586fc2e9a037f007fd868c5aa0be2f3f03e9603a66789"
Sep 09 20:11:08.194520 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:11:08.194485    2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/model-chainer-raw-61cac-6887f486b7-z4bk4"]
Sep 09 20:11:08.200172 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:11:08.200140    2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/model-chainer-raw-61cac-6887f486b7-z4bk4"]
Sep 09 20:11:08.490093 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:11:08.490015    2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="acc34d61-cf66-488a-8229-faeda1df33b6" path="/var/lib/kubelet/pods/acc34d61-cf66-488a-8229-faeda1df33b6/volumes"
Sep 09 20:11:10.086560 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:11:10.086524    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-5164f-predictor-6bbf8b7859-zrjtt" podUID="e13f2ebf-3cdb-4ed4-a94a-d35a1fe1573c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.28:8080: connect: connection refused"
Sep 09 20:11:20.086061 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:11:20.086022    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-5164f-predictor-6bbf8b7859-zrjtt" podUID="e13f2ebf-3cdb-4ed4-a94a-d35a1fe1573c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.28:8080: connect: connection refused"
Sep 09 20:11:30.086135 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:11:30.086095    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-5164f-predictor-6bbf8b7859-zrjtt" podUID="e13f2ebf-3cdb-4ed4-a94a-d35a1fe1573c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.28:8080: connect: connection refused"
Sep 09 20:11:40.086644 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:11:40.086548    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-5164f-predictor-6bbf8b7859-zrjtt" podUID="e13f2ebf-3cdb-4ed4-a94a-d35a1fe1573c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.28:8080: connect: connection refused"
Sep 09 20:11:50.086303 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:11:50.086263    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-5164f-predictor-6bbf8b7859-zrjtt" podUID="e13f2ebf-3cdb-4ed4-a94a-d35a1fe1573c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.28:8080: connect: connection refused"
Sep 09 20:12:00.086625 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:12:00.086591    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-5164f-predictor-6bbf8b7859-zrjtt"
Sep 09 20:12:17.328282 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:12:17.328242    2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/model-chainer-raw-hpa-5164f-679f86dddf-dj5jn"]
Sep 09 20:12:17.328781 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:12:17.328695    2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e6310063-1870-4f06-9116-a54299c215b2" containerName="kserve-container"
Sep 09 20:12:17.328781 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:12:17.328713    2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="e6310063-1870-4f06-9116-a54299c215b2" containerName="kserve-container"
Sep 09 20:12:17.328781 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:12:17.328735    2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="63e5f7d1-294c-455c-9653-c0194ff17e5f" containerName="storage-initializer"
Sep 09 20:12:17.328781 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:12:17.328744    2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="63e5f7d1-294c-455c-9653-c0194ff17e5f" containerName="storage-initializer"
Sep 09 20:12:17.328781 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:12:17.328759    2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e6310063-1870-4f06-9116-a54299c215b2" containerName="storage-initializer"
Sep 09 20:12:17.328781 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:12:17.328767    2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="e6310063-1870-4f06-9116-a54299c215b2" containerName="storage-initializer"
Sep 09 20:12:17.328781 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:12:17.328777    2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="acc34d61-cf66-488a-8229-faeda1df33b6" containerName="model-chainer-raw-61cac"
Sep 09 20:12:17.328781 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:12:17.328784    2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="acc34d61-cf66-488a-8229-faeda1df33b6" containerName="model-chainer-raw-61cac"
Sep 09 20:12:17.329196 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:12:17.328853    2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e6310063-1870-4f06-9116-a54299c215b2" containerName="kube-rbac-proxy"
Sep 09 20:12:17.329196 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:12:17.328863    2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="e6310063-1870-4f06-9116-a54299c215b2" containerName="kube-rbac-proxy"
Sep 09 20:12:17.329196 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:12:17.328874    2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="63e5f7d1-294c-455c-9653-c0194ff17e5f" containerName="kserve-container"
Sep 09 20:12:17.329196 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:12:17.328882    2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="63e5f7d1-294c-455c-9653-c0194ff17e5f" containerName="kserve-container"
Sep 09 20:12:17.329196 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:12:17.328905    2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="63e5f7d1-294c-455c-9653-c0194ff17e5f" containerName="kube-rbac-proxy"
Sep 09 20:12:17.329196 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:12:17.328915    2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="63e5f7d1-294c-455c-9653-c0194ff17e5f" containerName="kube-rbac-proxy"
Sep 09 20:12:17.329196 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:12:17.328994    2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="e6310063-1870-4f06-9116-a54299c215b2" containerName="kserve-container"
Sep 09 20:12:17.329196 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:12:17.329008    2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="acc34d61-cf66-488a-8229-faeda1df33b6" containerName="model-chainer-raw-61cac"
Sep 09 20:12:17.329196 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:12:17.329018    2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="63e5f7d1-294c-455c-9653-c0194ff17e5f" containerName="kube-rbac-proxy"
Sep 09 20:12:17.329196 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:12:17.329026    2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="63e5f7d1-294c-455c-9653-c0194ff17e5f" containerName="kserve-container"
Sep 09 20:12:17.329196 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:12:17.329035    2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="e6310063-1870-4f06-9116-a54299c215b2" containerName="kube-rbac-proxy"
Sep 09 20:12:17.332095 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:12:17.332075    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/model-chainer-raw-hpa-5164f-679f86dddf-dj5jn"
Sep 09 20:12:17.334183 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:12:17.334165    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"model-chainer-raw-hpa-5164f-serving-cert\""
Sep 09 20:12:17.334270 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:12:17.334181    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"model-chainer-raw-hpa-5164f-kube-rbac-proxy-sar-config\""
Sep 09 20:12:17.340080 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:12:17.340057    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/model-chainer-raw-hpa-5164f-679f86dddf-dj5jn"]
Sep 09 20:12:17.368171 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:12:17.368140    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ff9e45d8-005c-462e-a62b-834d0122de1a-openshift-service-ca-bundle\") pod \"model-chainer-raw-hpa-5164f-679f86dddf-dj5jn\" (UID: \"ff9e45d8-005c-462e-a62b-834d0122de1a\") " pod="kserve-ci-e2e-test/model-chainer-raw-hpa-5164f-679f86dddf-dj5jn"
Sep 09 20:12:17.368332 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:12:17.368213    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ff9e45d8-005c-462e-a62b-834d0122de1a-proxy-tls\") pod \"model-chainer-raw-hpa-5164f-679f86dddf-dj5jn\" (UID: \"ff9e45d8-005c-462e-a62b-834d0122de1a\") " pod="kserve-ci-e2e-test/model-chainer-raw-hpa-5164f-679f86dddf-dj5jn"
Sep 09 20:12:17.469239 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:12:17.469201    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ff9e45d8-005c-462e-a62b-834d0122de1a-openshift-service-ca-bundle\") pod \"model-chainer-raw-hpa-5164f-679f86dddf-dj5jn\" (UID: \"ff9e45d8-005c-462e-a62b-834d0122de1a\") " pod="kserve-ci-e2e-test/model-chainer-raw-hpa-5164f-679f86dddf-dj5jn"
Sep 09 20:12:17.469411 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:12:17.469257    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ff9e45d8-005c-462e-a62b-834d0122de1a-proxy-tls\") pod \"model-chainer-raw-hpa-5164f-679f86dddf-dj5jn\" (UID: \"ff9e45d8-005c-462e-a62b-834d0122de1a\") " pod="kserve-ci-e2e-test/model-chainer-raw-hpa-5164f-679f86dddf-dj5jn"
Sep 09 20:12:17.469411 ip-10-0-136-183 kubenswrapper[2574]: E0909 20:12:17.469348    2574 secret.go:189] Couldn't get secret kserve-ci-e2e-test/model-chainer-raw-hpa-5164f-serving-cert: secret "model-chainer-raw-hpa-5164f-serving-cert" not found
Sep 09 20:12:17.469411 ip-10-0-136-183 kubenswrapper[2574]: E0909 20:12:17.469411    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ff9e45d8-005c-462e-a62b-834d0122de1a-proxy-tls podName:ff9e45d8-005c-462e-a62b-834d0122de1a nodeName:}" failed. No retries permitted until 2026-09-09 20:12:17.969394494 +0000 UTC m=+1022.015647733 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/ff9e45d8-005c-462e-a62b-834d0122de1a-proxy-tls") pod "model-chainer-raw-hpa-5164f-679f86dddf-dj5jn" (UID: "ff9e45d8-005c-462e-a62b-834d0122de1a") : secret "model-chainer-raw-hpa-5164f-serving-cert" not found
Sep 09 20:12:17.469836 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:12:17.469788    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ff9e45d8-005c-462e-a62b-834d0122de1a-openshift-service-ca-bundle\") pod \"model-chainer-raw-hpa-5164f-679f86dddf-dj5jn\" (UID: \"ff9e45d8-005c-462e-a62b-834d0122de1a\") " pod="kserve-ci-e2e-test/model-chainer-raw-hpa-5164f-679f86dddf-dj5jn"
Sep 09 20:12:17.973600 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:12:17.973557    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ff9e45d8-005c-462e-a62b-834d0122de1a-proxy-tls\") pod \"model-chainer-raw-hpa-5164f-679f86dddf-dj5jn\" (UID: \"ff9e45d8-005c-462e-a62b-834d0122de1a\") " pod="kserve-ci-e2e-test/model-chainer-raw-hpa-5164f-679f86dddf-dj5jn"
Sep 09 20:12:17.976062 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:12:17.976033    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ff9e45d8-005c-462e-a62b-834d0122de1a-proxy-tls\") pod \"model-chainer-raw-hpa-5164f-679f86dddf-dj5jn\" (UID: \"ff9e45d8-005c-462e-a62b-834d0122de1a\") " pod="kserve-ci-e2e-test/model-chainer-raw-hpa-5164f-679f86dddf-dj5jn"
Sep 09 20:12:18.242402 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:12:18.242304    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/model-chainer-raw-hpa-5164f-679f86dddf-dj5jn"
Sep 09 20:12:18.514403 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:12:18.514318    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/model-chainer-raw-hpa-5164f-679f86dddf-dj5jn"]
Sep 09 20:12:18.517550 ip-10-0-136-183 kubenswrapper[2574]: W0909 20:12:18.517524    2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podff9e45d8_005c_462e_a62b_834d0122de1a.slice/crio-fdda40f09bc8564de3c99609e9d2ed2fc282421aa8b3e8e7da7860e173d6b2e0 WatchSource:0}: Error finding container fdda40f09bc8564de3c99609e9d2ed2fc282421aa8b3e8e7da7860e173d6b2e0: Status 404 returned error can't find the container with id fdda40f09bc8564de3c99609e9d2ed2fc282421aa8b3e8e7da7860e173d6b2e0
Sep 09 20:12:19.445734 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:12:19.445699    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/model-chainer-raw-hpa-5164f-679f86dddf-dj5jn" event={"ID":"ff9e45d8-005c-462e-a62b-834d0122de1a","Type":"ContainerStarted","Data":"63cc42afdf5b8a768b0b4c36426ea6508d45f917762e9b89d2a8be5a378dc600"}
Sep 09 20:12:19.445950 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:12:19.445751    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/model-chainer-raw-hpa-5164f-679f86dddf-dj5jn" event={"ID":"ff9e45d8-005c-462e-a62b-834d0122de1a","Type":"ContainerStarted","Data":"fdda40f09bc8564de3c99609e9d2ed2fc282421aa8b3e8e7da7860e173d6b2e0"}
Sep 09 20:12:19.445950 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:12:19.445871    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/model-chainer-raw-hpa-5164f-679f86dddf-dj5jn"
Sep 09 20:12:19.462174 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:12:19.462124    2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/model-chainer-raw-hpa-5164f-679f86dddf-dj5jn" podStartSLOduration=2.462107099 podStartE2EDuration="2.462107099s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 20:12:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:12:19.461604204 +0000 UTC m=+1023.507857464" watchObservedRunningTime="2026-09-09 20:12:19.462107099 +0000 UTC m=+1023.508360352"
Sep 09 20:12:25.456425 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:12:25.456395    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/model-chainer-raw-hpa-5164f-679f86dddf-dj5jn"
Sep 09 20:12:27.374687 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:12:27.374646    2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/model-chainer-raw-hpa-5164f-679f86dddf-dj5jn"]
Sep 09 20:12:27.375146 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:12:27.374884    2574 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/model-chainer-raw-hpa-5164f-679f86dddf-dj5jn" podUID="ff9e45d8-005c-462e-a62b-834d0122de1a" containerName="model-chainer-raw-hpa-5164f" containerID="cri-o://63cc42afdf5b8a768b0b4c36426ea6508d45f917762e9b89d2a8be5a378dc600" gracePeriod=30
Sep 09 20:12:27.497021 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:12:27.496984    2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-5164f-predictor-6bbf8b7859-zrjtt"]
Sep 09 20:12:27.497249 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:12:27.497228    2574 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-5164f-predictor-6bbf8b7859-zrjtt" podUID="e13f2ebf-3cdb-4ed4-a94a-d35a1fe1573c" containerName="kserve-container" containerID="cri-o://0da28e1b37cb97e78ce8dc562afba93b83c91176d6395f1cf31f5b143a226a3b" gracePeriod=30
Sep 09 20:12:27.497370 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:12:27.497336    2574 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-5164f-predictor-6bbf8b7859-zrjtt" podUID="e13f2ebf-3cdb-4ed4-a94a-d35a1fe1573c" containerName="kube-rbac-proxy" containerID="cri-o://1d69bb651bb4d6b325dcf5dbfe50924cd0b9eb19d2a0d78ccf0bb061f813b70e" gracePeriod=30
Sep 09 20:12:27.581873 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:12:27.581826    2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/message-dumper-raw-96f8b-predictor-7779456f4f-p56fj"]
Sep 09 20:12:27.589351 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:12:27.589313    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/message-dumper-raw-96f8b-predictor-7779456f4f-p56fj"
Sep 09 20:12:27.591290 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:12:27.591262    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"message-dumper-raw-96f8b-predictor-serving-cert\""
Sep 09 20:12:27.591443 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:12:27.591428    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"message-dumper-raw-96f8b-kube-rbac-proxy-sar-config\""
Sep 09 20:12:27.593843 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:12:27.593817    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/message-dumper-raw-96f8b-predictor-7779456f4f-p56fj"]
Sep 09 20:12:27.652431 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:12:27.652350    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qjvx4\" (UniqueName: \"kubernetes.io/projected/47632e59-c409-48e0-8d16-e23fa1f15697-kube-api-access-qjvx4\") pod \"message-dumper-raw-96f8b-predictor-7779456f4f-p56fj\" (UID: \"47632e59-c409-48e0-8d16-e23fa1f15697\") " pod="kserve-ci-e2e-test/message-dumper-raw-96f8b-predictor-7779456f4f-p56fj"
Sep 09 20:12:27.652431 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:12:27.652390    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/47632e59-c409-48e0-8d16-e23fa1f15697-kube-rbac-proxy-sar-config\") pod \"message-dumper-raw-96f8b-predictor-7779456f4f-p56fj\" (UID: \"47632e59-c409-48e0-8d16-e23fa1f15697\") " pod="kserve-ci-e2e-test/message-dumper-raw-96f8b-predictor-7779456f4f-p56fj"
Sep 09 20:12:27.652626 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:12:27.652494    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/47632e59-c409-48e0-8d16-e23fa1f15697-proxy-tls\") pod \"message-dumper-raw-96f8b-predictor-7779456f4f-p56fj\" (UID: \"47632e59-c409-48e0-8d16-e23fa1f15697\") " pod="kserve-ci-e2e-test/message-dumper-raw-96f8b-predictor-7779456f4f-p56fj"
Sep 09 20:12:27.753714 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:12:27.753669    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/47632e59-c409-48e0-8d16-e23fa1f15697-proxy-tls\") pod \"message-dumper-raw-96f8b-predictor-7779456f4f-p56fj\" (UID: \"47632e59-c409-48e0-8d16-e23fa1f15697\") " pod="kserve-ci-e2e-test/message-dumper-raw-96f8b-predictor-7779456f4f-p56fj"
Sep 09 20:12:27.753938 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:12:27.753740    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qjvx4\" (UniqueName: \"kubernetes.io/projected/47632e59-c409-48e0-8d16-e23fa1f15697-kube-api-access-qjvx4\") pod \"message-dumper-raw-96f8b-predictor-7779456f4f-p56fj\" (UID: \"47632e59-c409-48e0-8d16-e23fa1f15697\") " pod="kserve-ci-e2e-test/message-dumper-raw-96f8b-predictor-7779456f4f-p56fj"
Sep 09 20:12:27.753938 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:12:27.753772    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/47632e59-c409-48e0-8d16-e23fa1f15697-kube-rbac-proxy-sar-config\") pod \"message-dumper-raw-96f8b-predictor-7779456f4f-p56fj\" (UID: \"47632e59-c409-48e0-8d16-e23fa1f15697\") " pod="kserve-ci-e2e-test/message-dumper-raw-96f8b-predictor-7779456f4f-p56fj"
Sep 09 20:12:27.753938 ip-10-0-136-183 kubenswrapper[2574]: E0909 20:12:27.753869    2574 secret.go:189] Couldn't get secret kserve-ci-e2e-test/message-dumper-raw-96f8b-predictor-serving-cert: secret "message-dumper-raw-96f8b-predictor-serving-cert" not found
Sep 09 20:12:27.754079 ip-10-0-136-183 kubenswrapper[2574]: E0909 20:12:27.753942    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/47632e59-c409-48e0-8d16-e23fa1f15697-proxy-tls podName:47632e59-c409-48e0-8d16-e23fa1f15697 nodeName:}" failed. No retries permitted until 2026-09-09 20:12:28.253919221 +0000 UTC m=+1032.300172480 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/47632e59-c409-48e0-8d16-e23fa1f15697-proxy-tls") pod "message-dumper-raw-96f8b-predictor-7779456f4f-p56fj" (UID: "47632e59-c409-48e0-8d16-e23fa1f15697") : secret "message-dumper-raw-96f8b-predictor-serving-cert" not found
Sep 09 20:12:27.754452 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:12:27.754433    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/47632e59-c409-48e0-8d16-e23fa1f15697-kube-rbac-proxy-sar-config\") pod \"message-dumper-raw-96f8b-predictor-7779456f4f-p56fj\" (UID: \"47632e59-c409-48e0-8d16-e23fa1f15697\") " pod="kserve-ci-e2e-test/message-dumper-raw-96f8b-predictor-7779456f4f-p56fj"
Sep 09 20:12:27.762180 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:12:27.762150    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qjvx4\" (UniqueName: \"kubernetes.io/projected/47632e59-c409-48e0-8d16-e23fa1f15697-kube-api-access-qjvx4\") pod \"message-dumper-raw-96f8b-predictor-7779456f4f-p56fj\" (UID: \"47632e59-c409-48e0-8d16-e23fa1f15697\") " pod="kserve-ci-e2e-test/message-dumper-raw-96f8b-predictor-7779456f4f-p56fj"
Sep 09 20:12:28.257859 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:12:28.257790    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/47632e59-c409-48e0-8d16-e23fa1f15697-proxy-tls\") pod \"message-dumper-raw-96f8b-predictor-7779456f4f-p56fj\" (UID: \"47632e59-c409-48e0-8d16-e23fa1f15697\") " pod="kserve-ci-e2e-test/message-dumper-raw-96f8b-predictor-7779456f4f-p56fj"
Sep 09 20:12:28.260324 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:12:28.260293    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/47632e59-c409-48e0-8d16-e23fa1f15697-proxy-tls\") pod \"message-dumper-raw-96f8b-predictor-7779456f4f-p56fj\" (UID: \"47632e59-c409-48e0-8d16-e23fa1f15697\") " pod="kserve-ci-e2e-test/message-dumper-raw-96f8b-predictor-7779456f4f-p56fj"
Sep 09 20:12:28.484195 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:12:28.484157    2574 generic.go:356] "Generic (PLEG): container finished" podID="e13f2ebf-3cdb-4ed4-a94a-d35a1fe1573c" containerID="1d69bb651bb4d6b325dcf5dbfe50924cd0b9eb19d2a0d78ccf0bb061f813b70e" exitCode=2
Sep 09 20:12:28.484566 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:12:28.484213    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-5164f-predictor-6bbf8b7859-zrjtt" event={"ID":"e13f2ebf-3cdb-4ed4-a94a-d35a1fe1573c","Type":"ContainerDied","Data":"1d69bb651bb4d6b325dcf5dbfe50924cd0b9eb19d2a0d78ccf0bb061f813b70e"}
Sep 09 20:12:28.500753 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:12:28.500715    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/message-dumper-raw-96f8b-predictor-7779456f4f-p56fj"
Sep 09 20:12:28.772525 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:12:28.772305    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/message-dumper-raw-96f8b-predictor-7779456f4f-p56fj"]
Sep 09 20:12:28.775610 ip-10-0-136-183 kubenswrapper[2574]: W0909 20:12:28.775582    2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod47632e59_c409_48e0_8d16_e23fa1f15697.slice/crio-e5f03d4d8c4b1218da4a7d7b44c288e0d1f3f5aaec6a6b7aa23ac6af1baabfc5 WatchSource:0}: Error finding container e5f03d4d8c4b1218da4a7d7b44c288e0d1f3f5aaec6a6b7aa23ac6af1baabfc5: Status 404 returned error can't find the container with id e5f03d4d8c4b1218da4a7d7b44c288e0d1f3f5aaec6a6b7aa23ac6af1baabfc5
Sep 09 20:12:28.778136 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:12:28.778119    2574 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 09 20:12:29.489751 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:12:29.489721    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/message-dumper-raw-96f8b-predictor-7779456f4f-p56fj" event={"ID":"47632e59-c409-48e0-8d16-e23fa1f15697","Type":"ContainerStarted","Data":"e5f03d4d8c4b1218da4a7d7b44c288e0d1f3f5aaec6a6b7aa23ac6af1baabfc5"}
Sep 09 20:12:30.081984 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:12:30.081947    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-5164f-predictor-6bbf8b7859-zrjtt" podUID="e13f2ebf-3cdb-4ed4-a94a-d35a1fe1573c" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.28:8643/healthz\": dial tcp 10.133.0.28:8643: connect: connection refused"
Sep 09 20:12:30.086316 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:12:30.086285    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-5164f-predictor-6bbf8b7859-zrjtt" podUID="e13f2ebf-3cdb-4ed4-a94a-d35a1fe1573c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.28:8080: connect: connection refused"
Sep 09 20:12:30.455056 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:12:30.455010    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/model-chainer-raw-hpa-5164f-679f86dddf-dj5jn" podUID="ff9e45d8-005c-462e-a62b-834d0122de1a" containerName="model-chainer-raw-hpa-5164f" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 09 20:12:30.495263 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:12:30.495227    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/message-dumper-raw-96f8b-predictor-7779456f4f-p56fj" event={"ID":"47632e59-c409-48e0-8d16-e23fa1f15697","Type":"ContainerStarted","Data":"d53284c00b012a2328471cc047ca803f055fd8d78e255937a5b80d7b3783c143"}
Sep 09 20:12:30.495646 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:12:30.495297    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/message-dumper-raw-96f8b-predictor-7779456f4f-p56fj" event={"ID":"47632e59-c409-48e0-8d16-e23fa1f15697","Type":"ContainerStarted","Data":"43bf15c8e6446b5d3b0382790b4113d55c12946f298db929319a58c6c71e4aa5"}
Sep 09 20:12:30.495646 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:12:30.495330    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/message-dumper-raw-96f8b-predictor-7779456f4f-p56fj"
Sep 09 20:12:30.513972 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:12:30.513915    2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/message-dumper-raw-96f8b-predictor-7779456f4f-p56fj" podStartSLOduration=2.43877183 podStartE2EDuration="3.513896035s" totalImagesPullingTime="1.075124205s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 20:12:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:12:30.512779495 +0000 UTC m=+1034.559032758" watchObservedRunningTime="2026-09-09 20:12:30.513896035 +0000 UTC m=+1034.560149563"
Sep 09 20:12:31.499600 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:12:31.499563    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/message-dumper-raw-96f8b-predictor-7779456f4f-p56fj"
Sep 09 20:12:31.501549 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:12:31.501526    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/message-dumper-raw-96f8b-predictor-7779456f4f-p56fj"
Sep 09 20:12:32.351464 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:12:32.351437    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-5164f-predictor-6bbf8b7859-zrjtt"
Sep 09 20:12:32.491791 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:12:32.491707    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e13f2ebf-3cdb-4ed4-a94a-d35a1fe1573c-kserve-provision-location\") pod \"e13f2ebf-3cdb-4ed4-a94a-d35a1fe1573c\" (UID: \"e13f2ebf-3cdb-4ed4-a94a-d35a1fe1573c\") "
Sep 09 20:12:32.491791 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:12:32.491746    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e13f2ebf-3cdb-4ed4-a94a-d35a1fe1573c-kube-rbac-proxy-sar-config\") pod \"e13f2ebf-3cdb-4ed4-a94a-d35a1fe1573c\" (UID: \"e13f2ebf-3cdb-4ed4-a94a-d35a1fe1573c\") "
Sep 09 20:12:32.491791 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:12:32.491787    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/e13f2ebf-3cdb-4ed4-a94a-d35a1fe1573c-cabundle-cert\") pod \"e13f2ebf-3cdb-4ed4-a94a-d35a1fe1573c\" (UID: \"e13f2ebf-3cdb-4ed4-a94a-d35a1fe1573c\") "
Sep 09 20:12:32.492043 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:12:32.491831    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-z9tsb\" (UniqueName: \"kubernetes.io/projected/e13f2ebf-3cdb-4ed4-a94a-d35a1fe1573c-kube-api-access-z9tsb\") pod \"e13f2ebf-3cdb-4ed4-a94a-d35a1fe1573c\" (UID: \"e13f2ebf-3cdb-4ed4-a94a-d35a1fe1573c\") "
Sep 09 20:12:32.492043 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:12:32.491887    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e13f2ebf-3cdb-4ed4-a94a-d35a1fe1573c-proxy-tls\") pod \"e13f2ebf-3cdb-4ed4-a94a-d35a1fe1573c\" (UID: \"e13f2ebf-3cdb-4ed4-a94a-d35a1fe1573c\") "
Sep 09 20:12:32.492124 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:12:32.492077    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e13f2ebf-3cdb-4ed4-a94a-d35a1fe1573c-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "e13f2ebf-3cdb-4ed4-a94a-d35a1fe1573c" (UID: "e13f2ebf-3cdb-4ed4-a94a-d35a1fe1573c"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 20:12:32.492168 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:12:32.492147    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e13f2ebf-3cdb-4ed4-a94a-d35a1fe1573c-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "e13f2ebf-3cdb-4ed4-a94a-d35a1fe1573c" (UID: "e13f2ebf-3cdb-4ed4-a94a-d35a1fe1573c"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 20:12:32.492237 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:12:32.492210    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e13f2ebf-3cdb-4ed4-a94a-d35a1fe1573c-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "e13f2ebf-3cdb-4ed4-a94a-d35a1fe1573c" (UID: "e13f2ebf-3cdb-4ed4-a94a-d35a1fe1573c"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 20:12:32.494023 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:12:32.494005    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e13f2ebf-3cdb-4ed4-a94a-d35a1fe1573c-kube-api-access-z9tsb" (OuterVolumeSpecName: "kube-api-access-z9tsb") pod "e13f2ebf-3cdb-4ed4-a94a-d35a1fe1573c" (UID: "e13f2ebf-3cdb-4ed4-a94a-d35a1fe1573c"). InnerVolumeSpecName "kube-api-access-z9tsb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 20:12:32.494023 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:12:32.494011    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e13f2ebf-3cdb-4ed4-a94a-d35a1fe1573c-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "e13f2ebf-3cdb-4ed4-a94a-d35a1fe1573c" (UID: "e13f2ebf-3cdb-4ed4-a94a-d35a1fe1573c"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 20:12:32.505903 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:12:32.505880    2574 generic.go:356] "Generic (PLEG): container finished" podID="e13f2ebf-3cdb-4ed4-a94a-d35a1fe1573c" containerID="0da28e1b37cb97e78ce8dc562afba93b83c91176d6395f1cf31f5b143a226a3b" exitCode=0
Sep 09 20:12:32.506262 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:12:32.505934    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-5164f-predictor-6bbf8b7859-zrjtt"
Sep 09 20:12:32.506262 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:12:32.505942    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-5164f-predictor-6bbf8b7859-zrjtt" event={"ID":"e13f2ebf-3cdb-4ed4-a94a-d35a1fe1573c","Type":"ContainerDied","Data":"0da28e1b37cb97e78ce8dc562afba93b83c91176d6395f1cf31f5b143a226a3b"}
Sep 09 20:12:32.506262 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:12:32.506024    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-5164f-predictor-6bbf8b7859-zrjtt" event={"ID":"e13f2ebf-3cdb-4ed4-a94a-d35a1fe1573c","Type":"ContainerDied","Data":"312f5e2b5e66b60281d9d9bae0a41133c5d6e6f8e109fe033d529f6fbc1bb71b"}
Sep 09 20:12:32.506262 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:12:32.506075    2574 scope.go:117] "RemoveContainer" containerID="1d69bb651bb4d6b325dcf5dbfe50924cd0b9eb19d2a0d78ccf0bb061f813b70e"
Sep 09 20:12:32.517504 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:12:32.517483    2574 scope.go:117] "RemoveContainer" containerID="0da28e1b37cb97e78ce8dc562afba93b83c91176d6395f1cf31f5b143a226a3b"
Sep 09 20:12:32.525261 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:12:32.525243    2574 scope.go:117] "RemoveContainer" containerID="37ed6c1a64415dd9ef60b37dbb88d1082cb5fa60254c929a5546994edac6b8cc"
Sep 09 20:12:32.532144 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:12:32.532125    2574 scope.go:117] "RemoveContainer" containerID="1d69bb651bb4d6b325dcf5dbfe50924cd0b9eb19d2a0d78ccf0bb061f813b70e"
Sep 09 20:12:32.532420 ip-10-0-136-183 kubenswrapper[2574]: E0909 20:12:32.532398    2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1d69bb651bb4d6b325dcf5dbfe50924cd0b9eb19d2a0d78ccf0bb061f813b70e\": container with ID starting with 1d69bb651bb4d6b325dcf5dbfe50924cd0b9eb19d2a0d78ccf0bb061f813b70e not found: ID does not exist" containerID="1d69bb651bb4d6b325dcf5dbfe50924cd0b9eb19d2a0d78ccf0bb061f813b70e"
Sep 09 20:12:32.532495 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:12:32.532433    2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1d69bb651bb4d6b325dcf5dbfe50924cd0b9eb19d2a0d78ccf0bb061f813b70e"} err="failed to get container status \"1d69bb651bb4d6b325dcf5dbfe50924cd0b9eb19d2a0d78ccf0bb061f813b70e\": rpc error: code = NotFound desc = could not find container \"1d69bb651bb4d6b325dcf5dbfe50924cd0b9eb19d2a0d78ccf0bb061f813b70e\": container with ID starting with 1d69bb651bb4d6b325dcf5dbfe50924cd0b9eb19d2a0d78ccf0bb061f813b70e not found: ID does not exist"
Sep 09 20:12:32.532495 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:12:32.532457    2574 scope.go:117] "RemoveContainer" containerID="0da28e1b37cb97e78ce8dc562afba93b83c91176d6395f1cf31f5b143a226a3b"
Sep 09 20:12:32.532704 ip-10-0-136-183 kubenswrapper[2574]: E0909 20:12:32.532686    2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0da28e1b37cb97e78ce8dc562afba93b83c91176d6395f1cf31f5b143a226a3b\": container with ID starting with 0da28e1b37cb97e78ce8dc562afba93b83c91176d6395f1cf31f5b143a226a3b not found: ID does not exist" containerID="0da28e1b37cb97e78ce8dc562afba93b83c91176d6395f1cf31f5b143a226a3b"
Sep 09 20:12:32.532778 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:12:32.532756    2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0da28e1b37cb97e78ce8dc562afba93b83c91176d6395f1cf31f5b143a226a3b"} err="failed to get container status \"0da28e1b37cb97e78ce8dc562afba93b83c91176d6395f1cf31f5b143a226a3b\": rpc error: code = NotFound desc = could not find container \"0da28e1b37cb97e78ce8dc562afba93b83c91176d6395f1cf31f5b143a226a3b\": container with ID starting with 0da28e1b37cb97e78ce8dc562afba93b83c91176d6395f1cf31f5b143a226a3b not found: ID does not exist"
Sep 09 20:12:32.532928 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:12:32.532781    2574 scope.go:117] "RemoveContainer" containerID="37ed6c1a64415dd9ef60b37dbb88d1082cb5fa60254c929a5546994edac6b8cc"
Sep 09 20:12:32.533092 ip-10-0-136-183 kubenswrapper[2574]: E0909 20:12:32.533062    2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"37ed6c1a64415dd9ef60b37dbb88d1082cb5fa60254c929a5546994edac6b8cc\": container with ID starting with 37ed6c1a64415dd9ef60b37dbb88d1082cb5fa60254c929a5546994edac6b8cc not found: ID does not exist" containerID="37ed6c1a64415dd9ef60b37dbb88d1082cb5fa60254c929a5546994edac6b8cc"
Sep 09 20:12:32.533182 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:12:32.533091    2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"37ed6c1a64415dd9ef60b37dbb88d1082cb5fa60254c929a5546994edac6b8cc"} err="failed to get container status \"37ed6c1a64415dd9ef60b37dbb88d1082cb5fa60254c929a5546994edac6b8cc\": rpc error: code = NotFound desc = could not find container \"37ed6c1a64415dd9ef60b37dbb88d1082cb5fa60254c929a5546994edac6b8cc\": container with ID starting with 37ed6c1a64415dd9ef60b37dbb88d1082cb5fa60254c929a5546994edac6b8cc not found: ID does not exist"
Sep 09 20:12:32.536457 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:12:32.536437    2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-5164f-predictor-6bbf8b7859-zrjtt"]
Sep 09 20:12:32.540757 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:12:32.540737    2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-5164f-predictor-6bbf8b7859-zrjtt"]
Sep 09 20:12:32.592595 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:12:32.592554    2574 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e13f2ebf-3cdb-4ed4-a94a-d35a1fe1573c-kube-rbac-proxy-sar-config\") on node \"ip-10-0-136-183.ec2.internal\" DevicePath \"\""
Sep 09 20:12:32.592595 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:12:32.592588    2574 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/e13f2ebf-3cdb-4ed4-a94a-d35a1fe1573c-cabundle-cert\") on node \"ip-10-0-136-183.ec2.internal\" DevicePath \"\""
Sep 09 20:12:32.592595 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:12:32.592607    2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z9tsb\" (UniqueName: \"kubernetes.io/projected/e13f2ebf-3cdb-4ed4-a94a-d35a1fe1573c-kube-api-access-z9tsb\") on node \"ip-10-0-136-183.ec2.internal\" DevicePath \"\""
Sep 09 20:12:32.592911 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:12:32.592619    2574 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e13f2ebf-3cdb-4ed4-a94a-d35a1fe1573c-proxy-tls\") on node \"ip-10-0-136-183.ec2.internal\" DevicePath \"\""
Sep 09 20:12:32.592911 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:12:32.592631    2574 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e13f2ebf-3cdb-4ed4-a94a-d35a1fe1573c-kserve-provision-location\") on node \"ip-10-0-136-183.ec2.internal\" DevicePath \"\""
Sep 09 20:12:34.490963 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:12:34.490928    2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e13f2ebf-3cdb-4ed4-a94a-d35a1fe1573c" path="/var/lib/kubelet/pods/e13f2ebf-3cdb-4ed4-a94a-d35a1fe1573c/volumes"
Sep 09 20:12:35.455135 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:12:35.455097    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/model-chainer-raw-hpa-5164f-679f86dddf-dj5jn" podUID="ff9e45d8-005c-462e-a62b-834d0122de1a" containerName="model-chainer-raw-hpa-5164f" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 09 20:12:38.515976 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:12:38.515949    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/message-dumper-raw-96f8b-predictor-7779456f4f-p56fj"
Sep 09 20:12:40.454314 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:12:40.454271    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/model-chainer-raw-hpa-5164f-679f86dddf-dj5jn" podUID="ff9e45d8-005c-462e-a62b-834d0122de1a" containerName="model-chainer-raw-hpa-5164f" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 09 20:12:40.454718 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:12:40.454374    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/model-chainer-raw-hpa-5164f-679f86dddf-dj5jn"
Sep 09 20:12:45.454186 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:12:45.454147    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/model-chainer-raw-hpa-5164f-679f86dddf-dj5jn" podUID="ff9e45d8-005c-462e-a62b-834d0122de1a" containerName="model-chainer-raw-hpa-5164f" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 09 20:12:50.454664 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:12:50.454625    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/model-chainer-raw-hpa-5164f-679f86dddf-dj5jn" podUID="ff9e45d8-005c-462e-a62b-834d0122de1a" containerName="model-chainer-raw-hpa-5164f" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 09 20:12:55.454719 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:12:55.454670    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/model-chainer-raw-hpa-5164f-679f86dddf-dj5jn" podUID="ff9e45d8-005c-462e-a62b-834d0122de1a" containerName="model-chainer-raw-hpa-5164f" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 09 20:12:57.515185 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:12:57.515162    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/model-chainer-raw-hpa-5164f-679f86dddf-dj5jn"
Sep 09 20:12:57.603670 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:12:57.603633    2574 generic.go:356] "Generic (PLEG): container finished" podID="ff9e45d8-005c-462e-a62b-834d0122de1a" containerID="63cc42afdf5b8a768b0b4c36426ea6508d45f917762e9b89d2a8be5a378dc600" exitCode=0
Sep 09 20:12:57.603894 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:12:57.603677    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/model-chainer-raw-hpa-5164f-679f86dddf-dj5jn"
Sep 09 20:12:57.603894 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:12:57.603678    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/model-chainer-raw-hpa-5164f-679f86dddf-dj5jn" event={"ID":"ff9e45d8-005c-462e-a62b-834d0122de1a","Type":"ContainerDied","Data":"63cc42afdf5b8a768b0b4c36426ea6508d45f917762e9b89d2a8be5a378dc600"}
Sep 09 20:12:57.603894 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:12:57.603749    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/model-chainer-raw-hpa-5164f-679f86dddf-dj5jn" event={"ID":"ff9e45d8-005c-462e-a62b-834d0122de1a","Type":"ContainerDied","Data":"fdda40f09bc8564de3c99609e9d2ed2fc282421aa8b3e8e7da7860e173d6b2e0"}
Sep 09 20:12:57.603894 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:12:57.603781    2574 scope.go:117] "RemoveContainer" containerID="63cc42afdf5b8a768b0b4c36426ea6508d45f917762e9b89d2a8be5a378dc600"
Sep 09 20:12:57.611809 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:12:57.611776    2574 scope.go:117] "RemoveContainer" containerID="63cc42afdf5b8a768b0b4c36426ea6508d45f917762e9b89d2a8be5a378dc600"
Sep 09 20:12:57.612130 ip-10-0-136-183 kubenswrapper[2574]: E0909 20:12:57.612110    2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"63cc42afdf5b8a768b0b4c36426ea6508d45f917762e9b89d2a8be5a378dc600\": container with ID starting with 63cc42afdf5b8a768b0b4c36426ea6508d45f917762e9b89d2a8be5a378dc600 not found: ID does not exist" containerID="63cc42afdf5b8a768b0b4c36426ea6508d45f917762e9b89d2a8be5a378dc600"
Sep 09 20:12:57.612185 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:12:57.612141    2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"63cc42afdf5b8a768b0b4c36426ea6508d45f917762e9b89d2a8be5a378dc600"} err="failed to get container status \"63cc42afdf5b8a768b0b4c36426ea6508d45f917762e9b89d2a8be5a378dc600\": rpc error: code = NotFound desc = could not find container \"63cc42afdf5b8a768b0b4c36426ea6508d45f917762e9b89d2a8be5a378dc600\": container with ID starting with 63cc42afdf5b8a768b0b4c36426ea6508d45f917762e9b89d2a8be5a378dc600 not found: ID does not exist"
Sep 09 20:12:57.694279 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:12:57.694187    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ff9e45d8-005c-462e-a62b-834d0122de1a-proxy-tls\") pod \"ff9e45d8-005c-462e-a62b-834d0122de1a\" (UID: \"ff9e45d8-005c-462e-a62b-834d0122de1a\") "
Sep 09 20:12:57.694279 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:12:57.694274    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ff9e45d8-005c-462e-a62b-834d0122de1a-openshift-service-ca-bundle\") pod \"ff9e45d8-005c-462e-a62b-834d0122de1a\" (UID: \"ff9e45d8-005c-462e-a62b-834d0122de1a\") "
Sep 09 20:12:57.694581 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:12:57.694559    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ff9e45d8-005c-462e-a62b-834d0122de1a-openshift-service-ca-bundle" (OuterVolumeSpecName: "openshift-service-ca-bundle") pod "ff9e45d8-005c-462e-a62b-834d0122de1a" (UID: "ff9e45d8-005c-462e-a62b-834d0122de1a"). InnerVolumeSpecName "openshift-service-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 20:12:57.696232 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:12:57.696205    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ff9e45d8-005c-462e-a62b-834d0122de1a-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "ff9e45d8-005c-462e-a62b-834d0122de1a" (UID: "ff9e45d8-005c-462e-a62b-834d0122de1a"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 20:12:57.794944 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:12:57.794905    2574 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ff9e45d8-005c-462e-a62b-834d0122de1a-proxy-tls\") on node \"ip-10-0-136-183.ec2.internal\" DevicePath \"\""
Sep 09 20:12:57.794944 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:12:57.794937    2574 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ff9e45d8-005c-462e-a62b-834d0122de1a-openshift-service-ca-bundle\") on node \"ip-10-0-136-183.ec2.internal\" DevicePath \"\""
Sep 09 20:12:57.925811 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:12:57.924731    2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/model-chainer-raw-hpa-5164f-679f86dddf-dj5jn"]
Sep 09 20:12:57.928629 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:12:57.928600    2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/model-chainer-raw-hpa-5164f-679f86dddf-dj5jn"]
Sep 09 20:12:58.490146 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:12:58.490110    2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ff9e45d8-005c-462e-a62b-834d0122de1a" path="/var/lib/kubelet/pods/ff9e45d8-005c-462e-a62b-834d0122de1a/volumes"
Sep 09 20:13:42.556787 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:13:42.556744    2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-secondary-05a1b8-predictor-786c8fcd8f-bm4wv"]
Sep 09 20:13:42.557223 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:13:42.557130    2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ff9e45d8-005c-462e-a62b-834d0122de1a" containerName="model-chainer-raw-hpa-5164f"
Sep 09 20:13:42.557223 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:13:42.557145    2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="ff9e45d8-005c-462e-a62b-834d0122de1a" containerName="model-chainer-raw-hpa-5164f"
Sep 09 20:13:42.557223 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:13:42.557175    2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e13f2ebf-3cdb-4ed4-a94a-d35a1fe1573c" containerName="kserve-container"
Sep 09 20:13:42.557223 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:13:42.557182    2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="e13f2ebf-3cdb-4ed4-a94a-d35a1fe1573c" containerName="kserve-container"
Sep 09 20:13:42.557223 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:13:42.557188    2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e13f2ebf-3cdb-4ed4-a94a-d35a1fe1573c" containerName="kube-rbac-proxy"
Sep 09 20:13:42.557223 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:13:42.557194    2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="e13f2ebf-3cdb-4ed4-a94a-d35a1fe1573c" containerName="kube-rbac-proxy"
Sep 09 20:13:42.557223 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:13:42.557203    2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e13f2ebf-3cdb-4ed4-a94a-d35a1fe1573c" containerName="storage-initializer"
Sep 09 20:13:42.557223 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:13:42.557209    2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="e13f2ebf-3cdb-4ed4-a94a-d35a1fe1573c" containerName="storage-initializer"
Sep 09 20:13:42.557474 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:13:42.557260    2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="ff9e45d8-005c-462e-a62b-834d0122de1a" containerName="model-chainer-raw-hpa-5164f"
Sep 09 20:13:42.557474 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:13:42.557267    2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="e13f2ebf-3cdb-4ed4-a94a-d35a1fe1573c" containerName="kube-rbac-proxy"
Sep 09 20:13:42.557474 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:13:42.557273    2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="e13f2ebf-3cdb-4ed4-a94a-d35a1fe1573c" containerName="kserve-container"
Sep 09 20:13:42.561260 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:13:42.561244    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-secondary-05a1b8-predictor-786c8fcd8f-bm4wv"
Sep 09 20:13:42.563231 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:13:42.563204    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"invalid-s3-secret-05a1b8\""
Sep 09 20:13:42.563363 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:13:42.563243    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"invalid-s3-sa-05a1b8-dockercfg-r245b\""
Sep 09 20:13:42.563363 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:13:42.563249    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-secondary-05a1b8-kube-rbac-proxy-sar-config\""
Sep 09 20:13:42.563363 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:13:42.563258    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-secondary-05a1b8-predictor-serving-cert\""
Sep 09 20:13:42.563363 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:13:42.563216    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Sep 09 20:13:42.569441 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:13:42.569421    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-secondary-05a1b8-predictor-786c8fcd8f-bm4wv"]
Sep 09 20:13:42.608231 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:13:42.608194    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-plwsv\" (UniqueName: \"kubernetes.io/projected/96f5033a-c15e-4217-9151-84e55514a55f-kube-api-access-plwsv\") pod \"isvc-secondary-05a1b8-predictor-786c8fcd8f-bm4wv\" (UID: \"96f5033a-c15e-4217-9151-84e55514a55f\") " pod="kserve-ci-e2e-test/isvc-secondary-05a1b8-predictor-786c8fcd8f-bm4wv"
Sep 09 20:13:42.608391 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:13:42.608252    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/96f5033a-c15e-4217-9151-84e55514a55f-kserve-provision-location\") pod \"isvc-secondary-05a1b8-predictor-786c8fcd8f-bm4wv\" (UID: \"96f5033a-c15e-4217-9151-84e55514a55f\") " pod="kserve-ci-e2e-test/isvc-secondary-05a1b8-predictor-786c8fcd8f-bm4wv"
Sep 09 20:13:42.608391 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:13:42.608342    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/96f5033a-c15e-4217-9151-84e55514a55f-kube-rbac-proxy-sar-config\") pod \"isvc-secondary-05a1b8-predictor-786c8fcd8f-bm4wv\" (UID: \"96f5033a-c15e-4217-9151-84e55514a55f\") " pod="kserve-ci-e2e-test/isvc-secondary-05a1b8-predictor-786c8fcd8f-bm4wv"
Sep 09 20:13:42.608391 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:13:42.608380    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/96f5033a-c15e-4217-9151-84e55514a55f-proxy-tls\") pod \"isvc-secondary-05a1b8-predictor-786c8fcd8f-bm4wv\" (UID: \"96f5033a-c15e-4217-9151-84e55514a55f\") " pod="kserve-ci-e2e-test/isvc-secondary-05a1b8-predictor-786c8fcd8f-bm4wv"
Sep 09 20:13:42.608533 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:13:42.608514    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/96f5033a-c15e-4217-9151-84e55514a55f-cabundle-cert\") pod \"isvc-secondary-05a1b8-predictor-786c8fcd8f-bm4wv\" (UID: \"96f5033a-c15e-4217-9151-84e55514a55f\") " pod="kserve-ci-e2e-test/isvc-secondary-05a1b8-predictor-786c8fcd8f-bm4wv"
Sep 09 20:13:42.709927 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:13:42.709890    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/96f5033a-c15e-4217-9151-84e55514a55f-cabundle-cert\") pod \"isvc-secondary-05a1b8-predictor-786c8fcd8f-bm4wv\" (UID: \"96f5033a-c15e-4217-9151-84e55514a55f\") " pod="kserve-ci-e2e-test/isvc-secondary-05a1b8-predictor-786c8fcd8f-bm4wv"
Sep 09 20:13:42.710105 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:13:42.709948    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-plwsv\" (UniqueName: \"kubernetes.io/projected/96f5033a-c15e-4217-9151-84e55514a55f-kube-api-access-plwsv\") pod \"isvc-secondary-05a1b8-predictor-786c8fcd8f-bm4wv\" (UID: \"96f5033a-c15e-4217-9151-84e55514a55f\") " pod="kserve-ci-e2e-test/isvc-secondary-05a1b8-predictor-786c8fcd8f-bm4wv"
Sep 09 20:13:42.710105 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:13:42.709981    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/96f5033a-c15e-4217-9151-84e55514a55f-kserve-provision-location\") pod \"isvc-secondary-05a1b8-predictor-786c8fcd8f-bm4wv\" (UID: \"96f5033a-c15e-4217-9151-84e55514a55f\") " pod="kserve-ci-e2e-test/isvc-secondary-05a1b8-predictor-786c8fcd8f-bm4wv"
Sep 09 20:13:42.710105 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:13:42.710022    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/96f5033a-c15e-4217-9151-84e55514a55f-kube-rbac-proxy-sar-config\") pod \"isvc-secondary-05a1b8-predictor-786c8fcd8f-bm4wv\" (UID: \"96f5033a-c15e-4217-9151-84e55514a55f\") " pod="kserve-ci-e2e-test/isvc-secondary-05a1b8-predictor-786c8fcd8f-bm4wv"
Sep 09 20:13:42.710105 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:13:42.710053    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/96f5033a-c15e-4217-9151-84e55514a55f-proxy-tls\") pod \"isvc-secondary-05a1b8-predictor-786c8fcd8f-bm4wv\" (UID: \"96f5033a-c15e-4217-9151-84e55514a55f\") " pod="kserve-ci-e2e-test/isvc-secondary-05a1b8-predictor-786c8fcd8f-bm4wv"
Sep 09 20:13:42.710299 ip-10-0-136-183 kubenswrapper[2574]: E0909 20:13:42.710178    2574 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-secondary-05a1b8-predictor-serving-cert: secret "isvc-secondary-05a1b8-predictor-serving-cert" not found
Sep 09 20:13:42.710299 ip-10-0-136-183 kubenswrapper[2574]: E0909 20:13:42.710250    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/96f5033a-c15e-4217-9151-84e55514a55f-proxy-tls podName:96f5033a-c15e-4217-9151-84e55514a55f nodeName:}" failed. No retries permitted until 2026-09-09 20:13:43.210228096 +0000 UTC m=+1107.256481337 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/96f5033a-c15e-4217-9151-84e55514a55f-proxy-tls") pod "isvc-secondary-05a1b8-predictor-786c8fcd8f-bm4wv" (UID: "96f5033a-c15e-4217-9151-84e55514a55f") : secret "isvc-secondary-05a1b8-predictor-serving-cert" not found
Sep 09 20:13:42.710494 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:13:42.710470    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/96f5033a-c15e-4217-9151-84e55514a55f-kserve-provision-location\") pod \"isvc-secondary-05a1b8-predictor-786c8fcd8f-bm4wv\" (UID: \"96f5033a-c15e-4217-9151-84e55514a55f\") " pod="kserve-ci-e2e-test/isvc-secondary-05a1b8-predictor-786c8fcd8f-bm4wv"
Sep 09 20:13:42.710709 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:13:42.710688    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/96f5033a-c15e-4217-9151-84e55514a55f-cabundle-cert\") pod \"isvc-secondary-05a1b8-predictor-786c8fcd8f-bm4wv\" (UID: \"96f5033a-c15e-4217-9151-84e55514a55f\") " pod="kserve-ci-e2e-test/isvc-secondary-05a1b8-predictor-786c8fcd8f-bm4wv"
Sep 09 20:13:42.710858 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:13:42.710754    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/96f5033a-c15e-4217-9151-84e55514a55f-kube-rbac-proxy-sar-config\") pod \"isvc-secondary-05a1b8-predictor-786c8fcd8f-bm4wv\" (UID: \"96f5033a-c15e-4217-9151-84e55514a55f\") " pod="kserve-ci-e2e-test/isvc-secondary-05a1b8-predictor-786c8fcd8f-bm4wv"
Sep 09 20:13:42.720205 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:13:42.720177    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-plwsv\" (UniqueName: \"kubernetes.io/projected/96f5033a-c15e-4217-9151-84e55514a55f-kube-api-access-plwsv\") pod \"isvc-secondary-05a1b8-predictor-786c8fcd8f-bm4wv\" (UID: \"96f5033a-c15e-4217-9151-84e55514a55f\") " pod="kserve-ci-e2e-test/isvc-secondary-05a1b8-predictor-786c8fcd8f-bm4wv"
Sep 09 20:13:43.215084 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:13:43.215040    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/96f5033a-c15e-4217-9151-84e55514a55f-proxy-tls\") pod \"isvc-secondary-05a1b8-predictor-786c8fcd8f-bm4wv\" (UID: \"96f5033a-c15e-4217-9151-84e55514a55f\") " pod="kserve-ci-e2e-test/isvc-secondary-05a1b8-predictor-786c8fcd8f-bm4wv"
Sep 09 20:13:43.217465 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:13:43.217445    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/96f5033a-c15e-4217-9151-84e55514a55f-proxy-tls\") pod \"isvc-secondary-05a1b8-predictor-786c8fcd8f-bm4wv\" (UID: \"96f5033a-c15e-4217-9151-84e55514a55f\") " pod="kserve-ci-e2e-test/isvc-secondary-05a1b8-predictor-786c8fcd8f-bm4wv"
Sep 09 20:13:43.471585 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:13:43.471494    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-secondary-05a1b8-predictor-786c8fcd8f-bm4wv"
Sep 09 20:13:43.745653 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:13:43.745624    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-secondary-05a1b8-predictor-786c8fcd8f-bm4wv"]
Sep 09 20:13:43.748127 ip-10-0-136-183 kubenswrapper[2574]: W0909 20:13:43.748089    2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod96f5033a_c15e_4217_9151_84e55514a55f.slice/crio-ac8c141b15da5a450a9be4a91e925bfbfb599d36a49c108ca914711a36c7c4ab WatchSource:0}: Error finding container ac8c141b15da5a450a9be4a91e925bfbfb599d36a49c108ca914711a36c7c4ab: Status 404 returned error can't find the container with id ac8c141b15da5a450a9be4a91e925bfbfb599d36a49c108ca914711a36c7c4ab
Sep 09 20:13:43.786440 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:13:43.786408    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-secondary-05a1b8-predictor-786c8fcd8f-bm4wv" event={"ID":"96f5033a-c15e-4217-9151-84e55514a55f","Type":"ContainerStarted","Data":"ac8c141b15da5a450a9be4a91e925bfbfb599d36a49c108ca914711a36c7c4ab"}
Sep 09 20:13:44.792040 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:13:44.792005    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-secondary-05a1b8-predictor-786c8fcd8f-bm4wv" event={"ID":"96f5033a-c15e-4217-9151-84e55514a55f","Type":"ContainerStarted","Data":"2c3aad481254374b16bb5ad4e2bf924a49c8a5ef7e9cc87ba878fff0f5488089"}
Sep 09 20:13:46.800982 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:13:46.800951    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-secondary-05a1b8-predictor-786c8fcd8f-bm4wv_96f5033a-c15e-4217-9151-84e55514a55f/storage-initializer/0.log"
Sep 09 20:13:46.801395 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:13:46.800990    2574 generic.go:356] "Generic (PLEG): container finished" podID="96f5033a-c15e-4217-9151-84e55514a55f" containerID="2c3aad481254374b16bb5ad4e2bf924a49c8a5ef7e9cc87ba878fff0f5488089" exitCode=1
Sep 09 20:13:46.801395 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:13:46.801041    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-secondary-05a1b8-predictor-786c8fcd8f-bm4wv" event={"ID":"96f5033a-c15e-4217-9151-84e55514a55f","Type":"ContainerDied","Data":"2c3aad481254374b16bb5ad4e2bf924a49c8a5ef7e9cc87ba878fff0f5488089"}
Sep 09 20:13:47.806664 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:13:47.806637    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-secondary-05a1b8-predictor-786c8fcd8f-bm4wv_96f5033a-c15e-4217-9151-84e55514a55f/storage-initializer/0.log"
Sep 09 20:13:47.807086 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:13:47.806684    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-secondary-05a1b8-predictor-786c8fcd8f-bm4wv" event={"ID":"96f5033a-c15e-4217-9151-84e55514a55f","Type":"ContainerStarted","Data":"8dbd8710d256cc8f74c158d4eae306c97c87c92352a935ab92dd03b4251dcf9e"}
Sep 09 20:13:53.832205 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:13:53.832125    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-secondary-05a1b8-predictor-786c8fcd8f-bm4wv_96f5033a-c15e-4217-9151-84e55514a55f/storage-initializer/1.log"
Sep 09 20:13:53.832628 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:13:53.832547    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-secondary-05a1b8-predictor-786c8fcd8f-bm4wv_96f5033a-c15e-4217-9151-84e55514a55f/storage-initializer/0.log"
Sep 09 20:13:53.832628 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:13:53.832584    2574 generic.go:356] "Generic (PLEG): container finished" podID="96f5033a-c15e-4217-9151-84e55514a55f" containerID="8dbd8710d256cc8f74c158d4eae306c97c87c92352a935ab92dd03b4251dcf9e" exitCode=1
Sep 09 20:13:53.832628 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:13:53.832621    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-secondary-05a1b8-predictor-786c8fcd8f-bm4wv" event={"ID":"96f5033a-c15e-4217-9151-84e55514a55f","Type":"ContainerDied","Data":"8dbd8710d256cc8f74c158d4eae306c97c87c92352a935ab92dd03b4251dcf9e"}
Sep 09 20:13:53.832839 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:13:53.832693    2574 scope.go:117] "RemoveContainer" containerID="2c3aad481254374b16bb5ad4e2bf924a49c8a5ef7e9cc87ba878fff0f5488089"
Sep 09 20:13:53.833043 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:13:53.833025    2574 scope.go:117] "RemoveContainer" containerID="2c3aad481254374b16bb5ad4e2bf924a49c8a5ef7e9cc87ba878fff0f5488089"
Sep 09 20:13:53.843233 ip-10-0-136-183 kubenswrapper[2574]: E0909 20:13:53.843209    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"storage-initializer\" with CrashLoopBackOff: \"back-off 10s restarting failed container=storage-initializer pod=isvc-secondary-05a1b8-predictor-786c8fcd8f-bm4wv_kserve-ci-e2e-test(96f5033a-c15e-4217-9151-84e55514a55f)\"" pod="kserve-ci-e2e-test/isvc-secondary-05a1b8-predictor-786c8fcd8f-bm4wv" podUID="96f5033a-c15e-4217-9151-84e55514a55f"
Sep 09 20:13:53.843436 ip-10-0-136-183 kubenswrapper[2574]: E0909 20:13:53.843413    2574 log.go:32] "RemoveContainer from runtime service failed" err="rpc error: code = Unknown desc = failed to delete container k8s_storage-initializer_isvc-secondary-05a1b8-predictor-786c8fcd8f-bm4wv_kserve-ci-e2e-test_96f5033a-c15e-4217-9151-84e55514a55f_0 in pod sandbox ac8c141b15da5a450a9be4a91e925bfbfb599d36a49c108ca914711a36c7c4ab from index: no such id: '2c3aad481254374b16bb5ad4e2bf924a49c8a5ef7e9cc87ba878fff0f5488089'" containerID="2c3aad481254374b16bb5ad4e2bf924a49c8a5ef7e9cc87ba878fff0f5488089"
Sep 09 20:13:53.843497 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:13:53.843445    2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2c3aad481254374b16bb5ad4e2bf924a49c8a5ef7e9cc87ba878fff0f5488089"} err="rpc error: code = Unknown desc = failed to delete container k8s_storage-initializer_isvc-secondary-05a1b8-predictor-786c8fcd8f-bm4wv_kserve-ci-e2e-test_96f5033a-c15e-4217-9151-84e55514a55f_0 in pod sandbox ac8c141b15da5a450a9be4a91e925bfbfb599d36a49c108ca914711a36c7c4ab from index: no such id: '2c3aad481254374b16bb5ad4e2bf924a49c8a5ef7e9cc87ba878fff0f5488089'"
Sep 09 20:13:54.838505 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:13:54.838476    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-secondary-05a1b8-predictor-786c8fcd8f-bm4wv_96f5033a-c15e-4217-9151-84e55514a55f/storage-initializer/1.log"
Sep 09 20:13:58.579166 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:13:58.579132    2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-secondary-05a1b8-predictor-786c8fcd8f-bm4wv"]
Sep 09 20:13:58.702733 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:13:58.702708    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-secondary-05a1b8-predictor-786c8fcd8f-bm4wv_96f5033a-c15e-4217-9151-84e55514a55f/storage-initializer/1.log"
Sep 09 20:13:58.706445 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:13:58.706425    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-secondary-05a1b8-predictor-786c8fcd8f-bm4wv"
Sep 09 20:13:58.836660 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:13:58.836578    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/96f5033a-c15e-4217-9151-84e55514a55f-kserve-provision-location\") pod \"96f5033a-c15e-4217-9151-84e55514a55f\" (UID: \"96f5033a-c15e-4217-9151-84e55514a55f\") "
Sep 09 20:13:58.836660 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:13:58.836625    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/96f5033a-c15e-4217-9151-84e55514a55f-kube-rbac-proxy-sar-config\") pod \"96f5033a-c15e-4217-9151-84e55514a55f\" (UID: \"96f5033a-c15e-4217-9151-84e55514a55f\") "
Sep 09 20:13:58.836660 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:13:58.836647    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/96f5033a-c15e-4217-9151-84e55514a55f-proxy-tls\") pod \"96f5033a-c15e-4217-9151-84e55514a55f\" (UID: \"96f5033a-c15e-4217-9151-84e55514a55f\") "
Sep 09 20:13:58.836975 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:13:58.836692    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/96f5033a-c15e-4217-9151-84e55514a55f-cabundle-cert\") pod \"96f5033a-c15e-4217-9151-84e55514a55f\" (UID: \"96f5033a-c15e-4217-9151-84e55514a55f\") "
Sep 09 20:13:58.836975 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:13:58.836710    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-plwsv\" (UniqueName: \"kubernetes.io/projected/96f5033a-c15e-4217-9151-84e55514a55f-kube-api-access-plwsv\") pod \"96f5033a-c15e-4217-9151-84e55514a55f\" (UID: \"96f5033a-c15e-4217-9151-84e55514a55f\") "
Sep 09 20:13:58.836975 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:13:58.836880    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/96f5033a-c15e-4217-9151-84e55514a55f-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "96f5033a-c15e-4217-9151-84e55514a55f" (UID: "96f5033a-c15e-4217-9151-84e55514a55f"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 20:13:58.837139 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:13:58.837049    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/96f5033a-c15e-4217-9151-84e55514a55f-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "96f5033a-c15e-4217-9151-84e55514a55f" (UID: "96f5033a-c15e-4217-9151-84e55514a55f"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 20:13:58.837139 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:13:58.837102    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/96f5033a-c15e-4217-9151-84e55514a55f-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "96f5033a-c15e-4217-9151-84e55514a55f" (UID: "96f5033a-c15e-4217-9151-84e55514a55f"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 20:13:58.838868 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:13:58.838839    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/96f5033a-c15e-4217-9151-84e55514a55f-kube-api-access-plwsv" (OuterVolumeSpecName: "kube-api-access-plwsv") pod "96f5033a-c15e-4217-9151-84e55514a55f" (UID: "96f5033a-c15e-4217-9151-84e55514a55f"). InnerVolumeSpecName "kube-api-access-plwsv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 20:13:58.838993 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:13:58.838975    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/96f5033a-c15e-4217-9151-84e55514a55f-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "96f5033a-c15e-4217-9151-84e55514a55f" (UID: "96f5033a-c15e-4217-9151-84e55514a55f"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 20:13:58.855090 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:13:58.855071    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-secondary-05a1b8-predictor-786c8fcd8f-bm4wv_96f5033a-c15e-4217-9151-84e55514a55f/storage-initializer/1.log"
Sep 09 20:13:58.855200 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:13:58.855169    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-secondary-05a1b8-predictor-786c8fcd8f-bm4wv"
Sep 09 20:13:58.855200 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:13:58.855168    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-secondary-05a1b8-predictor-786c8fcd8f-bm4wv" event={"ID":"96f5033a-c15e-4217-9151-84e55514a55f","Type":"ContainerDied","Data":"ac8c141b15da5a450a9be4a91e925bfbfb599d36a49c108ca914711a36c7c4ab"}
Sep 09 20:13:58.855275 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:13:58.855241    2574 scope.go:117] "RemoveContainer" containerID="8dbd8710d256cc8f74c158d4eae306c97c87c92352a935ab92dd03b4251dcf9e"
Sep 09 20:13:58.893926 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:13:58.893899    2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-secondary-05a1b8-predictor-786c8fcd8f-bm4wv"]
Sep 09 20:13:58.897811 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:13:58.897776    2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-secondary-05a1b8-predictor-786c8fcd8f-bm4wv"]
Sep 09 20:13:58.937748 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:13:58.937709    2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-plwsv\" (UniqueName: \"kubernetes.io/projected/96f5033a-c15e-4217-9151-84e55514a55f-kube-api-access-plwsv\") on node \"ip-10-0-136-183.ec2.internal\" DevicePath \"\""
Sep 09 20:13:58.937748 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:13:58.937745    2574 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/96f5033a-c15e-4217-9151-84e55514a55f-kserve-provision-location\") on node \"ip-10-0-136-183.ec2.internal\" DevicePath \"\""
Sep 09 20:13:58.937748 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:13:58.937754    2574 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/96f5033a-c15e-4217-9151-84e55514a55f-kube-rbac-proxy-sar-config\") on node \"ip-10-0-136-183.ec2.internal\" DevicePath \"\""
Sep 09 20:13:58.938008 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:13:58.937764    2574 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/96f5033a-c15e-4217-9151-84e55514a55f-proxy-tls\") on node \"ip-10-0-136-183.ec2.internal\" DevicePath \"\""
Sep 09 20:13:58.938008 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:13:58.937777    2574 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/96f5033a-c15e-4217-9151-84e55514a55f-cabundle-cert\") on node \"ip-10-0-136-183.ec2.internal\" DevicePath \"\""
Sep 09 20:14:00.490043 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:14:00.490007    2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="96f5033a-c15e-4217-9151-84e55514a55f" path="/var/lib/kubelet/pods/96f5033a-c15e-4217-9151-84e55514a55f/volumes"
Sep 09 20:14:03.825909 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:14:03.825870    2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-init-fail-d5facb-predictor-7c8dff7f49-mcnqw"]
Sep 09 20:14:03.826288 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:14:03.826169    2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="96f5033a-c15e-4217-9151-84e55514a55f" containerName="storage-initializer"
Sep 09 20:14:03.826288 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:14:03.826180    2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="96f5033a-c15e-4217-9151-84e55514a55f" containerName="storage-initializer"
Sep 09 20:14:03.826288 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:14:03.826195    2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="96f5033a-c15e-4217-9151-84e55514a55f" containerName="storage-initializer"
Sep 09 20:14:03.826288 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:14:03.826202    2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="96f5033a-c15e-4217-9151-84e55514a55f" containerName="storage-initializer"
Sep 09 20:14:03.826288 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:14:03.826255    2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="96f5033a-c15e-4217-9151-84e55514a55f" containerName="storage-initializer"
Sep 09 20:14:03.826449 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:14:03.826359    2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="96f5033a-c15e-4217-9151-84e55514a55f" containerName="storage-initializer"
Sep 09 20:14:03.830948 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:14:03.830920    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-init-fail-d5facb-predictor-7c8dff7f49-mcnqw"
Sep 09 20:14:03.833042 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:14:03.833004    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-init-fail-d5facb-kube-rbac-proxy-sar-config\""
Sep 09 20:14:03.833042 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:14:03.833022    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Sep 09 20:14:03.833291 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:14:03.833176    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"fail-s3-secret-d5facb\""
Sep 09 20:14:03.833291 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:14:03.833254    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-init-fail-d5facb-predictor-serving-cert\""
Sep 09 20:14:03.833621 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:14:03.833593    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"fail-s3-sa-d5facb-dockercfg-4ks8j\""
Sep 09 20:14:03.839408 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:14:03.839172    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-init-fail-d5facb-predictor-7c8dff7f49-mcnqw"]
Sep 09 20:14:03.978550 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:14:03.978513    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e9441890-355d-4bae-9e45-a31071c9894e-proxy-tls\") pod \"isvc-init-fail-d5facb-predictor-7c8dff7f49-mcnqw\" (UID: \"e9441890-355d-4bae-9e45-a31071c9894e\") " pod="kserve-ci-e2e-test/isvc-init-fail-d5facb-predictor-7c8dff7f49-mcnqw"
Sep 09 20:14:03.978550 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:14:03.978555    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mmxrv\" (UniqueName: \"kubernetes.io/projected/e9441890-355d-4bae-9e45-a31071c9894e-kube-api-access-mmxrv\") pod \"isvc-init-fail-d5facb-predictor-7c8dff7f49-mcnqw\" (UID: \"e9441890-355d-4bae-9e45-a31071c9894e\") " pod="kserve-ci-e2e-test/isvc-init-fail-d5facb-predictor-7c8dff7f49-mcnqw"
Sep 09 20:14:03.978745 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:14:03.978573    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e9441890-355d-4bae-9e45-a31071c9894e-kserve-provision-location\") pod \"isvc-init-fail-d5facb-predictor-7c8dff7f49-mcnqw\" (UID: \"e9441890-355d-4bae-9e45-a31071c9894e\") " pod="kserve-ci-e2e-test/isvc-init-fail-d5facb-predictor-7c8dff7f49-mcnqw"
Sep 09 20:14:03.978745 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:14:03.978698    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e9441890-355d-4bae-9e45-a31071c9894e-kube-rbac-proxy-sar-config\") pod \"isvc-init-fail-d5facb-predictor-7c8dff7f49-mcnqw\" (UID: \"e9441890-355d-4bae-9e45-a31071c9894e\") " pod="kserve-ci-e2e-test/isvc-init-fail-d5facb-predictor-7c8dff7f49-mcnqw"
Sep 09 20:14:03.978745 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:14:03.978731    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/e9441890-355d-4bae-9e45-a31071c9894e-cabundle-cert\") pod \"isvc-init-fail-d5facb-predictor-7c8dff7f49-mcnqw\" (UID: \"e9441890-355d-4bae-9e45-a31071c9894e\") " pod="kserve-ci-e2e-test/isvc-init-fail-d5facb-predictor-7c8dff7f49-mcnqw"
Sep 09 20:14:04.079824 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:14:04.079694    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mmxrv\" (UniqueName: \"kubernetes.io/projected/e9441890-355d-4bae-9e45-a31071c9894e-kube-api-access-mmxrv\") pod \"isvc-init-fail-d5facb-predictor-7c8dff7f49-mcnqw\" (UID: \"e9441890-355d-4bae-9e45-a31071c9894e\") " pod="kserve-ci-e2e-test/isvc-init-fail-d5facb-predictor-7c8dff7f49-mcnqw"
Sep 09 20:14:04.079824 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:14:04.079741    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e9441890-355d-4bae-9e45-a31071c9894e-kserve-provision-location\") pod \"isvc-init-fail-d5facb-predictor-7c8dff7f49-mcnqw\" (UID: \"e9441890-355d-4bae-9e45-a31071c9894e\") " pod="kserve-ci-e2e-test/isvc-init-fail-d5facb-predictor-7c8dff7f49-mcnqw"
Sep 09 20:14:04.079824 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:14:04.079823    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e9441890-355d-4bae-9e45-a31071c9894e-kube-rbac-proxy-sar-config\") pod \"isvc-init-fail-d5facb-predictor-7c8dff7f49-mcnqw\" (UID: \"e9441890-355d-4bae-9e45-a31071c9894e\") " pod="kserve-ci-e2e-test/isvc-init-fail-d5facb-predictor-7c8dff7f49-mcnqw"
Sep 09 20:14:04.080118 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:14:04.079852    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/e9441890-355d-4bae-9e45-a31071c9894e-cabundle-cert\") pod \"isvc-init-fail-d5facb-predictor-7c8dff7f49-mcnqw\" (UID: \"e9441890-355d-4bae-9e45-a31071c9894e\") " pod="kserve-ci-e2e-test/isvc-init-fail-d5facb-predictor-7c8dff7f49-mcnqw"
Sep 09 20:14:04.080118 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:14:04.079896    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e9441890-355d-4bae-9e45-a31071c9894e-proxy-tls\") pod \"isvc-init-fail-d5facb-predictor-7c8dff7f49-mcnqw\" (UID: \"e9441890-355d-4bae-9e45-a31071c9894e\") " pod="kserve-ci-e2e-test/isvc-init-fail-d5facb-predictor-7c8dff7f49-mcnqw"
Sep 09 20:14:04.080249 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:14:04.080228    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e9441890-355d-4bae-9e45-a31071c9894e-kserve-provision-location\") pod \"isvc-init-fail-d5facb-predictor-7c8dff7f49-mcnqw\" (UID: \"e9441890-355d-4bae-9e45-a31071c9894e\") " pod="kserve-ci-e2e-test/isvc-init-fail-d5facb-predictor-7c8dff7f49-mcnqw"
Sep 09 20:14:04.080625 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:14:04.080598    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/e9441890-355d-4bae-9e45-a31071c9894e-cabundle-cert\") pod \"isvc-init-fail-d5facb-predictor-7c8dff7f49-mcnqw\" (UID: \"e9441890-355d-4bae-9e45-a31071c9894e\") " pod="kserve-ci-e2e-test/isvc-init-fail-d5facb-predictor-7c8dff7f49-mcnqw"
Sep 09 20:14:04.080734 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:14:04.080597    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e9441890-355d-4bae-9e45-a31071c9894e-kube-rbac-proxy-sar-config\") pod \"isvc-init-fail-d5facb-predictor-7c8dff7f49-mcnqw\" (UID: \"e9441890-355d-4bae-9e45-a31071c9894e\") " pod="kserve-ci-e2e-test/isvc-init-fail-d5facb-predictor-7c8dff7f49-mcnqw"
Sep 09 20:14:04.082456 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:14:04.082437    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e9441890-355d-4bae-9e45-a31071c9894e-proxy-tls\") pod \"isvc-init-fail-d5facb-predictor-7c8dff7f49-mcnqw\" (UID: \"e9441890-355d-4bae-9e45-a31071c9894e\") " pod="kserve-ci-e2e-test/isvc-init-fail-d5facb-predictor-7c8dff7f49-mcnqw"
Sep 09 20:14:04.089869 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:14:04.089839    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mmxrv\" (UniqueName: \"kubernetes.io/projected/e9441890-355d-4bae-9e45-a31071c9894e-kube-api-access-mmxrv\") pod \"isvc-init-fail-d5facb-predictor-7c8dff7f49-mcnqw\" (UID: \"e9441890-355d-4bae-9e45-a31071c9894e\") " pod="kserve-ci-e2e-test/isvc-init-fail-d5facb-predictor-7c8dff7f49-mcnqw"
Sep 09 20:14:04.143621 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:14:04.143581    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-init-fail-d5facb-predictor-7c8dff7f49-mcnqw"
Sep 09 20:14:04.416038 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:14:04.416009    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-init-fail-d5facb-predictor-7c8dff7f49-mcnqw"]
Sep 09 20:14:04.418523 ip-10-0-136-183 kubenswrapper[2574]: W0909 20:14:04.418496    2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode9441890_355d_4bae_9e45_a31071c9894e.slice/crio-df1452274e0c8bfabb1293863c23ab60d25646106a52dc983ef603b71b05631f WatchSource:0}: Error finding container df1452274e0c8bfabb1293863c23ab60d25646106a52dc983ef603b71b05631f: Status 404 returned error can't find the container with id df1452274e0c8bfabb1293863c23ab60d25646106a52dc983ef603b71b05631f
Sep 09 20:14:04.882069 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:14:04.882025    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-init-fail-d5facb-predictor-7c8dff7f49-mcnqw" event={"ID":"e9441890-355d-4bae-9e45-a31071c9894e","Type":"ContainerStarted","Data":"2b74c2d92dcc9b19eff2943f8d8f2dd2113634df7ae662afe2b04133dd590f54"}
Sep 09 20:14:04.882523 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:14:04.882128    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-init-fail-d5facb-predictor-7c8dff7f49-mcnqw" event={"ID":"e9441890-355d-4bae-9e45-a31071c9894e","Type":"ContainerStarted","Data":"df1452274e0c8bfabb1293863c23ab60d25646106a52dc983ef603b71b05631f"}
Sep 09 20:14:07.895547 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:14:07.895519    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-init-fail-d5facb-predictor-7c8dff7f49-mcnqw_e9441890-355d-4bae-9e45-a31071c9894e/storage-initializer/0.log"
Sep 09 20:14:07.895929 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:14:07.895558    2574 generic.go:356] "Generic (PLEG): container finished" podID="e9441890-355d-4bae-9e45-a31071c9894e" containerID="2b74c2d92dcc9b19eff2943f8d8f2dd2113634df7ae662afe2b04133dd590f54" exitCode=1
Sep 09 20:14:07.895929 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:14:07.895593    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-init-fail-d5facb-predictor-7c8dff7f49-mcnqw" event={"ID":"e9441890-355d-4bae-9e45-a31071c9894e","Type":"ContainerDied","Data":"2b74c2d92dcc9b19eff2943f8d8f2dd2113634df7ae662afe2b04133dd590f54"}
Sep 09 20:14:08.794619 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:14:08.794587    2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-init-fail-d5facb-predictor-7c8dff7f49-mcnqw"]
Sep 09 20:14:08.900854 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:14:08.900826    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-init-fail-d5facb-predictor-7c8dff7f49-mcnqw_e9441890-355d-4bae-9e45-a31071c9894e/storage-initializer/0.log"
Sep 09 20:14:08.901288 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:14:08.900885    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-init-fail-d5facb-predictor-7c8dff7f49-mcnqw" event={"ID":"e9441890-355d-4bae-9e45-a31071c9894e","Type":"ContainerStarted","Data":"75b3d745ffcbc75fd5a5ad5b3ec147227ba16533fd7e48eca95351912e69a050"}
Sep 09 20:14:08.901288 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:14:08.901000    2574 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-init-fail-d5facb-predictor-7c8dff7f49-mcnqw" podUID="e9441890-355d-4bae-9e45-a31071c9894e" containerName="storage-initializer" containerID="cri-o://75b3d745ffcbc75fd5a5ad5b3ec147227ba16533fd7e48eca95351912e69a050" gracePeriod=30
Sep 09 20:14:12.444177 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:14:12.444153    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-init-fail-d5facb-predictor-7c8dff7f49-mcnqw_e9441890-355d-4bae-9e45-a31071c9894e/storage-initializer/1.log"
Sep 09 20:14:12.444524 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:14:12.444511    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-init-fail-d5facb-predictor-7c8dff7f49-mcnqw_e9441890-355d-4bae-9e45-a31071c9894e/storage-initializer/0.log"
Sep 09 20:14:12.446943 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:14:12.446929    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-init-fail-d5facb-predictor-7c8dff7f49-mcnqw"
Sep 09 20:14:12.551642 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:14:12.551554    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mmxrv\" (UniqueName: \"kubernetes.io/projected/e9441890-355d-4bae-9e45-a31071c9894e-kube-api-access-mmxrv\") pod \"e9441890-355d-4bae-9e45-a31071c9894e\" (UID: \"e9441890-355d-4bae-9e45-a31071c9894e\") "
Sep 09 20:14:12.551642 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:14:12.551596    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e9441890-355d-4bae-9e45-a31071c9894e-kserve-provision-location\") pod \"e9441890-355d-4bae-9e45-a31071c9894e\" (UID: \"e9441890-355d-4bae-9e45-a31071c9894e\") "
Sep 09 20:14:12.551642 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:14:12.551621    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/e9441890-355d-4bae-9e45-a31071c9894e-cabundle-cert\") pod \"e9441890-355d-4bae-9e45-a31071c9894e\" (UID: \"e9441890-355d-4bae-9e45-a31071c9894e\") "
Sep 09 20:14:12.551936 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:14:12.551660    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e9441890-355d-4bae-9e45-a31071c9894e-proxy-tls\") pod \"e9441890-355d-4bae-9e45-a31071c9894e\" (UID: \"e9441890-355d-4bae-9e45-a31071c9894e\") "
Sep 09 20:14:12.551936 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:14:12.551686    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e9441890-355d-4bae-9e45-a31071c9894e-kube-rbac-proxy-sar-config\") pod \"e9441890-355d-4bae-9e45-a31071c9894e\" (UID: \"e9441890-355d-4bae-9e45-a31071c9894e\") "
Sep 09 20:14:12.552037 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:14:12.551963    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e9441890-355d-4bae-9e45-a31071c9894e-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "e9441890-355d-4bae-9e45-a31071c9894e" (UID: "e9441890-355d-4bae-9e45-a31071c9894e"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 20:14:12.552100 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:14:12.552032    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e9441890-355d-4bae-9e45-a31071c9894e-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "e9441890-355d-4bae-9e45-a31071c9894e" (UID: "e9441890-355d-4bae-9e45-a31071c9894e"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 20:14:12.552203 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:14:12.552182    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e9441890-355d-4bae-9e45-a31071c9894e-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "e9441890-355d-4bae-9e45-a31071c9894e" (UID: "e9441890-355d-4bae-9e45-a31071c9894e"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 20:14:12.553833 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:14:12.553794    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e9441890-355d-4bae-9e45-a31071c9894e-kube-api-access-mmxrv" (OuterVolumeSpecName: "kube-api-access-mmxrv") pod "e9441890-355d-4bae-9e45-a31071c9894e" (UID: "e9441890-355d-4bae-9e45-a31071c9894e"). InnerVolumeSpecName "kube-api-access-mmxrv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 20:14:12.553833 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:14:12.553818    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e9441890-355d-4bae-9e45-a31071c9894e-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "e9441890-355d-4bae-9e45-a31071c9894e" (UID: "e9441890-355d-4bae-9e45-a31071c9894e"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 20:14:12.652544 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:14:12.652503    2574 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e9441890-355d-4bae-9e45-a31071c9894e-proxy-tls\") on node \"ip-10-0-136-183.ec2.internal\" DevicePath \"\""
Sep 09 20:14:12.652544 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:14:12.652538    2574 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e9441890-355d-4bae-9e45-a31071c9894e-kube-rbac-proxy-sar-config\") on node \"ip-10-0-136-183.ec2.internal\" DevicePath \"\""
Sep 09 20:14:12.652544 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:14:12.652548    2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mmxrv\" (UniqueName: \"kubernetes.io/projected/e9441890-355d-4bae-9e45-a31071c9894e-kube-api-access-mmxrv\") on node \"ip-10-0-136-183.ec2.internal\" DevicePath \"\""
Sep 09 20:14:12.652784 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:14:12.652560    2574 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e9441890-355d-4bae-9e45-a31071c9894e-kserve-provision-location\") on node \"ip-10-0-136-183.ec2.internal\" DevicePath \"\""
Sep 09 20:14:12.652784 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:14:12.652573    2574 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/e9441890-355d-4bae-9e45-a31071c9894e-cabundle-cert\") on node \"ip-10-0-136-183.ec2.internal\" DevicePath \"\""
Sep 09 20:14:12.918625 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:14:12.918595    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-init-fail-d5facb-predictor-7c8dff7f49-mcnqw_e9441890-355d-4bae-9e45-a31071c9894e/storage-initializer/1.log"
Sep 09 20:14:12.919050 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:14:12.919028    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-init-fail-d5facb-predictor-7c8dff7f49-mcnqw_e9441890-355d-4bae-9e45-a31071c9894e/storage-initializer/0.log"
Sep 09 20:14:12.919167 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:14:12.919075    2574 generic.go:356] "Generic (PLEG): container finished" podID="e9441890-355d-4bae-9e45-a31071c9894e" containerID="75b3d745ffcbc75fd5a5ad5b3ec147227ba16533fd7e48eca95351912e69a050" exitCode=1
Sep 09 20:14:12.919167 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:14:12.919124    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-init-fail-d5facb-predictor-7c8dff7f49-mcnqw" event={"ID":"e9441890-355d-4bae-9e45-a31071c9894e","Type":"ContainerDied","Data":"75b3d745ffcbc75fd5a5ad5b3ec147227ba16533fd7e48eca95351912e69a050"}
Sep 09 20:14:12.919276 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:14:12.919185    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-init-fail-d5facb-predictor-7c8dff7f49-mcnqw" event={"ID":"e9441890-355d-4bae-9e45-a31071c9894e","Type":"ContainerDied","Data":"df1452274e0c8bfabb1293863c23ab60d25646106a52dc983ef603b71b05631f"}
Sep 09 20:14:12.919276 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:14:12.919130    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-init-fail-d5facb-predictor-7c8dff7f49-mcnqw"
Sep 09 20:14:12.919276 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:14:12.919227    2574 scope.go:117] "RemoveContainer" containerID="75b3d745ffcbc75fd5a5ad5b3ec147227ba16533fd7e48eca95351912e69a050"
Sep 09 20:14:12.928764 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:14:12.928737    2574 scope.go:117] "RemoveContainer" containerID="2b74c2d92dcc9b19eff2943f8d8f2dd2113634df7ae662afe2b04133dd590f54"
Sep 09 20:14:12.936397 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:14:12.936379    2574 scope.go:117] "RemoveContainer" containerID="75b3d745ffcbc75fd5a5ad5b3ec147227ba16533fd7e48eca95351912e69a050"
Sep 09 20:14:12.936692 ip-10-0-136-183 kubenswrapper[2574]: E0909 20:14:12.936672    2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"75b3d745ffcbc75fd5a5ad5b3ec147227ba16533fd7e48eca95351912e69a050\": container with ID starting with 75b3d745ffcbc75fd5a5ad5b3ec147227ba16533fd7e48eca95351912e69a050 not found: ID does not exist" containerID="75b3d745ffcbc75fd5a5ad5b3ec147227ba16533fd7e48eca95351912e69a050"
Sep 09 20:14:12.936752 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:14:12.936699    2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"75b3d745ffcbc75fd5a5ad5b3ec147227ba16533fd7e48eca95351912e69a050"} err="failed to get container status \"75b3d745ffcbc75fd5a5ad5b3ec147227ba16533fd7e48eca95351912e69a050\": rpc error: code = NotFound desc = could not find container \"75b3d745ffcbc75fd5a5ad5b3ec147227ba16533fd7e48eca95351912e69a050\": container with ID starting with 75b3d745ffcbc75fd5a5ad5b3ec147227ba16533fd7e48eca95351912e69a050 not found: ID does not exist"
Sep 09 20:14:12.936752 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:14:12.936717    2574 scope.go:117] "RemoveContainer" containerID="2b74c2d92dcc9b19eff2943f8d8f2dd2113634df7ae662afe2b04133dd590f54"
Sep 09 20:14:12.937384 ip-10-0-136-183 kubenswrapper[2574]: E0909 20:14:12.937368    2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2b74c2d92dcc9b19eff2943f8d8f2dd2113634df7ae662afe2b04133dd590f54\": container with ID starting with 2b74c2d92dcc9b19eff2943f8d8f2dd2113634df7ae662afe2b04133dd590f54 not found: ID does not exist" containerID="2b74c2d92dcc9b19eff2943f8d8f2dd2113634df7ae662afe2b04133dd590f54"
Sep 09 20:14:12.937433 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:14:12.937388    2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2b74c2d92dcc9b19eff2943f8d8f2dd2113634df7ae662afe2b04133dd590f54"} err="failed to get container status \"2b74c2d92dcc9b19eff2943f8d8f2dd2113634df7ae662afe2b04133dd590f54\": rpc error: code = NotFound desc = could not find container \"2b74c2d92dcc9b19eff2943f8d8f2dd2113634df7ae662afe2b04133dd590f54\": container with ID starting with 2b74c2d92dcc9b19eff2943f8d8f2dd2113634df7ae662afe2b04133dd590f54 not found: ID does not exist"
Sep 09 20:14:12.969047 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:14:12.969009    2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-init-fail-d5facb-predictor-7c8dff7f49-mcnqw"]
Sep 09 20:14:12.972438 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:14:12.972408    2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-init-fail-d5facb-predictor-7c8dff7f49-mcnqw"]
Sep 09 20:14:14.048863 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:14:14.048831    2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/raw-sklearn-ca752-predictor-768df8d5ff-kmb78"]
Sep 09 20:14:14.049251 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:14:14.049147    2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e9441890-355d-4bae-9e45-a31071c9894e" containerName="storage-initializer"
Sep 09 20:14:14.049251 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:14:14.049159    2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="e9441890-355d-4bae-9e45-a31071c9894e" containerName="storage-initializer"
Sep 09 20:14:14.049251 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:14:14.049223    2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="e9441890-355d-4bae-9e45-a31071c9894e" containerName="storage-initializer"
Sep 09 20:14:14.049251 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:14:14.049232    2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="e9441890-355d-4bae-9e45-a31071c9894e" containerName="storage-initializer"
Sep 09 20:14:14.049378 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:14:14.049293    2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e9441890-355d-4bae-9e45-a31071c9894e" containerName="storage-initializer"
Sep 09 20:14:14.049378 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:14:14.049301    2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="e9441890-355d-4bae-9e45-a31071c9894e" containerName="storage-initializer"
Sep 09 20:14:14.053911 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:14:14.053892    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/raw-sklearn-ca752-predictor-768df8d5ff-kmb78"
Sep 09 20:14:14.056074 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:14:14.056052    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"raw-sklearn-ca752-predictor-serving-cert\""
Sep 09 20:14:14.056195 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:14:14.056074    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"raw-sklearn-ca752-kube-rbac-proxy-sar-config\""
Sep 09 20:14:14.056195 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:14:14.056115    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Sep 09 20:14:14.056284 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:14:14.056258    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Sep 09 20:14:14.065066 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:14:14.065043    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/raw-sklearn-ca752-predictor-768df8d5ff-kmb78"]
Sep 09 20:14:14.166571 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:14:14.166538    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/dc10e362-8836-47e1-8b31-2d71f5cb82bd-cabundle-cert\") pod \"raw-sklearn-ca752-predictor-768df8d5ff-kmb78\" (UID: \"dc10e362-8836-47e1-8b31-2d71f5cb82bd\") " pod="kserve-ci-e2e-test/raw-sklearn-ca752-predictor-768df8d5ff-kmb78"
Sep 09 20:14:14.166735 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:14:14.166574    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/dc10e362-8836-47e1-8b31-2d71f5cb82bd-kube-rbac-proxy-sar-config\") pod \"raw-sklearn-ca752-predictor-768df8d5ff-kmb78\" (UID: \"dc10e362-8836-47e1-8b31-2d71f5cb82bd\") " pod="kserve-ci-e2e-test/raw-sklearn-ca752-predictor-768df8d5ff-kmb78"
Sep 09 20:14:14.166735 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:14:14.166614    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/dc10e362-8836-47e1-8b31-2d71f5cb82bd-kserve-provision-location\") pod \"raw-sklearn-ca752-predictor-768df8d5ff-kmb78\" (UID: \"dc10e362-8836-47e1-8b31-2d71f5cb82bd\") " pod="kserve-ci-e2e-test/raw-sklearn-ca752-predictor-768df8d5ff-kmb78"
Sep 09 20:14:14.166735 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:14:14.166635    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/dc10e362-8836-47e1-8b31-2d71f5cb82bd-proxy-tls\") pod \"raw-sklearn-ca752-predictor-768df8d5ff-kmb78\" (UID: \"dc10e362-8836-47e1-8b31-2d71f5cb82bd\") " pod="kserve-ci-e2e-test/raw-sklearn-ca752-predictor-768df8d5ff-kmb78"
Sep 09 20:14:14.166735 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:14:14.166709    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-674qx\" (UniqueName: \"kubernetes.io/projected/dc10e362-8836-47e1-8b31-2d71f5cb82bd-kube-api-access-674qx\") pod \"raw-sklearn-ca752-predictor-768df8d5ff-kmb78\" (UID: \"dc10e362-8836-47e1-8b31-2d71f5cb82bd\") " pod="kserve-ci-e2e-test/raw-sklearn-ca752-predictor-768df8d5ff-kmb78"
Sep 09 20:14:14.267145 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:14:14.267109    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-674qx\" (UniqueName: \"kubernetes.io/projected/dc10e362-8836-47e1-8b31-2d71f5cb82bd-kube-api-access-674qx\") pod \"raw-sklearn-ca752-predictor-768df8d5ff-kmb78\" (UID: \"dc10e362-8836-47e1-8b31-2d71f5cb82bd\") " pod="kserve-ci-e2e-test/raw-sklearn-ca752-predictor-768df8d5ff-kmb78"
Sep 09 20:14:14.267268 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:14:14.267159    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/dc10e362-8836-47e1-8b31-2d71f5cb82bd-cabundle-cert\") pod \"raw-sklearn-ca752-predictor-768df8d5ff-kmb78\" (UID: \"dc10e362-8836-47e1-8b31-2d71f5cb82bd\") " pod="kserve-ci-e2e-test/raw-sklearn-ca752-predictor-768df8d5ff-kmb78"
Sep 09 20:14:14.267268 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:14:14.267179    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/dc10e362-8836-47e1-8b31-2d71f5cb82bd-kube-rbac-proxy-sar-config\") pod \"raw-sklearn-ca752-predictor-768df8d5ff-kmb78\" (UID: \"dc10e362-8836-47e1-8b31-2d71f5cb82bd\") " pod="kserve-ci-e2e-test/raw-sklearn-ca752-predictor-768df8d5ff-kmb78"
Sep 09 20:14:14.267268 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:14:14.267219    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/dc10e362-8836-47e1-8b31-2d71f5cb82bd-kserve-provision-location\") pod \"raw-sklearn-ca752-predictor-768df8d5ff-kmb78\" (UID: \"dc10e362-8836-47e1-8b31-2d71f5cb82bd\") " pod="kserve-ci-e2e-test/raw-sklearn-ca752-predictor-768df8d5ff-kmb78"
Sep 09 20:14:14.267268 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:14:14.267239    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/dc10e362-8836-47e1-8b31-2d71f5cb82bd-proxy-tls\") pod \"raw-sklearn-ca752-predictor-768df8d5ff-kmb78\" (UID: \"dc10e362-8836-47e1-8b31-2d71f5cb82bd\") " pod="kserve-ci-e2e-test/raw-sklearn-ca752-predictor-768df8d5ff-kmb78"
Sep 09 20:14:14.267670 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:14:14.267647    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/dc10e362-8836-47e1-8b31-2d71f5cb82bd-kserve-provision-location\") pod \"raw-sklearn-ca752-predictor-768df8d5ff-kmb78\" (UID: \"dc10e362-8836-47e1-8b31-2d71f5cb82bd\") " pod="kserve-ci-e2e-test/raw-sklearn-ca752-predictor-768df8d5ff-kmb78"
Sep 09 20:14:14.267839 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:14:14.267821    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/dc10e362-8836-47e1-8b31-2d71f5cb82bd-cabundle-cert\") pod \"raw-sklearn-ca752-predictor-768df8d5ff-kmb78\" (UID: \"dc10e362-8836-47e1-8b31-2d71f5cb82bd\") " pod="kserve-ci-e2e-test/raw-sklearn-ca752-predictor-768df8d5ff-kmb78"
Sep 09 20:14:14.267933 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:14:14.267917    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/dc10e362-8836-47e1-8b31-2d71f5cb82bd-kube-rbac-proxy-sar-config\") pod \"raw-sklearn-ca752-predictor-768df8d5ff-kmb78\" (UID: \"dc10e362-8836-47e1-8b31-2d71f5cb82bd\") " pod="kserve-ci-e2e-test/raw-sklearn-ca752-predictor-768df8d5ff-kmb78"
Sep 09 20:14:14.269661 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:14:14.269636    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/dc10e362-8836-47e1-8b31-2d71f5cb82bd-proxy-tls\") pod \"raw-sklearn-ca752-predictor-768df8d5ff-kmb78\" (UID: \"dc10e362-8836-47e1-8b31-2d71f5cb82bd\") " pod="kserve-ci-e2e-test/raw-sklearn-ca752-predictor-768df8d5ff-kmb78"
Sep 09 20:14:14.281203 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:14:14.281175    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-674qx\" (UniqueName: \"kubernetes.io/projected/dc10e362-8836-47e1-8b31-2d71f5cb82bd-kube-api-access-674qx\") pod \"raw-sklearn-ca752-predictor-768df8d5ff-kmb78\" (UID: \"dc10e362-8836-47e1-8b31-2d71f5cb82bd\") " pod="kserve-ci-e2e-test/raw-sklearn-ca752-predictor-768df8d5ff-kmb78"
Sep 09 20:14:14.364713 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:14:14.364672    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/raw-sklearn-ca752-predictor-768df8d5ff-kmb78"
Sep 09 20:14:14.490989 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:14:14.490946    2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e9441890-355d-4bae-9e45-a31071c9894e" path="/var/lib/kubelet/pods/e9441890-355d-4bae-9e45-a31071c9894e/volumes"
Sep 09 20:14:14.642664 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:14:14.642537    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/raw-sklearn-ca752-predictor-768df8d5ff-kmb78"]
Sep 09 20:14:14.645439 ip-10-0-136-183 kubenswrapper[2574]: W0909 20:14:14.645409    2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddc10e362_8836_47e1_8b31_2d71f5cb82bd.slice/crio-6e0e91c2eaf15c06e4d453e1de5fc4078cf0ba41dbef375824a96d74504bdfb7 WatchSource:0}: Error finding container 6e0e91c2eaf15c06e4d453e1de5fc4078cf0ba41dbef375824a96d74504bdfb7: Status 404 returned error can't find the container with id 6e0e91c2eaf15c06e4d453e1de5fc4078cf0ba41dbef375824a96d74504bdfb7
Sep 09 20:14:14.931815 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:14:14.931715    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/raw-sklearn-ca752-predictor-768df8d5ff-kmb78" event={"ID":"dc10e362-8836-47e1-8b31-2d71f5cb82bd","Type":"ContainerStarted","Data":"c49d7a86289811dd28bdac8a10b9120c4e11e7f1cbd8b0be69cdaba786e94ccd"}
Sep 09 20:14:14.931815 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:14:14.931771    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/raw-sklearn-ca752-predictor-768df8d5ff-kmb78" event={"ID":"dc10e362-8836-47e1-8b31-2d71f5cb82bd","Type":"ContainerStarted","Data":"6e0e91c2eaf15c06e4d453e1de5fc4078cf0ba41dbef375824a96d74504bdfb7"}
Sep 09 20:14:18.949681 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:14:18.949644    2574 generic.go:356] "Generic (PLEG): container finished" podID="dc10e362-8836-47e1-8b31-2d71f5cb82bd" containerID="c49d7a86289811dd28bdac8a10b9120c4e11e7f1cbd8b0be69cdaba786e94ccd" exitCode=0
Sep 09 20:14:18.950129 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:14:18.949723    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/raw-sklearn-ca752-predictor-768df8d5ff-kmb78" event={"ID":"dc10e362-8836-47e1-8b31-2d71f5cb82bd","Type":"ContainerDied","Data":"c49d7a86289811dd28bdac8a10b9120c4e11e7f1cbd8b0be69cdaba786e94ccd"}
Sep 09 20:14:19.959164 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:14:19.959123    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/raw-sklearn-ca752-predictor-768df8d5ff-kmb78" event={"ID":"dc10e362-8836-47e1-8b31-2d71f5cb82bd","Type":"ContainerStarted","Data":"6b44bfd1e502163c0f34e9716b02aab9df3cf5fe39049501d95237e5e3250e13"}
Sep 09 20:14:19.959648 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:14:19.959230    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/raw-sklearn-ca752-predictor-768df8d5ff-kmb78" event={"ID":"dc10e362-8836-47e1-8b31-2d71f5cb82bd","Type":"ContainerStarted","Data":"91b16af588a0ce6944e78c2401998c6a79c9f5998f7fcd548a45ba003728688e"}
Sep 09 20:14:19.959648 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:14:19.959443    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/raw-sklearn-ca752-predictor-768df8d5ff-kmb78"
Sep 09 20:14:19.978419 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:14:19.978360    2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/raw-sklearn-ca752-predictor-768df8d5ff-kmb78" podStartSLOduration=2.241596906 podStartE2EDuration="5.978342737s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.736745831s" isStatefulPod=false podCreationTimestamp="2026-09-09 20:14:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:14:19.976731654 +0000 UTC m=+1144.022984918" watchObservedRunningTime="2026-09-09 20:14:19.978342737 +0000 UTC m=+1144.024595999"
Sep 09 20:14:20.963536 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:14:20.963501    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/raw-sklearn-ca752-predictor-768df8d5ff-kmb78"
Sep 09 20:14:20.964599 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:14:20.964570    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-ca752-predictor-768df8d5ff-kmb78" podUID="dc10e362-8836-47e1-8b31-2d71f5cb82bd" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.33:8080: connect: connection refused"
Sep 09 20:14:21.967030 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:14:21.966990    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-ca752-predictor-768df8d5ff-kmb78" podUID="dc10e362-8836-47e1-8b31-2d71f5cb82bd" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.33:8080: connect: connection refused"
Sep 09 20:14:22.636173 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:14:22.636143    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_message-dumper-raw-96f8b-predictor-7779456f4f-p56fj_47632e59-c409-48e0-8d16-e23fa1f15697/kserve-container/0.log"
Sep 09 20:14:22.856341 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:14:22.856308    2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/message-dumper-raw-96f8b-predictor-7779456f4f-p56fj"]
Sep 09 20:14:22.856570 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:14:22.856542    2574 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/message-dumper-raw-96f8b-predictor-7779456f4f-p56fj" podUID="47632e59-c409-48e0-8d16-e23fa1f15697" containerName="kserve-container" containerID="cri-o://43bf15c8e6446b5d3b0382790b4113d55c12946f298db929319a58c6c71e4aa5" gracePeriod=30
Sep 09 20:14:22.856638 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:14:22.856584    2574 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/message-dumper-raw-96f8b-predictor-7779456f4f-p56fj" podUID="47632e59-c409-48e0-8d16-e23fa1f15697" containerName="kube-rbac-proxy" containerID="cri-o://d53284c00b012a2328471cc047ca803f055fd8d78e255937a5b80d7b3783c143" gracePeriod=30
Sep 09 20:14:22.971348 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:14:22.971315    2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-scale-raw-0b35a-predictor-8587c87585-8q262"]
Sep 09 20:14:22.972747 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:14:22.972719    2574 generic.go:356] "Generic (PLEG): container finished" podID="47632e59-c409-48e0-8d16-e23fa1f15697" containerID="d53284c00b012a2328471cc047ca803f055fd8d78e255937a5b80d7b3783c143" exitCode=2
Sep 09 20:14:22.972747 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:14:22.972744    2574 generic.go:356] "Generic (PLEG): container finished" podID="47632e59-c409-48e0-8d16-e23fa1f15697" containerID="43bf15c8e6446b5d3b0382790b4113d55c12946f298db929319a58c6c71e4aa5" exitCode=2
Sep 09 20:14:22.974568 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:14:22.974546    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/message-dumper-raw-96f8b-predictor-7779456f4f-p56fj" event={"ID":"47632e59-c409-48e0-8d16-e23fa1f15697","Type":"ContainerDied","Data":"d53284c00b012a2328471cc047ca803f055fd8d78e255937a5b80d7b3783c143"}
Sep 09 20:14:22.974653 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:14:22.974585    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/message-dumper-raw-96f8b-predictor-7779456f4f-p56fj" event={"ID":"47632e59-c409-48e0-8d16-e23fa1f15697","Type":"ContainerDied","Data":"43bf15c8e6446b5d3b0382790b4113d55c12946f298db929319a58c6c71e4aa5"}
Sep 09 20:14:22.974694 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:14:22.974674    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-0b35a-predictor-8587c87585-8q262"
Sep 09 20:14:22.978042 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:14:22.978024    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-scale-raw-0b35a-kube-rbac-proxy-sar-config\""
Sep 09 20:14:22.979410 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:14:22.979393    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-scale-raw-0b35a-predictor-serving-cert\""
Sep 09 20:14:22.985021 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:14:22.985001    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-scale-raw-0b35a-predictor-8587c87585-8q262"]
Sep 09 20:14:23.111638 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:14:23.111613    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/message-dumper-raw-96f8b-predictor-7779456f4f-p56fj"
Sep 09 20:14:23.136904 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:14:23.136864    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/2c75cf85-1abd-4fe0-bdf6-b405acf5d913-kserve-provision-location\") pod \"isvc-sklearn-scale-raw-0b35a-predictor-8587c87585-8q262\" (UID: \"2c75cf85-1abd-4fe0-bdf6-b405acf5d913\") " pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-0b35a-predictor-8587c87585-8q262"
Sep 09 20:14:23.137073 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:14:23.136927    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/2c75cf85-1abd-4fe0-bdf6-b405acf5d913-cabundle-cert\") pod \"isvc-sklearn-scale-raw-0b35a-predictor-8587c87585-8q262\" (UID: \"2c75cf85-1abd-4fe0-bdf6-b405acf5d913\") " pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-0b35a-predictor-8587c87585-8q262"
Sep 09 20:14:23.137073 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:14:23.136976    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/2c75cf85-1abd-4fe0-bdf6-b405acf5d913-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-scale-raw-0b35a-predictor-8587c87585-8q262\" (UID: \"2c75cf85-1abd-4fe0-bdf6-b405acf5d913\") " pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-0b35a-predictor-8587c87585-8q262"
Sep 09 20:14:23.137073 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:14:23.137056    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/2c75cf85-1abd-4fe0-bdf6-b405acf5d913-proxy-tls\") pod \"isvc-sklearn-scale-raw-0b35a-predictor-8587c87585-8q262\" (UID: \"2c75cf85-1abd-4fe0-bdf6-b405acf5d913\") " pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-0b35a-predictor-8587c87585-8q262"
Sep 09 20:14:23.137216 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:14:23.137091    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lgv94\" (UniqueName: \"kubernetes.io/projected/2c75cf85-1abd-4fe0-bdf6-b405acf5d913-kube-api-access-lgv94\") pod \"isvc-sklearn-scale-raw-0b35a-predictor-8587c87585-8q262\" (UID: \"2c75cf85-1abd-4fe0-bdf6-b405acf5d913\") " pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-0b35a-predictor-8587c87585-8q262"
Sep 09 20:14:23.238162 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:14:23.238058    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qjvx4\" (UniqueName: \"kubernetes.io/projected/47632e59-c409-48e0-8d16-e23fa1f15697-kube-api-access-qjvx4\") pod \"47632e59-c409-48e0-8d16-e23fa1f15697\" (UID: \"47632e59-c409-48e0-8d16-e23fa1f15697\") "
Sep 09 20:14:23.238162 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:14:23.238134    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/47632e59-c409-48e0-8d16-e23fa1f15697-kube-rbac-proxy-sar-config\") pod \"47632e59-c409-48e0-8d16-e23fa1f15697\" (UID: \"47632e59-c409-48e0-8d16-e23fa1f15697\") "
Sep 09 20:14:23.238162 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:14:23.238158    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/47632e59-c409-48e0-8d16-e23fa1f15697-proxy-tls\") pod \"47632e59-c409-48e0-8d16-e23fa1f15697\" (UID: \"47632e59-c409-48e0-8d16-e23fa1f15697\") "
Sep 09 20:14:23.238456 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:14:23.238243    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/2c75cf85-1abd-4fe0-bdf6-b405acf5d913-kserve-provision-location\") pod \"isvc-sklearn-scale-raw-0b35a-predictor-8587c87585-8q262\" (UID: \"2c75cf85-1abd-4fe0-bdf6-b405acf5d913\") " pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-0b35a-predictor-8587c87585-8q262"
Sep 09 20:14:23.238456 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:14:23.238268    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/2c75cf85-1abd-4fe0-bdf6-b405acf5d913-cabundle-cert\") pod \"isvc-sklearn-scale-raw-0b35a-predictor-8587c87585-8q262\" (UID: \"2c75cf85-1abd-4fe0-bdf6-b405acf5d913\") " pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-0b35a-predictor-8587c87585-8q262"
Sep 09 20:14:23.238456 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:14:23.238293    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/2c75cf85-1abd-4fe0-bdf6-b405acf5d913-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-scale-raw-0b35a-predictor-8587c87585-8q262\" (UID: \"2c75cf85-1abd-4fe0-bdf6-b405acf5d913\") " pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-0b35a-predictor-8587c87585-8q262"
Sep 09 20:14:23.238610 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:14:23.238554    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/47632e59-c409-48e0-8d16-e23fa1f15697-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "47632e59-c409-48e0-8d16-e23fa1f15697" (UID: "47632e59-c409-48e0-8d16-e23fa1f15697"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 20:14:23.238656 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:14:23.238615    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/2c75cf85-1abd-4fe0-bdf6-b405acf5d913-proxy-tls\") pod \"isvc-sklearn-scale-raw-0b35a-predictor-8587c87585-8q262\" (UID: \"2c75cf85-1abd-4fe0-bdf6-b405acf5d913\") " pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-0b35a-predictor-8587c87585-8q262"
Sep 09 20:14:23.238701 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:14:23.238670    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lgv94\" (UniqueName: \"kubernetes.io/projected/2c75cf85-1abd-4fe0-bdf6-b405acf5d913-kube-api-access-lgv94\") pod \"isvc-sklearn-scale-raw-0b35a-predictor-8587c87585-8q262\" (UID: \"2c75cf85-1abd-4fe0-bdf6-b405acf5d913\") " pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-0b35a-predictor-8587c87585-8q262"
Sep 09 20:14:23.238701 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:14:23.238681    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/2c75cf85-1abd-4fe0-bdf6-b405acf5d913-kserve-provision-location\") pod \"isvc-sklearn-scale-raw-0b35a-predictor-8587c87585-8q262\" (UID: \"2c75cf85-1abd-4fe0-bdf6-b405acf5d913\") " pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-0b35a-predictor-8587c87585-8q262"
Sep 09 20:14:23.238831 ip-10-0-136-183 kubenswrapper[2574]: E0909 20:14:23.238775    2574 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-sklearn-scale-raw-0b35a-predictor-serving-cert: secret "isvc-sklearn-scale-raw-0b35a-predictor-serving-cert" not found
Sep 09 20:14:23.238831 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:14:23.238784    2574 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/47632e59-c409-48e0-8d16-e23fa1f15697-kube-rbac-proxy-sar-config\") on node \"ip-10-0-136-183.ec2.internal\" DevicePath \"\""
Sep 09 20:14:23.238933 ip-10-0-136-183 kubenswrapper[2574]: E0909 20:14:23.238862    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2c75cf85-1abd-4fe0-bdf6-b405acf5d913-proxy-tls podName:2c75cf85-1abd-4fe0-bdf6-b405acf5d913 nodeName:}" failed. No retries permitted until 2026-09-09 20:14:23.738841343 +0000 UTC m=+1147.785094600 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/2c75cf85-1abd-4fe0-bdf6-b405acf5d913-proxy-tls") pod "isvc-sklearn-scale-raw-0b35a-predictor-8587c87585-8q262" (UID: "2c75cf85-1abd-4fe0-bdf6-b405acf5d913") : secret "isvc-sklearn-scale-raw-0b35a-predictor-serving-cert" not found
Sep 09 20:14:23.239101 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:14:23.239083    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/2c75cf85-1abd-4fe0-bdf6-b405acf5d913-cabundle-cert\") pod \"isvc-sklearn-scale-raw-0b35a-predictor-8587c87585-8q262\" (UID: \"2c75cf85-1abd-4fe0-bdf6-b405acf5d913\") " pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-0b35a-predictor-8587c87585-8q262"
Sep 09 20:14:23.239144 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:14:23.239124    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/2c75cf85-1abd-4fe0-bdf6-b405acf5d913-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-scale-raw-0b35a-predictor-8587c87585-8q262\" (UID: \"2c75cf85-1abd-4fe0-bdf6-b405acf5d913\") " pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-0b35a-predictor-8587c87585-8q262"
Sep 09 20:14:23.240451 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:14:23.240430    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/47632e59-c409-48e0-8d16-e23fa1f15697-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "47632e59-c409-48e0-8d16-e23fa1f15697" (UID: "47632e59-c409-48e0-8d16-e23fa1f15697"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 20:14:23.240535 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:14:23.240444    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/47632e59-c409-48e0-8d16-e23fa1f15697-kube-api-access-qjvx4" (OuterVolumeSpecName: "kube-api-access-qjvx4") pod "47632e59-c409-48e0-8d16-e23fa1f15697" (UID: "47632e59-c409-48e0-8d16-e23fa1f15697"). InnerVolumeSpecName "kube-api-access-qjvx4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 20:14:23.248098 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:14:23.248077    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lgv94\" (UniqueName: \"kubernetes.io/projected/2c75cf85-1abd-4fe0-bdf6-b405acf5d913-kube-api-access-lgv94\") pod \"isvc-sklearn-scale-raw-0b35a-predictor-8587c87585-8q262\" (UID: \"2c75cf85-1abd-4fe0-bdf6-b405acf5d913\") " pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-0b35a-predictor-8587c87585-8q262"
Sep 09 20:14:23.339369 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:14:23.339333    2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qjvx4\" (UniqueName: \"kubernetes.io/projected/47632e59-c409-48e0-8d16-e23fa1f15697-kube-api-access-qjvx4\") on node \"ip-10-0-136-183.ec2.internal\" DevicePath \"\""
Sep 09 20:14:23.339369 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:14:23.339362    2574 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/47632e59-c409-48e0-8d16-e23fa1f15697-proxy-tls\") on node \"ip-10-0-136-183.ec2.internal\" DevicePath \"\""
Sep 09 20:14:23.743670 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:14:23.743621    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/2c75cf85-1abd-4fe0-bdf6-b405acf5d913-proxy-tls\") pod \"isvc-sklearn-scale-raw-0b35a-predictor-8587c87585-8q262\" (UID: \"2c75cf85-1abd-4fe0-bdf6-b405acf5d913\") " pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-0b35a-predictor-8587c87585-8q262"
Sep 09 20:14:23.746059 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:14:23.746030    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/2c75cf85-1abd-4fe0-bdf6-b405acf5d913-proxy-tls\") pod \"isvc-sklearn-scale-raw-0b35a-predictor-8587c87585-8q262\" (UID: \"2c75cf85-1abd-4fe0-bdf6-b405acf5d913\") " pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-0b35a-predictor-8587c87585-8q262"
Sep 09 20:14:23.885440 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:14:23.885405    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-0b35a-predictor-8587c87585-8q262"
Sep 09 20:14:23.978544 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:14:23.978504    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/message-dumper-raw-96f8b-predictor-7779456f4f-p56fj" event={"ID":"47632e59-c409-48e0-8d16-e23fa1f15697","Type":"ContainerDied","Data":"e5f03d4d8c4b1218da4a7d7b44c288e0d1f3f5aaec6a6b7aa23ac6af1baabfc5"}
Sep 09 20:14:23.978544 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:14:23.978533    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/message-dumper-raw-96f8b-predictor-7779456f4f-p56fj"
Sep 09 20:14:23.979023 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:14:23.978594    2574 scope.go:117] "RemoveContainer" containerID="d53284c00b012a2328471cc047ca803f055fd8d78e255937a5b80d7b3783c143"
Sep 09 20:14:23.987002 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:14:23.986981    2574 scope.go:117] "RemoveContainer" containerID="43bf15c8e6446b5d3b0382790b4113d55c12946f298db929319a58c6c71e4aa5"
Sep 09 20:14:24.000267 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:14:24.000244    2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/message-dumper-raw-96f8b-predictor-7779456f4f-p56fj"]
Sep 09 20:14:24.005600 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:14:24.005577    2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/message-dumper-raw-96f8b-predictor-7779456f4f-p56fj"]
Sep 09 20:14:24.162117 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:14:24.162089    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-scale-raw-0b35a-predictor-8587c87585-8q262"]
Sep 09 20:14:24.164449 ip-10-0-136-183 kubenswrapper[2574]: W0909 20:14:24.164421    2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2c75cf85_1abd_4fe0_bdf6_b405acf5d913.slice/crio-0acc97198c8d1aa14695b03aec1240761a5a88f7bea2d826f59706ddb12670da WatchSource:0}: Error finding container 0acc97198c8d1aa14695b03aec1240761a5a88f7bea2d826f59706ddb12670da: Status 404 returned error can't find the container with id 0acc97198c8d1aa14695b03aec1240761a5a88f7bea2d826f59706ddb12670da
Sep 09 20:14:24.492000 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:14:24.491958    2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="47632e59-c409-48e0-8d16-e23fa1f15697" path="/var/lib/kubelet/pods/47632e59-c409-48e0-8d16-e23fa1f15697/volumes"
Sep 09 20:14:24.984552 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:14:24.984512    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-0b35a-predictor-8587c87585-8q262" event={"ID":"2c75cf85-1abd-4fe0-bdf6-b405acf5d913","Type":"ContainerStarted","Data":"212d4e09d417a2ceadb9a7e3821ba1527407a1f7f915ca20ebe4f6de2da1574c"}
Sep 09 20:14:24.985053 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:14:24.984578    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-0b35a-predictor-8587c87585-8q262" event={"ID":"2c75cf85-1abd-4fe0-bdf6-b405acf5d913","Type":"ContainerStarted","Data":"0acc97198c8d1aa14695b03aec1240761a5a88f7bea2d826f59706ddb12670da"}
Sep 09 20:14:26.971947 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:14:26.971918    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/raw-sklearn-ca752-predictor-768df8d5ff-kmb78"
Sep 09 20:14:26.972406 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:14:26.972380    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-ca752-predictor-768df8d5ff-kmb78" podUID="dc10e362-8836-47e1-8b31-2d71f5cb82bd" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.33:8080: connect: connection refused"
Sep 09 20:14:29.004012 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:14:29.003976    2574 generic.go:356] "Generic (PLEG): container finished" podID="2c75cf85-1abd-4fe0-bdf6-b405acf5d913" containerID="212d4e09d417a2ceadb9a7e3821ba1527407a1f7f915ca20ebe4f6de2da1574c" exitCode=0
Sep 09 20:14:29.004375 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:14:29.004033    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-0b35a-predictor-8587c87585-8q262" event={"ID":"2c75cf85-1abd-4fe0-bdf6-b405acf5d913","Type":"ContainerDied","Data":"212d4e09d417a2ceadb9a7e3821ba1527407a1f7f915ca20ebe4f6de2da1574c"}
Sep 09 20:14:30.009699 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:14:30.009662    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-0b35a-predictor-8587c87585-8q262" event={"ID":"2c75cf85-1abd-4fe0-bdf6-b405acf5d913","Type":"ContainerStarted","Data":"cb85f6de203abb32e2f25742368e4db643fcfbd21046dbfec9e36af39a9bd60f"}
Sep 09 20:14:30.010151 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:14:30.009723    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-0b35a-predictor-8587c87585-8q262" event={"ID":"2c75cf85-1abd-4fe0-bdf6-b405acf5d913","Type":"ContainerStarted","Data":"33889ab127b9c7a3432996255792d2625fce079ccae22abc81baedc7af3e92ec"}
Sep 09 20:14:30.010151 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:14:30.010003    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-0b35a-predictor-8587c87585-8q262"
Sep 09 20:14:30.010151 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:14:30.010047    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-0b35a-predictor-8587c87585-8q262"
Sep 09 20:14:30.011393 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:14:30.011364    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-0b35a-predictor-8587c87585-8q262" podUID="2c75cf85-1abd-4fe0-bdf6-b405acf5d913" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.34:8080: connect: connection refused"
Sep 09 20:14:30.036027 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:14:30.035975    2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-0b35a-predictor-8587c87585-8q262" podStartSLOduration=4.239364392 podStartE2EDuration="8.035957436s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.796593044s" isStatefulPod=false podCreationTimestamp="2026-09-09 20:14:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:14:30.035953788 +0000 UTC m=+1154.082207051" watchObservedRunningTime="2026-09-09 20:14:30.035957436 +0000 UTC m=+1154.082210699"
Sep 09 20:14:31.014319 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:14:31.014275    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-0b35a-predictor-8587c87585-8q262" podUID="2c75cf85-1abd-4fe0-bdf6-b405acf5d913" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.34:8080: connect: connection refused"
Sep 09 20:14:36.018721 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:14:36.018692    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-0b35a-predictor-8587c87585-8q262"
Sep 09 20:14:36.019255 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:14:36.019221    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-0b35a-predictor-8587c87585-8q262" podUID="2c75cf85-1abd-4fe0-bdf6-b405acf5d913" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.34:8080: connect: connection refused"
Sep 09 20:14:36.973172 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:14:36.973123    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-ca752-predictor-768df8d5ff-kmb78" podUID="dc10e362-8836-47e1-8b31-2d71f5cb82bd" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.33:8080: connect: connection refused"
Sep 09 20:14:46.019625 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:14:46.019541    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-0b35a-predictor-8587c87585-8q262" podUID="2c75cf85-1abd-4fe0-bdf6-b405acf5d913" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.34:8080: connect: connection refused"
Sep 09 20:14:46.972459 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:14:46.972416    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-ca752-predictor-768df8d5ff-kmb78" podUID="dc10e362-8836-47e1-8b31-2d71f5cb82bd" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.33:8080: connect: connection refused"
Sep 09 20:14:56.019196 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:14:56.019149    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-0b35a-predictor-8587c87585-8q262" podUID="2c75cf85-1abd-4fe0-bdf6-b405acf5d913" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.34:8080: connect: connection refused"
Sep 09 20:14:56.973314 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:14:56.973272    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-ca752-predictor-768df8d5ff-kmb78" podUID="dc10e362-8836-47e1-8b31-2d71f5cb82bd" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.33:8080: connect: connection refused"
Sep 09 20:15:06.020132 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:15:06.020091    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-0b35a-predictor-8587c87585-8q262" podUID="2c75cf85-1abd-4fe0-bdf6-b405acf5d913" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.34:8080: connect: connection refused"
Sep 09 20:15:06.972997 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:15:06.972957    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-ca752-predictor-768df8d5ff-kmb78" podUID="dc10e362-8836-47e1-8b31-2d71f5cb82bd" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.33:8080: connect: connection refused"
Sep 09 20:15:16.019291 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:15:16.019249    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-0b35a-predictor-8587c87585-8q262" podUID="2c75cf85-1abd-4fe0-bdf6-b405acf5d913" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.34:8080: connect: connection refused"
Sep 09 20:15:16.972581 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:15:16.972541    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-ca752-predictor-768df8d5ff-kmb78" podUID="dc10e362-8836-47e1-8b31-2d71f5cb82bd" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.33:8080: connect: connection refused"
Sep 09 20:15:26.019990 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:15:26.019947    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-0b35a-predictor-8587c87585-8q262" podUID="2c75cf85-1abd-4fe0-bdf6-b405acf5d913" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.34:8080: connect: connection refused"
Sep 09 20:15:26.973264 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:15:26.973225    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-ca752-predictor-768df8d5ff-kmb78" podUID="dc10e362-8836-47e1-8b31-2d71f5cb82bd" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.33:8080: connect: connection refused"
Sep 09 20:15:36.020052 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:15:36.020008    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-0b35a-predictor-8587c87585-8q262" podUID="2c75cf85-1abd-4fe0-bdf6-b405acf5d913" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.34:8080: connect: connection refused"
Sep 09 20:15:36.973994 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:15:36.973962    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/raw-sklearn-ca752-predictor-768df8d5ff-kmb78"
Sep 09 20:15:38.486639 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:15:38.486598    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-0b35a-predictor-8587c87585-8q262" podUID="2c75cf85-1abd-4fe0-bdf6-b405acf5d913" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.34:8080: connect: connection refused"
Sep 09 20:15:44.115234 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:15:44.115199    2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/raw-sklearn-ca752-predictor-768df8d5ff-kmb78"]
Sep 09 20:15:44.115713 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:15:44.115463    2574 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/raw-sklearn-ca752-predictor-768df8d5ff-kmb78" podUID="dc10e362-8836-47e1-8b31-2d71f5cb82bd" containerName="kserve-container" containerID="cri-o://91b16af588a0ce6944e78c2401998c6a79c9f5998f7fcd548a45ba003728688e" gracePeriod=30
Sep 09 20:15:44.115713 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:15:44.115528    2574 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/raw-sklearn-ca752-predictor-768df8d5ff-kmb78" podUID="dc10e362-8836-47e1-8b31-2d71f5cb82bd" containerName="kube-rbac-proxy" containerID="cri-o://6b44bfd1e502163c0f34e9716b02aab9df3cf5fe39049501d95237e5e3250e13" gracePeriod=30
Sep 09 20:15:44.309056 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:15:44.309024    2574 generic.go:356] "Generic (PLEG): container finished" podID="dc10e362-8836-47e1-8b31-2d71f5cb82bd" containerID="6b44bfd1e502163c0f34e9716b02aab9df3cf5fe39049501d95237e5e3250e13" exitCode=2
Sep 09 20:15:44.309225 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:15:44.309096    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/raw-sklearn-ca752-predictor-768df8d5ff-kmb78" event={"ID":"dc10e362-8836-47e1-8b31-2d71f5cb82bd","Type":"ContainerDied","Data":"6b44bfd1e502163c0f34e9716b02aab9df3cf5fe39049501d95237e5e3250e13"}
Sep 09 20:15:46.968171 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:15:46.968115    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-ca752-predictor-768df8d5ff-kmb78" podUID="dc10e362-8836-47e1-8b31-2d71f5cb82bd" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.33:8643/healthz\": dial tcp 10.133.0.33:8643: connect: connection refused"
Sep 09 20:15:46.972588 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:15:46.972560    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-ca752-predictor-768df8d5ff-kmb78" podUID="dc10e362-8836-47e1-8b31-2d71f5cb82bd" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.33:8080: connect: connection refused"
Sep 09 20:15:48.486655 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:15:48.486613    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-0b35a-predictor-8587c87585-8q262" podUID="2c75cf85-1abd-4fe0-bdf6-b405acf5d913" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.34:8080: connect: connection refused"
Sep 09 20:15:49.158716 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:15:49.158688    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/raw-sklearn-ca752-predictor-768df8d5ff-kmb78"
Sep 09 20:15:49.198717 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:15:49.198687    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-674qx\" (UniqueName: \"kubernetes.io/projected/dc10e362-8836-47e1-8b31-2d71f5cb82bd-kube-api-access-674qx\") pod \"dc10e362-8836-47e1-8b31-2d71f5cb82bd\" (UID: \"dc10e362-8836-47e1-8b31-2d71f5cb82bd\") "
Sep 09 20:15:49.198918 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:15:49.198732    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/dc10e362-8836-47e1-8b31-2d71f5cb82bd-kube-rbac-proxy-sar-config\") pod \"dc10e362-8836-47e1-8b31-2d71f5cb82bd\" (UID: \"dc10e362-8836-47e1-8b31-2d71f5cb82bd\") "
Sep 09 20:15:49.198918 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:15:49.198776    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/dc10e362-8836-47e1-8b31-2d71f5cb82bd-kserve-provision-location\") pod \"dc10e362-8836-47e1-8b31-2d71f5cb82bd\" (UID: \"dc10e362-8836-47e1-8b31-2d71f5cb82bd\") "
Sep 09 20:15:49.198918 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:15:49.198825    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/dc10e362-8836-47e1-8b31-2d71f5cb82bd-cabundle-cert\") pod \"dc10e362-8836-47e1-8b31-2d71f5cb82bd\" (UID: \"dc10e362-8836-47e1-8b31-2d71f5cb82bd\") "
Sep 09 20:15:49.198918 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:15:49.198881    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/dc10e362-8836-47e1-8b31-2d71f5cb82bd-proxy-tls\") pod \"dc10e362-8836-47e1-8b31-2d71f5cb82bd\" (UID: \"dc10e362-8836-47e1-8b31-2d71f5cb82bd\") "
Sep 09 20:15:49.199206 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:15:49.199172    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dc10e362-8836-47e1-8b31-2d71f5cb82bd-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "dc10e362-8836-47e1-8b31-2d71f5cb82bd" (UID: "dc10e362-8836-47e1-8b31-2d71f5cb82bd"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 20:15:49.199354 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:15:49.199208    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dc10e362-8836-47e1-8b31-2d71f5cb82bd-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "dc10e362-8836-47e1-8b31-2d71f5cb82bd" (UID: "dc10e362-8836-47e1-8b31-2d71f5cb82bd"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 20:15:49.199530 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:15:49.199510    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dc10e362-8836-47e1-8b31-2d71f5cb82bd-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "dc10e362-8836-47e1-8b31-2d71f5cb82bd" (UID: "dc10e362-8836-47e1-8b31-2d71f5cb82bd"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 20:15:49.201058 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:15:49.201028    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dc10e362-8836-47e1-8b31-2d71f5cb82bd-kube-api-access-674qx" (OuterVolumeSpecName: "kube-api-access-674qx") pod "dc10e362-8836-47e1-8b31-2d71f5cb82bd" (UID: "dc10e362-8836-47e1-8b31-2d71f5cb82bd"). InnerVolumeSpecName "kube-api-access-674qx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 20:15:49.201495 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:15:49.201479    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/dc10e362-8836-47e1-8b31-2d71f5cb82bd-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "dc10e362-8836-47e1-8b31-2d71f5cb82bd" (UID: "dc10e362-8836-47e1-8b31-2d71f5cb82bd"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 20:15:49.299884 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:15:49.299792    2574 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/dc10e362-8836-47e1-8b31-2d71f5cb82bd-kube-rbac-proxy-sar-config\") on node \"ip-10-0-136-183.ec2.internal\" DevicePath \"\""
Sep 09 20:15:49.299884 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:15:49.299833    2574 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/dc10e362-8836-47e1-8b31-2d71f5cb82bd-kserve-provision-location\") on node \"ip-10-0-136-183.ec2.internal\" DevicePath \"\""
Sep 09 20:15:49.299884 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:15:49.299843    2574 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/dc10e362-8836-47e1-8b31-2d71f5cb82bd-cabundle-cert\") on node \"ip-10-0-136-183.ec2.internal\" DevicePath \"\""
Sep 09 20:15:49.299884 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:15:49.299854    2574 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/dc10e362-8836-47e1-8b31-2d71f5cb82bd-proxy-tls\") on node \"ip-10-0-136-183.ec2.internal\" DevicePath \"\""
Sep 09 20:15:49.299884 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:15:49.299862    2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-674qx\" (UniqueName: \"kubernetes.io/projected/dc10e362-8836-47e1-8b31-2d71f5cb82bd-kube-api-access-674qx\") on node \"ip-10-0-136-183.ec2.internal\" DevicePath \"\""
Sep 09 20:15:49.331197 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:15:49.331157    2574 generic.go:356] "Generic (PLEG): container finished" podID="dc10e362-8836-47e1-8b31-2d71f5cb82bd" containerID="91b16af588a0ce6944e78c2401998c6a79c9f5998f7fcd548a45ba003728688e" exitCode=0
Sep 09 20:15:49.331347 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:15:49.331204    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/raw-sklearn-ca752-predictor-768df8d5ff-kmb78" event={"ID":"dc10e362-8836-47e1-8b31-2d71f5cb82bd","Type":"ContainerDied","Data":"91b16af588a0ce6944e78c2401998c6a79c9f5998f7fcd548a45ba003728688e"}
Sep 09 20:15:49.331347 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:15:49.331233    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/raw-sklearn-ca752-predictor-768df8d5ff-kmb78"
Sep 09 20:15:49.331347 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:15:49.331296    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/raw-sklearn-ca752-predictor-768df8d5ff-kmb78" event={"ID":"dc10e362-8836-47e1-8b31-2d71f5cb82bd","Type":"ContainerDied","Data":"6e0e91c2eaf15c06e4d453e1de5fc4078cf0ba41dbef375824a96d74504bdfb7"}
Sep 09 20:15:49.331347 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:15:49.331333    2574 scope.go:117] "RemoveContainer" containerID="6b44bfd1e502163c0f34e9716b02aab9df3cf5fe39049501d95237e5e3250e13"
Sep 09 20:15:49.340818 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:15:49.340791    2574 scope.go:117] "RemoveContainer" containerID="91b16af588a0ce6944e78c2401998c6a79c9f5998f7fcd548a45ba003728688e"
Sep 09 20:15:49.348098 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:15:49.348081    2574 scope.go:117] "RemoveContainer" containerID="c49d7a86289811dd28bdac8a10b9120c4e11e7f1cbd8b0be69cdaba786e94ccd"
Sep 09 20:15:49.355013 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:15:49.354986    2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/raw-sklearn-ca752-predictor-768df8d5ff-kmb78"]
Sep 09 20:15:49.356052 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:15:49.356037    2574 scope.go:117] "RemoveContainer" containerID="6b44bfd1e502163c0f34e9716b02aab9df3cf5fe39049501d95237e5e3250e13"
Sep 09 20:15:49.356620 ip-10-0-136-183 kubenswrapper[2574]: E0909 20:15:49.356421    2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6b44bfd1e502163c0f34e9716b02aab9df3cf5fe39049501d95237e5e3250e13\": container with ID starting with 6b44bfd1e502163c0f34e9716b02aab9df3cf5fe39049501d95237e5e3250e13 not found: ID does not exist" containerID="6b44bfd1e502163c0f34e9716b02aab9df3cf5fe39049501d95237e5e3250e13"
Sep 09 20:15:49.356737 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:15:49.356639    2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6b44bfd1e502163c0f34e9716b02aab9df3cf5fe39049501d95237e5e3250e13"} err="failed to get container status \"6b44bfd1e502163c0f34e9716b02aab9df3cf5fe39049501d95237e5e3250e13\": rpc error: code = NotFound desc = could not find container \"6b44bfd1e502163c0f34e9716b02aab9df3cf5fe39049501d95237e5e3250e13\": container with ID starting with 6b44bfd1e502163c0f34e9716b02aab9df3cf5fe39049501d95237e5e3250e13 not found: ID does not exist"
Sep 09 20:15:49.356737 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:15:49.356669    2574 scope.go:117] "RemoveContainer" containerID="91b16af588a0ce6944e78c2401998c6a79c9f5998f7fcd548a45ba003728688e"
Sep 09 20:15:49.357096 ip-10-0-136-183 kubenswrapper[2574]: E0909 20:15:49.357072    2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"91b16af588a0ce6944e78c2401998c6a79c9f5998f7fcd548a45ba003728688e\": container with ID starting with 91b16af588a0ce6944e78c2401998c6a79c9f5998f7fcd548a45ba003728688e not found: ID does not exist" containerID="91b16af588a0ce6944e78c2401998c6a79c9f5998f7fcd548a45ba003728688e"
Sep 09 20:15:49.357166 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:15:49.357105    2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"91b16af588a0ce6944e78c2401998c6a79c9f5998f7fcd548a45ba003728688e"} err="failed to get container status \"91b16af588a0ce6944e78c2401998c6a79c9f5998f7fcd548a45ba003728688e\": rpc error: code = NotFound desc = could not find container \"91b16af588a0ce6944e78c2401998c6a79c9f5998f7fcd548a45ba003728688e\": container with ID starting with 91b16af588a0ce6944e78c2401998c6a79c9f5998f7fcd548a45ba003728688e not found: ID does not exist"
Sep 09 20:15:49.357166 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:15:49.357122    2574 scope.go:117] "RemoveContainer" containerID="c49d7a86289811dd28bdac8a10b9120c4e11e7f1cbd8b0be69cdaba786e94ccd"
Sep 09 20:15:49.357390 ip-10-0-136-183 kubenswrapper[2574]: E0909 20:15:49.357369    2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c49d7a86289811dd28bdac8a10b9120c4e11e7f1cbd8b0be69cdaba786e94ccd\": container with ID starting with c49d7a86289811dd28bdac8a10b9120c4e11e7f1cbd8b0be69cdaba786e94ccd not found: ID does not exist" containerID="c49d7a86289811dd28bdac8a10b9120c4e11e7f1cbd8b0be69cdaba786e94ccd"
Sep 09 20:15:49.357491 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:15:49.357396    2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c49d7a86289811dd28bdac8a10b9120c4e11e7f1cbd8b0be69cdaba786e94ccd"} err="failed to get container status \"c49d7a86289811dd28bdac8a10b9120c4e11e7f1cbd8b0be69cdaba786e94ccd\": rpc error: code = NotFound desc = could not find container \"c49d7a86289811dd28bdac8a10b9120c4e11e7f1cbd8b0be69cdaba786e94ccd\": container with ID starting with c49d7a86289811dd28bdac8a10b9120c4e11e7f1cbd8b0be69cdaba786e94ccd not found: ID does not exist"
Sep 09 20:15:49.358496 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:15:49.358475    2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/raw-sklearn-ca752-predictor-768df8d5ff-kmb78"]
Sep 09 20:15:50.490549 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:15:50.490514    2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="dc10e362-8836-47e1-8b31-2d71f5cb82bd" path="/var/lib/kubelet/pods/dc10e362-8836-47e1-8b31-2d71f5cb82bd/volumes"
Sep 09 20:15:58.486946 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:15:58.486899    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-0b35a-predictor-8587c87585-8q262" podUID="2c75cf85-1abd-4fe0-bdf6-b405acf5d913" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.34:8080: connect: connection refused"
Sep 09 20:16:08.487172 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:16:08.487073    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-0b35a-predictor-8587c87585-8q262" podUID="2c75cf85-1abd-4fe0-bdf6-b405acf5d913" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.34:8080: connect: connection refused"
Sep 09 20:16:18.486501 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:16:18.486452    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-0b35a-predictor-8587c87585-8q262" podUID="2c75cf85-1abd-4fe0-bdf6-b405acf5d913" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.34:8080: connect: connection refused"
Sep 09 20:16:28.486517 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:16:28.486471    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-0b35a-predictor-8587c87585-8q262" podUID="2c75cf85-1abd-4fe0-bdf6-b405acf5d913" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.34:8080: connect: connection refused"
Sep 09 20:16:38.487059 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:16:38.487017    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-0b35a-predictor-8587c87585-8q262" podUID="2c75cf85-1abd-4fe0-bdf6-b405acf5d913" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.34:8080: connect: connection refused"
Sep 09 20:16:45.486094 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:16:45.486043    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-0b35a-predictor-8587c87585-8q262" podUID="2c75cf85-1abd-4fe0-bdf6-b405acf5d913" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.34:8080: connect: connection refused"
Sep 09 20:16:55.486237 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:16:55.486193    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-0b35a-predictor-8587c87585-8q262" podUID="2c75cf85-1abd-4fe0-bdf6-b405acf5d913" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.34:8080: connect: connection refused"
Sep 09 20:17:05.486834 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:17:05.486776    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-0b35a-predictor-8587c87585-8q262"
Sep 09 20:17:13.011356 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:17:13.011316    2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-scale-raw-0b35a-predictor-8587c87585-8q262"]
Sep 09 20:17:13.011898 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:17:13.011622    2574 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-0b35a-predictor-8587c87585-8q262" podUID="2c75cf85-1abd-4fe0-bdf6-b405acf5d913" containerName="kserve-container" containerID="cri-o://33889ab127b9c7a3432996255792d2625fce079ccae22abc81baedc7af3e92ec" gracePeriod=30
Sep 09 20:17:13.011898 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:17:13.011679    2574 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-0b35a-predictor-8587c87585-8q262" podUID="2c75cf85-1abd-4fe0-bdf6-b405acf5d913" containerName="kube-rbac-proxy" containerID="cri-o://cb85f6de203abb32e2f25742368e4db643fcfbd21046dbfec9e36af39a9bd60f" gracePeriod=30
Sep 09 20:17:13.655597 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:17:13.655556    2574 generic.go:356] "Generic (PLEG): container finished" podID="2c75cf85-1abd-4fe0-bdf6-b405acf5d913" containerID="cb85f6de203abb32e2f25742368e4db643fcfbd21046dbfec9e36af39a9bd60f" exitCode=2
Sep 09 20:17:13.655769 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:17:13.655625    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-0b35a-predictor-8587c87585-8q262" event={"ID":"2c75cf85-1abd-4fe0-bdf6-b405acf5d913","Type":"ContainerDied","Data":"cb85f6de203abb32e2f25742368e4db643fcfbd21046dbfec9e36af39a9bd60f"}
Sep 09 20:17:15.486388 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:17:15.486344    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-0b35a-predictor-8587c87585-8q262" podUID="2c75cf85-1abd-4fe0-bdf6-b405acf5d913" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.34:8080: connect: connection refused"
Sep 09 20:17:16.015508 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:17:16.015465    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-0b35a-predictor-8587c87585-8q262" podUID="2c75cf85-1abd-4fe0-bdf6-b405acf5d913" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.34:8643/healthz\": dial tcp 10.133.0.34:8643: connect: connection refused"
Sep 09 20:17:18.192598 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:17:18.192560    2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-v2-predictor-579784d7d7-rztjm"]
Sep 09 20:17:18.193040 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:17:18.193022    2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="47632e59-c409-48e0-8d16-e23fa1f15697" containerName="kube-rbac-proxy"
Sep 09 20:17:18.193085 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:17:18.193043    2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="47632e59-c409-48e0-8d16-e23fa1f15697" containerName="kube-rbac-proxy"
Sep 09 20:17:18.193085 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:17:18.193057    2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="47632e59-c409-48e0-8d16-e23fa1f15697" containerName="kserve-container"
Sep 09 20:17:18.193085 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:17:18.193066    2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="47632e59-c409-48e0-8d16-e23fa1f15697" containerName="kserve-container"
Sep 09 20:17:18.193183 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:17:18.193085    2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="dc10e362-8836-47e1-8b31-2d71f5cb82bd" containerName="kserve-container"
Sep 09 20:17:18.193183 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:17:18.193094    2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="dc10e362-8836-47e1-8b31-2d71f5cb82bd" containerName="kserve-container"
Sep 09 20:17:18.193183 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:17:18.193111    2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="dc10e362-8836-47e1-8b31-2d71f5cb82bd" containerName="storage-initializer"
Sep 09 20:17:18.193183 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:17:18.193119    2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="dc10e362-8836-47e1-8b31-2d71f5cb82bd" containerName="storage-initializer"
Sep 09 20:17:18.193183 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:17:18.193135    2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="dc10e362-8836-47e1-8b31-2d71f5cb82bd" containerName="kube-rbac-proxy"
Sep 09 20:17:18.193183 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:17:18.193145    2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="dc10e362-8836-47e1-8b31-2d71f5cb82bd" containerName="kube-rbac-proxy"
Sep 09 20:17:18.193364 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:17:18.193213    2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="47632e59-c409-48e0-8d16-e23fa1f15697" containerName="kserve-container"
Sep 09 20:17:18.193364 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:17:18.193224    2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="dc10e362-8836-47e1-8b31-2d71f5cb82bd" containerName="kube-rbac-proxy"
Sep 09 20:17:18.193364 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:17:18.193237    2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="47632e59-c409-48e0-8d16-e23fa1f15697" containerName="kube-rbac-proxy"
Sep 09 20:17:18.193364 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:17:18.193246    2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="dc10e362-8836-47e1-8b31-2d71f5cb82bd" containerName="kserve-container"
Sep 09 20:17:18.196746 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:17:18.196726    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-579784d7d7-rztjm"
Sep 09 20:17:18.198543 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:17:18.198522    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-create-v2-predictor-serving-cert\""
Sep 09 20:17:18.198645 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:17:18.198577    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-create-kube-rbac-proxy-sar-config\""
Sep 09 20:17:18.204191 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:17:18.204168    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-v2-predictor-579784d7d7-rztjm"]
Sep 09 20:17:18.281711 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:17:18.281675    2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-predictor-68984b9498-8pzgp"]
Sep 09 20:17:18.283291 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:17:18.283269    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/20bf029a-9d4f-4e81-a894-cf19df755f7e-cabundle-cert\") pod \"isvc-canary-create-v2-predictor-579784d7d7-rztjm\" (UID: \"20bf029a-9d4f-4e81-a894-cf19df755f7e\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-579784d7d7-rztjm"
Sep 09 20:17:18.283388 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:17:18.283300    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/20bf029a-9d4f-4e81-a894-cf19df755f7e-proxy-tls\") pod \"isvc-canary-create-v2-predictor-579784d7d7-rztjm\" (UID: \"20bf029a-9d4f-4e81-a894-cf19df755f7e\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-579784d7d7-rztjm"
Sep 09 20:17:18.283388 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:17:18.283325    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hsffd\" (UniqueName: \"kubernetes.io/projected/20bf029a-9d4f-4e81-a894-cf19df755f7e-kube-api-access-hsffd\") pod \"isvc-canary-create-v2-predictor-579784d7d7-rztjm\" (UID: \"20bf029a-9d4f-4e81-a894-cf19df755f7e\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-579784d7d7-rztjm"
Sep 09 20:17:18.283466 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:17:18.283418    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/20bf029a-9d4f-4e81-a894-cf19df755f7e-kube-rbac-proxy-sar-config\") pod \"isvc-canary-create-v2-predictor-579784d7d7-rztjm\" (UID: \"20bf029a-9d4f-4e81-a894-cf19df755f7e\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-579784d7d7-rztjm"
Sep 09 20:17:18.283466 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:17:18.283460    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/20bf029a-9d4f-4e81-a894-cf19df755f7e-kserve-provision-location\") pod \"isvc-canary-create-v2-predictor-579784d7d7-rztjm\" (UID: \"20bf029a-9d4f-4e81-a894-cf19df755f7e\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-579784d7d7-rztjm"
Sep 09 20:17:18.284933 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:17:18.284919    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-68984b9498-8pzgp"
Sep 09 20:17:18.287280 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:17:18.287258    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-create-predictor-serving-cert\""
Sep 09 20:17:18.300252 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:17:18.300229    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-predictor-68984b9498-8pzgp"]
Sep 09 20:17:18.384187 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:17:18.384156    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/20bf029a-9d4f-4e81-a894-cf19df755f7e-kserve-provision-location\") pod \"isvc-canary-create-v2-predictor-579784d7d7-rztjm\" (UID: \"20bf029a-9d4f-4e81-a894-cf19df755f7e\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-579784d7d7-rztjm"
Sep 09 20:17:18.384446 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:17:18.384197    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c5d108da-0802-4311-9fc0-24e4fb6bb64e-proxy-tls\") pod \"isvc-canary-create-predictor-68984b9498-8pzgp\" (UID: \"c5d108da-0802-4311-9fc0-24e4fb6bb64e\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-68984b9498-8pzgp"
Sep 09 20:17:18.384446 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:17:18.384233    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/c5d108da-0802-4311-9fc0-24e4fb6bb64e-kube-rbac-proxy-sar-config\") pod \"isvc-canary-create-predictor-68984b9498-8pzgp\" (UID: \"c5d108da-0802-4311-9fc0-24e4fb6bb64e\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-68984b9498-8pzgp"
Sep 09 20:17:18.384446 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:17:18.384305    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/20bf029a-9d4f-4e81-a894-cf19df755f7e-cabundle-cert\") pod \"isvc-canary-create-v2-predictor-579784d7d7-rztjm\" (UID: \"20bf029a-9d4f-4e81-a894-cf19df755f7e\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-579784d7d7-rztjm"
Sep 09 20:17:18.384446 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:17:18.384426    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/20bf029a-9d4f-4e81-a894-cf19df755f7e-proxy-tls\") pod \"isvc-canary-create-v2-predictor-579784d7d7-rztjm\" (UID: \"20bf029a-9d4f-4e81-a894-cf19df755f7e\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-579784d7d7-rztjm"
Sep 09 20:17:18.384609 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:17:18.384470    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lvknl\" (UniqueName: \"kubernetes.io/projected/c5d108da-0802-4311-9fc0-24e4fb6bb64e-kube-api-access-lvknl\") pod \"isvc-canary-create-predictor-68984b9498-8pzgp\" (UID: \"c5d108da-0802-4311-9fc0-24e4fb6bb64e\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-68984b9498-8pzgp"
Sep 09 20:17:18.384609 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:17:18.384490    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hsffd\" (UniqueName: \"kubernetes.io/projected/20bf029a-9d4f-4e81-a894-cf19df755f7e-kube-api-access-hsffd\") pod \"isvc-canary-create-v2-predictor-579784d7d7-rztjm\" (UID: \"20bf029a-9d4f-4e81-a894-cf19df755f7e\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-579784d7d7-rztjm"
Sep 09 20:17:18.384609 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:17:18.384523    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/20bf029a-9d4f-4e81-a894-cf19df755f7e-kube-rbac-proxy-sar-config\") pod \"isvc-canary-create-v2-predictor-579784d7d7-rztjm\" (UID: \"20bf029a-9d4f-4e81-a894-cf19df755f7e\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-579784d7d7-rztjm"
Sep 09 20:17:18.384609 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:17:18.384525    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/20bf029a-9d4f-4e81-a894-cf19df755f7e-kserve-provision-location\") pod \"isvc-canary-create-v2-predictor-579784d7d7-rztjm\" (UID: \"20bf029a-9d4f-4e81-a894-cf19df755f7e\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-579784d7d7-rztjm"
Sep 09 20:17:18.384609 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:17:18.384570    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c5d108da-0802-4311-9fc0-24e4fb6bb64e-kserve-provision-location\") pod \"isvc-canary-create-predictor-68984b9498-8pzgp\" (UID: \"c5d108da-0802-4311-9fc0-24e4fb6bb64e\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-68984b9498-8pzgp"
Sep 09 20:17:18.384609 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:17:18.384603    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/c5d108da-0802-4311-9fc0-24e4fb6bb64e-cabundle-cert\") pod \"isvc-canary-create-predictor-68984b9498-8pzgp\" (UID: \"c5d108da-0802-4311-9fc0-24e4fb6bb64e\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-68984b9498-8pzgp"
Sep 09 20:17:18.385043 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:17:18.385011    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/20bf029a-9d4f-4e81-a894-cf19df755f7e-cabundle-cert\") pod \"isvc-canary-create-v2-predictor-579784d7d7-rztjm\" (UID: \"20bf029a-9d4f-4e81-a894-cf19df755f7e\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-579784d7d7-rztjm"
Sep 09 20:17:18.385204 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:17:18.385188    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/20bf029a-9d4f-4e81-a894-cf19df755f7e-kube-rbac-proxy-sar-config\") pod \"isvc-canary-create-v2-predictor-579784d7d7-rztjm\" (UID: \"20bf029a-9d4f-4e81-a894-cf19df755f7e\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-579784d7d7-rztjm"
Sep 09 20:17:18.386860 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:17:18.386839    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/20bf029a-9d4f-4e81-a894-cf19df755f7e-proxy-tls\") pod \"isvc-canary-create-v2-predictor-579784d7d7-rztjm\" (UID: \"20bf029a-9d4f-4e81-a894-cf19df755f7e\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-579784d7d7-rztjm"
Sep 09 20:17:18.394689 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:17:18.394668    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hsffd\" (UniqueName: \"kubernetes.io/projected/20bf029a-9d4f-4e81-a894-cf19df755f7e-kube-api-access-hsffd\") pod \"isvc-canary-create-v2-predictor-579784d7d7-rztjm\" (UID: \"20bf029a-9d4f-4e81-a894-cf19df755f7e\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-579784d7d7-rztjm"
Sep 09 20:17:18.485575 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:17:18.485476    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lvknl\" (UniqueName: \"kubernetes.io/projected/c5d108da-0802-4311-9fc0-24e4fb6bb64e-kube-api-access-lvknl\") pod \"isvc-canary-create-predictor-68984b9498-8pzgp\" (UID: \"c5d108da-0802-4311-9fc0-24e4fb6bb64e\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-68984b9498-8pzgp"
Sep 09 20:17:18.485575 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:17:18.485570    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c5d108da-0802-4311-9fc0-24e4fb6bb64e-kserve-provision-location\") pod \"isvc-canary-create-predictor-68984b9498-8pzgp\" (UID: \"c5d108da-0802-4311-9fc0-24e4fb6bb64e\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-68984b9498-8pzgp"
Sep 09 20:17:18.485869 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:17:18.485599    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/c5d108da-0802-4311-9fc0-24e4fb6bb64e-cabundle-cert\") pod \"isvc-canary-create-predictor-68984b9498-8pzgp\" (UID: \"c5d108da-0802-4311-9fc0-24e4fb6bb64e\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-68984b9498-8pzgp"
Sep 09 20:17:18.485869 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:17:18.485660    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c5d108da-0802-4311-9fc0-24e4fb6bb64e-proxy-tls\") pod \"isvc-canary-create-predictor-68984b9498-8pzgp\" (UID: \"c5d108da-0802-4311-9fc0-24e4fb6bb64e\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-68984b9498-8pzgp"
Sep 09 20:17:18.485869 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:17:18.485687    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/c5d108da-0802-4311-9fc0-24e4fb6bb64e-kube-rbac-proxy-sar-config\") pod \"isvc-canary-create-predictor-68984b9498-8pzgp\" (UID: \"c5d108da-0802-4311-9fc0-24e4fb6bb64e\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-68984b9498-8pzgp"
Sep 09 20:17:18.486055 ip-10-0-136-183 kubenswrapper[2574]: E0909 20:17:18.486022    2574 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-canary-create-predictor-serving-cert: secret "isvc-canary-create-predictor-serving-cert" not found
Sep 09 20:17:18.486055 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:17:18.486045    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c5d108da-0802-4311-9fc0-24e4fb6bb64e-kserve-provision-location\") pod \"isvc-canary-create-predictor-68984b9498-8pzgp\" (UID: \"c5d108da-0802-4311-9fc0-24e4fb6bb64e\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-68984b9498-8pzgp"
Sep 09 20:17:18.486171 ip-10-0-136-183 kubenswrapper[2574]: E0909 20:17:18.486085    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c5d108da-0802-4311-9fc0-24e4fb6bb64e-proxy-tls podName:c5d108da-0802-4311-9fc0-24e4fb6bb64e nodeName:}" failed. No retries permitted until 2026-09-09 20:17:18.986066154 +0000 UTC m=+1323.032319408 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/c5d108da-0802-4311-9fc0-24e4fb6bb64e-proxy-tls") pod "isvc-canary-create-predictor-68984b9498-8pzgp" (UID: "c5d108da-0802-4311-9fc0-24e4fb6bb64e") : secret "isvc-canary-create-predictor-serving-cert" not found
Sep 09 20:17:18.486352 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:17:18.486330    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/c5d108da-0802-4311-9fc0-24e4fb6bb64e-kube-rbac-proxy-sar-config\") pod \"isvc-canary-create-predictor-68984b9498-8pzgp\" (UID: \"c5d108da-0802-4311-9fc0-24e4fb6bb64e\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-68984b9498-8pzgp"
Sep 09 20:17:18.486520 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:17:18.486497    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/c5d108da-0802-4311-9fc0-24e4fb6bb64e-cabundle-cert\") pod \"isvc-canary-create-predictor-68984b9498-8pzgp\" (UID: \"c5d108da-0802-4311-9fc0-24e4fb6bb64e\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-68984b9498-8pzgp"
Sep 09 20:17:18.495455 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:17:18.495430    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lvknl\" (UniqueName: \"kubernetes.io/projected/c5d108da-0802-4311-9fc0-24e4fb6bb64e-kube-api-access-lvknl\") pod \"isvc-canary-create-predictor-68984b9498-8pzgp\" (UID: \"c5d108da-0802-4311-9fc0-24e4fb6bb64e\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-68984b9498-8pzgp"
Sep 09 20:17:18.508320 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:17:18.508295    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-579784d7d7-rztjm"
Sep 09 20:17:18.781418 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:17:18.781334    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-v2-predictor-579784d7d7-rztjm"]
Sep 09 20:17:18.784461 ip-10-0-136-183 kubenswrapper[2574]: W0909 20:17:18.784432    2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod20bf029a_9d4f_4e81_a894_cf19df755f7e.slice/crio-30525e0e026cb817ee2dbdb8e35aad8c18a98822d4cdadefaf3f0536639a1834 WatchSource:0}: Error finding container 30525e0e026cb817ee2dbdb8e35aad8c18a98822d4cdadefaf3f0536639a1834: Status 404 returned error can't find the container with id 30525e0e026cb817ee2dbdb8e35aad8c18a98822d4cdadefaf3f0536639a1834
Sep 09 20:17:18.990845 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:17:18.990791    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c5d108da-0802-4311-9fc0-24e4fb6bb64e-proxy-tls\") pod \"isvc-canary-create-predictor-68984b9498-8pzgp\" (UID: \"c5d108da-0802-4311-9fc0-24e4fb6bb64e\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-68984b9498-8pzgp"
Sep 09 20:17:18.993181 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:17:18.993156    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c5d108da-0802-4311-9fc0-24e4fb6bb64e-proxy-tls\") pod \"isvc-canary-create-predictor-68984b9498-8pzgp\" (UID: \"c5d108da-0802-4311-9fc0-24e4fb6bb64e\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-68984b9498-8pzgp"
Sep 09 20:17:19.195517 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:17:19.195479    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-68984b9498-8pzgp"
Sep 09 20:17:19.481154 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:17:19.481095    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-predictor-68984b9498-8pzgp"]
Sep 09 20:17:19.483727 ip-10-0-136-183 kubenswrapper[2574]: W0909 20:17:19.483698    2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc5d108da_0802_4311_9fc0_24e4fb6bb64e.slice/crio-5f4c73acaad7902383ca0005938553ee1e8d17859b6badd2778af4f450463a34 WatchSource:0}: Error finding container 5f4c73acaad7902383ca0005938553ee1e8d17859b6badd2778af4f450463a34: Status 404 returned error can't find the container with id 5f4c73acaad7902383ca0005938553ee1e8d17859b6badd2778af4f450463a34
Sep 09 20:17:19.680615 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:17:19.680576    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-579784d7d7-rztjm" event={"ID":"20bf029a-9d4f-4e81-a894-cf19df755f7e","Type":"ContainerStarted","Data":"6bffa6d14c51f973cfe9228ad42c3b863bd8036b3620de258795c98725fa39b4"}
Sep 09 20:17:19.680854 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:17:19.680654    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-579784d7d7-rztjm" event={"ID":"20bf029a-9d4f-4e81-a894-cf19df755f7e","Type":"ContainerStarted","Data":"30525e0e026cb817ee2dbdb8e35aad8c18a98822d4cdadefaf3f0536639a1834"}
Sep 09 20:17:19.682171 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:17:19.682137    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-68984b9498-8pzgp" event={"ID":"c5d108da-0802-4311-9fc0-24e4fb6bb64e","Type":"ContainerStarted","Data":"d50490f2e7ec0589939e801824b2e31de6afb01a8c084ab800378a67ca0b02ae"}
Sep 09 20:17:19.682283 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:17:19.682186    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-68984b9498-8pzgp" event={"ID":"c5d108da-0802-4311-9fc0-24e4fb6bb64e","Type":"ContainerStarted","Data":"5f4c73acaad7902383ca0005938553ee1e8d17859b6badd2778af4f450463a34"}
Sep 09 20:17:21.015038 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:17:21.014993    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-0b35a-predictor-8587c87585-8q262" podUID="2c75cf85-1abd-4fe0-bdf6-b405acf5d913" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.34:8643/healthz\": dial tcp 10.133.0.34:8643: connect: connection refused"
Sep 09 20:17:23.157633 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:17:23.157607    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-0b35a-predictor-8587c87585-8q262"
Sep 09 20:17:23.225241 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:17:23.225206    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/2c75cf85-1abd-4fe0-bdf6-b405acf5d913-kserve-provision-location\") pod \"2c75cf85-1abd-4fe0-bdf6-b405acf5d913\" (UID: \"2c75cf85-1abd-4fe0-bdf6-b405acf5d913\") "
Sep 09 20:17:23.225357 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:17:23.225244    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/2c75cf85-1abd-4fe0-bdf6-b405acf5d913-kube-rbac-proxy-sar-config\") pod \"2c75cf85-1abd-4fe0-bdf6-b405acf5d913\" (UID: \"2c75cf85-1abd-4fe0-bdf6-b405acf5d913\") "
Sep 09 20:17:23.225357 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:17:23.225287    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/2c75cf85-1abd-4fe0-bdf6-b405acf5d913-proxy-tls\") pod \"2c75cf85-1abd-4fe0-bdf6-b405acf5d913\" (UID: \"2c75cf85-1abd-4fe0-bdf6-b405acf5d913\") "
Sep 09 20:17:23.225357 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:17:23.225308    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/2c75cf85-1abd-4fe0-bdf6-b405acf5d913-cabundle-cert\") pod \"2c75cf85-1abd-4fe0-bdf6-b405acf5d913\" (UID: \"2c75cf85-1abd-4fe0-bdf6-b405acf5d913\") "
Sep 09 20:17:23.225357 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:17:23.225326    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-lgv94\" (UniqueName: \"kubernetes.io/projected/2c75cf85-1abd-4fe0-bdf6-b405acf5d913-kube-api-access-lgv94\") pod \"2c75cf85-1abd-4fe0-bdf6-b405acf5d913\" (UID: \"2c75cf85-1abd-4fe0-bdf6-b405acf5d913\") "
Sep 09 20:17:23.225649 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:17:23.225553    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2c75cf85-1abd-4fe0-bdf6-b405acf5d913-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "2c75cf85-1abd-4fe0-bdf6-b405acf5d913" (UID: "2c75cf85-1abd-4fe0-bdf6-b405acf5d913"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 20:17:23.225722 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:17:23.225682    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2c75cf85-1abd-4fe0-bdf6-b405acf5d913-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "2c75cf85-1abd-4fe0-bdf6-b405acf5d913" (UID: "2c75cf85-1abd-4fe0-bdf6-b405acf5d913"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 20:17:23.225722 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:17:23.225688    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2c75cf85-1abd-4fe0-bdf6-b405acf5d913-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "2c75cf85-1abd-4fe0-bdf6-b405acf5d913" (UID: "2c75cf85-1abd-4fe0-bdf6-b405acf5d913"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 20:17:23.227306 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:17:23.227285    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2c75cf85-1abd-4fe0-bdf6-b405acf5d913-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "2c75cf85-1abd-4fe0-bdf6-b405acf5d913" (UID: "2c75cf85-1abd-4fe0-bdf6-b405acf5d913"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 20:17:23.227369 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:17:23.227350    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2c75cf85-1abd-4fe0-bdf6-b405acf5d913-kube-api-access-lgv94" (OuterVolumeSpecName: "kube-api-access-lgv94") pod "2c75cf85-1abd-4fe0-bdf6-b405acf5d913" (UID: "2c75cf85-1abd-4fe0-bdf6-b405acf5d913"). InnerVolumeSpecName "kube-api-access-lgv94". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 20:17:23.326292 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:17:23.326203    2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lgv94\" (UniqueName: \"kubernetes.io/projected/2c75cf85-1abd-4fe0-bdf6-b405acf5d913-kube-api-access-lgv94\") on node \"ip-10-0-136-183.ec2.internal\" DevicePath \"\""
Sep 09 20:17:23.326292 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:17:23.326232    2574 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/2c75cf85-1abd-4fe0-bdf6-b405acf5d913-kserve-provision-location\") on node \"ip-10-0-136-183.ec2.internal\" DevicePath \"\""
Sep 09 20:17:23.326292 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:17:23.326241    2574 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/2c75cf85-1abd-4fe0-bdf6-b405acf5d913-kube-rbac-proxy-sar-config\") on node \"ip-10-0-136-183.ec2.internal\" DevicePath \"\""
Sep 09 20:17:23.326292 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:17:23.326251    2574 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/2c75cf85-1abd-4fe0-bdf6-b405acf5d913-proxy-tls\") on node \"ip-10-0-136-183.ec2.internal\" DevicePath \"\""
Sep 09 20:17:23.326292 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:17:23.326259    2574 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/2c75cf85-1abd-4fe0-bdf6-b405acf5d913-cabundle-cert\") on node \"ip-10-0-136-183.ec2.internal\" DevicePath \"\""
Sep 09 20:17:23.699160 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:17:23.699126    2574 generic.go:356] "Generic (PLEG): container finished" podID="2c75cf85-1abd-4fe0-bdf6-b405acf5d913" containerID="33889ab127b9c7a3432996255792d2625fce079ccae22abc81baedc7af3e92ec" exitCode=0
Sep 09 20:17:23.699326 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:17:23.699180    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-0b35a-predictor-8587c87585-8q262"
Sep 09 20:17:23.699326 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:17:23.699201    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-0b35a-predictor-8587c87585-8q262" event={"ID":"2c75cf85-1abd-4fe0-bdf6-b405acf5d913","Type":"ContainerDied","Data":"33889ab127b9c7a3432996255792d2625fce079ccae22abc81baedc7af3e92ec"}
Sep 09 20:17:23.699326 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:17:23.699252    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-0b35a-predictor-8587c87585-8q262" event={"ID":"2c75cf85-1abd-4fe0-bdf6-b405acf5d913","Type":"ContainerDied","Data":"0acc97198c8d1aa14695b03aec1240761a5a88f7bea2d826f59706ddb12670da"}
Sep 09 20:17:23.699326 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:17:23.699279    2574 scope.go:117] "RemoveContainer" containerID="cb85f6de203abb32e2f25742368e4db643fcfbd21046dbfec9e36af39a9bd60f"
Sep 09 20:17:23.700826 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:17:23.700779    2574 generic.go:356] "Generic (PLEG): container finished" podID="20bf029a-9d4f-4e81-a894-cf19df755f7e" containerID="6bffa6d14c51f973cfe9228ad42c3b863bd8036b3620de258795c98725fa39b4" exitCode=0
Sep 09 20:17:23.700936 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:17:23.700846    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-579784d7d7-rztjm" event={"ID":"20bf029a-9d4f-4e81-a894-cf19df755f7e","Type":"ContainerDied","Data":"6bffa6d14c51f973cfe9228ad42c3b863bd8036b3620de258795c98725fa39b4"}
Sep 09 20:17:23.702648 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:17:23.702624    2574 generic.go:356] "Generic (PLEG): container finished" podID="c5d108da-0802-4311-9fc0-24e4fb6bb64e" containerID="d50490f2e7ec0589939e801824b2e31de6afb01a8c084ab800378a67ca0b02ae" exitCode=0
Sep 09 20:17:23.702738 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:17:23.702689    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-68984b9498-8pzgp" event={"ID":"c5d108da-0802-4311-9fc0-24e4fb6bb64e","Type":"ContainerDied","Data":"d50490f2e7ec0589939e801824b2e31de6afb01a8c084ab800378a67ca0b02ae"}
Sep 09 20:17:23.708897 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:17:23.708827    2574 scope.go:117] "RemoveContainer" containerID="33889ab127b9c7a3432996255792d2625fce079ccae22abc81baedc7af3e92ec"
Sep 09 20:17:23.716695 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:17:23.716678    2574 scope.go:117] "RemoveContainer" containerID="212d4e09d417a2ceadb9a7e3821ba1527407a1f7f915ca20ebe4f6de2da1574c"
Sep 09 20:17:23.737851 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:17:23.737793    2574 scope.go:117] "RemoveContainer" containerID="cb85f6de203abb32e2f25742368e4db643fcfbd21046dbfec9e36af39a9bd60f"
Sep 09 20:17:23.738185 ip-10-0-136-183 kubenswrapper[2574]: E0909 20:17:23.738165    2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cb85f6de203abb32e2f25742368e4db643fcfbd21046dbfec9e36af39a9bd60f\": container with ID starting with cb85f6de203abb32e2f25742368e4db643fcfbd21046dbfec9e36af39a9bd60f not found: ID does not exist" containerID="cb85f6de203abb32e2f25742368e4db643fcfbd21046dbfec9e36af39a9bd60f"
Sep 09 20:17:23.738273 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:17:23.738200    2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cb85f6de203abb32e2f25742368e4db643fcfbd21046dbfec9e36af39a9bd60f"} err="failed to get container status \"cb85f6de203abb32e2f25742368e4db643fcfbd21046dbfec9e36af39a9bd60f\": rpc error: code = NotFound desc = could not find container \"cb85f6de203abb32e2f25742368e4db643fcfbd21046dbfec9e36af39a9bd60f\": container with ID starting with cb85f6de203abb32e2f25742368e4db643fcfbd21046dbfec9e36af39a9bd60f not found: ID does not exist"
Sep 09 20:17:23.738273 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:17:23.738225    2574 scope.go:117] "RemoveContainer" containerID="33889ab127b9c7a3432996255792d2625fce079ccae22abc81baedc7af3e92ec"
Sep 09 20:17:23.738495 ip-10-0-136-183 kubenswrapper[2574]: E0909 20:17:23.738476    2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"33889ab127b9c7a3432996255792d2625fce079ccae22abc81baedc7af3e92ec\": container with ID starting with 33889ab127b9c7a3432996255792d2625fce079ccae22abc81baedc7af3e92ec not found: ID does not exist" containerID="33889ab127b9c7a3432996255792d2625fce079ccae22abc81baedc7af3e92ec"
Sep 09 20:17:23.738534 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:17:23.738501    2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"33889ab127b9c7a3432996255792d2625fce079ccae22abc81baedc7af3e92ec"} err="failed to get container status \"33889ab127b9c7a3432996255792d2625fce079ccae22abc81baedc7af3e92ec\": rpc error: code = NotFound desc = could not find container \"33889ab127b9c7a3432996255792d2625fce079ccae22abc81baedc7af3e92ec\": container with ID starting with 33889ab127b9c7a3432996255792d2625fce079ccae22abc81baedc7af3e92ec not found: ID does not exist"
Sep 09 20:17:23.738534 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:17:23.738515    2574 scope.go:117] "RemoveContainer" containerID="212d4e09d417a2ceadb9a7e3821ba1527407a1f7f915ca20ebe4f6de2da1574c"
Sep 09 20:17:23.738744 ip-10-0-136-183 kubenswrapper[2574]: E0909 20:17:23.738727    2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"212d4e09d417a2ceadb9a7e3821ba1527407a1f7f915ca20ebe4f6de2da1574c\": container with ID starting with 212d4e09d417a2ceadb9a7e3821ba1527407a1f7f915ca20ebe4f6de2da1574c not found: ID does not exist" containerID="212d4e09d417a2ceadb9a7e3821ba1527407a1f7f915ca20ebe4f6de2da1574c"
Sep 09 20:17:23.738813 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:17:23.738751    2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"212d4e09d417a2ceadb9a7e3821ba1527407a1f7f915ca20ebe4f6de2da1574c"} err="failed to get container status \"212d4e09d417a2ceadb9a7e3821ba1527407a1f7f915ca20ebe4f6de2da1574c\": rpc error: code = NotFound desc = could not find container \"212d4e09d417a2ceadb9a7e3821ba1527407a1f7f915ca20ebe4f6de2da1574c\": container with ID starting with 212d4e09d417a2ceadb9a7e3821ba1527407a1f7f915ca20ebe4f6de2da1574c not found: ID does not exist"
Sep 09 20:17:23.752938 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:17:23.752915    2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-scale-raw-0b35a-predictor-8587c87585-8q262"]
Sep 09 20:17:23.757689 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:17:23.757667    2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-scale-raw-0b35a-predictor-8587c87585-8q262"]
Sep 09 20:17:24.490918 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:17:24.490877    2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2c75cf85-1abd-4fe0-bdf6-b405acf5d913" path="/var/lib/kubelet/pods/2c75cf85-1abd-4fe0-bdf6-b405acf5d913/volumes"
Sep 09 20:17:24.710443 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:17:24.710409    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-579784d7d7-rztjm" event={"ID":"20bf029a-9d4f-4e81-a894-cf19df755f7e","Type":"ContainerStarted","Data":"baec69383ad1b7c26350dfeb837c4fff1b3bc6bdc1bc65304495a35ee0f6f47a"}
Sep 09 20:17:24.710601 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:17:24.710472    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-579784d7d7-rztjm" event={"ID":"20bf029a-9d4f-4e81-a894-cf19df755f7e","Type":"ContainerStarted","Data":"2a719f612c149e193bb8eb5dc83bc5691d1f8a948a5d8cbb76a759db732e84b6"}
Sep 09 20:17:24.710776 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:17:24.710753    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-579784d7d7-rztjm"
Sep 09 20:17:24.710920 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:17:24.710836    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-579784d7d7-rztjm"
Sep 09 20:17:24.711976 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:17:24.711941    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-579784d7d7-rztjm" podUID="20bf029a-9d4f-4e81-a894-cf19df755f7e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.35:8080: connect: connection refused"
Sep 09 20:17:24.712566 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:17:24.712543    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-68984b9498-8pzgp" event={"ID":"c5d108da-0802-4311-9fc0-24e4fb6bb64e","Type":"ContainerStarted","Data":"d1c33395cb78f6681d76c9fa305b614278388e752e62fb467ec97dd8d1d65450"}
Sep 09 20:17:24.712656 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:17:24.712602    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-68984b9498-8pzgp" event={"ID":"c5d108da-0802-4311-9fc0-24e4fb6bb64e","Type":"ContainerStarted","Data":"50da26306e84d5404c61a078030ecca7fe20114024c5bced8340b85bfff62036"}
Sep 09 20:17:24.712877 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:17:24.712858    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-68984b9498-8pzgp"
Sep 09 20:17:24.712962 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:17:24.712897    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-68984b9498-8pzgp"
Sep 09 20:17:24.713930 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:17:24.713907    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-68984b9498-8pzgp" podUID="c5d108da-0802-4311-9fc0-24e4fb6bb64e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.36:8080: connect: connection refused"
Sep 09 20:17:24.731241 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:17:24.731195    2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-579784d7d7-rztjm" podStartSLOduration=2.86094957 podStartE2EDuration="6.731180296s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.870230726s" isStatefulPod=false podCreationTimestamp="2026-09-09 20:17:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:17:24.729092487 +0000 UTC m=+1328.775345748" watchObservedRunningTime="2026-09-09 20:17:24.731180296 +0000 UTC m=+1328.777433558"
Sep 09 20:17:24.746168 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:17:24.746068    2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-68984b9498-8pzgp" podStartSLOduration=3.092977618 podStartE2EDuration="6.746052s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.653074382s" isStatefulPod=false podCreationTimestamp="2026-09-09 20:17:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:17:24.744454739 +0000 UTC m=+1328.790708016" watchObservedRunningTime="2026-09-09 20:17:24.746052 +0000 UTC m=+1328.792305262"
Sep 09 20:17:25.716480 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:17:25.716433    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-68984b9498-8pzgp" podUID="c5d108da-0802-4311-9fc0-24e4fb6bb64e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.36:8080: connect: connection refused"
Sep 09 20:17:25.716878 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:17:25.716554    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-579784d7d7-rztjm" podUID="20bf029a-9d4f-4e81-a894-cf19df755f7e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.35:8080: connect: connection refused"
Sep 09 20:17:30.721678 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:17:30.721648    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-68984b9498-8pzgp"
Sep 09 20:17:30.722226 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:17:30.722201    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-579784d7d7-rztjm"
Sep 09 20:17:30.722302 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:17:30.722235    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-68984b9498-8pzgp" podUID="c5d108da-0802-4311-9fc0-24e4fb6bb64e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.36:8080: connect: connection refused"
Sep 09 20:17:30.722723 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:17:30.722699    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-579784d7d7-rztjm" podUID="20bf029a-9d4f-4e81-a894-cf19df755f7e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.35:8080: connect: connection refused"
Sep 09 20:17:40.722422 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:17:40.722336    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-68984b9498-8pzgp" podUID="c5d108da-0802-4311-9fc0-24e4fb6bb64e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.36:8080: connect: connection refused"
Sep 09 20:17:40.722788 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:17:40.722588    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-579784d7d7-rztjm" podUID="20bf029a-9d4f-4e81-a894-cf19df755f7e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.35:8080: connect: connection refused"
Sep 09 20:17:50.722311 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:17:50.722268    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-68984b9498-8pzgp" podUID="c5d108da-0802-4311-9fc0-24e4fb6bb64e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.36:8080: connect: connection refused"
Sep 09 20:17:50.722776 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:17:50.722597    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-579784d7d7-rztjm" podUID="20bf029a-9d4f-4e81-a894-cf19df755f7e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.35:8080: connect: connection refused"
Sep 09 20:18:00.722997 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:18:00.722951    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-68984b9498-8pzgp" podUID="c5d108da-0802-4311-9fc0-24e4fb6bb64e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.36:8080: connect: connection refused"
Sep 09 20:18:00.723402 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:18:00.722962    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-579784d7d7-rztjm" podUID="20bf029a-9d4f-4e81-a894-cf19df755f7e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.35:8080: connect: connection refused"
Sep 09 20:18:10.722295 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:18:10.722253    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-68984b9498-8pzgp" podUID="c5d108da-0802-4311-9fc0-24e4fb6bb64e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.36:8080: connect: connection refused"
Sep 09 20:18:10.722700 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:18:10.722572    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-579784d7d7-rztjm" podUID="20bf029a-9d4f-4e81-a894-cf19df755f7e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.35:8080: connect: connection refused"
Sep 09 20:18:20.722760 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:18:20.722714    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-68984b9498-8pzgp" podUID="c5d108da-0802-4311-9fc0-24e4fb6bb64e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.36:8080: connect: connection refused"
Sep 09 20:18:20.723178 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:18:20.722714    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-579784d7d7-rztjm" podUID="20bf029a-9d4f-4e81-a894-cf19df755f7e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.35:8080: connect: connection refused"
Sep 09 20:18:30.723185 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:18:30.723140    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-68984b9498-8pzgp" podUID="c5d108da-0802-4311-9fc0-24e4fb6bb64e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.36:8080: connect: connection refused"
Sep 09 20:18:30.723666 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:18:30.723140    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-579784d7d7-rztjm" podUID="20bf029a-9d4f-4e81-a894-cf19df755f7e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.35:8080: connect: connection refused"
Sep 09 20:18:40.722985 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:18:40.722949    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-579784d7d7-rztjm"
Sep 09 20:18:40.723433 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:18:40.723083    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-68984b9498-8pzgp"
Sep 09 20:18:48.288997 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:18:48.288958    2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-predictor-68984b9498-8pzgp"]
Sep 09 20:18:48.289426 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:18:48.289281    2574 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-68984b9498-8pzgp" podUID="c5d108da-0802-4311-9fc0-24e4fb6bb64e" containerName="kserve-container" containerID="cri-o://50da26306e84d5404c61a078030ecca7fe20114024c5bced8340b85bfff62036" gracePeriod=30
Sep 09 20:18:48.289426 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:18:48.289344    2574 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-68984b9498-8pzgp" podUID="c5d108da-0802-4311-9fc0-24e4fb6bb64e" containerName="kube-rbac-proxy" containerID="cri-o://d1c33395cb78f6681d76c9fa305b614278388e752e62fb467ec97dd8d1d65450" gracePeriod=30
Sep 09 20:18:48.296471 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:18:48.296443    2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-v2-predictor-579784d7d7-rztjm"]
Sep 09 20:18:48.296775 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:18:48.296725    2574 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-579784d7d7-rztjm" podUID="20bf029a-9d4f-4e81-a894-cf19df755f7e" containerName="kserve-container" containerID="cri-o://2a719f612c149e193bb8eb5dc83bc5691d1f8a948a5d8cbb76a759db732e84b6" gracePeriod=30
Sep 09 20:18:48.297080 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:18:48.296819    2574 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-579784d7d7-rztjm" podUID="20bf029a-9d4f-4e81-a894-cf19df755f7e" containerName="kube-rbac-proxy" containerID="cri-o://baec69383ad1b7c26350dfeb837c4fff1b3bc6bdc1bc65304495a35ee0f6f47a" gracePeriod=30
Sep 09 20:18:49.040733 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:18:49.040696    2574 generic.go:356] "Generic (PLEG): container finished" podID="c5d108da-0802-4311-9fc0-24e4fb6bb64e" containerID="d1c33395cb78f6681d76c9fa305b614278388e752e62fb467ec97dd8d1d65450" exitCode=2
Sep 09 20:18:49.040940 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:18:49.040770    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-68984b9498-8pzgp" event={"ID":"c5d108da-0802-4311-9fc0-24e4fb6bb64e","Type":"ContainerDied","Data":"d1c33395cb78f6681d76c9fa305b614278388e752e62fb467ec97dd8d1d65450"}
Sep 09 20:18:49.042782 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:18:49.042763    2574 generic.go:356] "Generic (PLEG): container finished" podID="20bf029a-9d4f-4e81-a894-cf19df755f7e" containerID="baec69383ad1b7c26350dfeb837c4fff1b3bc6bdc1bc65304495a35ee0f6f47a" exitCode=2
Sep 09 20:18:49.042897 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:18:49.042818    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-579784d7d7-rztjm" event={"ID":"20bf029a-9d4f-4e81-a894-cf19df755f7e","Type":"ContainerDied","Data":"baec69383ad1b7c26350dfeb837c4fff1b3bc6bdc1bc65304495a35ee0f6f47a"}
Sep 09 20:18:50.717688 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:18:50.717639    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-68984b9498-8pzgp" podUID="c5d108da-0802-4311-9fc0-24e4fb6bb64e" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.36:8643/healthz\": dial tcp 10.133.0.36:8643: connect: connection refused"
Sep 09 20:18:50.718116 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:18:50.717637    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-579784d7d7-rztjm" podUID="20bf029a-9d4f-4e81-a894-cf19df755f7e" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.35:8643/healthz\": dial tcp 10.133.0.35:8643: connect: connection refused"
Sep 09 20:18:50.723011 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:18:50.722979    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-579784d7d7-rztjm" podUID="20bf029a-9d4f-4e81-a894-cf19df755f7e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.35:8080: connect: connection refused"
Sep 09 20:18:50.723134 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:18:50.722979    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-68984b9498-8pzgp" podUID="c5d108da-0802-4311-9fc0-24e4fb6bb64e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.36:8080: connect: connection refused"
Sep 09 20:18:53.371049 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:18:53.371020    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-68984b9498-8pzgp"
Sep 09 20:18:53.373495 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:18:53.373472    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-579784d7d7-rztjm"
Sep 09 20:18:53.464291 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:18:53.464213    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c5d108da-0802-4311-9fc0-24e4fb6bb64e-proxy-tls\") pod \"c5d108da-0802-4311-9fc0-24e4fb6bb64e\" (UID: \"c5d108da-0802-4311-9fc0-24e4fb6bb64e\") "
Sep 09 20:18:53.464291 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:18:53.464258    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c5d108da-0802-4311-9fc0-24e4fb6bb64e-kserve-provision-location\") pod \"c5d108da-0802-4311-9fc0-24e4fb6bb64e\" (UID: \"c5d108da-0802-4311-9fc0-24e4fb6bb64e\") "
Sep 09 20:18:53.464291 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:18:53.464290    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hsffd\" (UniqueName: \"kubernetes.io/projected/20bf029a-9d4f-4e81-a894-cf19df755f7e-kube-api-access-hsffd\") pod \"20bf029a-9d4f-4e81-a894-cf19df755f7e\" (UID: \"20bf029a-9d4f-4e81-a894-cf19df755f7e\") "
Sep 09 20:18:53.464563 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:18:53.464314    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/20bf029a-9d4f-4e81-a894-cf19df755f7e-cabundle-cert\") pod \"20bf029a-9d4f-4e81-a894-cf19df755f7e\" (UID: \"20bf029a-9d4f-4e81-a894-cf19df755f7e\") "
Sep 09 20:18:53.464563 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:18:53.464337    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/20bf029a-9d4f-4e81-a894-cf19df755f7e-kserve-provision-location\") pod \"20bf029a-9d4f-4e81-a894-cf19df755f7e\" (UID: \"20bf029a-9d4f-4e81-a894-cf19df755f7e\") "
Sep 09 20:18:53.464563 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:18:53.464352    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/c5d108da-0802-4311-9fc0-24e4fb6bb64e-cabundle-cert\") pod \"c5d108da-0802-4311-9fc0-24e4fb6bb64e\" (UID: \"c5d108da-0802-4311-9fc0-24e4fb6bb64e\") "
Sep 09 20:18:53.464563 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:18:53.464384    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/c5d108da-0802-4311-9fc0-24e4fb6bb64e-kube-rbac-proxy-sar-config\") pod \"c5d108da-0802-4311-9fc0-24e4fb6bb64e\" (UID: \"c5d108da-0802-4311-9fc0-24e4fb6bb64e\") "
Sep 09 20:18:53.464563 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:18:53.464411    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-lvknl\" (UniqueName: \"kubernetes.io/projected/c5d108da-0802-4311-9fc0-24e4fb6bb64e-kube-api-access-lvknl\") pod \"c5d108da-0802-4311-9fc0-24e4fb6bb64e\" (UID: \"c5d108da-0802-4311-9fc0-24e4fb6bb64e\") "
Sep 09 20:18:53.464563 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:18:53.464440    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/20bf029a-9d4f-4e81-a894-cf19df755f7e-kube-rbac-proxy-sar-config\") pod \"20bf029a-9d4f-4e81-a894-cf19df755f7e\" (UID: \"20bf029a-9d4f-4e81-a894-cf19df755f7e\") "
Sep 09 20:18:53.464563 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:18:53.464474    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/20bf029a-9d4f-4e81-a894-cf19df755f7e-proxy-tls\") pod \"20bf029a-9d4f-4e81-a894-cf19df755f7e\" (UID: \"20bf029a-9d4f-4e81-a894-cf19df755f7e\") "
Sep 09 20:18:53.464946 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:18:53.464680    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c5d108da-0802-4311-9fc0-24e4fb6bb64e-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "c5d108da-0802-4311-9fc0-24e4fb6bb64e" (UID: "c5d108da-0802-4311-9fc0-24e4fb6bb64e"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 20:18:53.464946 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:18:53.464685    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/20bf029a-9d4f-4e81-a894-cf19df755f7e-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "20bf029a-9d4f-4e81-a894-cf19df755f7e" (UID: "20bf029a-9d4f-4e81-a894-cf19df755f7e"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 20:18:53.464946 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:18:53.464741    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/20bf029a-9d4f-4e81-a894-cf19df755f7e-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "20bf029a-9d4f-4e81-a894-cf19df755f7e" (UID: "20bf029a-9d4f-4e81-a894-cf19df755f7e"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 20:18:53.464946 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:18:53.464770    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c5d108da-0802-4311-9fc0-24e4fb6bb64e-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "c5d108da-0802-4311-9fc0-24e4fb6bb64e" (UID: "c5d108da-0802-4311-9fc0-24e4fb6bb64e"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 20:18:53.465159 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:18:53.465099    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c5d108da-0802-4311-9fc0-24e4fb6bb64e-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "c5d108da-0802-4311-9fc0-24e4fb6bb64e" (UID: "c5d108da-0802-4311-9fc0-24e4fb6bb64e"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 20:18:53.465214 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:18:53.465169    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/20bf029a-9d4f-4e81-a894-cf19df755f7e-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "20bf029a-9d4f-4e81-a894-cf19df755f7e" (UID: "20bf029a-9d4f-4e81-a894-cf19df755f7e"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 20:18:53.466789 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:18:53.466760    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/20bf029a-9d4f-4e81-a894-cf19df755f7e-kube-api-access-hsffd" (OuterVolumeSpecName: "kube-api-access-hsffd") pod "20bf029a-9d4f-4e81-a894-cf19df755f7e" (UID: "20bf029a-9d4f-4e81-a894-cf19df755f7e"). InnerVolumeSpecName "kube-api-access-hsffd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 20:18:53.467075 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:18:53.466977    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c5d108da-0802-4311-9fc0-24e4fb6bb64e-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "c5d108da-0802-4311-9fc0-24e4fb6bb64e" (UID: "c5d108da-0802-4311-9fc0-24e4fb6bb64e"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 20:18:53.467156 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:18:53.467105    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c5d108da-0802-4311-9fc0-24e4fb6bb64e-kube-api-access-lvknl" (OuterVolumeSpecName: "kube-api-access-lvknl") pod "c5d108da-0802-4311-9fc0-24e4fb6bb64e" (UID: "c5d108da-0802-4311-9fc0-24e4fb6bb64e"). InnerVolumeSpecName "kube-api-access-lvknl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 20:18:53.467156 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:18:53.467145    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/20bf029a-9d4f-4e81-a894-cf19df755f7e-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "20bf029a-9d4f-4e81-a894-cf19df755f7e" (UID: "20bf029a-9d4f-4e81-a894-cf19df755f7e"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 20:18:53.480610 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:18:53.480582    2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-c6f7b798f-7srgk"]
Sep 09 20:18:53.480949 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:18:53.480933    2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="20bf029a-9d4f-4e81-a894-cf19df755f7e" containerName="storage-initializer"
Sep 09 20:18:53.481006 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:18:53.480951    2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="20bf029a-9d4f-4e81-a894-cf19df755f7e" containerName="storage-initializer"
Sep 09 20:18:53.481006 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:18:53.480960    2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="20bf029a-9d4f-4e81-a894-cf19df755f7e" containerName="kserve-container"
Sep 09 20:18:53.481006 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:18:53.480966    2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="20bf029a-9d4f-4e81-a894-cf19df755f7e" containerName="kserve-container"
Sep 09 20:18:53.481006 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:18:53.480973    2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2c75cf85-1abd-4fe0-bdf6-b405acf5d913" containerName="storage-initializer"
Sep 09 20:18:53.481006 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:18:53.480978    2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="2c75cf85-1abd-4fe0-bdf6-b405acf5d913" containerName="storage-initializer"
Sep 09 20:18:53.481006 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:18:53.480985    2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="20bf029a-9d4f-4e81-a894-cf19df755f7e" containerName="kube-rbac-proxy"
Sep 09 20:18:53.481006 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:18:53.480990    2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="20bf029a-9d4f-4e81-a894-cf19df755f7e" containerName="kube-rbac-proxy"
Sep 09 20:18:53.481006 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:18:53.480998    2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c5d108da-0802-4311-9fc0-24e4fb6bb64e" containerName="storage-initializer"
Sep 09 20:18:53.481006 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:18:53.481003    2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="c5d108da-0802-4311-9fc0-24e4fb6bb64e" containerName="storage-initializer"
Sep 09 20:18:53.481272 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:18:53.481019    2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2c75cf85-1abd-4fe0-bdf6-b405acf5d913" containerName="kserve-container"
Sep 09 20:18:53.481272 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:18:53.481024    2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="2c75cf85-1abd-4fe0-bdf6-b405acf5d913" containerName="kserve-container"
Sep 09 20:18:53.481272 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:18:53.481029    2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c5d108da-0802-4311-9fc0-24e4fb6bb64e" containerName="kube-rbac-proxy"
Sep 09 20:18:53.481272 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:18:53.481034    2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="c5d108da-0802-4311-9fc0-24e4fb6bb64e" containerName="kube-rbac-proxy"
Sep 09 20:18:53.481272 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:18:53.481043    2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c5d108da-0802-4311-9fc0-24e4fb6bb64e" containerName="kserve-container"
Sep 09 20:18:53.481272 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:18:53.481048    2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="c5d108da-0802-4311-9fc0-24e4fb6bb64e" containerName="kserve-container"
Sep 09 20:18:53.481272 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:18:53.481054    2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2c75cf85-1abd-4fe0-bdf6-b405acf5d913" containerName="kube-rbac-proxy"
Sep 09 20:18:53.481272 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:18:53.481059    2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="2c75cf85-1abd-4fe0-bdf6-b405acf5d913" containerName="kube-rbac-proxy"
Sep 09 20:18:53.481272 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:18:53.481107    2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="c5d108da-0802-4311-9fc0-24e4fb6bb64e" containerName="kserve-container"
Sep 09 20:18:53.481272 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:18:53.481116    2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="20bf029a-9d4f-4e81-a894-cf19df755f7e" containerName="kube-rbac-proxy"
Sep 09 20:18:53.481272 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:18:53.481123    2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="c5d108da-0802-4311-9fc0-24e4fb6bb64e" containerName="kube-rbac-proxy"
Sep 09 20:18:53.481272 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:18:53.481128    2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="2c75cf85-1abd-4fe0-bdf6-b405acf5d913" containerName="kube-rbac-proxy"
Sep 09 20:18:53.481272 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:18:53.481135    2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="20bf029a-9d4f-4e81-a894-cf19df755f7e" containerName="kserve-container"
Sep 09 20:18:53.481272 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:18:53.481140    2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="2c75cf85-1abd-4fe0-bdf6-b405acf5d913" containerName="kserve-container"
Sep 09 20:18:53.485671 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:18:53.485653    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-c6f7b798f-7srgk"
Sep 09 20:18:53.487734 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:18:53.487714    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-promote-v2-predictor-serving-cert\""
Sep 09 20:18:53.487862 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:18:53.487744    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-promote-kube-rbac-proxy-sar-config\""
Sep 09 20:18:53.493547 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:18:53.493527    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-c6f7b798f-7srgk"]
Sep 09 20:18:53.531814 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:18:53.531760    2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-predictor-cb75746f8-fbb4s"]
Sep 09 20:18:53.535234 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:18:53.535217    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-cb75746f8-fbb4s"
Sep 09 20:18:53.537054 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:18:53.537035    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-promote-predictor-serving-cert\""
Sep 09 20:18:53.545095 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:18:53.545070    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-predictor-cb75746f8-fbb4s"]
Sep 09 20:18:53.566041 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:18:53.566016    2574 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/20bf029a-9d4f-4e81-a894-cf19df755f7e-proxy-tls\") on node \"ip-10-0-136-183.ec2.internal\" DevicePath \"\""
Sep 09 20:18:53.566041 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:18:53.566041    2574 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c5d108da-0802-4311-9fc0-24e4fb6bb64e-proxy-tls\") on node \"ip-10-0-136-183.ec2.internal\" DevicePath \"\""
Sep 09 20:18:53.566233 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:18:53.566053    2574 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c5d108da-0802-4311-9fc0-24e4fb6bb64e-kserve-provision-location\") on node \"ip-10-0-136-183.ec2.internal\" DevicePath \"\""
Sep 09 20:18:53.566233 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:18:53.566067    2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hsffd\" (UniqueName: \"kubernetes.io/projected/20bf029a-9d4f-4e81-a894-cf19df755f7e-kube-api-access-hsffd\") on node \"ip-10-0-136-183.ec2.internal\" DevicePath \"\""
Sep 09 20:18:53.566233 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:18:53.566081    2574 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/20bf029a-9d4f-4e81-a894-cf19df755f7e-cabundle-cert\") on node \"ip-10-0-136-183.ec2.internal\" DevicePath \"\""
Sep 09 20:18:53.566233 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:18:53.566092    2574 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/20bf029a-9d4f-4e81-a894-cf19df755f7e-kserve-provision-location\") on node \"ip-10-0-136-183.ec2.internal\" DevicePath \"\""
Sep 09 20:18:53.566233 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:18:53.566105    2574 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/c5d108da-0802-4311-9fc0-24e4fb6bb64e-cabundle-cert\") on node \"ip-10-0-136-183.ec2.internal\" DevicePath \"\""
Sep 09 20:18:53.566233 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:18:53.566117    2574 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/c5d108da-0802-4311-9fc0-24e4fb6bb64e-kube-rbac-proxy-sar-config\") on node \"ip-10-0-136-183.ec2.internal\" DevicePath \"\""
Sep 09 20:18:53.566233 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:18:53.566128    2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lvknl\" (UniqueName: \"kubernetes.io/projected/c5d108da-0802-4311-9fc0-24e4fb6bb64e-kube-api-access-lvknl\") on node \"ip-10-0-136-183.ec2.internal\" DevicePath \"\""
Sep 09 20:18:53.566233 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:18:53.566140    2574 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/20bf029a-9d4f-4e81-a894-cf19df755f7e-kube-rbac-proxy-sar-config\") on node \"ip-10-0-136-183.ec2.internal\" DevicePath \"\""
Sep 09 20:18:53.666654 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:18:53.666615    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c0780517-e231-497d-8fad-ec175d8fe67a-proxy-tls\") pod \"isvc-canary-promote-predictor-cb75746f8-fbb4s\" (UID: \"c0780517-e231-497d-8fad-ec175d8fe67a\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-cb75746f8-fbb4s"
Sep 09 20:18:53.666654 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:18:53.666657    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/fc5e9aa4-28b7-49e6-a6b7-7a23e90b4354-cabundle-cert\") pod \"isvc-canary-promote-v2-predictor-c6f7b798f-7srgk\" (UID: \"fc5e9aa4-28b7-49e6-a6b7-7a23e90b4354\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-c6f7b798f-7srgk"
Sep 09 20:18:53.666885 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:18:53.666677    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/c0780517-e231-497d-8fad-ec175d8fe67a-kube-rbac-proxy-sar-config\") pod \"isvc-canary-promote-predictor-cb75746f8-fbb4s\" (UID: \"c0780517-e231-497d-8fad-ec175d8fe67a\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-cb75746f8-fbb4s"
Sep 09 20:18:53.666885 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:18:53.666760    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-czgk8\" (UniqueName: \"kubernetes.io/projected/fc5e9aa4-28b7-49e6-a6b7-7a23e90b4354-kube-api-access-czgk8\") pod \"isvc-canary-promote-v2-predictor-c6f7b798f-7srgk\" (UID: \"fc5e9aa4-28b7-49e6-a6b7-7a23e90b4354\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-c6f7b798f-7srgk"
Sep 09 20:18:53.666885 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:18:53.666789    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/fc5e9aa4-28b7-49e6-a6b7-7a23e90b4354-kserve-provision-location\") pod \"isvc-canary-promote-v2-predictor-c6f7b798f-7srgk\" (UID: \"fc5e9aa4-28b7-49e6-a6b7-7a23e90b4354\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-c6f7b798f-7srgk"
Sep 09 20:18:53.666885 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:18:53.666836    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/c0780517-e231-497d-8fad-ec175d8fe67a-cabundle-cert\") pod \"isvc-canary-promote-predictor-cb75746f8-fbb4s\" (UID: \"c0780517-e231-497d-8fad-ec175d8fe67a\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-cb75746f8-fbb4s"
Sep 09 20:18:53.666885 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:18:53.666858    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lmcvw\" (UniqueName: \"kubernetes.io/projected/c0780517-e231-497d-8fad-ec175d8fe67a-kube-api-access-lmcvw\") pod \"isvc-canary-promote-predictor-cb75746f8-fbb4s\" (UID: \"c0780517-e231-497d-8fad-ec175d8fe67a\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-cb75746f8-fbb4s"
Sep 09 20:18:53.666885 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:18:53.666887    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/fc5e9aa4-28b7-49e6-a6b7-7a23e90b4354-proxy-tls\") pod \"isvc-canary-promote-v2-predictor-c6f7b798f-7srgk\" (UID: \"fc5e9aa4-28b7-49e6-a6b7-7a23e90b4354\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-c6f7b798f-7srgk"
Sep 09 20:18:53.667092 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:18:53.666936    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c0780517-e231-497d-8fad-ec175d8fe67a-kserve-provision-location\") pod \"isvc-canary-promote-predictor-cb75746f8-fbb4s\" (UID: \"c0780517-e231-497d-8fad-ec175d8fe67a\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-cb75746f8-fbb4s"
Sep 09 20:18:53.667092 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:18:53.666991    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/fc5e9aa4-28b7-49e6-a6b7-7a23e90b4354-kube-rbac-proxy-sar-config\") pod \"isvc-canary-promote-v2-predictor-c6f7b798f-7srgk\" (UID: \"fc5e9aa4-28b7-49e6-a6b7-7a23e90b4354\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-c6f7b798f-7srgk"
Sep 09 20:18:53.768283 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:18:53.768187    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/fc5e9aa4-28b7-49e6-a6b7-7a23e90b4354-kube-rbac-proxy-sar-config\") pod \"isvc-canary-promote-v2-predictor-c6f7b798f-7srgk\" (UID: \"fc5e9aa4-28b7-49e6-a6b7-7a23e90b4354\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-c6f7b798f-7srgk"
Sep 09 20:18:53.768283 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:18:53.768236    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c0780517-e231-497d-8fad-ec175d8fe67a-proxy-tls\") pod \"isvc-canary-promote-predictor-cb75746f8-fbb4s\" (UID: \"c0780517-e231-497d-8fad-ec175d8fe67a\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-cb75746f8-fbb4s"
Sep 09 20:18:53.768283 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:18:53.768266    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/fc5e9aa4-28b7-49e6-a6b7-7a23e90b4354-cabundle-cert\") pod \"isvc-canary-promote-v2-predictor-c6f7b798f-7srgk\" (UID: \"fc5e9aa4-28b7-49e6-a6b7-7a23e90b4354\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-c6f7b798f-7srgk"
Sep 09 20:18:53.768572 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:18:53.768294    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/c0780517-e231-497d-8fad-ec175d8fe67a-kube-rbac-proxy-sar-config\") pod \"isvc-canary-promote-predictor-cb75746f8-fbb4s\" (UID: \"c0780517-e231-497d-8fad-ec175d8fe67a\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-cb75746f8-fbb4s"
Sep 09 20:18:53.768572 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:18:53.768333    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-czgk8\" (UniqueName: \"kubernetes.io/projected/fc5e9aa4-28b7-49e6-a6b7-7a23e90b4354-kube-api-access-czgk8\") pod \"isvc-canary-promote-v2-predictor-c6f7b798f-7srgk\" (UID: \"fc5e9aa4-28b7-49e6-a6b7-7a23e90b4354\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-c6f7b798f-7srgk"
Sep 09 20:18:53.768572 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:18:53.768357    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/fc5e9aa4-28b7-49e6-a6b7-7a23e90b4354-kserve-provision-location\") pod \"isvc-canary-promote-v2-predictor-c6f7b798f-7srgk\" (UID: \"fc5e9aa4-28b7-49e6-a6b7-7a23e90b4354\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-c6f7b798f-7srgk"
Sep 09 20:18:53.768572 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:18:53.768380    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/c0780517-e231-497d-8fad-ec175d8fe67a-cabundle-cert\") pod \"isvc-canary-promote-predictor-cb75746f8-fbb4s\" (UID: \"c0780517-e231-497d-8fad-ec175d8fe67a\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-cb75746f8-fbb4s"
Sep 09 20:18:53.768572 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:18:53.768409    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lmcvw\" (UniqueName: \"kubernetes.io/projected/c0780517-e231-497d-8fad-ec175d8fe67a-kube-api-access-lmcvw\") pod \"isvc-canary-promote-predictor-cb75746f8-fbb4s\" (UID: \"c0780517-e231-497d-8fad-ec175d8fe67a\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-cb75746f8-fbb4s"
Sep 09 20:18:53.768572 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:18:53.768450    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/fc5e9aa4-28b7-49e6-a6b7-7a23e90b4354-proxy-tls\") pod \"isvc-canary-promote-v2-predictor-c6f7b798f-7srgk\" (UID: \"fc5e9aa4-28b7-49e6-a6b7-7a23e90b4354\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-c6f7b798f-7srgk"
Sep 09 20:18:53.768572 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:18:53.768482    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c0780517-e231-497d-8fad-ec175d8fe67a-kserve-provision-location\") pod \"isvc-canary-promote-predictor-cb75746f8-fbb4s\" (UID: \"c0780517-e231-497d-8fad-ec175d8fe67a\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-cb75746f8-fbb4s"
Sep 09 20:18:53.768988 ip-10-0-136-183 kubenswrapper[2574]: E0909 20:18:53.768664    2574 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-serving-cert: secret "isvc-canary-promote-v2-predictor-serving-cert" not found
Sep 09 20:18:53.768988 ip-10-0-136-183 kubenswrapper[2574]: E0909 20:18:53.768732    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fc5e9aa4-28b7-49e6-a6b7-7a23e90b4354-proxy-tls podName:fc5e9aa4-28b7-49e6-a6b7-7a23e90b4354 nodeName:}" failed. No retries permitted until 2026-09-09 20:18:54.268709936 +0000 UTC m=+1418.314963191 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/fc5e9aa4-28b7-49e6-a6b7-7a23e90b4354-proxy-tls") pod "isvc-canary-promote-v2-predictor-c6f7b798f-7srgk" (UID: "fc5e9aa4-28b7-49e6-a6b7-7a23e90b4354") : secret "isvc-canary-promote-v2-predictor-serving-cert" not found
Sep 09 20:18:53.768988 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:18:53.768859    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c0780517-e231-497d-8fad-ec175d8fe67a-kserve-provision-location\") pod \"isvc-canary-promote-predictor-cb75746f8-fbb4s\" (UID: \"c0780517-e231-497d-8fad-ec175d8fe67a\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-cb75746f8-fbb4s"
Sep 09 20:18:53.768988 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:18:53.768860    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/fc5e9aa4-28b7-49e6-a6b7-7a23e90b4354-kserve-provision-location\") pod \"isvc-canary-promote-v2-predictor-c6f7b798f-7srgk\" (UID: \"fc5e9aa4-28b7-49e6-a6b7-7a23e90b4354\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-c6f7b798f-7srgk"
Sep 09 20:18:53.769242 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:18:53.769178    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/c0780517-e231-497d-8fad-ec175d8fe67a-kube-rbac-proxy-sar-config\") pod \"isvc-canary-promote-predictor-cb75746f8-fbb4s\" (UID: \"c0780517-e231-497d-8fad-ec175d8fe67a\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-cb75746f8-fbb4s"
Sep 09 20:18:53.769303 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:18:53.769219    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/fc5e9aa4-28b7-49e6-a6b7-7a23e90b4354-kube-rbac-proxy-sar-config\") pod \"isvc-canary-promote-v2-predictor-c6f7b798f-7srgk\" (UID: \"fc5e9aa4-28b7-49e6-a6b7-7a23e90b4354\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-c6f7b798f-7srgk"
Sep 09 20:18:53.769303 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:18:53.769263    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/fc5e9aa4-28b7-49e6-a6b7-7a23e90b4354-cabundle-cert\") pod \"isvc-canary-promote-v2-predictor-c6f7b798f-7srgk\" (UID: \"fc5e9aa4-28b7-49e6-a6b7-7a23e90b4354\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-c6f7b798f-7srgk"
Sep 09 20:18:53.769303 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:18:53.769274    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/c0780517-e231-497d-8fad-ec175d8fe67a-cabundle-cert\") pod \"isvc-canary-promote-predictor-cb75746f8-fbb4s\" (UID: \"c0780517-e231-497d-8fad-ec175d8fe67a\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-cb75746f8-fbb4s"
Sep 09 20:18:53.770874 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:18:53.770857    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c0780517-e231-497d-8fad-ec175d8fe67a-proxy-tls\") pod \"isvc-canary-promote-predictor-cb75746f8-fbb4s\" (UID: \"c0780517-e231-497d-8fad-ec175d8fe67a\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-cb75746f8-fbb4s"
Sep 09 20:18:53.776407 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:18:53.776380    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lmcvw\" (UniqueName: \"kubernetes.io/projected/c0780517-e231-497d-8fad-ec175d8fe67a-kube-api-access-lmcvw\") pod \"isvc-canary-promote-predictor-cb75746f8-fbb4s\" (UID: \"c0780517-e231-497d-8fad-ec175d8fe67a\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-cb75746f8-fbb4s"
Sep 09 20:18:53.776504 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:18:53.776435    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-czgk8\" (UniqueName: \"kubernetes.io/projected/fc5e9aa4-28b7-49e6-a6b7-7a23e90b4354-kube-api-access-czgk8\") pod \"isvc-canary-promote-v2-predictor-c6f7b798f-7srgk\" (UID: \"fc5e9aa4-28b7-49e6-a6b7-7a23e90b4354\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-c6f7b798f-7srgk"
Sep 09 20:18:53.845755 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:18:53.845693    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-cb75746f8-fbb4s"
Sep 09 20:18:54.070780 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:18:54.070681    2574 generic.go:356] "Generic (PLEG): container finished" podID="c5d108da-0802-4311-9fc0-24e4fb6bb64e" containerID="50da26306e84d5404c61a078030ecca7fe20114024c5bced8340b85bfff62036" exitCode=0
Sep 09 20:18:54.070942 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:18:54.070771    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-68984b9498-8pzgp" event={"ID":"c5d108da-0802-4311-9fc0-24e4fb6bb64e","Type":"ContainerDied","Data":"50da26306e84d5404c61a078030ecca7fe20114024c5bced8340b85bfff62036"}
Sep 09 20:18:54.070942 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:18:54.070780    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-68984b9498-8pzgp"
Sep 09 20:18:54.070942 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:18:54.070888    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-68984b9498-8pzgp" event={"ID":"c5d108da-0802-4311-9fc0-24e4fb6bb64e","Type":"ContainerDied","Data":"5f4c73acaad7902383ca0005938553ee1e8d17859b6badd2778af4f450463a34"}
Sep 09 20:18:54.070942 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:18:54.070931    2574 scope.go:117] "RemoveContainer" containerID="d1c33395cb78f6681d76c9fa305b614278388e752e62fb467ec97dd8d1d65450"
Sep 09 20:18:54.072908 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:18:54.072884    2574 generic.go:356] "Generic (PLEG): container finished" podID="20bf029a-9d4f-4e81-a894-cf19df755f7e" containerID="2a719f612c149e193bb8eb5dc83bc5691d1f8a948a5d8cbb76a759db732e84b6" exitCode=0
Sep 09 20:18:54.073030 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:18:54.072912    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-579784d7d7-rztjm" event={"ID":"20bf029a-9d4f-4e81-a894-cf19df755f7e","Type":"ContainerDied","Data":"2a719f612c149e193bb8eb5dc83bc5691d1f8a948a5d8cbb76a759db732e84b6"}
Sep 09 20:18:54.073030 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:18:54.072936    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-579784d7d7-rztjm"
Sep 09 20:18:54.073030 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:18:54.072949    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-579784d7d7-rztjm" event={"ID":"20bf029a-9d4f-4e81-a894-cf19df755f7e","Type":"ContainerDied","Data":"30525e0e026cb817ee2dbdb8e35aad8c18a98822d4cdadefaf3f0536639a1834"}
Sep 09 20:18:54.079461 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:18:54.079437    2574 scope.go:117] "RemoveContainer" containerID="50da26306e84d5404c61a078030ecca7fe20114024c5bced8340b85bfff62036"
Sep 09 20:18:54.086846 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:18:54.086784    2574 scope.go:117] "RemoveContainer" containerID="d50490f2e7ec0589939e801824b2e31de6afb01a8c084ab800378a67ca0b02ae"
Sep 09 20:18:54.094185 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:18:54.094165    2574 scope.go:117] "RemoveContainer" containerID="d1c33395cb78f6681d76c9fa305b614278388e752e62fb467ec97dd8d1d65450"
Sep 09 20:18:54.094463 ip-10-0-136-183 kubenswrapper[2574]: E0909 20:18:54.094444    2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d1c33395cb78f6681d76c9fa305b614278388e752e62fb467ec97dd8d1d65450\": container with ID starting with d1c33395cb78f6681d76c9fa305b614278388e752e62fb467ec97dd8d1d65450 not found: ID does not exist" containerID="d1c33395cb78f6681d76c9fa305b614278388e752e62fb467ec97dd8d1d65450"
Sep 09 20:18:54.094519 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:18:54.094472    2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d1c33395cb78f6681d76c9fa305b614278388e752e62fb467ec97dd8d1d65450"} err="failed to get container status \"d1c33395cb78f6681d76c9fa305b614278388e752e62fb467ec97dd8d1d65450\": rpc error: code = NotFound desc = could not find container \"d1c33395cb78f6681d76c9fa305b614278388e752e62fb467ec97dd8d1d65450\": container with ID starting with d1c33395cb78f6681d76c9fa305b614278388e752e62fb467ec97dd8d1d65450 not found: ID does not exist"
Sep 09 20:18:54.094519 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:18:54.094490    2574 scope.go:117] "RemoveContainer" containerID="50da26306e84d5404c61a078030ecca7fe20114024c5bced8340b85bfff62036"
Sep 09 20:18:54.094606 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:18:54.094593    2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-v2-predictor-579784d7d7-rztjm"]
Sep 09 20:18:54.094742 ip-10-0-136-183 kubenswrapper[2574]: E0909 20:18:54.094725    2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"50da26306e84d5404c61a078030ecca7fe20114024c5bced8340b85bfff62036\": container with ID starting with 50da26306e84d5404c61a078030ecca7fe20114024c5bced8340b85bfff62036 not found: ID does not exist" containerID="50da26306e84d5404c61a078030ecca7fe20114024c5bced8340b85bfff62036"
Sep 09 20:18:54.094779 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:18:54.094747    2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"50da26306e84d5404c61a078030ecca7fe20114024c5bced8340b85bfff62036"} err="failed to get container status \"50da26306e84d5404c61a078030ecca7fe20114024c5bced8340b85bfff62036\": rpc error: code = NotFound desc = could not find container \"50da26306e84d5404c61a078030ecca7fe20114024c5bced8340b85bfff62036\": container with ID starting with 50da26306e84d5404c61a078030ecca7fe20114024c5bced8340b85bfff62036 not found: ID does not exist"
Sep 09 20:18:54.094779 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:18:54.094764    2574 scope.go:117] "RemoveContainer" containerID="d50490f2e7ec0589939e801824b2e31de6afb01a8c084ab800378a67ca0b02ae"
Sep 09 20:18:54.095026 ip-10-0-136-183 kubenswrapper[2574]: E0909 20:18:54.095011    2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d50490f2e7ec0589939e801824b2e31de6afb01a8c084ab800378a67ca0b02ae\": container with ID starting with d50490f2e7ec0589939e801824b2e31de6afb01a8c084ab800378a67ca0b02ae not found: ID does not exist" containerID="d50490f2e7ec0589939e801824b2e31de6afb01a8c084ab800378a67ca0b02ae"
Sep 09 20:18:54.095070 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:18:54.095031    2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d50490f2e7ec0589939e801824b2e31de6afb01a8c084ab800378a67ca0b02ae"} err="failed to get container status \"d50490f2e7ec0589939e801824b2e31de6afb01a8c084ab800378a67ca0b02ae\": rpc error: code = NotFound desc = could not find container \"d50490f2e7ec0589939e801824b2e31de6afb01a8c084ab800378a67ca0b02ae\": container with ID starting with d50490f2e7ec0589939e801824b2e31de6afb01a8c084ab800378a67ca0b02ae not found: ID does not exist"
Sep 09 20:18:54.095070 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:18:54.095043    2574 scope.go:117] "RemoveContainer" containerID="baec69383ad1b7c26350dfeb837c4fff1b3bc6bdc1bc65304495a35ee0f6f47a"
Sep 09 20:18:54.099198 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:18:54.099176    2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-v2-predictor-579784d7d7-rztjm"]
Sep 09 20:18:54.102959 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:18:54.102943    2574 scope.go:117] "RemoveContainer" containerID="2a719f612c149e193bb8eb5dc83bc5691d1f8a948a5d8cbb76a759db732e84b6"
Sep 09 20:18:54.110200 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:18:54.110183    2574 scope.go:117] "RemoveContainer" containerID="6bffa6d14c51f973cfe9228ad42c3b863bd8036b3620de258795c98725fa39b4"
Sep 09 20:18:54.110751 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:18:54.110727    2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-predictor-68984b9498-8pzgp"]
Sep 09 20:18:54.113687 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:18:54.113665    2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-predictor-68984b9498-8pzgp"]
Sep 09 20:18:54.117525 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:18:54.117460    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-predictor-cb75746f8-fbb4s"]
Sep 09 20:18:54.118032 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:18:54.117971    2574 scope.go:117] "RemoveContainer" containerID="baec69383ad1b7c26350dfeb837c4fff1b3bc6bdc1bc65304495a35ee0f6f47a"
Sep 09 20:18:54.118250 ip-10-0-136-183 kubenswrapper[2574]: E0909 20:18:54.118232    2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"baec69383ad1b7c26350dfeb837c4fff1b3bc6bdc1bc65304495a35ee0f6f47a\": container with ID starting with baec69383ad1b7c26350dfeb837c4fff1b3bc6bdc1bc65304495a35ee0f6f47a not found: ID does not exist" containerID="baec69383ad1b7c26350dfeb837c4fff1b3bc6bdc1bc65304495a35ee0f6f47a"
Sep 09 20:18:54.118297 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:18:54.118257    2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"baec69383ad1b7c26350dfeb837c4fff1b3bc6bdc1bc65304495a35ee0f6f47a"} err="failed to get container status \"baec69383ad1b7c26350dfeb837c4fff1b3bc6bdc1bc65304495a35ee0f6f47a\": rpc error: code = NotFound desc = could not find container \"baec69383ad1b7c26350dfeb837c4fff1b3bc6bdc1bc65304495a35ee0f6f47a\": container with ID starting with baec69383ad1b7c26350dfeb837c4fff1b3bc6bdc1bc65304495a35ee0f6f47a not found: ID does not exist"
Sep 09 20:18:54.118297 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:18:54.118276    2574 scope.go:117] "RemoveContainer" containerID="2a719f612c149e193bb8eb5dc83bc5691d1f8a948a5d8cbb76a759db732e84b6"
Sep 09 20:18:54.118518 ip-10-0-136-183 kubenswrapper[2574]: E0909 20:18:54.118501    2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2a719f612c149e193bb8eb5dc83bc5691d1f8a948a5d8cbb76a759db732e84b6\": container with ID starting with 2a719f612c149e193bb8eb5dc83bc5691d1f8a948a5d8cbb76a759db732e84b6 not found: ID does not exist" containerID="2a719f612c149e193bb8eb5dc83bc5691d1f8a948a5d8cbb76a759db732e84b6"
Sep 09 20:18:54.118594 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:18:54.118527    2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2a719f612c149e193bb8eb5dc83bc5691d1f8a948a5d8cbb76a759db732e84b6"} err="failed to get container status \"2a719f612c149e193bb8eb5dc83bc5691d1f8a948a5d8cbb76a759db732e84b6\": rpc error: code = NotFound desc = could not find container \"2a719f612c149e193bb8eb5dc83bc5691d1f8a948a5d8cbb76a759db732e84b6\": container with ID starting with 2a719f612c149e193bb8eb5dc83bc5691d1f8a948a5d8cbb76a759db732e84b6 not found: ID does not exist"
Sep 09 20:18:54.118594 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:18:54.118549    2574 scope.go:117] "RemoveContainer" containerID="6bffa6d14c51f973cfe9228ad42c3b863bd8036b3620de258795c98725fa39b4"
Sep 09 20:18:54.118862 ip-10-0-136-183 kubenswrapper[2574]: E0909 20:18:54.118846    2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6bffa6d14c51f973cfe9228ad42c3b863bd8036b3620de258795c98725fa39b4\": container with ID starting with 6bffa6d14c51f973cfe9228ad42c3b863bd8036b3620de258795c98725fa39b4 not found: ID does not exist" containerID="6bffa6d14c51f973cfe9228ad42c3b863bd8036b3620de258795c98725fa39b4"
Sep 09 20:18:54.118919 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:18:54.118868    2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6bffa6d14c51f973cfe9228ad42c3b863bd8036b3620de258795c98725fa39b4"} err="failed to get container status \"6bffa6d14c51f973cfe9228ad42c3b863bd8036b3620de258795c98725fa39b4\": rpc error: code = NotFound desc = could not find container \"6bffa6d14c51f973cfe9228ad42c3b863bd8036b3620de258795c98725fa39b4\": container with ID starting with 6bffa6d14c51f973cfe9228ad42c3b863bd8036b3620de258795c98725fa39b4 not found: ID does not exist"
Sep 09 20:18:54.120192 ip-10-0-136-183 kubenswrapper[2574]: W0909 20:18:54.120175    2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc0780517_e231_497d_8fad_ec175d8fe67a.slice/crio-ef490fb58e39dbe97b6514f1db99a8bb2097ba14c5d23ee9896ac377b70a0024 WatchSource:0}: Error finding container ef490fb58e39dbe97b6514f1db99a8bb2097ba14c5d23ee9896ac377b70a0024: Status 404 returned error can't find the container with id ef490fb58e39dbe97b6514f1db99a8bb2097ba14c5d23ee9896ac377b70a0024
Sep 09 20:18:54.273543 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:18:54.273501    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/fc5e9aa4-28b7-49e6-a6b7-7a23e90b4354-proxy-tls\") pod \"isvc-canary-promote-v2-predictor-c6f7b798f-7srgk\" (UID: \"fc5e9aa4-28b7-49e6-a6b7-7a23e90b4354\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-c6f7b798f-7srgk"
Sep 09 20:18:54.276302 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:18:54.276264    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/fc5e9aa4-28b7-49e6-a6b7-7a23e90b4354-proxy-tls\") pod \"isvc-canary-promote-v2-predictor-c6f7b798f-7srgk\" (UID: \"fc5e9aa4-28b7-49e6-a6b7-7a23e90b4354\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-c6f7b798f-7srgk"
Sep 09 20:18:54.397183 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:18:54.397147    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-c6f7b798f-7srgk"
Sep 09 20:18:54.490300 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:18:54.490267    2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="20bf029a-9d4f-4e81-a894-cf19df755f7e" path="/var/lib/kubelet/pods/20bf029a-9d4f-4e81-a894-cf19df755f7e/volumes"
Sep 09 20:18:54.490752 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:18:54.490738    2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c5d108da-0802-4311-9fc0-24e4fb6bb64e" path="/var/lib/kubelet/pods/c5d108da-0802-4311-9fc0-24e4fb6bb64e/volumes"
Sep 09 20:18:54.669718 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:18:54.669636    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-c6f7b798f-7srgk"]
Sep 09 20:18:54.672616 ip-10-0-136-183 kubenswrapper[2574]: W0909 20:18:54.672593    2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfc5e9aa4_28b7_49e6_a6b7_7a23e90b4354.slice/crio-91a2782caa83fc48b72faa81e37eb99020ada979b16cc2022ce7d4a6dc46c706 WatchSource:0}: Error finding container 91a2782caa83fc48b72faa81e37eb99020ada979b16cc2022ce7d4a6dc46c706: Status 404 returned error can't find the container with id 91a2782caa83fc48b72faa81e37eb99020ada979b16cc2022ce7d4a6dc46c706
Sep 09 20:18:55.079314 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:18:55.079223    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-cb75746f8-fbb4s" event={"ID":"c0780517-e231-497d-8fad-ec175d8fe67a","Type":"ContainerStarted","Data":"e4b7057ed8e85d67f944c99330cf9cb291505ff787ee79ec1341a9631aa91171"}
Sep 09 20:18:55.079314 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:18:55.079290    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-cb75746f8-fbb4s" event={"ID":"c0780517-e231-497d-8fad-ec175d8fe67a","Type":"ContainerStarted","Data":"ef490fb58e39dbe97b6514f1db99a8bb2097ba14c5d23ee9896ac377b70a0024"}
Sep 09 20:18:55.080860 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:18:55.080827    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-c6f7b798f-7srgk" event={"ID":"fc5e9aa4-28b7-49e6-a6b7-7a23e90b4354","Type":"ContainerStarted","Data":"b76cf4d922b57c8143258c9373d627e4a192528254693ed71b141755ae5f9cf1"}
Sep 09 20:18:55.080986 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:18:55.080896    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-c6f7b798f-7srgk" event={"ID":"fc5e9aa4-28b7-49e6-a6b7-7a23e90b4354","Type":"ContainerStarted","Data":"91a2782caa83fc48b72faa81e37eb99020ada979b16cc2022ce7d4a6dc46c706"}
Sep 09 20:18:58.095140 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:18:58.095103    2574 generic.go:356] "Generic (PLEG): container finished" podID="c0780517-e231-497d-8fad-ec175d8fe67a" containerID="e4b7057ed8e85d67f944c99330cf9cb291505ff787ee79ec1341a9631aa91171" exitCode=0
Sep 09 20:18:58.095492 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:18:58.095176    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-cb75746f8-fbb4s" event={"ID":"c0780517-e231-497d-8fad-ec175d8fe67a","Type":"ContainerDied","Data":"e4b7057ed8e85d67f944c99330cf9cb291505ff787ee79ec1341a9631aa91171"}
Sep 09 20:18:59.100846 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:18:59.100811    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-cb75746f8-fbb4s" event={"ID":"c0780517-e231-497d-8fad-ec175d8fe67a","Type":"ContainerStarted","Data":"b847115b80f82d4320ea56ca0c5418963d4c7be74a1a9e5dc8308cfb1a1ab048"}
Sep 09 20:18:59.101279 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:18:59.100876    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-cb75746f8-fbb4s" event={"ID":"c0780517-e231-497d-8fad-ec175d8fe67a","Type":"ContainerStarted","Data":"d150279a11e4e4c314109ba3b74809142da247d9df4dc27f4d1cf673ec6efa26"}
Sep 09 20:18:59.101279 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:18:59.101149    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-cb75746f8-fbb4s"
Sep 09 20:18:59.101393 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:18:59.101301    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-cb75746f8-fbb4s"
Sep 09 20:18:59.102440 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:18:59.102415    2574 generic.go:356] "Generic (PLEG): container finished" podID="fc5e9aa4-28b7-49e6-a6b7-7a23e90b4354" containerID="b76cf4d922b57c8143258c9373d627e4a192528254693ed71b141755ae5f9cf1" exitCode=0
Sep 09 20:18:59.102560 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:18:59.102486    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-c6f7b798f-7srgk" event={"ID":"fc5e9aa4-28b7-49e6-a6b7-7a23e90b4354","Type":"ContainerDied","Data":"b76cf4d922b57c8143258c9373d627e4a192528254693ed71b141755ae5f9cf1"}
Sep 09 20:18:59.102759 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:18:59.102738    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-cb75746f8-fbb4s" podUID="c0780517-e231-497d-8fad-ec175d8fe67a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.38:8080: connect: connection refused"
Sep 09 20:18:59.119692 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:18:59.119646    2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-cb75746f8-fbb4s" podStartSLOduration=2.374118586 podStartE2EDuration="6.119628836s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.74551025s" isStatefulPod=false podCreationTimestamp="2026-09-09 20:18:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:18:59.118437162 +0000 UTC m=+1423.164690425" watchObservedRunningTime="2026-09-09 20:18:59.119628836 +0000 UTC m=+1423.165882097"
Sep 09 20:19:00.109581 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:19:00.109541    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-c6f7b798f-7srgk" event={"ID":"fc5e9aa4-28b7-49e6-a6b7-7a23e90b4354","Type":"ContainerStarted","Data":"42207dda5ae8f35d78bea05e7a2d2ddd5054aede5cd4e23fed18ed4e449767a6"}
Sep 09 20:19:00.110001 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:19:00.109615    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-c6f7b798f-7srgk" event={"ID":"fc5e9aa4-28b7-49e6-a6b7-7a23e90b4354","Type":"ContainerStarted","Data":"45dd16a818fafdd6b66cc563c6e6c2cdcdeda5c188b3b00dc225b79c10ecd088"}
Sep 09 20:19:00.110105 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:19:00.110051    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-c6f7b798f-7srgk"
Sep 09 20:19:00.110105 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:19:00.110064    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-cb75746f8-fbb4s" podUID="c0780517-e231-497d-8fad-ec175d8fe67a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.38:8080: connect: connection refused"
Sep 09 20:19:00.110105 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:19:00.110105    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-c6f7b798f-7srgk"
Sep 09 20:19:00.111479 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:19:00.111450    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-c6f7b798f-7srgk" podUID="fc5e9aa4-28b7-49e6-a6b7-7a23e90b4354" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.37:8080: connect: connection refused"
Sep 09 20:19:00.128690 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:19:00.128648    2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-c6f7b798f-7srgk" podStartSLOduration=3.4297811830000002 podStartE2EDuration="7.128636321s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.698855138s" isStatefulPod=false podCreationTimestamp="2026-09-09 20:18:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:19:00.127490095 +0000 UTC m=+1424.173743358" watchObservedRunningTime="2026-09-09 20:19:00.128636321 +0000 UTC m=+1424.174889582"
Sep 09 20:19:01.113848 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:19:01.113784    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-c6f7b798f-7srgk" podUID="fc5e9aa4-28b7-49e6-a6b7-7a23e90b4354" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.37:8080: connect: connection refused"
Sep 09 20:19:05.114352 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:19:05.114324    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-cb75746f8-fbb4s"
Sep 09 20:19:05.114860 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:19:05.114835    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-cb75746f8-fbb4s" podUID="c0780517-e231-497d-8fad-ec175d8fe67a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.38:8080: connect: connection refused"
Sep 09 20:19:06.119354 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:19:06.119324    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-c6f7b798f-7srgk"
Sep 09 20:19:06.120068 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:19:06.120038    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-c6f7b798f-7srgk" podUID="fc5e9aa4-28b7-49e6-a6b7-7a23e90b4354" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.37:8080: connect: connection refused"
Sep 09 20:19:15.115616 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:19:15.115519    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-cb75746f8-fbb4s" podUID="c0780517-e231-497d-8fad-ec175d8fe67a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.38:8080: connect: connection refused"
Sep 09 20:19:16.120822 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:19:16.120753    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-c6f7b798f-7srgk" podUID="fc5e9aa4-28b7-49e6-a6b7-7a23e90b4354" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.37:8080: connect: connection refused"
Sep 09 20:19:25.115258 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:19:25.115212    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-cb75746f8-fbb4s" podUID="c0780517-e231-497d-8fad-ec175d8fe67a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.38:8080: connect: connection refused"
Sep 09 20:19:26.120708 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:19:26.120664    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-c6f7b798f-7srgk" podUID="fc5e9aa4-28b7-49e6-a6b7-7a23e90b4354" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.37:8080: connect: connection refused"
Sep 09 20:19:35.115243 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:19:35.115202    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-cb75746f8-fbb4s" podUID="c0780517-e231-497d-8fad-ec175d8fe67a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.38:8080: connect: connection refused"
Sep 09 20:19:36.120695 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:19:36.120653    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-c6f7b798f-7srgk" podUID="fc5e9aa4-28b7-49e6-a6b7-7a23e90b4354" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.37:8080: connect: connection refused"
Sep 09 20:19:45.115641 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:19:45.115602    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-cb75746f8-fbb4s" podUID="c0780517-e231-497d-8fad-ec175d8fe67a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.38:8080: connect: connection refused"
Sep 09 20:19:46.120092 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:19:46.120048    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-c6f7b798f-7srgk" podUID="fc5e9aa4-28b7-49e6-a6b7-7a23e90b4354" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.37:8080: connect: connection refused"
Sep 09 20:19:55.115843 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:19:55.115780    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-cb75746f8-fbb4s" podUID="c0780517-e231-497d-8fad-ec175d8fe67a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.38:8080: connect: connection refused"
Sep 09 20:19:56.120777 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:19:56.120732    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-c6f7b798f-7srgk" podUID="fc5e9aa4-28b7-49e6-a6b7-7a23e90b4354" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.37:8080: connect: connection refused"
Sep 09 20:20:05.115258 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:20:05.115213    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-cb75746f8-fbb4s" podUID="c0780517-e231-497d-8fad-ec175d8fe67a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.38:8080: connect: connection refused"
Sep 09 20:20:06.120695 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:20:06.120653    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-c6f7b798f-7srgk" podUID="fc5e9aa4-28b7-49e6-a6b7-7a23e90b4354" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.37:8080: connect: connection refused"
Sep 09 20:20:15.115811 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:20:15.115769    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-cb75746f8-fbb4s"
Sep 09 20:20:16.120862 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:20:16.120831    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-c6f7b798f-7srgk"
Sep 09 20:20:23.531977 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:20:23.531927    2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-predictor-cb75746f8-fbb4s"]
Sep 09 20:20:23.532394 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:20:23.532215    2574 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-cb75746f8-fbb4s" podUID="c0780517-e231-497d-8fad-ec175d8fe67a" containerName="kserve-container" containerID="cri-o://d150279a11e4e4c314109ba3b74809142da247d9df4dc27f4d1cf673ec6efa26" gracePeriod=30
Sep 09 20:20:23.532394 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:20:23.532245    2574 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-cb75746f8-fbb4s" podUID="c0780517-e231-497d-8fad-ec175d8fe67a" containerName="kube-rbac-proxy" containerID="cri-o://b847115b80f82d4320ea56ca0c5418963d4c7be74a1a9e5dc8308cfb1a1ab048" gracePeriod=30
Sep 09 20:20:24.446741 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:20:24.446700    2574 generic.go:356] "Generic (PLEG): container finished" podID="c0780517-e231-497d-8fad-ec175d8fe67a" containerID="b847115b80f82d4320ea56ca0c5418963d4c7be74a1a9e5dc8308cfb1a1ab048" exitCode=2
Sep 09 20:20:24.446926 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:20:24.446771    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-cb75746f8-fbb4s" event={"ID":"c0780517-e231-497d-8fad-ec175d8fe67a","Type":"ContainerDied","Data":"b847115b80f82d4320ea56ca0c5418963d4c7be74a1a9e5dc8308cfb1a1ab048"}
Sep 09 20:20:25.110756 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:20:25.110717    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-cb75746f8-fbb4s" podUID="c0780517-e231-497d-8fad-ec175d8fe67a" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.38:8643/healthz\": dial tcp 10.133.0.38:8643: connect: connection refused"
Sep 09 20:20:25.115030 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:20:25.115004    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-cb75746f8-fbb4s" podUID="c0780517-e231-497d-8fad-ec175d8fe67a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.38:8080: connect: connection refused"
Sep 09 20:20:28.371657 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:20:28.371630    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-cb75746f8-fbb4s"
Sep 09 20:20:28.410530 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:20:28.410495    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/c0780517-e231-497d-8fad-ec175d8fe67a-kube-rbac-proxy-sar-config\") pod \"c0780517-e231-497d-8fad-ec175d8fe67a\" (UID: \"c0780517-e231-497d-8fad-ec175d8fe67a\") "
Sep 09 20:20:28.410530 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:20:28.410533    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-lmcvw\" (UniqueName: \"kubernetes.io/projected/c0780517-e231-497d-8fad-ec175d8fe67a-kube-api-access-lmcvw\") pod \"c0780517-e231-497d-8fad-ec175d8fe67a\" (UID: \"c0780517-e231-497d-8fad-ec175d8fe67a\") "
Sep 09 20:20:28.410781 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:20:28.410559    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c0780517-e231-497d-8fad-ec175d8fe67a-proxy-tls\") pod \"c0780517-e231-497d-8fad-ec175d8fe67a\" (UID: \"c0780517-e231-497d-8fad-ec175d8fe67a\") "
Sep 09 20:20:28.410781 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:20:28.410579    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/c0780517-e231-497d-8fad-ec175d8fe67a-cabundle-cert\") pod \"c0780517-e231-497d-8fad-ec175d8fe67a\" (UID: \"c0780517-e231-497d-8fad-ec175d8fe67a\") "
Sep 09 20:20:28.410781 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:20:28.410596    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c0780517-e231-497d-8fad-ec175d8fe67a-kserve-provision-location\") pod \"c0780517-e231-497d-8fad-ec175d8fe67a\" (UID: \"c0780517-e231-497d-8fad-ec175d8fe67a\") "
Sep 09 20:20:28.410968 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:20:28.410913    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c0780517-e231-497d-8fad-ec175d8fe67a-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "c0780517-e231-497d-8fad-ec175d8fe67a" (UID: "c0780517-e231-497d-8fad-ec175d8fe67a"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 20:20:28.411034 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:20:28.411010    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c0780517-e231-497d-8fad-ec175d8fe67a-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "c0780517-e231-497d-8fad-ec175d8fe67a" (UID: "c0780517-e231-497d-8fad-ec175d8fe67a"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 20:20:28.411097 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:20:28.411039    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c0780517-e231-497d-8fad-ec175d8fe67a-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "c0780517-e231-497d-8fad-ec175d8fe67a" (UID: "c0780517-e231-497d-8fad-ec175d8fe67a"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 20:20:28.412813 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:20:28.412772    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c0780517-e231-497d-8fad-ec175d8fe67a-kube-api-access-lmcvw" (OuterVolumeSpecName: "kube-api-access-lmcvw") pod "c0780517-e231-497d-8fad-ec175d8fe67a" (UID: "c0780517-e231-497d-8fad-ec175d8fe67a"). InnerVolumeSpecName "kube-api-access-lmcvw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 20:20:28.412922 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:20:28.412852    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c0780517-e231-497d-8fad-ec175d8fe67a-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "c0780517-e231-497d-8fad-ec175d8fe67a" (UID: "c0780517-e231-497d-8fad-ec175d8fe67a"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 20:20:28.464346 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:20:28.464254    2574 generic.go:356] "Generic (PLEG): container finished" podID="c0780517-e231-497d-8fad-ec175d8fe67a" containerID="d150279a11e4e4c314109ba3b74809142da247d9df4dc27f4d1cf673ec6efa26" exitCode=0
Sep 09 20:20:28.464346 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:20:28.464303    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-cb75746f8-fbb4s" event={"ID":"c0780517-e231-497d-8fad-ec175d8fe67a","Type":"ContainerDied","Data":"d150279a11e4e4c314109ba3b74809142da247d9df4dc27f4d1cf673ec6efa26"}
Sep 09 20:20:28.464346 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:20:28.464322    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-cb75746f8-fbb4s"
Sep 09 20:20:28.464536 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:20:28.464399    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-cb75746f8-fbb4s" event={"ID":"c0780517-e231-497d-8fad-ec175d8fe67a","Type":"ContainerDied","Data":"ef490fb58e39dbe97b6514f1db99a8bb2097ba14c5d23ee9896ac377b70a0024"}
Sep 09 20:20:28.464536 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:20:28.464442    2574 scope.go:117] "RemoveContainer" containerID="b847115b80f82d4320ea56ca0c5418963d4c7be74a1a9e5dc8308cfb1a1ab048"
Sep 09 20:20:28.473645 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:20:28.473625    2574 scope.go:117] "RemoveContainer" containerID="d150279a11e4e4c314109ba3b74809142da247d9df4dc27f4d1cf673ec6efa26"
Sep 09 20:20:28.481731 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:20:28.481714    2574 scope.go:117] "RemoveContainer" containerID="e4b7057ed8e85d67f944c99330cf9cb291505ff787ee79ec1341a9631aa91171"
Sep 09 20:20:28.485434 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:20:28.485411    2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-predictor-cb75746f8-fbb4s"]
Sep 09 20:20:28.490193 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:20:28.490168    2574 scope.go:117] "RemoveContainer" containerID="b847115b80f82d4320ea56ca0c5418963d4c7be74a1a9e5dc8308cfb1a1ab048"
Sep 09 20:20:28.490488 ip-10-0-136-183 kubenswrapper[2574]: E0909 20:20:28.490466    2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b847115b80f82d4320ea56ca0c5418963d4c7be74a1a9e5dc8308cfb1a1ab048\": container with ID starting with b847115b80f82d4320ea56ca0c5418963d4c7be74a1a9e5dc8308cfb1a1ab048 not found: ID does not exist" containerID="b847115b80f82d4320ea56ca0c5418963d4c7be74a1a9e5dc8308cfb1a1ab048"
Sep 09 20:20:28.490561 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:20:28.490497    2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b847115b80f82d4320ea56ca0c5418963d4c7be74a1a9e5dc8308cfb1a1ab048"} err="failed to get container status \"b847115b80f82d4320ea56ca0c5418963d4c7be74a1a9e5dc8308cfb1a1ab048\": rpc error: code = NotFound desc = could not find container \"b847115b80f82d4320ea56ca0c5418963d4c7be74a1a9e5dc8308cfb1a1ab048\": container with ID starting with b847115b80f82d4320ea56ca0c5418963d4c7be74a1a9e5dc8308cfb1a1ab048 not found: ID does not exist"
Sep 09 20:20:28.490561 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:20:28.490517    2574 scope.go:117] "RemoveContainer" containerID="d150279a11e4e4c314109ba3b74809142da247d9df4dc27f4d1cf673ec6efa26"
Sep 09 20:20:28.490775 ip-10-0-136-183 kubenswrapper[2574]: E0909 20:20:28.490755    2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d150279a11e4e4c314109ba3b74809142da247d9df4dc27f4d1cf673ec6efa26\": container with ID starting with d150279a11e4e4c314109ba3b74809142da247d9df4dc27f4d1cf673ec6efa26 not found: ID does not exist" containerID="d150279a11e4e4c314109ba3b74809142da247d9df4dc27f4d1cf673ec6efa26"
Sep 09 20:20:28.490869 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:20:28.490786    2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d150279a11e4e4c314109ba3b74809142da247d9df4dc27f4d1cf673ec6efa26"} err="failed to get container status \"d150279a11e4e4c314109ba3b74809142da247d9df4dc27f4d1cf673ec6efa26\": rpc error: code = NotFound desc = could not find container \"d150279a11e4e4c314109ba3b74809142da247d9df4dc27f4d1cf673ec6efa26\": container with ID starting with d150279a11e4e4c314109ba3b74809142da247d9df4dc27f4d1cf673ec6efa26 not found: ID does not exist"
Sep 09 20:20:28.490869 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:20:28.490824    2574 scope.go:117] "RemoveContainer" containerID="e4b7057ed8e85d67f944c99330cf9cb291505ff787ee79ec1341a9631aa91171"
Sep 09 20:20:28.491073 ip-10-0-136-183 kubenswrapper[2574]: E0909 20:20:28.491055    2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e4b7057ed8e85d67f944c99330cf9cb291505ff787ee79ec1341a9631aa91171\": container with ID starting with e4b7057ed8e85d67f944c99330cf9cb291505ff787ee79ec1341a9631aa91171 not found: ID does not exist" containerID="e4b7057ed8e85d67f944c99330cf9cb291505ff787ee79ec1341a9631aa91171"
Sep 09 20:20:28.491113 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:20:28.491079    2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e4b7057ed8e85d67f944c99330cf9cb291505ff787ee79ec1341a9631aa91171"} err="failed to get container status \"e4b7057ed8e85d67f944c99330cf9cb291505ff787ee79ec1341a9631aa91171\": rpc error: code = NotFound desc = could not find container \"e4b7057ed8e85d67f944c99330cf9cb291505ff787ee79ec1341a9631aa91171\": container with ID starting with e4b7057ed8e85d67f944c99330cf9cb291505ff787ee79ec1341a9631aa91171 not found: ID does not exist"
Sep 09 20:20:28.491547 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:20:28.491531    2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-predictor-cb75746f8-fbb4s"]
Sep 09 20:20:28.511087 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:20:28.511064    2574 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c0780517-e231-497d-8fad-ec175d8fe67a-proxy-tls\") on node \"ip-10-0-136-183.ec2.internal\" DevicePath \"\""
Sep 09 20:20:28.511087 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:20:28.511086    2574 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/c0780517-e231-497d-8fad-ec175d8fe67a-cabundle-cert\") on node \"ip-10-0-136-183.ec2.internal\" DevicePath \"\""
Sep 09 20:20:28.511237 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:20:28.511095    2574 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c0780517-e231-497d-8fad-ec175d8fe67a-kserve-provision-location\") on node \"ip-10-0-136-183.ec2.internal\" DevicePath \"\""
Sep 09 20:20:28.511237 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:20:28.511105    2574 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/c0780517-e231-497d-8fad-ec175d8fe67a-kube-rbac-proxy-sar-config\") on node \"ip-10-0-136-183.ec2.internal\" DevicePath \"\""
Sep 09 20:20:28.511237 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:20:28.511115    2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lmcvw\" (UniqueName: \"kubernetes.io/projected/c0780517-e231-497d-8fad-ec175d8fe67a-kube-api-access-lmcvw\") on node \"ip-10-0-136-183.ec2.internal\" DevicePath \"\""
Sep 09 20:20:30.489511 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:20:30.489473    2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c0780517-e231-497d-8fad-ec175d8fe67a" path="/var/lib/kubelet/pods/c0780517-e231-497d-8fad-ec175d8fe67a/volumes"
Sep 09 20:21:53.788505 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:21:53.788474    2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-c6f7b798f-7srgk"]
Sep 09 20:21:53.789042 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:21:53.788726    2574 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-c6f7b798f-7srgk" podUID="fc5e9aa4-28b7-49e6-a6b7-7a23e90b4354" containerName="kserve-container" containerID="cri-o://45dd16a818fafdd6b66cc563c6e6c2cdcdeda5c188b3b00dc225b79c10ecd088" gracePeriod=30
Sep 09 20:21:53.789042 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:21:53.788745    2574 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-c6f7b798f-7srgk" podUID="fc5e9aa4-28b7-49e6-a6b7-7a23e90b4354" containerName="kube-rbac-proxy" containerID="cri-o://42207dda5ae8f35d78bea05e7a2d2ddd5054aede5cd4e23fed18ed4e449767a6" gracePeriod=30
Sep 09 20:21:54.786811 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:21:54.786761    2574 generic.go:356] "Generic (PLEG): container finished" podID="fc5e9aa4-28b7-49e6-a6b7-7a23e90b4354" containerID="42207dda5ae8f35d78bea05e7a2d2ddd5054aede5cd4e23fed18ed4e449767a6" exitCode=2
Sep 09 20:21:54.786993 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:21:54.786841    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-c6f7b798f-7srgk" event={"ID":"fc5e9aa4-28b7-49e6-a6b7-7a23e90b4354","Type":"ContainerDied","Data":"42207dda5ae8f35d78bea05e7a2d2ddd5054aede5cd4e23fed18ed4e449767a6"}
Sep 09 20:21:56.114284 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:21:56.114246    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-c6f7b798f-7srgk" podUID="fc5e9aa4-28b7-49e6-a6b7-7a23e90b4354" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.37:8643/healthz\": dial tcp 10.133.0.37:8643: connect: connection refused"
Sep 09 20:21:56.120238 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:21:56.120208    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-c6f7b798f-7srgk" podUID="fc5e9aa4-28b7-49e6-a6b7-7a23e90b4354" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.37:8080: connect: connection refused"
Sep 09 20:21:58.805709 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:21:58.805666    2574 generic.go:356] "Generic (PLEG): container finished" podID="fc5e9aa4-28b7-49e6-a6b7-7a23e90b4354" containerID="45dd16a818fafdd6b66cc563c6e6c2cdcdeda5c188b3b00dc225b79c10ecd088" exitCode=0
Sep 09 20:21:58.806121 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:21:58.805736    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-c6f7b798f-7srgk" event={"ID":"fc5e9aa4-28b7-49e6-a6b7-7a23e90b4354","Type":"ContainerDied","Data":"45dd16a818fafdd6b66cc563c6e6c2cdcdeda5c188b3b00dc225b79c10ecd088"}
Sep 09 20:21:58.832421 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:21:58.832399    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-c6f7b798f-7srgk"
Sep 09 20:21:58.911237 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:21:58.911203    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/fc5e9aa4-28b7-49e6-a6b7-7a23e90b4354-cabundle-cert\") pod \"fc5e9aa4-28b7-49e6-a6b7-7a23e90b4354\" (UID: \"fc5e9aa4-28b7-49e6-a6b7-7a23e90b4354\") "
Sep 09 20:21:58.911461 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:21:58.911259    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-czgk8\" (UniqueName: \"kubernetes.io/projected/fc5e9aa4-28b7-49e6-a6b7-7a23e90b4354-kube-api-access-czgk8\") pod \"fc5e9aa4-28b7-49e6-a6b7-7a23e90b4354\" (UID: \"fc5e9aa4-28b7-49e6-a6b7-7a23e90b4354\") "
Sep 09 20:21:58.911461 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:21:58.911309    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/fc5e9aa4-28b7-49e6-a6b7-7a23e90b4354-kserve-provision-location\") pod \"fc5e9aa4-28b7-49e6-a6b7-7a23e90b4354\" (UID: \"fc5e9aa4-28b7-49e6-a6b7-7a23e90b4354\") "
Sep 09 20:21:58.911461 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:21:58.911331    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/fc5e9aa4-28b7-49e6-a6b7-7a23e90b4354-proxy-tls\") pod \"fc5e9aa4-28b7-49e6-a6b7-7a23e90b4354\" (UID: \"fc5e9aa4-28b7-49e6-a6b7-7a23e90b4354\") "
Sep 09 20:21:58.911461 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:21:58.911348    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/fc5e9aa4-28b7-49e6-a6b7-7a23e90b4354-kube-rbac-proxy-sar-config\") pod \"fc5e9aa4-28b7-49e6-a6b7-7a23e90b4354\" (UID: \"fc5e9aa4-28b7-49e6-a6b7-7a23e90b4354\") "
Sep 09 20:21:58.911677 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:21:58.911651    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fc5e9aa4-28b7-49e6-a6b7-7a23e90b4354-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "fc5e9aa4-28b7-49e6-a6b7-7a23e90b4354" (UID: "fc5e9aa4-28b7-49e6-a6b7-7a23e90b4354"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 20:21:58.911742 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:21:58.911656    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fc5e9aa4-28b7-49e6-a6b7-7a23e90b4354-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "fc5e9aa4-28b7-49e6-a6b7-7a23e90b4354" (UID: "fc5e9aa4-28b7-49e6-a6b7-7a23e90b4354"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 20:21:58.911860 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:21:58.911832    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fc5e9aa4-28b7-49e6-a6b7-7a23e90b4354-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "fc5e9aa4-28b7-49e6-a6b7-7a23e90b4354" (UID: "fc5e9aa4-28b7-49e6-a6b7-7a23e90b4354"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 20:21:58.913550 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:21:58.913520    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fc5e9aa4-28b7-49e6-a6b7-7a23e90b4354-kube-api-access-czgk8" (OuterVolumeSpecName: "kube-api-access-czgk8") pod "fc5e9aa4-28b7-49e6-a6b7-7a23e90b4354" (UID: "fc5e9aa4-28b7-49e6-a6b7-7a23e90b4354"). InnerVolumeSpecName "kube-api-access-czgk8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 20:21:58.913655 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:21:58.913563    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fc5e9aa4-28b7-49e6-a6b7-7a23e90b4354-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "fc5e9aa4-28b7-49e6-a6b7-7a23e90b4354" (UID: "fc5e9aa4-28b7-49e6-a6b7-7a23e90b4354"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 20:21:59.012868 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:21:59.012829    2574 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/fc5e9aa4-28b7-49e6-a6b7-7a23e90b4354-cabundle-cert\") on node \"ip-10-0-136-183.ec2.internal\" DevicePath \"\""
Sep 09 20:21:59.012868 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:21:59.012859    2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-czgk8\" (UniqueName: \"kubernetes.io/projected/fc5e9aa4-28b7-49e6-a6b7-7a23e90b4354-kube-api-access-czgk8\") on node \"ip-10-0-136-183.ec2.internal\" DevicePath \"\""
Sep 09 20:21:59.012868 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:21:59.012870    2574 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/fc5e9aa4-28b7-49e6-a6b7-7a23e90b4354-kserve-provision-location\") on node \"ip-10-0-136-183.ec2.internal\" DevicePath \"\""
Sep 09 20:21:59.013106 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:21:59.012882    2574 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/fc5e9aa4-28b7-49e6-a6b7-7a23e90b4354-proxy-tls\") on node \"ip-10-0-136-183.ec2.internal\" DevicePath \"\""
Sep 09 20:21:59.013106 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:21:59.012890    2574 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/fc5e9aa4-28b7-49e6-a6b7-7a23e90b4354-kube-rbac-proxy-sar-config\") on node \"ip-10-0-136-183.ec2.internal\" DevicePath \"\""
Sep 09 20:21:59.811599 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:21:59.811563    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-c6f7b798f-7srgk" event={"ID":"fc5e9aa4-28b7-49e6-a6b7-7a23e90b4354","Type":"ContainerDied","Data":"91a2782caa83fc48b72faa81e37eb99020ada979b16cc2022ce7d4a6dc46c706"}
Sep 09 20:21:59.812159 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:21:59.811608    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-c6f7b798f-7srgk"
Sep 09 20:21:59.812159 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:21:59.811631    2574 scope.go:117] "RemoveContainer" containerID="42207dda5ae8f35d78bea05e7a2d2ddd5054aede5cd4e23fed18ed4e449767a6"
Sep 09 20:21:59.820222 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:21:59.820206    2574 scope.go:117] "RemoveContainer" containerID="45dd16a818fafdd6b66cc563c6e6c2cdcdeda5c188b3b00dc225b79c10ecd088"
Sep 09 20:21:59.827210 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:21:59.827195    2574 scope.go:117] "RemoveContainer" containerID="b76cf4d922b57c8143258c9373d627e4a192528254693ed71b141755ae5f9cf1"
Sep 09 20:21:59.831553 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:21:59.831529    2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-c6f7b798f-7srgk"]
Sep 09 20:21:59.838441 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:21:59.838421    2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-c6f7b798f-7srgk"]
Sep 09 20:22:00.490225 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:22:00.490195    2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="fc5e9aa4-28b7-49e6-a6b7-7a23e90b4354" path="/var/lib/kubelet/pods/fc5e9aa4-28b7-49e6-a6b7-7a23e90b4354/volumes"
Sep 09 20:24:04.312634 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:24:04.312540    2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-g9ns7/must-gather-7wmk7"]
Sep 09 20:24:04.313254 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:24:04.313012    2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c0780517-e231-497d-8fad-ec175d8fe67a" containerName="kserve-container"
Sep 09 20:24:04.313254 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:24:04.313031    2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="c0780517-e231-497d-8fad-ec175d8fe67a" containerName="kserve-container"
Sep 09 20:24:04.313254 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:24:04.313045    2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fc5e9aa4-28b7-49e6-a6b7-7a23e90b4354" containerName="kserve-container"
Sep 09 20:24:04.313254 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:24:04.313053    2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="fc5e9aa4-28b7-49e6-a6b7-7a23e90b4354" containerName="kserve-container"
Sep 09 20:24:04.313254 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:24:04.313065    2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fc5e9aa4-28b7-49e6-a6b7-7a23e90b4354" containerName="kube-rbac-proxy"
Sep 09 20:24:04.313254 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:24:04.313073    2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="fc5e9aa4-28b7-49e6-a6b7-7a23e90b4354" containerName="kube-rbac-proxy"
Sep 09 20:24:04.313254 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:24:04.313083    2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c0780517-e231-497d-8fad-ec175d8fe67a" containerName="storage-initializer"
Sep 09 20:24:04.313254 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:24:04.313092    2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="c0780517-e231-497d-8fad-ec175d8fe67a" containerName="storage-initializer"
Sep 09 20:24:04.313254 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:24:04.313100    2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fc5e9aa4-28b7-49e6-a6b7-7a23e90b4354" containerName="storage-initializer"
Sep 09 20:24:04.313254 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:24:04.313108    2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="fc5e9aa4-28b7-49e6-a6b7-7a23e90b4354" containerName="storage-initializer"
Sep 09 20:24:04.313254 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:24:04.313128    2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c0780517-e231-497d-8fad-ec175d8fe67a" containerName="kube-rbac-proxy"
Sep 09 20:24:04.313254 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:24:04.313136    2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="c0780517-e231-497d-8fad-ec175d8fe67a" containerName="kube-rbac-proxy"
Sep 09 20:24:04.313254 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:24:04.313207    2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="fc5e9aa4-28b7-49e6-a6b7-7a23e90b4354" containerName="kube-rbac-proxy"
Sep 09 20:24:04.313254 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:24:04.313221    2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="fc5e9aa4-28b7-49e6-a6b7-7a23e90b4354" containerName="kserve-container"
Sep 09 20:24:04.313254 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:24:04.313231    2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="c0780517-e231-497d-8fad-ec175d8fe67a" containerName="kube-rbac-proxy"
Sep 09 20:24:04.313254 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:24:04.313240    2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="c0780517-e231-497d-8fad-ec175d8fe67a" containerName="kserve-container"
Sep 09 20:24:04.316456 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:24:04.316430    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-g9ns7/must-gather-7wmk7"
Sep 09 20:24:04.318849 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:24:04.318819    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-g9ns7\"/\"openshift-service-ca.crt\""
Sep 09 20:24:04.319119 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:24:04.319102    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-g9ns7\"/\"default-dockercfg-c5qww\""
Sep 09 20:24:04.319394 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:24:04.319372    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-g9ns7\"/\"kube-root-ca.crt\""
Sep 09 20:24:04.327470 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:24:04.327444    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-g9ns7/must-gather-7wmk7"]
Sep 09 20:24:04.408012 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:24:04.407974    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/88d24ed7-f300-4053-86f9-2e60cdf544fb-must-gather-output\") pod \"must-gather-7wmk7\" (UID: \"88d24ed7-f300-4053-86f9-2e60cdf544fb\") " pod="openshift-must-gather-g9ns7/must-gather-7wmk7"
Sep 09 20:24:04.408200 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:24:04.408042    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7nrz2\" (UniqueName: \"kubernetes.io/projected/88d24ed7-f300-4053-86f9-2e60cdf544fb-kube-api-access-7nrz2\") pod \"must-gather-7wmk7\" (UID: \"88d24ed7-f300-4053-86f9-2e60cdf544fb\") " pod="openshift-must-gather-g9ns7/must-gather-7wmk7"
Sep 09 20:24:04.509479 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:24:04.509442    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7nrz2\" (UniqueName: \"kubernetes.io/projected/88d24ed7-f300-4053-86f9-2e60cdf544fb-kube-api-access-7nrz2\") pod \"must-gather-7wmk7\" (UID: \"88d24ed7-f300-4053-86f9-2e60cdf544fb\") " pod="openshift-must-gather-g9ns7/must-gather-7wmk7"
Sep 09 20:24:04.509660 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:24:04.509499    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/88d24ed7-f300-4053-86f9-2e60cdf544fb-must-gather-output\") pod \"must-gather-7wmk7\" (UID: \"88d24ed7-f300-4053-86f9-2e60cdf544fb\") " pod="openshift-must-gather-g9ns7/must-gather-7wmk7"
Sep 09 20:24:04.509839 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:24:04.509794    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/88d24ed7-f300-4053-86f9-2e60cdf544fb-must-gather-output\") pod \"must-gather-7wmk7\" (UID: \"88d24ed7-f300-4053-86f9-2e60cdf544fb\") " pod="openshift-must-gather-g9ns7/must-gather-7wmk7"
Sep 09 20:24:04.518599 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:24:04.518571    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7nrz2\" (UniqueName: \"kubernetes.io/projected/88d24ed7-f300-4053-86f9-2e60cdf544fb-kube-api-access-7nrz2\") pod \"must-gather-7wmk7\" (UID: \"88d24ed7-f300-4053-86f9-2e60cdf544fb\") " pod="openshift-must-gather-g9ns7/must-gather-7wmk7"
Sep 09 20:24:04.639619 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:24:04.639589    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-g9ns7/must-gather-7wmk7"
Sep 09 20:24:04.907179 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:24:04.907104    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-g9ns7/must-gather-7wmk7"]
Sep 09 20:24:04.910046 ip-10-0-136-183 kubenswrapper[2574]: W0909 20:24:04.910018    2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod88d24ed7_f300_4053_86f9_2e60cdf544fb.slice/crio-db493d212e74633bd4be1a1c0da1c9489d833efda3883a55be706b891d0af282 WatchSource:0}: Error finding container db493d212e74633bd4be1a1c0da1c9489d833efda3883a55be706b891d0af282: Status 404 returned error can't find the container with id db493d212e74633bd4be1a1c0da1c9489d833efda3883a55be706b891d0af282
Sep 09 20:24:04.911986 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:24:04.911966    2574 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 09 20:24:05.273747 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:24:05.273648    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-g9ns7/must-gather-7wmk7" event={"ID":"88d24ed7-f300-4053-86f9-2e60cdf544fb","Type":"ContainerStarted","Data":"db493d212e74633bd4be1a1c0da1c9489d833efda3883a55be706b891d0af282"}
Sep 09 20:24:11.304493 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:24:11.304387    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-g9ns7/must-gather-7wmk7" event={"ID":"88d24ed7-f300-4053-86f9-2e60cdf544fb","Type":"ContainerStarted","Data":"f8b98b15105ed7c97be4d78e78cc37752f52d00e36c33c714e2d8b9fa0769971"}
Sep 09 20:24:11.304493 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:24:11.304434    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-g9ns7/must-gather-7wmk7" event={"ID":"88d24ed7-f300-4053-86f9-2e60cdf544fb","Type":"ContainerStarted","Data":"a1c695a54f0dd59c1d16c3286a972be2696bf37c1125151d00c15bf8299a2eac"}
Sep 09 20:24:11.319754 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:24:11.319704    2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-g9ns7/must-gather-7wmk7" podStartSLOduration=1.192622173 podStartE2EDuration="7.319688542s" totalImagesPullingTime="6.127066369s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 20:24:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:24:11.31925482 +0000 UTC m=+1735.365508072" watchObservedRunningTime="2026-09-09 20:24:11.319688542 +0000 UTC m=+1735.365941803"
Sep 09 20:24:34.406236 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:24:34.406200    2574 generic.go:356] "Generic (PLEG): container finished" podID="88d24ed7-f300-4053-86f9-2e60cdf544fb" containerID="a1c695a54f0dd59c1d16c3286a972be2696bf37c1125151d00c15bf8299a2eac" exitCode=0
Sep 09 20:24:34.406691 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:24:34.406263    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-g9ns7/must-gather-7wmk7" event={"ID":"88d24ed7-f300-4053-86f9-2e60cdf544fb","Type":"ContainerDied","Data":"a1c695a54f0dd59c1d16c3286a972be2696bf37c1125151d00c15bf8299a2eac"}
Sep 09 20:24:34.406691 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:24:34.406596    2574 scope.go:117] "RemoveContainer" containerID="a1c695a54f0dd59c1d16c3286a972be2696bf37c1125151d00c15bf8299a2eac"
Sep 09 20:24:34.997743 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:24:34.997714    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-must-gather-g9ns7_must-gather-7wmk7_88d24ed7-f300-4053-86f9-2e60cdf544fb/gather/0.log"
Sep 09 20:24:38.645046 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:24:38.645015    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-gjc9x_c756f4c9-2360-4fff-a136-b6869b02977e/global-pull-secret-syncer/0.log"
Sep 09 20:24:38.757750 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:24:38.757719    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-d9ww4_2d0e0ff3-612a-4eb3-b2df-342af115b6c3/konnectivity-agent/0.log"
Sep 09 20:24:38.841294 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:24:38.841262    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-136-183.ec2.internal_c82ea50b31881ad827f1ff32ee67e2c3/haproxy/0.log"
Sep 09 20:24:40.459010 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:24:40.458975    2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-must-gather-g9ns7/must-gather-7wmk7"]
Sep 09 20:24:40.459517 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:24:40.459169    2574 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-must-gather-g9ns7/must-gather-7wmk7" podUID="88d24ed7-f300-4053-86f9-2e60cdf544fb" containerName="copy" containerID="cri-o://f8b98b15105ed7c97be4d78e78cc37752f52d00e36c33c714e2d8b9fa0769971" gracePeriod=2
Sep 09 20:24:40.460907 ip-10-0-136-183 kubenswrapper[2574]: E0909 20:24:40.460871    2574 status_manager.go:1018] "Failed to get status for pod" err="pods \"must-gather-7wmk7\" is forbidden: User \"system:node:ip-10-0-136-183.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"openshift-must-gather-g9ns7\": no relationship found between node 'ip-10-0-136-183.ec2.internal' and this object" podUID="88d24ed7-f300-4053-86f9-2e60cdf544fb" pod="openshift-must-gather-g9ns7/must-gather-7wmk7"
Sep 09 20:24:40.461432 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:24:40.461412    2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-must-gather-g9ns7/must-gather-7wmk7"]
Sep 09 20:24:40.689115 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:24:40.689092    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-must-gather-g9ns7_must-gather-7wmk7_88d24ed7-f300-4053-86f9-2e60cdf544fb/copy/0.log"
Sep 09 20:24:40.690651 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:24:40.690631    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-g9ns7/must-gather-7wmk7"
Sep 09 20:24:40.802446 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:24:40.802361    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/88d24ed7-f300-4053-86f9-2e60cdf544fb-must-gather-output\") pod \"88d24ed7-f300-4053-86f9-2e60cdf544fb\" (UID: \"88d24ed7-f300-4053-86f9-2e60cdf544fb\") "
Sep 09 20:24:40.802446 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:24:40.802396    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7nrz2\" (UniqueName: \"kubernetes.io/projected/88d24ed7-f300-4053-86f9-2e60cdf544fb-kube-api-access-7nrz2\") pod \"88d24ed7-f300-4053-86f9-2e60cdf544fb\" (UID: \"88d24ed7-f300-4053-86f9-2e60cdf544fb\") "
Sep 09 20:24:40.804463 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:24:40.804430    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/88d24ed7-f300-4053-86f9-2e60cdf544fb-must-gather-output" (OuterVolumeSpecName: "must-gather-output") pod "88d24ed7-f300-4053-86f9-2e60cdf544fb" (UID: "88d24ed7-f300-4053-86f9-2e60cdf544fb"). InnerVolumeSpecName "must-gather-output". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 20:24:40.804660 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:24:40.804639    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/88d24ed7-f300-4053-86f9-2e60cdf544fb-kube-api-access-7nrz2" (OuterVolumeSpecName: "kube-api-access-7nrz2") pod "88d24ed7-f300-4053-86f9-2e60cdf544fb" (UID: "88d24ed7-f300-4053-86f9-2e60cdf544fb"). InnerVolumeSpecName "kube-api-access-7nrz2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 20:24:40.903333 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:24:40.903287    2574 reconciler_common.go:299] "Volume detached for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/88d24ed7-f300-4053-86f9-2e60cdf544fb-must-gather-output\") on node \"ip-10-0-136-183.ec2.internal\" DevicePath \"\""
Sep 09 20:24:40.903333 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:24:40.903331    2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7nrz2\" (UniqueName: \"kubernetes.io/projected/88d24ed7-f300-4053-86f9-2e60cdf544fb-kube-api-access-7nrz2\") on node \"ip-10-0-136-183.ec2.internal\" DevicePath \"\""
Sep 09 20:24:41.433762 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:24:41.433735    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-must-gather-g9ns7_must-gather-7wmk7_88d24ed7-f300-4053-86f9-2e60cdf544fb/copy/0.log"
Sep 09 20:24:41.434082 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:24:41.434057    2574 generic.go:356] "Generic (PLEG): container finished" podID="88d24ed7-f300-4053-86f9-2e60cdf544fb" containerID="f8b98b15105ed7c97be4d78e78cc37752f52d00e36c33c714e2d8b9fa0769971" exitCode=143
Sep 09 20:24:41.434182 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:24:41.434104    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-g9ns7/must-gather-7wmk7"
Sep 09 20:24:41.434182 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:24:41.434167    2574 scope.go:117] "RemoveContainer" containerID="f8b98b15105ed7c97be4d78e78cc37752f52d00e36c33c714e2d8b9fa0769971"
Sep 09 20:24:41.443760 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:24:41.443743    2574 scope.go:117] "RemoveContainer" containerID="a1c695a54f0dd59c1d16c3286a972be2696bf37c1125151d00c15bf8299a2eac"
Sep 09 20:24:41.457451 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:24:41.457427    2574 scope.go:117] "RemoveContainer" containerID="f8b98b15105ed7c97be4d78e78cc37752f52d00e36c33c714e2d8b9fa0769971"
Sep 09 20:24:41.457815 ip-10-0-136-183 kubenswrapper[2574]: E0909 20:24:41.457774    2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f8b98b15105ed7c97be4d78e78cc37752f52d00e36c33c714e2d8b9fa0769971\": container with ID starting with f8b98b15105ed7c97be4d78e78cc37752f52d00e36c33c714e2d8b9fa0769971 not found: ID does not exist" containerID="f8b98b15105ed7c97be4d78e78cc37752f52d00e36c33c714e2d8b9fa0769971"
Sep 09 20:24:41.457907 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:24:41.457830    2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f8b98b15105ed7c97be4d78e78cc37752f52d00e36c33c714e2d8b9fa0769971"} err="failed to get container status \"f8b98b15105ed7c97be4d78e78cc37752f52d00e36c33c714e2d8b9fa0769971\": rpc error: code = NotFound desc = could not find container \"f8b98b15105ed7c97be4d78e78cc37752f52d00e36c33c714e2d8b9fa0769971\": container with ID starting with f8b98b15105ed7c97be4d78e78cc37752f52d00e36c33c714e2d8b9fa0769971 not found: ID does not exist"
Sep 09 20:24:41.457907 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:24:41.457865    2574 scope.go:117] "RemoveContainer" containerID="a1c695a54f0dd59c1d16c3286a972be2696bf37c1125151d00c15bf8299a2eac"
Sep 09 20:24:41.458165 ip-10-0-136-183 kubenswrapper[2574]: E0909 20:24:41.458147    2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a1c695a54f0dd59c1d16c3286a972be2696bf37c1125151d00c15bf8299a2eac\": container with ID starting with a1c695a54f0dd59c1d16c3286a972be2696bf37c1125151d00c15bf8299a2eac not found: ID does not exist" containerID="a1c695a54f0dd59c1d16c3286a972be2696bf37c1125151d00c15bf8299a2eac"
Sep 09 20:24:41.458208 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:24:41.458172    2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a1c695a54f0dd59c1d16c3286a972be2696bf37c1125151d00c15bf8299a2eac"} err="failed to get container status \"a1c695a54f0dd59c1d16c3286a972be2696bf37c1125151d00c15bf8299a2eac\": rpc error: code = NotFound desc = could not find container \"a1c695a54f0dd59c1d16c3286a972be2696bf37c1125151d00c15bf8299a2eac\": container with ID starting with a1c695a54f0dd59c1d16c3286a972be2696bf37c1125151d00c15bf8299a2eac not found: ID does not exist"
Sep 09 20:24:42.489715 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:24:42.489678    2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="88d24ed7-f300-4053-86f9-2e60cdf544fb" path="/var/lib/kubelet/pods/88d24ed7-f300-4053-86f9-2e60cdf544fb/volumes"
Sep 09 20:24:43.767923 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:24:43.767887    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-8b862_06886459-8d92-4c11-826a-a879fb96aa71/node-exporter/0.log"
Sep 09 20:24:43.794960 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:24:43.794911    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-8b862_06886459-8d92-4c11-826a-a879fb96aa71/kube-rbac-proxy/0.log"
Sep 09 20:24:43.823894 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:24:43.823865    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-8b862_06886459-8d92-4c11-826a-a879fb96aa71/init-textfile/0.log"
Sep 09 20:24:44.151892 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:24:44.151863    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-6f4946db56-ht869_09d0544d-3823-4cb2-8611-82a74d206a45/prometheus-operator/0.log"
Sep 09 20:24:44.171848 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:24:44.171825    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-6f4946db56-ht869_09d0544d-3823-4cb2-8611-82a74d206a45/kube-rbac-proxy/0.log"
Sep 09 20:24:44.345425 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:24:44.345401    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-65c468894b-882qv_6dcf380f-2f53-4047-8a59-8e93030d4c24/thanos-query/0.log"
Sep 09 20:24:44.373778 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:24:44.373743    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-65c468894b-882qv_6dcf380f-2f53-4047-8a59-8e93030d4c24/kube-rbac-proxy-web/0.log"
Sep 09 20:24:44.402658 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:24:44.402583    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-65c468894b-882qv_6dcf380f-2f53-4047-8a59-8e93030d4c24/kube-rbac-proxy/0.log"
Sep 09 20:24:44.427841 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:24:44.427791    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-65c468894b-882qv_6dcf380f-2f53-4047-8a59-8e93030d4c24/prom-label-proxy/0.log"
Sep 09 20:24:44.455570 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:24:44.455535    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-65c468894b-882qv_6dcf380f-2f53-4047-8a59-8e93030d4c24/kube-rbac-proxy-rules/0.log"
Sep 09 20:24:44.486260 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:24:44.486203    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-65c468894b-882qv_6dcf380f-2f53-4047-8a59-8e93030d4c24/kube-rbac-proxy-metrics/0.log"
Sep 09 20:24:45.757437 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:24:45.757408    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-console_networking-console-plugin-79977f9844-4c8qr_12ae69a5-0173-433e-977f-1b13cc984906/networking-console-plugin/0.log"
Sep 09 20:24:46.820029 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:24:46.819993    2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-cwwj7/perf-node-gather-daemonset-6d457"]
Sep 09 20:24:46.820461 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:24:46.820337    2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="88d24ed7-f300-4053-86f9-2e60cdf544fb" containerName="copy"
Sep 09 20:24:46.820461 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:24:46.820353    2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="88d24ed7-f300-4053-86f9-2e60cdf544fb" containerName="copy"
Sep 09 20:24:46.820461 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:24:46.820367    2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="88d24ed7-f300-4053-86f9-2e60cdf544fb" containerName="gather"
Sep 09 20:24:46.820461 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:24:46.820373    2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="88d24ed7-f300-4053-86f9-2e60cdf544fb" containerName="gather"
Sep 09 20:24:46.820461 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:24:46.820428    2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="88d24ed7-f300-4053-86f9-2e60cdf544fb" containerName="gather"
Sep 09 20:24:46.820461 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:24:46.820438    2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="88d24ed7-f300-4053-86f9-2e60cdf544fb" containerName="copy"
Sep 09 20:24:46.824354 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:24:46.824320    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-cwwj7/perf-node-gather-daemonset-6d457"
Sep 09 20:24:46.826410 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:24:46.826384    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-cwwj7\"/\"default-dockercfg-54whg\""
Sep 09 20:24:46.826976 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:24:46.826954    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-cwwj7\"/\"kube-root-ca.crt\""
Sep 09 20:24:46.827247 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:24:46.826976    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-cwwj7\"/\"openshift-service-ca.crt\""
Sep 09 20:24:46.831815 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:24:46.831772    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-cwwj7/perf-node-gather-daemonset-6d457"]
Sep 09 20:24:46.954889 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:24:46.954851    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/9ba4e4d7-0b74-4894-87d7-762ecd010e47-podres\") pod \"perf-node-gather-daemonset-6d457\" (UID: \"9ba4e4d7-0b74-4894-87d7-762ecd010e47\") " pod="openshift-must-gather-cwwj7/perf-node-gather-daemonset-6d457"
Sep 09 20:24:46.954889 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:24:46.954895    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/9ba4e4d7-0b74-4894-87d7-762ecd010e47-sys\") pod \"perf-node-gather-daemonset-6d457\" (UID: \"9ba4e4d7-0b74-4894-87d7-762ecd010e47\") " pod="openshift-must-gather-cwwj7/perf-node-gather-daemonset-6d457"
Sep 09 20:24:46.955111 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:24:46.954911    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/9ba4e4d7-0b74-4894-87d7-762ecd010e47-lib-modules\") pod \"perf-node-gather-daemonset-6d457\" (UID: \"9ba4e4d7-0b74-4894-87d7-762ecd010e47\") " pod="openshift-must-gather-cwwj7/perf-node-gather-daemonset-6d457"
Sep 09 20:24:46.955111 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:24:46.954942    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/9ba4e4d7-0b74-4894-87d7-762ecd010e47-proc\") pod \"perf-node-gather-daemonset-6d457\" (UID: \"9ba4e4d7-0b74-4894-87d7-762ecd010e47\") " pod="openshift-must-gather-cwwj7/perf-node-gather-daemonset-6d457"
Sep 09 20:24:46.955111 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:24:46.954977    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r68kg\" (UniqueName: \"kubernetes.io/projected/9ba4e4d7-0b74-4894-87d7-762ecd010e47-kube-api-access-r68kg\") pod \"perf-node-gather-daemonset-6d457\" (UID: \"9ba4e4d7-0b74-4894-87d7-762ecd010e47\") " pod="openshift-must-gather-cwwj7/perf-node-gather-daemonset-6d457"
Sep 09 20:24:47.055642 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:24:47.055604    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/9ba4e4d7-0b74-4894-87d7-762ecd010e47-sys\") pod \"perf-node-gather-daemonset-6d457\" (UID: \"9ba4e4d7-0b74-4894-87d7-762ecd010e47\") " pod="openshift-must-gather-cwwj7/perf-node-gather-daemonset-6d457"
Sep 09 20:24:47.055642 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:24:47.055641    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/9ba4e4d7-0b74-4894-87d7-762ecd010e47-lib-modules\") pod \"perf-node-gather-daemonset-6d457\" (UID: \"9ba4e4d7-0b74-4894-87d7-762ecd010e47\") " pod="openshift-must-gather-cwwj7/perf-node-gather-daemonset-6d457"
Sep 09 20:24:47.055936 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:24:47.055664    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/9ba4e4d7-0b74-4894-87d7-762ecd010e47-proc\") pod \"perf-node-gather-daemonset-6d457\" (UID: \"9ba4e4d7-0b74-4894-87d7-762ecd010e47\") " pod="openshift-must-gather-cwwj7/perf-node-gather-daemonset-6d457"
Sep 09 20:24:47.055936 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:24:47.055707    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r68kg\" (UniqueName: \"kubernetes.io/projected/9ba4e4d7-0b74-4894-87d7-762ecd010e47-kube-api-access-r68kg\") pod \"perf-node-gather-daemonset-6d457\" (UID: \"9ba4e4d7-0b74-4894-87d7-762ecd010e47\") " pod="openshift-must-gather-cwwj7/perf-node-gather-daemonset-6d457"
Sep 09 20:24:47.055936 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:24:47.055754    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/9ba4e4d7-0b74-4894-87d7-762ecd010e47-sys\") pod \"perf-node-gather-daemonset-6d457\" (UID: \"9ba4e4d7-0b74-4894-87d7-762ecd010e47\") " pod="openshift-must-gather-cwwj7/perf-node-gather-daemonset-6d457"
Sep 09 20:24:47.055936 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:24:47.055765    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/9ba4e4d7-0b74-4894-87d7-762ecd010e47-podres\") pod \"perf-node-gather-daemonset-6d457\" (UID: \"9ba4e4d7-0b74-4894-87d7-762ecd010e47\") " pod="openshift-must-gather-cwwj7/perf-node-gather-daemonset-6d457"
Sep 09 20:24:47.055936 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:24:47.055829    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/9ba4e4d7-0b74-4894-87d7-762ecd010e47-lib-modules\") pod \"perf-node-gather-daemonset-6d457\" (UID: \"9ba4e4d7-0b74-4894-87d7-762ecd010e47\") " pod="openshift-must-gather-cwwj7/perf-node-gather-daemonset-6d457"
Sep 09 20:24:47.055936 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:24:47.055835    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/9ba4e4d7-0b74-4894-87d7-762ecd010e47-proc\") pod \"perf-node-gather-daemonset-6d457\" (UID: \"9ba4e4d7-0b74-4894-87d7-762ecd010e47\") " pod="openshift-must-gather-cwwj7/perf-node-gather-daemonset-6d457"
Sep 09 20:24:47.055936 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:24:47.055898    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/9ba4e4d7-0b74-4894-87d7-762ecd010e47-podres\") pod \"perf-node-gather-daemonset-6d457\" (UID: \"9ba4e4d7-0b74-4894-87d7-762ecd010e47\") " pod="openshift-must-gather-cwwj7/perf-node-gather-daemonset-6d457"
Sep 09 20:24:47.068706 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:24:47.068679    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-r68kg\" (UniqueName: \"kubernetes.io/projected/9ba4e4d7-0b74-4894-87d7-762ecd010e47-kube-api-access-r68kg\") pod \"perf-node-gather-daemonset-6d457\" (UID: \"9ba4e4d7-0b74-4894-87d7-762ecd010e47\") " pod="openshift-must-gather-cwwj7/perf-node-gather-daemonset-6d457"
Sep 09 20:24:47.134263 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:24:47.134224    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-cwwj7/perf-node-gather-daemonset-6d457"
Sep 09 20:24:47.404120 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:24:47.404041    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-cwwj7/perf-node-gather-daemonset-6d457"]
Sep 09 20:24:47.407635 ip-10-0-136-183 kubenswrapper[2574]: W0909 20:24:47.407593    2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod9ba4e4d7_0b74_4894_87d7_762ecd010e47.slice/crio-ca7f89278b00d1b0a445b90b78dbd7c2142e2bb38f4191666a5a91f752dff5d4 WatchSource:0}: Error finding container ca7f89278b00d1b0a445b90b78dbd7c2142e2bb38f4191666a5a91f752dff5d4: Status 404 returned error can't find the container with id ca7f89278b00d1b0a445b90b78dbd7c2142e2bb38f4191666a5a91f752dff5d4
Sep 09 20:24:47.462635 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:24:47.462596    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-cwwj7/perf-node-gather-daemonset-6d457" event={"ID":"9ba4e4d7-0b74-4894-87d7-762ecd010e47","Type":"ContainerStarted","Data":"ca7f89278b00d1b0a445b90b78dbd7c2142e2bb38f4191666a5a91f752dff5d4"}
Sep 09 20:24:48.289505 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:24:48.289474    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-4blgq_3a81bb5a-0683-466e-b32e-960f4ca22ae6/dns/0.log"
Sep 09 20:24:48.314761 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:24:48.314733    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-4blgq_3a81bb5a-0683-466e-b32e-960f4ca22ae6/kube-rbac-proxy/0.log"
Sep 09 20:24:48.468053 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:24:48.468012    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-cwwj7/perf-node-gather-daemonset-6d457" event={"ID":"9ba4e4d7-0b74-4894-87d7-762ecd010e47","Type":"ContainerStarted","Data":"36509e41e647bdea3417ddd81d603e8f6a60d27b6162c0b947553434010bfc41"}
Sep 09 20:24:48.468231 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:24:48.468142    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-cwwj7/perf-node-gather-daemonset-6d457"
Sep 09 20:24:48.477908 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:24:48.477879    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-q8krg_496a3333-1b9c-4eb2-8896-b6586eaf6877/dns-node-resolver/0.log"
Sep 09 20:24:48.484275 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:24:48.484235    2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-cwwj7/perf-node-gather-daemonset-6d457" podStartSLOduration=2.484221721 podStartE2EDuration="2.484221721s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-09 20:24:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:24:48.482760922 +0000 UTC m=+1772.529014183" watchObservedRunningTime="2026-09-09 20:24:48.484221721 +0000 UTC m=+1772.530474983"
Sep 09 20:24:48.971308 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:24:48.971277    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_image-registry-5f4c5fd4d7-jtgss_d69a5037-dd5e-4065-a877-9350010e5bb9/registry/0.log"
Sep 09 20:24:49.021185 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:24:49.021151    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-mgg8m_39099fe9-527d-4802-b40b-1bf535221165/node-ca/0.log"
Sep 09 20:24:49.809118 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:24:49.809086    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-58574c9bd-vdsvv_85e55211-a74b-4907-bc5b-887db353bdd7/router/0.log"
Sep 09 20:24:50.193317 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:24:50.193283    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-57cmp_ceff0c1c-cb80-437c-a942-9ebe28442933/serve-healthcheck-canary/0.log"
Sep 09 20:24:50.631710 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:24:50.631681    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-k6ctg_2eee1bd1-9d73-45d9-acf7-323648f0d6ae/kube-rbac-proxy/0.log"
Sep 09 20:24:50.655566 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:24:50.655533    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-k6ctg_2eee1bd1-9d73-45d9-acf7-323648f0d6ae/exporter/0.log"
Sep 09 20:24:50.678771 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:24:50.678741    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-k6ctg_2eee1bd1-9d73-45d9-acf7-323648f0d6ae/extractor/0.log"
Sep 09 20:24:53.118394 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:24:53.118359    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_kserve-controller-manager-7676c798d9-b9cfb_93002f61-5ef9-4f71-baf0-75011154f51b/manager/0.log"
Sep 09 20:24:53.156727 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:24:53.156687    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_llmisvc-controller-manager-645f9c8fd6-lg7w2_fe0462c4-ccd9-41fd-a85e-1fc4f3ba500d/manager/0.log"
Sep 09 20:24:54.481786 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:24:54.481758    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-cwwj7/perf-node-gather-daemonset-6d457"
Sep 09 20:24:57.737144 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:24:57.737047    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-fdbb8fdff-94tx6_afb6a238-0f7b-4545-8d8c-434c1c329064/migrator/0.log"
Sep 09 20:24:57.761739 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:24:57.761699    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-fdbb8fdff-94tx6_afb6a238-0f7b-4545-8d8c-434c1c329064/graceful-termination/0.log"
Sep 09 20:24:59.507605 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:24:59.507575    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-gzqh4_bed9028e-2a22-4c8f-a637-bb5ad9204adc/kube-multus-additional-cni-plugins/0.log"
Sep 09 20:24:59.530725 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:24:59.530687    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-gzqh4_bed9028e-2a22-4c8f-a637-bb5ad9204adc/egress-router-binary-copy/0.log"
Sep 09 20:24:59.553212 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:24:59.553179    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-gzqh4_bed9028e-2a22-4c8f-a637-bb5ad9204adc/cni-plugins/0.log"
Sep 09 20:24:59.575251 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:24:59.575219    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-gzqh4_bed9028e-2a22-4c8f-a637-bb5ad9204adc/bond-cni-plugin/0.log"
Sep 09 20:24:59.598754 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:24:59.598719    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-gzqh4_bed9028e-2a22-4c8f-a637-bb5ad9204adc/routeoverride-cni/0.log"
Sep 09 20:24:59.620527 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:24:59.620496    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-gzqh4_bed9028e-2a22-4c8f-a637-bb5ad9204adc/whereabouts-cni-bincopy/0.log"
Sep 09 20:24:59.642585 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:24:59.642554    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-gzqh4_bed9028e-2a22-4c8f-a637-bb5ad9204adc/whereabouts-cni/0.log"
Sep 09 20:24:59.795234 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:24:59.795145    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-sbx29_9c19e4c1-5ffa-4fb8-88f0-93f1b32a591b/kube-multus/0.log"
Sep 09 20:24:59.868364 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:24:59.868335    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-mdrck_47c2b5b9-456a-4528-8d83-7ce3269cc16e/network-metrics-daemon/0.log"
Sep 09 20:24:59.888701 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:24:59.888673    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-mdrck_47c2b5b9-456a-4528-8d83-7ce3269cc16e/kube-rbac-proxy/0.log"
Sep 09 20:25:00.919361 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:25:00.919304    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-jcr9m_2eb82961-9118-4b08-9d97-8d5cbac8cac3/ovn-controller/0.log"
Sep 09 20:25:00.938233 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:25:00.938206    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-jcr9m_2eb82961-9118-4b08-9d97-8d5cbac8cac3/ovn-acl-logging/0.log"
Sep 09 20:25:00.965619 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:25:00.965588    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-jcr9m_2eb82961-9118-4b08-9d97-8d5cbac8cac3/kube-rbac-proxy-node/0.log"
Sep 09 20:25:00.989972 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:25:00.989935    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-jcr9m_2eb82961-9118-4b08-9d97-8d5cbac8cac3/kube-rbac-proxy-ovn-metrics/0.log"
Sep 09 20:25:01.010344 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:25:01.010318    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-jcr9m_2eb82961-9118-4b08-9d97-8d5cbac8cac3/northd/0.log"
Sep 09 20:25:01.032224 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:25:01.032195    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-jcr9m_2eb82961-9118-4b08-9d97-8d5cbac8cac3/nbdb/0.log"
Sep 09 20:25:01.055299 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:25:01.055269    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-jcr9m_2eb82961-9118-4b08-9d97-8d5cbac8cac3/sbdb/0.log"
Sep 09 20:25:01.111586 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:25:01.111555    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-jcr9m_2eb82961-9118-4b08-9d97-8d5cbac8cac3/ovnkube-controller/0.log"
Sep 09 20:25:02.302567 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:25:02.302535    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-diagnostics_network-check-source-685456655f-cf5xl_7c707e1b-d4bd-4a62-9828-07976f8b7b7d/check-endpoints/0.log"
Sep 09 20:25:02.328335 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:25:02.328307    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-diagnostics_network-check-target-4czxv_54458a6d-921b-4637-addc-7c4ee839b592/network-check-target-container/0.log"
Sep 09 20:25:03.420439 ip-10-0-136-183 kubenswrapper[2574]: I0909 20:25:03.420391    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-operator_iptables-alerter-gvsgq_1f0e15c0-0c09-4135-baf4-767e800917d5/iptables-alerter/0.log"