Sep 16 12:45:58.866815 ip-10-0-133-218 systemd[1]: kubelet.service: Failed to load environment files: No such file or directory
Sep 16 12:45:58.866826 ip-10-0-133-218 systemd[1]: kubelet.service: Failed to run 'start-pre' task: No such file or directory
Sep 16 12:45:58.866834 ip-10-0-133-218 systemd[1]: kubelet.service: Failed with result 'resources'.
Sep 16 12:45:58.866997 ip-10-0-133-218 systemd[1]: Failed to start Kubernetes Kubelet.
Sep 16 12:46:08.962136 ip-10-0-133-218 systemd[1]: kubelet.service: Failed to schedule restart job: Unit crio.service not found.
Sep 16 12:46:08.962157 ip-10-0-133-218 systemd[1]: kubelet.service: Failed with result 'resources'.
-- Boot fdd0d3de6b4447f695fbd61861261980 --
Sep 16 12:47:40.965784 ip-10-0-133-218 systemd[1]: Starting Kubernetes Kubelet...
Sep 16 12:47:41.393060 ip-10-0-133-218 kubenswrapper[2580]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 16 12:47:41.393060 ip-10-0-133-218 kubenswrapper[2580]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Sep 16 12:47:41.393060 ip-10-0-133-218 kubenswrapper[2580]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 16 12:47:41.393060 ip-10-0-133-218 kubenswrapper[2580]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI.
Sep 16 12:47:41.393060 ip-10-0-133-218 kubenswrapper[2580]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 16 12:47:41.395999 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:47:41.395912    2580 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime"
Sep 16 12:47:41.403152 ip-10-0-133-218 kubenswrapper[2580]: W0916 12:47:41.403130    2580 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 16 12:47:41.403152 ip-10-0-133-218 kubenswrapper[2580]: W0916 12:47:41.403148    2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 16 12:47:41.403152 ip-10-0-133-218 kubenswrapper[2580]: W0916 12:47:41.403152    2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 16 12:47:41.403152 ip-10-0-133-218 kubenswrapper[2580]: W0916 12:47:41.403155    2580 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 16 12:47:41.403152 ip-10-0-133-218 kubenswrapper[2580]: W0916 12:47:41.403158    2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 16 12:47:41.403152 ip-10-0-133-218 kubenswrapper[2580]: W0916 12:47:41.403161    2580 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 16 12:47:41.403358 ip-10-0-133-218 kubenswrapper[2580]: W0916 12:47:41.403165    2580 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 16 12:47:41.403358 ip-10-0-133-218 kubenswrapper[2580]: W0916 12:47:41.403168    2580 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 16 12:47:41.403358 ip-10-0-133-218 kubenswrapper[2580]: W0916 12:47:41.403171    2580 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 16 12:47:41.403358 ip-10-0-133-218 kubenswrapper[2580]: W0916 12:47:41.403174    2580 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 16 12:47:41.403358 ip-10-0-133-218 kubenswrapper[2580]: W0916 12:47:41.403176    2580 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 16 12:47:41.403358 ip-10-0-133-218 kubenswrapper[2580]: W0916 12:47:41.403179    2580 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 16 12:47:41.403358 ip-10-0-133-218 kubenswrapper[2580]: W0916 12:47:41.403182    2580 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 16 12:47:41.403358 ip-10-0-133-218 kubenswrapper[2580]: W0916 12:47:41.403184    2580 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 16 12:47:41.403358 ip-10-0-133-218 kubenswrapper[2580]: W0916 12:47:41.403187    2580 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 16 12:47:41.403358 ip-10-0-133-218 kubenswrapper[2580]: W0916 12:47:41.403190    2580 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 16 12:47:41.403358 ip-10-0-133-218 kubenswrapper[2580]: W0916 12:47:41.403193    2580 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 16 12:47:41.403358 ip-10-0-133-218 kubenswrapper[2580]: W0916 12:47:41.403195    2580 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 16 12:47:41.403358 ip-10-0-133-218 kubenswrapper[2580]: W0916 12:47:41.403198    2580 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 16 12:47:41.403358 ip-10-0-133-218 kubenswrapper[2580]: W0916 12:47:41.403201    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 16 12:47:41.403358 ip-10-0-133-218 kubenswrapper[2580]: W0916 12:47:41.403203    2580 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 16 12:47:41.403358 ip-10-0-133-218 kubenswrapper[2580]: W0916 12:47:41.403208    2580 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 16 12:47:41.403358 ip-10-0-133-218 kubenswrapper[2580]: W0916 12:47:41.403212    2580 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 16 12:47:41.403358 ip-10-0-133-218 kubenswrapper[2580]: W0916 12:47:41.403215    2580 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 16 12:47:41.403358 ip-10-0-133-218 kubenswrapper[2580]: W0916 12:47:41.403218    2580 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 16 12:47:41.403358 ip-10-0-133-218 kubenswrapper[2580]: W0916 12:47:41.403221    2580 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 16 12:47:41.403862 ip-10-0-133-218 kubenswrapper[2580]: W0916 12:47:41.403224    2580 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 16 12:47:41.403862 ip-10-0-133-218 kubenswrapper[2580]: W0916 12:47:41.403226    2580 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 16 12:47:41.403862 ip-10-0-133-218 kubenswrapper[2580]: W0916 12:47:41.403229    2580 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 16 12:47:41.403862 ip-10-0-133-218 kubenswrapper[2580]: W0916 12:47:41.403232    2580 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 16 12:47:41.403862 ip-10-0-133-218 kubenswrapper[2580]: W0916 12:47:41.403235    2580 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 16 12:47:41.403862 ip-10-0-133-218 kubenswrapper[2580]: W0916 12:47:41.403238    2580 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 16 12:47:41.403862 ip-10-0-133-218 kubenswrapper[2580]: W0916 12:47:41.403240    2580 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 16 12:47:41.403862 ip-10-0-133-218 kubenswrapper[2580]: W0916 12:47:41.403243    2580 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 16 12:47:41.403862 ip-10-0-133-218 kubenswrapper[2580]: W0916 12:47:41.403245    2580 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 16 12:47:41.403862 ip-10-0-133-218 kubenswrapper[2580]: W0916 12:47:41.403248    2580 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 16 12:47:41.403862 ip-10-0-133-218 kubenswrapper[2580]: W0916 12:47:41.403250    2580 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 16 12:47:41.403862 ip-10-0-133-218 kubenswrapper[2580]: W0916 12:47:41.403253    2580 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 16 12:47:41.403862 ip-10-0-133-218 kubenswrapper[2580]: W0916 12:47:41.403255    2580 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 16 12:47:41.403862 ip-10-0-133-218 kubenswrapper[2580]: W0916 12:47:41.403259    2580 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 16 12:47:41.403862 ip-10-0-133-218 kubenswrapper[2580]: W0916 12:47:41.403262    2580 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 16 12:47:41.403862 ip-10-0-133-218 kubenswrapper[2580]: W0916 12:47:41.403264    2580 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 16 12:47:41.403862 ip-10-0-133-218 kubenswrapper[2580]: W0916 12:47:41.403267    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 16 12:47:41.403862 ip-10-0-133-218 kubenswrapper[2580]: W0916 12:47:41.403270    2580 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 16 12:47:41.403862 ip-10-0-133-218 kubenswrapper[2580]: W0916 12:47:41.403273    2580 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 16 12:47:41.403862 ip-10-0-133-218 kubenswrapper[2580]: W0916 12:47:41.403275    2580 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 16 12:47:41.404359 ip-10-0-133-218 kubenswrapper[2580]: W0916 12:47:41.403278    2580 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 16 12:47:41.404359 ip-10-0-133-218 kubenswrapper[2580]: W0916 12:47:41.403281    2580 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 16 12:47:41.404359 ip-10-0-133-218 kubenswrapper[2580]: W0916 12:47:41.403284    2580 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 16 12:47:41.404359 ip-10-0-133-218 kubenswrapper[2580]: W0916 12:47:41.403286    2580 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 16 12:47:41.404359 ip-10-0-133-218 kubenswrapper[2580]: W0916 12:47:41.403291    2580 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 16 12:47:41.404359 ip-10-0-133-218 kubenswrapper[2580]: W0916 12:47:41.403295    2580 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 16 12:47:41.404359 ip-10-0-133-218 kubenswrapper[2580]: W0916 12:47:41.403298    2580 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 16 12:47:41.404359 ip-10-0-133-218 kubenswrapper[2580]: W0916 12:47:41.403301    2580 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 16 12:47:41.404359 ip-10-0-133-218 kubenswrapper[2580]: W0916 12:47:41.403304    2580 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 16 12:47:41.404359 ip-10-0-133-218 kubenswrapper[2580]: W0916 12:47:41.403307    2580 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 16 12:47:41.404359 ip-10-0-133-218 kubenswrapper[2580]: W0916 12:47:41.403309    2580 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 16 12:47:41.404359 ip-10-0-133-218 kubenswrapper[2580]: W0916 12:47:41.403312    2580 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 16 12:47:41.404359 ip-10-0-133-218 kubenswrapper[2580]: W0916 12:47:41.403314    2580 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 16 12:47:41.404359 ip-10-0-133-218 kubenswrapper[2580]: W0916 12:47:41.403317    2580 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 16 12:47:41.404359 ip-10-0-133-218 kubenswrapper[2580]: W0916 12:47:41.403320    2580 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 16 12:47:41.404359 ip-10-0-133-218 kubenswrapper[2580]: W0916 12:47:41.403322    2580 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 16 12:47:41.404359 ip-10-0-133-218 kubenswrapper[2580]: W0916 12:47:41.403325    2580 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 16 12:47:41.404359 ip-10-0-133-218 kubenswrapper[2580]: W0916 12:47:41.403327    2580 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 16 12:47:41.404359 ip-10-0-133-218 kubenswrapper[2580]: W0916 12:47:41.403331    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 16 12:47:41.404838 ip-10-0-133-218 kubenswrapper[2580]: W0916 12:47:41.403335    2580 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 16 12:47:41.404838 ip-10-0-133-218 kubenswrapper[2580]: W0916 12:47:41.403339    2580 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 16 12:47:41.404838 ip-10-0-133-218 kubenswrapper[2580]: W0916 12:47:41.403342    2580 feature_gate.go:328] unrecognized feature gate: Example
Sep 16 12:47:41.404838 ip-10-0-133-218 kubenswrapper[2580]: W0916 12:47:41.403345    2580 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 16 12:47:41.404838 ip-10-0-133-218 kubenswrapper[2580]: W0916 12:47:41.403348    2580 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 16 12:47:41.404838 ip-10-0-133-218 kubenswrapper[2580]: W0916 12:47:41.403351    2580 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 16 12:47:41.404838 ip-10-0-133-218 kubenswrapper[2580]: W0916 12:47:41.403355    2580 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 16 12:47:41.404838 ip-10-0-133-218 kubenswrapper[2580]: W0916 12:47:41.403358    2580 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 16 12:47:41.404838 ip-10-0-133-218 kubenswrapper[2580]: W0916 12:47:41.403361    2580 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 16 12:47:41.404838 ip-10-0-133-218 kubenswrapper[2580]: W0916 12:47:41.403364    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 16 12:47:41.404838 ip-10-0-133-218 kubenswrapper[2580]: W0916 12:47:41.403367    2580 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 16 12:47:41.404838 ip-10-0-133-218 kubenswrapper[2580]: W0916 12:47:41.403370    2580 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 16 12:47:41.404838 ip-10-0-133-218 kubenswrapper[2580]: W0916 12:47:41.403372    2580 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 16 12:47:41.404838 ip-10-0-133-218 kubenswrapper[2580]: W0916 12:47:41.403375    2580 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 16 12:47:41.404838 ip-10-0-133-218 kubenswrapper[2580]: W0916 12:47:41.403378    2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 16 12:47:41.404838 ip-10-0-133-218 kubenswrapper[2580]: W0916 12:47:41.403380    2580 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 16 12:47:41.404838 ip-10-0-133-218 kubenswrapper[2580]: W0916 12:47:41.403384    2580 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 16 12:47:41.404838 ip-10-0-133-218 kubenswrapper[2580]: W0916 12:47:41.403386    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 16 12:47:41.404838 ip-10-0-133-218 kubenswrapper[2580]: W0916 12:47:41.403390    2580 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 16 12:47:41.405303 ip-10-0-133-218 kubenswrapper[2580]: W0916 12:47:41.403393    2580 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 16 12:47:41.405303 ip-10-0-133-218 kubenswrapper[2580]: W0916 12:47:41.403395    2580 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 16 12:47:41.405303 ip-10-0-133-218 kubenswrapper[2580]: W0916 12:47:41.403398    2580 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 16 12:47:41.405303 ip-10-0-133-218 kubenswrapper[2580]: W0916 12:47:41.403401    2580 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 16 12:47:41.405303 ip-10-0-133-218 kubenswrapper[2580]: W0916 12:47:41.403404    2580 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 16 12:47:41.405303 ip-10-0-133-218 kubenswrapper[2580]: W0916 12:47:41.403406    2580 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 16 12:47:41.405303 ip-10-0-133-218 kubenswrapper[2580]: W0916 12:47:41.403409    2580 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 16 12:47:41.405303 ip-10-0-133-218 kubenswrapper[2580]: W0916 12:47:41.403411    2580 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 16 12:47:41.405303 ip-10-0-133-218 kubenswrapper[2580]: W0916 12:47:41.403414    2580 feature_gate.go:328] unrecognized feature gate: Example2
Sep 16 12:47:41.405303 ip-10-0-133-218 kubenswrapper[2580]: W0916 12:47:41.403417    2580 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 16 12:47:41.405303 ip-10-0-133-218 kubenswrapper[2580]: W0916 12:47:41.403419    2580 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 16 12:47:41.405303 ip-10-0-133-218 kubenswrapper[2580]: W0916 12:47:41.403421    2580 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 16 12:47:41.405303 ip-10-0-133-218 kubenswrapper[2580]: W0916 12:47:41.403424    2580 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 16 12:47:41.405303 ip-10-0-133-218 kubenswrapper[2580]: W0916 12:47:41.403427    2580 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 16 12:47:41.405303 ip-10-0-133-218 kubenswrapper[2580]: W0916 12:47:41.403429    2580 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 16 12:47:41.405303 ip-10-0-133-218 kubenswrapper[2580]: W0916 12:47:41.404144    2580 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 16 12:47:41.405303 ip-10-0-133-218 kubenswrapper[2580]: W0916 12:47:41.404151    2580 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 16 12:47:41.405303 ip-10-0-133-218 kubenswrapper[2580]: W0916 12:47:41.404154    2580 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 16 12:47:41.405303 ip-10-0-133-218 kubenswrapper[2580]: W0916 12:47:41.404158    2580 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 16 12:47:41.405303 ip-10-0-133-218 kubenswrapper[2580]: W0916 12:47:41.404161    2580 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 16 12:47:41.405798 ip-10-0-133-218 kubenswrapper[2580]: W0916 12:47:41.404164    2580 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 16 12:47:41.405798 ip-10-0-133-218 kubenswrapper[2580]: W0916 12:47:41.404167    2580 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 16 12:47:41.405798 ip-10-0-133-218 kubenswrapper[2580]: W0916 12:47:41.404169    2580 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 16 12:47:41.405798 ip-10-0-133-218 kubenswrapper[2580]: W0916 12:47:41.404172    2580 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 16 12:47:41.405798 ip-10-0-133-218 kubenswrapper[2580]: W0916 12:47:41.404175    2580 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 16 12:47:41.405798 ip-10-0-133-218 kubenswrapper[2580]: W0916 12:47:41.404178    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 16 12:47:41.405798 ip-10-0-133-218 kubenswrapper[2580]: W0916 12:47:41.404180    2580 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 16 12:47:41.405798 ip-10-0-133-218 kubenswrapper[2580]: W0916 12:47:41.404183    2580 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 16 12:47:41.405798 ip-10-0-133-218 kubenswrapper[2580]: W0916 12:47:41.404186    2580 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 16 12:47:41.405798 ip-10-0-133-218 kubenswrapper[2580]: W0916 12:47:41.404189    2580 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 16 12:47:41.405798 ip-10-0-133-218 kubenswrapper[2580]: W0916 12:47:41.404191    2580 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 16 12:47:41.405798 ip-10-0-133-218 kubenswrapper[2580]: W0916 12:47:41.404195    2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 16 12:47:41.405798 ip-10-0-133-218 kubenswrapper[2580]: W0916 12:47:41.404197    2580 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 16 12:47:41.405798 ip-10-0-133-218 kubenswrapper[2580]: W0916 12:47:41.404200    2580 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 16 12:47:41.405798 ip-10-0-133-218 kubenswrapper[2580]: W0916 12:47:41.404203    2580 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 16 12:47:41.405798 ip-10-0-133-218 kubenswrapper[2580]: W0916 12:47:41.404206    2580 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 16 12:47:41.405798 ip-10-0-133-218 kubenswrapper[2580]: W0916 12:47:41.404208    2580 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 16 12:47:41.405798 ip-10-0-133-218 kubenswrapper[2580]: W0916 12:47:41.404213    2580 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 16 12:47:41.405798 ip-10-0-133-218 kubenswrapper[2580]: W0916 12:47:41.404216    2580 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 16 12:47:41.406293 ip-10-0-133-218 kubenswrapper[2580]: W0916 12:47:41.404219    2580 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 16 12:47:41.406293 ip-10-0-133-218 kubenswrapper[2580]: W0916 12:47:41.404222    2580 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 16 12:47:41.406293 ip-10-0-133-218 kubenswrapper[2580]: W0916 12:47:41.404225    2580 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 16 12:47:41.406293 ip-10-0-133-218 kubenswrapper[2580]: W0916 12:47:41.404227    2580 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 16 12:47:41.406293 ip-10-0-133-218 kubenswrapper[2580]: W0916 12:47:41.404230    2580 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 16 12:47:41.406293 ip-10-0-133-218 kubenswrapper[2580]: W0916 12:47:41.404233    2580 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 16 12:47:41.406293 ip-10-0-133-218 kubenswrapper[2580]: W0916 12:47:41.404236    2580 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 16 12:47:41.406293 ip-10-0-133-218 kubenswrapper[2580]: W0916 12:47:41.404238    2580 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 16 12:47:41.406293 ip-10-0-133-218 kubenswrapper[2580]: W0916 12:47:41.404241    2580 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 16 12:47:41.406293 ip-10-0-133-218 kubenswrapper[2580]: W0916 12:47:41.404243    2580 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 16 12:47:41.406293 ip-10-0-133-218 kubenswrapper[2580]: W0916 12:47:41.404248    2580 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 16 12:47:41.406293 ip-10-0-133-218 kubenswrapper[2580]: W0916 12:47:41.404251    2580 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 16 12:47:41.406293 ip-10-0-133-218 kubenswrapper[2580]: W0916 12:47:41.404255    2580 feature_gate.go:328] unrecognized feature gate: Example2
Sep 16 12:47:41.406293 ip-10-0-133-218 kubenswrapper[2580]: W0916 12:47:41.404258    2580 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 16 12:47:41.406293 ip-10-0-133-218 kubenswrapper[2580]: W0916 12:47:41.404261    2580 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 16 12:47:41.406293 ip-10-0-133-218 kubenswrapper[2580]: W0916 12:47:41.404264    2580 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 16 12:47:41.406293 ip-10-0-133-218 kubenswrapper[2580]: W0916 12:47:41.404268    2580 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 16 12:47:41.406293 ip-10-0-133-218 kubenswrapper[2580]: W0916 12:47:41.404271    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 16 12:47:41.406293 ip-10-0-133-218 kubenswrapper[2580]: W0916 12:47:41.404275    2580 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 16 12:47:41.406788 ip-10-0-133-218 kubenswrapper[2580]: W0916 12:47:41.404278    2580 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 16 12:47:41.406788 ip-10-0-133-218 kubenswrapper[2580]: W0916 12:47:41.404281    2580 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 16 12:47:41.406788 ip-10-0-133-218 kubenswrapper[2580]: W0916 12:47:41.404284    2580 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 16 12:47:41.406788 ip-10-0-133-218 kubenswrapper[2580]: W0916 12:47:41.404286    2580 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 16 12:47:41.406788 ip-10-0-133-218 kubenswrapper[2580]: W0916 12:47:41.404289    2580 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 16 12:47:41.406788 ip-10-0-133-218 kubenswrapper[2580]: W0916 12:47:41.404292    2580 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 16 12:47:41.406788 ip-10-0-133-218 kubenswrapper[2580]: W0916 12:47:41.404294    2580 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 16 12:47:41.406788 ip-10-0-133-218 kubenswrapper[2580]: W0916 12:47:41.404297    2580 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 16 12:47:41.406788 ip-10-0-133-218 kubenswrapper[2580]: W0916 12:47:41.404300    2580 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 16 12:47:41.406788 ip-10-0-133-218 kubenswrapper[2580]: W0916 12:47:41.404303    2580 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 16 12:47:41.406788 ip-10-0-133-218 kubenswrapper[2580]: W0916 12:47:41.404305    2580 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 16 12:47:41.406788 ip-10-0-133-218 kubenswrapper[2580]: W0916 12:47:41.404308    2580 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 16 12:47:41.406788 ip-10-0-133-218 kubenswrapper[2580]: W0916 12:47:41.404310    2580 feature_gate.go:328] unrecognized feature gate: Example
Sep 16 12:47:41.406788 ip-10-0-133-218 kubenswrapper[2580]: W0916 12:47:41.404313    2580 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 16 12:47:41.406788 ip-10-0-133-218 kubenswrapper[2580]: W0916 12:47:41.404316    2580 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 16 12:47:41.406788 ip-10-0-133-218 kubenswrapper[2580]: W0916 12:47:41.404318    2580 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 16 12:47:41.406788 ip-10-0-133-218 kubenswrapper[2580]: W0916 12:47:41.404322    2580 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 16 12:47:41.406788 ip-10-0-133-218 kubenswrapper[2580]: W0916 12:47:41.404325    2580 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 16 12:47:41.406788 ip-10-0-133-218 kubenswrapper[2580]: W0916 12:47:41.404328    2580 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 16 12:47:41.406788 ip-10-0-133-218 kubenswrapper[2580]: W0916 12:47:41.404330    2580 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 16 12:47:41.407297 ip-10-0-133-218 kubenswrapper[2580]: W0916 12:47:41.404333    2580 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 16 12:47:41.407297 ip-10-0-133-218 kubenswrapper[2580]: W0916 12:47:41.404335    2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 16 12:47:41.407297 ip-10-0-133-218 kubenswrapper[2580]: W0916 12:47:41.404338    2580 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 16 12:47:41.407297 ip-10-0-133-218 kubenswrapper[2580]: W0916 12:47:41.404340    2580 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 16 12:47:41.407297 ip-10-0-133-218 kubenswrapper[2580]: W0916 12:47:41.404343    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 16 12:47:41.407297 ip-10-0-133-218 kubenswrapper[2580]: W0916 12:47:41.404346    2580 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 16 12:47:41.407297 ip-10-0-133-218 kubenswrapper[2580]: W0916 12:47:41.404349    2580 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 16 12:47:41.407297 ip-10-0-133-218 kubenswrapper[2580]: W0916 12:47:41.404351    2580 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 16 12:47:41.407297 ip-10-0-133-218 kubenswrapper[2580]: W0916 12:47:41.404354    2580 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 16 12:47:41.407297 ip-10-0-133-218 kubenswrapper[2580]: W0916 12:47:41.404356    2580 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 16 12:47:41.407297 ip-10-0-133-218 kubenswrapper[2580]: W0916 12:47:41.404359    2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 16 12:47:41.407297 ip-10-0-133-218 kubenswrapper[2580]: W0916 12:47:41.404362    2580 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 16 12:47:41.407297 ip-10-0-133-218 kubenswrapper[2580]: W0916 12:47:41.404364    2580 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 16 12:47:41.407297 ip-10-0-133-218 kubenswrapper[2580]: W0916 12:47:41.404367    2580 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 16 12:47:41.407297 ip-10-0-133-218 kubenswrapper[2580]: W0916 12:47:41.404370    2580 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 16 12:47:41.407297 ip-10-0-133-218 kubenswrapper[2580]: W0916 12:47:41.404372    2580 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 16 12:47:41.407297 ip-10-0-133-218 kubenswrapper[2580]: W0916 12:47:41.404375    2580 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 16 12:47:41.407297 ip-10-0-133-218 kubenswrapper[2580]: W0916 12:47:41.404378    2580 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 16 12:47:41.407297 ip-10-0-133-218 kubenswrapper[2580]: W0916 12:47:41.404380    2580 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 16 12:47:41.407297 ip-10-0-133-218 kubenswrapper[2580]: W0916 12:47:41.404383    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 16 12:47:41.407809 ip-10-0-133-218 kubenswrapper[2580]: W0916 12:47:41.404385    2580 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 16 12:47:41.407809 ip-10-0-133-218 kubenswrapper[2580]: W0916 12:47:41.404388    2580 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 16 12:47:41.407809 ip-10-0-133-218 kubenswrapper[2580]: W0916 12:47:41.404391    2580 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 16 12:47:41.407809 ip-10-0-133-218 kubenswrapper[2580]: W0916 12:47:41.404393    2580 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 16 12:47:41.407809 ip-10-0-133-218 kubenswrapper[2580]: W0916 12:47:41.404396    2580 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 16 12:47:41.407809 ip-10-0-133-218 kubenswrapper[2580]: W0916 12:47:41.404398    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 16 12:47:41.407809 ip-10-0-133-218 kubenswrapper[2580]: W0916 12:47:41.404401    2580 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 16 12:47:41.407809 ip-10-0-133-218 kubenswrapper[2580]: W0916 12:47:41.404403    2580 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 16 12:47:41.407809 ip-10-0-133-218 kubenswrapper[2580]: W0916 12:47:41.404406    2580 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 16 12:47:41.407809 ip-10-0-133-218 kubenswrapper[2580]: W0916 12:47:41.404408    2580 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 16 12:47:41.407809 ip-10-0-133-218 kubenswrapper[2580]: W0916 12:47:41.404411    2580 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 16 12:47:41.407809 ip-10-0-133-218 kubenswrapper[2580]: W0916 12:47:41.404414    2580 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 16 12:47:41.407809 ip-10-0-133-218 kubenswrapper[2580]: W0916 12:47:41.404416    2580 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 16 12:47:41.407809 ip-10-0-133-218 kubenswrapper[2580]: W0916 12:47:41.404419    2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 16 12:47:41.407809 ip-10-0-133-218 kubenswrapper[2580]: W0916 12:47:41.404421    2580 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 16 12:47:41.407809 ip-10-0-133-218 kubenswrapper[2580]: W0916 12:47:41.404424    2580 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 16 12:47:41.407809 ip-10-0-133-218 kubenswrapper[2580]: W0916 12:47:41.404843    2580 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 16 12:47:41.407809 ip-10-0-133-218 kubenswrapper[2580]: W0916 12:47:41.404849    2580 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 16 12:47:41.407809 ip-10-0-133-218 kubenswrapper[2580]: W0916 12:47:41.404853    2580 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 16 12:47:41.407809 ip-10-0-133-218 kubenswrapper[2580]: W0916 12:47:41.404856    2580 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 16 12:47:41.408294 ip-10-0-133-218 kubenswrapper[2580]: W0916 12:47:41.404859    2580 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 16 12:47:41.408294 ip-10-0-133-218 kubenswrapper[2580]: W0916 12:47:41.404862    2580 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 16 12:47:41.408294 ip-10-0-133-218 kubenswrapper[2580]: W0916 12:47:41.404865    2580 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 16 12:47:41.408294 ip-10-0-133-218 kubenswrapper[2580]: W0916 12:47:41.404867    2580 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 16 12:47:41.408294 ip-10-0-133-218 kubenswrapper[2580]: W0916 12:47:41.404870    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 16 12:47:41.408294 ip-10-0-133-218 kubenswrapper[2580]: W0916 12:47:41.404873    2580 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 16 12:47:41.408294 ip-10-0-133-218 kubenswrapper[2580]: W0916 12:47:41.404876    2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 16 12:47:41.408294 ip-10-0-133-218 kubenswrapper[2580]: W0916 12:47:41.404878    2580 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 16 12:47:41.408294 ip-10-0-133-218 kubenswrapper[2580]: W0916 12:47:41.404881    2580 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 16 12:47:41.408294 ip-10-0-133-218 kubenswrapper[2580]: W0916 12:47:41.404884    2580 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 16 12:47:41.408294 ip-10-0-133-218 kubenswrapper[2580]: W0916 12:47:41.404886    2580 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 16 12:47:41.408294 ip-10-0-133-218 kubenswrapper[2580]: W0916 12:47:41.404889    2580 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 16 12:47:41.408294 ip-10-0-133-218 kubenswrapper[2580]: W0916 12:47:41.404892    2580 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 16 12:47:41.408294 ip-10-0-133-218 kubenswrapper[2580]: W0916 12:47:41.404895    2580 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 16 12:47:41.408294 ip-10-0-133-218 kubenswrapper[2580]: W0916 12:47:41.404898    2580 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 16 12:47:41.408294 ip-10-0-133-218 kubenswrapper[2580]: W0916 12:47:41.404900    2580 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 16 12:47:41.408294 ip-10-0-133-218 kubenswrapper[2580]: W0916 12:47:41.404903    2580 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 16 12:47:41.408294 ip-10-0-133-218 kubenswrapper[2580]: W0916 12:47:41.404906    2580 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 16 12:47:41.408294 ip-10-0-133-218 kubenswrapper[2580]: W0916 12:47:41.404908    2580 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 16 12:47:41.408294 ip-10-0-133-218 kubenswrapper[2580]: W0916 12:47:41.404910    2580 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 16 12:47:41.408835 ip-10-0-133-218 kubenswrapper[2580]: W0916 12:47:41.404913    2580 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 16 12:47:41.408835 ip-10-0-133-218 kubenswrapper[2580]: W0916 12:47:41.404916    2580 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 16 12:47:41.408835 ip-10-0-133-218 kubenswrapper[2580]: W0916 12:47:41.404919    2580 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 16 12:47:41.408835 ip-10-0-133-218 kubenswrapper[2580]: W0916 12:47:41.404922    2580 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 16 12:47:41.408835 ip-10-0-133-218 kubenswrapper[2580]: W0916 12:47:41.404924    2580 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 16 12:47:41.408835 ip-10-0-133-218 kubenswrapper[2580]: W0916 12:47:41.404927    2580 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 16 12:47:41.408835 ip-10-0-133-218 kubenswrapper[2580]: W0916 12:47:41.404930    2580 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 16 12:47:41.408835 ip-10-0-133-218 kubenswrapper[2580]: W0916 12:47:41.404933    2580 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 16 12:47:41.408835 ip-10-0-133-218 kubenswrapper[2580]: W0916 12:47:41.404936    2580 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 16 12:47:41.408835 ip-10-0-133-218 kubenswrapper[2580]: W0916 12:47:41.404940    2580 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 16 12:47:41.408835 ip-10-0-133-218 kubenswrapper[2580]: W0916 12:47:41.404944    2580 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 16 12:47:41.408835 ip-10-0-133-218 kubenswrapper[2580]: W0916 12:47:41.404947    2580 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 16 12:47:41.408835 ip-10-0-133-218 kubenswrapper[2580]: W0916 12:47:41.404950    2580 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 16 12:47:41.408835 ip-10-0-133-218 kubenswrapper[2580]: W0916 12:47:41.404952    2580 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 16 12:47:41.408835 ip-10-0-133-218 kubenswrapper[2580]: W0916 12:47:41.404955    2580 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 16 12:47:41.408835 ip-10-0-133-218 kubenswrapper[2580]: W0916 12:47:41.404957    2580 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 16 12:47:41.408835 ip-10-0-133-218 kubenswrapper[2580]: W0916 12:47:41.404960    2580 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 16 12:47:41.408835 ip-10-0-133-218 kubenswrapper[2580]: W0916 12:47:41.404962    2580 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 16 12:47:41.408835 ip-10-0-133-218 kubenswrapper[2580]: W0916 12:47:41.404965    2580 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 16 12:47:41.409308 ip-10-0-133-218 kubenswrapper[2580]: W0916 12:47:41.404968    2580 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 16 12:47:41.409308 ip-10-0-133-218 kubenswrapper[2580]: W0916 12:47:41.404971    2580 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 16 12:47:41.409308 ip-10-0-133-218 kubenswrapper[2580]: W0916 12:47:41.404973    2580 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 16 12:47:41.409308 ip-10-0-133-218 kubenswrapper[2580]: W0916 12:47:41.404976    2580 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 16 12:47:41.409308 ip-10-0-133-218 kubenswrapper[2580]: W0916 12:47:41.404978    2580 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 16 12:47:41.409308 ip-10-0-133-218 kubenswrapper[2580]: W0916 12:47:41.404981    2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 16 12:47:41.409308 ip-10-0-133-218 kubenswrapper[2580]: W0916 12:47:41.404984    2580 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 16 12:47:41.409308 ip-10-0-133-218 kubenswrapper[2580]: W0916 12:47:41.404987    2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 16 12:47:41.409308 ip-10-0-133-218 kubenswrapper[2580]: W0916 12:47:41.404991    2580 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 16 12:47:41.409308 ip-10-0-133-218 kubenswrapper[2580]: W0916 12:47:41.404994    2580 feature_gate.go:328] unrecognized feature gate: Example
Sep 16 12:47:41.409308 ip-10-0-133-218 kubenswrapper[2580]: W0916 12:47:41.404997    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 16 12:47:41.409308 ip-10-0-133-218 kubenswrapper[2580]: W0916 12:47:41.405000    2580 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 16 12:47:41.409308 ip-10-0-133-218 kubenswrapper[2580]: W0916 12:47:41.405004    2580 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 16 12:47:41.409308 ip-10-0-133-218 kubenswrapper[2580]: W0916 12:47:41.405007    2580 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 16 12:47:41.409308 ip-10-0-133-218 kubenswrapper[2580]: W0916 12:47:41.405010    2580 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 16 12:47:41.409308 ip-10-0-133-218 kubenswrapper[2580]: W0916 12:47:41.405013    2580 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 16 12:47:41.409308 ip-10-0-133-218 kubenswrapper[2580]: W0916 12:47:41.405016    2580 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 16 12:47:41.409308 ip-10-0-133-218 kubenswrapper[2580]: W0916 12:47:41.405019    2580 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 16 12:47:41.409308 ip-10-0-133-218 kubenswrapper[2580]: W0916 12:47:41.405022    2580 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 16 12:47:41.409784 ip-10-0-133-218 kubenswrapper[2580]: W0916 12:47:41.405024    2580 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 16 12:47:41.409784 ip-10-0-133-218 kubenswrapper[2580]: W0916 12:47:41.405027    2580 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 16 12:47:41.409784 ip-10-0-133-218 kubenswrapper[2580]: W0916 12:47:41.405032    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 16 12:47:41.409784 ip-10-0-133-218 kubenswrapper[2580]: W0916 12:47:41.405035    2580 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 16 12:47:41.409784 ip-10-0-133-218 kubenswrapper[2580]: W0916 12:47:41.405038    2580 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 16 12:47:41.409784 ip-10-0-133-218 kubenswrapper[2580]: W0916 12:47:41.405040    2580 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 16 12:47:41.409784 ip-10-0-133-218 kubenswrapper[2580]: W0916 12:47:41.405043    2580 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 16 12:47:41.409784 ip-10-0-133-218 kubenswrapper[2580]: W0916 12:47:41.405045    2580 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 16 12:47:41.409784 ip-10-0-133-218 kubenswrapper[2580]: W0916 12:47:41.405048    2580 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 16 12:47:41.409784 ip-10-0-133-218 kubenswrapper[2580]: W0916 12:47:41.405050    2580 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 16 12:47:41.409784 ip-10-0-133-218 kubenswrapper[2580]: W0916 12:47:41.405053    2580 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 16 12:47:41.409784 ip-10-0-133-218 kubenswrapper[2580]: W0916 12:47:41.405055    2580 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 16 12:47:41.409784 ip-10-0-133-218 kubenswrapper[2580]: W0916 12:47:41.405058    2580 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 16 12:47:41.409784 ip-10-0-133-218 kubenswrapper[2580]: W0916 12:47:41.405060    2580 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 16 12:47:41.409784 ip-10-0-133-218 kubenswrapper[2580]: W0916 12:47:41.405063    2580 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 16 12:47:41.409784 ip-10-0-133-218 kubenswrapper[2580]: W0916 12:47:41.405066    2580 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 16 12:47:41.409784 ip-10-0-133-218 kubenswrapper[2580]: W0916 12:47:41.405068    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 16 12:47:41.409784 ip-10-0-133-218 kubenswrapper[2580]: W0916 12:47:41.405071    2580 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 16 12:47:41.409784 ip-10-0-133-218 kubenswrapper[2580]: W0916 12:47:41.405073    2580 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 16 12:47:41.409784 ip-10-0-133-218 kubenswrapper[2580]: W0916 12:47:41.405076    2580 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 16 12:47:41.410323 ip-10-0-133-218 kubenswrapper[2580]: W0916 12:47:41.405080    2580 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 16 12:47:41.410323 ip-10-0-133-218 kubenswrapper[2580]: W0916 12:47:41.405082    2580 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 16 12:47:41.410323 ip-10-0-133-218 kubenswrapper[2580]: W0916 12:47:41.405085    2580 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 16 12:47:41.410323 ip-10-0-133-218 kubenswrapper[2580]: W0916 12:47:41.405087    2580 feature_gate.go:328] unrecognized feature gate: Example2
Sep 16 12:47:41.410323 ip-10-0-133-218 kubenswrapper[2580]: W0916 12:47:41.405090    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 16 12:47:41.410323 ip-10-0-133-218 kubenswrapper[2580]: W0916 12:47:41.405092    2580 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 16 12:47:41.410323 ip-10-0-133-218 kubenswrapper[2580]: W0916 12:47:41.405095    2580 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 16 12:47:41.410323 ip-10-0-133-218 kubenswrapper[2580]: W0916 12:47:41.405097    2580 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 16 12:47:41.410323 ip-10-0-133-218 kubenswrapper[2580]: W0916 12:47:41.405100    2580 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 16 12:47:41.410323 ip-10-0-133-218 kubenswrapper[2580]: W0916 12:47:41.405103    2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 16 12:47:41.410323 ip-10-0-133-218 kubenswrapper[2580]: W0916 12:47:41.405105    2580 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 16 12:47:41.410323 ip-10-0-133-218 kubenswrapper[2580]: W0916 12:47:41.405108    2580 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 16 12:47:41.410323 ip-10-0-133-218 kubenswrapper[2580]: W0916 12:47:41.405111    2580 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 16 12:47:41.410323 ip-10-0-133-218 kubenswrapper[2580]: W0916 12:47:41.405114    2580 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 16 12:47:41.410323 ip-10-0-133-218 kubenswrapper[2580]: W0916 12:47:41.405116    2580 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 16 12:47:41.410323 ip-10-0-133-218 kubenswrapper[2580]: W0916 12:47:41.405119    2580 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 16 12:47:41.410323 ip-10-0-133-218 kubenswrapper[2580]: W0916 12:47:41.405122    2580 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 16 12:47:41.410323 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:47:41.405194    2580 flags.go:64] FLAG: --address="0.0.0.0"
Sep 16 12:47:41.410323 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:47:41.405227    2580 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Sep 16 12:47:41.410323 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:47:41.405233    2580 flags.go:64] FLAG: --anonymous-auth="true"
Sep 16 12:47:41.410323 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:47:41.405238    2580 flags.go:64] FLAG: --application-metrics-count-limit="100"
Sep 16 12:47:41.410853 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:47:41.405242    2580 flags.go:64] FLAG: --authentication-token-webhook="false"
Sep 16 12:47:41.410853 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:47:41.405246    2580 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Sep 16 12:47:41.410853 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:47:41.405250    2580 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Sep 16 12:47:41.410853 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:47:41.405257    2580 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Sep 16 12:47:41.410853 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:47:41.405261    2580 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Sep 16 12:47:41.410853 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:47:41.405264    2580 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Sep 16 12:47:41.410853 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:47:41.405267    2580 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Sep 16 12:47:41.410853 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:47:41.405270    2580 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Sep 16 12:47:41.410853 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:47:41.405273    2580 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Sep 16 12:47:41.410853 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:47:41.405276    2580 flags.go:64] FLAG: --cgroup-root=""
Sep 16 12:47:41.410853 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:47:41.405279    2580 flags.go:64] FLAG: --cgroups-per-qos="true"
Sep 16 12:47:41.410853 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:47:41.405282    2580 flags.go:64] FLAG: --client-ca-file=""
Sep 16 12:47:41.410853 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:47:41.405286    2580 flags.go:64] FLAG: --cloud-provider="external"
Sep 16 12:47:41.410853 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:47:41.405289    2580 flags.go:64] FLAG: --cluster-dns="[]"
Sep 16 12:47:41.410853 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:47:41.405293    2580 flags.go:64] FLAG: --cluster-domain=""
Sep 16 12:47:41.410853 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:47:41.405296    2580 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Sep 16 12:47:41.410853 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:47:41.405299    2580 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Sep 16 12:47:41.410853 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:47:41.405302    2580 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Sep 16 12:47:41.410853 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:47:41.405306    2580 flags.go:64] FLAG: --container-log-max-files="5"
Sep 16 12:47:41.410853 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:47:41.405314    2580 flags.go:64] FLAG: --container-log-max-size="10Mi"
Sep 16 12:47:41.410853 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:47:41.405317    2580 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Sep 16 12:47:41.410853 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:47:41.405320    2580 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Sep 16 12:47:41.410853 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:47:41.405325    2580 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Sep 16 12:47:41.411419 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:47:41.405328    2580 flags.go:64] FLAG: --contention-profiling="false"
Sep 16 12:47:41.411419 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:47:41.405331    2580 flags.go:64] FLAG: --cpu-cfs-quota="true"
Sep 16 12:47:41.411419 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:47:41.405334    2580 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Sep 16 12:47:41.411419 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:47:41.405338    2580 flags.go:64] FLAG: --cpu-manager-policy="none"
Sep 16 12:47:41.411419 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:47:41.405341    2580 flags.go:64] FLAG: --cpu-manager-policy-options=""
Sep 16 12:47:41.411419 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:47:41.405344    2580 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Sep 16 12:47:41.411419 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:47:41.405347    2580 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Sep 16 12:47:41.411419 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:47:41.405350    2580 flags.go:64] FLAG: --enable-debugging-handlers="true"
Sep 16 12:47:41.411419 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:47:41.405353    2580 flags.go:64] FLAG: --enable-load-reader="false"
Sep 16 12:47:41.411419 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:47:41.405356    2580 flags.go:64] FLAG: --enable-server="true"
Sep 16 12:47:41.411419 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:47:41.405359    2580 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Sep 16 12:47:41.411419 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:47:41.405363    2580 flags.go:64] FLAG: --event-burst="100"
Sep 16 12:47:41.411419 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:47:41.405368    2580 flags.go:64] FLAG: --event-qps="50"
Sep 16 12:47:41.411419 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:47:41.405371    2580 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Sep 16 12:47:41.411419 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:47:41.405374    2580 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Sep 16 12:47:41.411419 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:47:41.405377    2580 flags.go:64] FLAG: --eviction-hard=""
Sep 16 12:47:41.411419 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:47:41.405380    2580 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Sep 16 12:47:41.411419 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:47:41.405383    2580 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Sep 16 12:47:41.411419 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:47:41.405386    2580 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Sep 16 12:47:41.411419 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:47:41.405389    2580 flags.go:64] FLAG: --eviction-soft=""
Sep 16 12:47:41.411419 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:47:41.405392    2580 flags.go:64] FLAG: --eviction-soft-grace-period=""
Sep 16 12:47:41.411419 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:47:41.405395    2580 flags.go:64] FLAG: --exit-on-lock-contention="false"
Sep 16 12:47:41.411419 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:47:41.405399    2580 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Sep 16 12:47:41.411419 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:47:41.405402    2580 flags.go:64] FLAG: --experimental-mounter-path=""
Sep 16 12:47:41.411419 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:47:41.405405    2580 flags.go:64] FLAG: --fail-cgroupv1="false"
Sep 16 12:47:41.412069 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:47:41.405408    2580 flags.go:64] FLAG: --fail-swap-on="true"
Sep 16 12:47:41.412069 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:47:41.405411    2580 flags.go:64] FLAG: --feature-gates=""
Sep 16 12:47:41.412069 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:47:41.405414    2580 flags.go:64] FLAG: --file-check-frequency="20s"
Sep 16 12:47:41.412069 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:47:41.405417    2580 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Sep 16 12:47:41.412069 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:47:41.405421    2580 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Sep 16 12:47:41.412069 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:47:41.405424    2580 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Sep 16 12:47:41.412069 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:47:41.405428    2580 flags.go:64] FLAG: --healthz-port="10248"
Sep 16 12:47:41.412069 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:47:41.405431    2580 flags.go:64] FLAG: --help="false"
Sep 16 12:47:41.412069 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:47:41.405434    2580 flags.go:64] FLAG: --hostname-override="ip-10-0-133-218.ec2.internal"
Sep 16 12:47:41.412069 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:47:41.405437    2580 flags.go:64] FLAG: --housekeeping-interval="10s"
Sep 16 12:47:41.412069 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:47:41.405440    2580 flags.go:64] FLAG: --http-check-frequency="20s"
Sep 16 12:47:41.412069 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:47:41.405443    2580 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Sep 16 12:47:41.412069 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:47:41.405447    2580 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Sep 16 12:47:41.412069 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:47:41.405450    2580 flags.go:64] FLAG: --image-gc-high-threshold="85"
Sep 16 12:47:41.412069 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:47:41.405453    2580 flags.go:64] FLAG: --image-gc-low-threshold="80"
Sep 16 12:47:41.412069 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:47:41.405456    2580 flags.go:64] FLAG: --image-service-endpoint=""
Sep 16 12:47:41.412069 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:47:41.405458    2580 flags.go:64] FLAG: --kernel-memcg-notification="false"
Sep 16 12:47:41.412069 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:47:41.405461    2580 flags.go:64] FLAG: --kube-api-burst="100"
Sep 16 12:47:41.412069 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:47:41.405464    2580 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Sep 16 12:47:41.412069 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:47:41.405468    2580 flags.go:64] FLAG: --kube-api-qps="50"
Sep 16 12:47:41.412069 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:47:41.405471    2580 flags.go:64] FLAG: --kube-reserved=""
Sep 16 12:47:41.412069 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:47:41.405474    2580 flags.go:64] FLAG: --kube-reserved-cgroup=""
Sep 16 12:47:41.412069 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:47:41.405477    2580 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Sep 16 12:47:41.412069 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:47:41.405480    2580 flags.go:64] FLAG: --kubelet-cgroups=""
Sep 16 12:47:41.412688 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:47:41.405483    2580 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Sep 16 12:47:41.412688 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:47:41.405503    2580 flags.go:64] FLAG: --lock-file=""
Sep 16 12:47:41.412688 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:47:41.405506    2580 flags.go:64] FLAG: --log-cadvisor-usage="false"
Sep 16 12:47:41.412688 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:47:41.405509    2580 flags.go:64] FLAG: --log-flush-frequency="5s"
Sep 16 12:47:41.412688 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:47:41.405512    2580 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Sep 16 12:47:41.412688 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:47:41.405517    2580 flags.go:64] FLAG: --log-json-split-stream="false"
Sep 16 12:47:41.412688 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:47:41.405521    2580 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Sep 16 12:47:41.412688 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:47:41.405524    2580 flags.go:64] FLAG: --log-text-split-stream="false"
Sep 16 12:47:41.412688 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:47:41.405527    2580 flags.go:64] FLAG: --logging-format="text"
Sep 16 12:47:41.412688 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:47:41.405530    2580 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Sep 16 12:47:41.412688 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:47:41.405533    2580 flags.go:64] FLAG: --make-iptables-util-chains="true"
Sep 16 12:47:41.412688 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:47:41.405536    2580 flags.go:64] FLAG: --manifest-url=""
Sep 16 12:47:41.412688 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:47:41.405539    2580 flags.go:64] FLAG: --manifest-url-header=""
Sep 16 12:47:41.412688 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:47:41.405543    2580 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Sep 16 12:47:41.412688 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:47:41.405548    2580 flags.go:64] FLAG: --max-open-files="1000000"
Sep 16 12:47:41.412688 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:47:41.405552    2580 flags.go:64] FLAG: --max-pods="110"
Sep 16 12:47:41.412688 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:47:41.405555    2580 flags.go:64] FLAG: --maximum-dead-containers="-1"
Sep 16 12:47:41.412688 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:47:41.405558    2580 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Sep 16 12:47:41.412688 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:47:41.405561    2580 flags.go:64] FLAG: --memory-manager-policy="None"
Sep 16 12:47:41.412688 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:47:41.405564    2580 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Sep 16 12:47:41.412688 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:47:41.405567    2580 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Sep 16 12:47:41.412688 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:47:41.405570    2580 flags.go:64] FLAG: --node-ip="0.0.0.0"
Sep 16 12:47:41.412688 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:47:41.405574    2580 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Sep 16 12:47:41.412688 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:47:41.405583    2580 flags.go:64] FLAG: --node-status-max-images="50"
Sep 16 12:47:41.412688 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:47:41.405586    2580 flags.go:64] FLAG: --node-status-update-frequency="10s"
Sep 16 12:47:41.413334 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:47:41.405589    2580 flags.go:64] FLAG: --oom-score-adj="-999"
Sep 16 12:47:41.413334 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:47:41.405592    2580 flags.go:64] FLAG: --pod-cidr=""
Sep 16 12:47:41.413334 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:47:41.405596    2580 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:eee392dbe3ac2ed4a38bca29ae3bd486f5b7780406c62c600154cd1906871b10"
Sep 16 12:47:41.413334 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:47:41.405602    2580 flags.go:64] FLAG: --pod-manifest-path=""
Sep 16 12:47:41.413334 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:47:41.405605    2580 flags.go:64] FLAG: --pod-max-pids="-1"
Sep 16 12:47:41.413334 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:47:41.405608    2580 flags.go:64] FLAG: --pods-per-core="0"
Sep 16 12:47:41.413334 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:47:41.405611    2580 flags.go:64] FLAG: --port="10250"
Sep 16 12:47:41.413334 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:47:41.405614    2580 flags.go:64] FLAG: --protect-kernel-defaults="false"
Sep 16 12:47:41.413334 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:47:41.405617    2580 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-0639fdd15a3cf38e7"
Sep 16 12:47:41.413334 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:47:41.405620    2580 flags.go:64] FLAG: --qos-reserved=""
Sep 16 12:47:41.413334 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:47:41.405623    2580 flags.go:64] FLAG: --read-only-port="10255"
Sep 16 12:47:41.413334 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:47:41.405626    2580 flags.go:64] FLAG: --register-node="true"
Sep 16 12:47:41.413334 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:47:41.405629    2580 flags.go:64] FLAG: --register-with-taints=""
Sep 16 12:47:41.413334 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:47:41.405633    2580 flags.go:64] FLAG: --registry-burst="10"
Sep 16 12:47:41.413334 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:47:41.405636    2580 flags.go:64] FLAG: --registry-qps="5"
Sep 16 12:47:41.413334 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:47:41.405639    2580 flags.go:64] FLAG: --reserved-cpus=""
Sep 16 12:47:41.413334 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:47:41.405642    2580 flags.go:64] FLAG: --reserved-memory=""
Sep 16 12:47:41.413334 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:47:41.405646    2580 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Sep 16 12:47:41.413334 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:47:41.405649    2580 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Sep 16 12:47:41.413334 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:47:41.405651    2580 flags.go:64] FLAG: --rotate-certificates="false"
Sep 16 12:47:41.413334 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:47:41.405655    2580 flags.go:64] FLAG: --rotate-server-certificates="false"
Sep 16 12:47:41.413334 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:47:41.405658    2580 flags.go:64] FLAG: --runonce="false"
Sep 16 12:47:41.413334 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:47:41.405662    2580 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Sep 16 12:47:41.413334 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:47:41.405665    2580 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Sep 16 12:47:41.413334 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:47:41.405668    2580 flags.go:64] FLAG: --seccomp-default="false"
Sep 16 12:47:41.414019 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:47:41.405671    2580 flags.go:64] FLAG: --serialize-image-pulls="true"
Sep 16 12:47:41.414019 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:47:41.405673    2580 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Sep 16 12:47:41.414019 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:47:41.405676    2580 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Sep 16 12:47:41.414019 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:47:41.405679    2580 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Sep 16 12:47:41.414019 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:47:41.405682    2580 flags.go:64] FLAG: --storage-driver-password="root"
Sep 16 12:47:41.414019 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:47:41.405685    2580 flags.go:64] FLAG: --storage-driver-secure="false"
Sep 16 12:47:41.414019 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:47:41.405688    2580 flags.go:64] FLAG: --storage-driver-table="stats"
Sep 16 12:47:41.414019 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:47:41.405691    2580 flags.go:64] FLAG: --storage-driver-user="root"
Sep 16 12:47:41.414019 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:47:41.405694    2580 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Sep 16 12:47:41.414019 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:47:41.405698    2580 flags.go:64] FLAG: --sync-frequency="1m0s"
Sep 16 12:47:41.414019 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:47:41.405701    2580 flags.go:64] FLAG: --system-cgroups=""
Sep 16 12:47:41.414019 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:47:41.405704    2580 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Sep 16 12:47:41.414019 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:47:41.405709    2580 flags.go:64] FLAG: --system-reserved-cgroup=""
Sep 16 12:47:41.414019 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:47:41.405712    2580 flags.go:64] FLAG: --tls-cert-file=""
Sep 16 12:47:41.414019 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:47:41.405715    2580 flags.go:64] FLAG: --tls-cipher-suites="[]"
Sep 16 12:47:41.414019 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:47:41.405719    2580 flags.go:64] FLAG: --tls-min-version=""
Sep 16 12:47:41.414019 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:47:41.405722    2580 flags.go:64] FLAG: --tls-private-key-file=""
Sep 16 12:47:41.414019 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:47:41.405725    2580 flags.go:64] FLAG: --topology-manager-policy="none"
Sep 16 12:47:41.414019 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:47:41.405728    2580 flags.go:64] FLAG: --topology-manager-policy-options=""
Sep 16 12:47:41.414019 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:47:41.405731    2580 flags.go:64] FLAG: --topology-manager-scope="container"
Sep 16 12:47:41.414019 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:47:41.405734    2580 flags.go:64] FLAG: --v="2"
Sep 16 12:47:41.414019 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:47:41.405737    2580 flags.go:64] FLAG: --version="false"
Sep 16 12:47:41.414019 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:47:41.405742    2580 flags.go:64] FLAG: --vmodule=""
Sep 16 12:47:41.414019 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:47:41.405746    2580 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Sep 16 12:47:41.414019 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:47:41.405749    2580 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Sep 16 12:47:41.414833 ip-10-0-133-218 kubenswrapper[2580]: W0916 12:47:41.405882    2580 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 16 12:47:41.414833 ip-10-0-133-218 kubenswrapper[2580]: W0916 12:47:41.405886    2580 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 16 12:47:41.414833 ip-10-0-133-218 kubenswrapper[2580]: W0916 12:47:41.405889    2580 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 16 12:47:41.414833 ip-10-0-133-218 kubenswrapper[2580]: W0916 12:47:41.405892    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 16 12:47:41.414833 ip-10-0-133-218 kubenswrapper[2580]: W0916 12:47:41.405896    2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 16 12:47:41.414833 ip-10-0-133-218 kubenswrapper[2580]: W0916 12:47:41.405899    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 16 12:47:41.414833 ip-10-0-133-218 kubenswrapper[2580]: W0916 12:47:41.405902    2580 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 16 12:47:41.414833 ip-10-0-133-218 kubenswrapper[2580]: W0916 12:47:41.405906    2580 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 16 12:47:41.414833 ip-10-0-133-218 kubenswrapper[2580]: W0916 12:47:41.405910    2580 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 16 12:47:41.414833 ip-10-0-133-218 kubenswrapper[2580]: W0916 12:47:41.405913    2580 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 16 12:47:41.414833 ip-10-0-133-218 kubenswrapper[2580]: W0916 12:47:41.405916    2580 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 16 12:47:41.414833 ip-10-0-133-218 kubenswrapper[2580]: W0916 12:47:41.405928    2580 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 16 12:47:41.414833 ip-10-0-133-218 kubenswrapper[2580]: W0916 12:47:41.405932    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 16 12:47:41.414833 ip-10-0-133-218 kubenswrapper[2580]: W0916 12:47:41.405935    2580 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 16 12:47:41.414833 ip-10-0-133-218 kubenswrapper[2580]: W0916 12:47:41.405938    2580 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 16 12:47:41.414833 ip-10-0-133-218 kubenswrapper[2580]: W0916 12:47:41.405941    2580 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 16 12:47:41.414833 ip-10-0-133-218 kubenswrapper[2580]: W0916 12:47:41.405947    2580 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 16 12:47:41.414833 ip-10-0-133-218 kubenswrapper[2580]: W0916 12:47:41.405950    2580 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 16 12:47:41.415319 ip-10-0-133-218 kubenswrapper[2580]: W0916 12:47:41.405953    2580 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 16 12:47:41.415319 ip-10-0-133-218 kubenswrapper[2580]: W0916 12:47:41.405956    2580 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 16 12:47:41.415319 ip-10-0-133-218 kubenswrapper[2580]: W0916 12:47:41.405959    2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 16 12:47:41.415319 ip-10-0-133-218 kubenswrapper[2580]: W0916 12:47:41.405962    2580 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 16 12:47:41.415319 ip-10-0-133-218 kubenswrapper[2580]: W0916 12:47:41.405964    2580 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 16 12:47:41.415319 ip-10-0-133-218 kubenswrapper[2580]: W0916 12:47:41.405967    2580 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 16 12:47:41.415319 ip-10-0-133-218 kubenswrapper[2580]: W0916 12:47:41.405970    2580 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 16 12:47:41.415319 ip-10-0-133-218 kubenswrapper[2580]: W0916 12:47:41.405972    2580 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 16 12:47:41.415319 ip-10-0-133-218 kubenswrapper[2580]: W0916 12:47:41.405975    2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 16 12:47:41.415319 ip-10-0-133-218 kubenswrapper[2580]: W0916 12:47:41.405978    2580 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 16 12:47:41.415319 ip-10-0-133-218 kubenswrapper[2580]: W0916 12:47:41.405980    2580 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 16 12:47:41.415319 ip-10-0-133-218 kubenswrapper[2580]: W0916 12:47:41.405983    2580 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 16 12:47:41.415319 ip-10-0-133-218 kubenswrapper[2580]: W0916 12:47:41.405987    2580 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 16 12:47:41.415319 ip-10-0-133-218 kubenswrapper[2580]: W0916 12:47:41.405989    2580 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 16 12:47:41.415319 ip-10-0-133-218 kubenswrapper[2580]: W0916 12:47:41.405992    2580 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 16 12:47:41.415319 ip-10-0-133-218 kubenswrapper[2580]: W0916 12:47:41.405994    2580 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 16 12:47:41.415319 ip-10-0-133-218 kubenswrapper[2580]: W0916 12:47:41.405997    2580 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 16 12:47:41.415319 ip-10-0-133-218 kubenswrapper[2580]: W0916 12:47:41.406000    2580 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 16 12:47:41.415319 ip-10-0-133-218 kubenswrapper[2580]: W0916 12:47:41.406004    2580 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 16 12:47:41.415319 ip-10-0-133-218 kubenswrapper[2580]: W0916 12:47:41.406006    2580 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 16 12:47:41.415839 ip-10-0-133-218 kubenswrapper[2580]: W0916 12:47:41.406009    2580 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 16 12:47:41.415839 ip-10-0-133-218 kubenswrapper[2580]: W0916 12:47:41.406011    2580 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 16 12:47:41.415839 ip-10-0-133-218 kubenswrapper[2580]: W0916 12:47:41.406014    2580 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 16 12:47:41.415839 ip-10-0-133-218 kubenswrapper[2580]: W0916 12:47:41.406017    2580 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 16 12:47:41.415839 ip-10-0-133-218 kubenswrapper[2580]: W0916 12:47:41.406019    2580 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 16 12:47:41.415839 ip-10-0-133-218 kubenswrapper[2580]: W0916 12:47:41.406022    2580 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 16 12:47:41.415839 ip-10-0-133-218 kubenswrapper[2580]: W0916 12:47:41.406025    2580 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 16 12:47:41.415839 ip-10-0-133-218 kubenswrapper[2580]: W0916 12:47:41.406027    2580 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 16 12:47:41.415839 ip-10-0-133-218 kubenswrapper[2580]: W0916 12:47:41.406030    2580 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 16 12:47:41.415839 ip-10-0-133-218 kubenswrapper[2580]: W0916 12:47:41.406032    2580 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 16 12:47:41.415839 ip-10-0-133-218 kubenswrapper[2580]: W0916 12:47:41.406037    2580 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 16 12:47:41.415839 ip-10-0-133-218 kubenswrapper[2580]: W0916 12:47:41.406039    2580 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 16 12:47:41.415839 ip-10-0-133-218 kubenswrapper[2580]: W0916 12:47:41.406042    2580 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 16 12:47:41.415839 ip-10-0-133-218 kubenswrapper[2580]: W0916 12:47:41.406045    2580 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 16 12:47:41.415839 ip-10-0-133-218 kubenswrapper[2580]: W0916 12:47:41.406048    2580 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 16 12:47:41.415839 ip-10-0-133-218 kubenswrapper[2580]: W0916 12:47:41.406050    2580 feature_gate.go:328] unrecognized feature gate: Example2
Sep 16 12:47:41.415839 ip-10-0-133-218 kubenswrapper[2580]: W0916 12:47:41.406053    2580 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 16 12:47:41.415839 ip-10-0-133-218 kubenswrapper[2580]: W0916 12:47:41.406055    2580 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 16 12:47:41.415839 ip-10-0-133-218 kubenswrapper[2580]: W0916 12:47:41.406058    2580 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 16 12:47:41.415839 ip-10-0-133-218 kubenswrapper[2580]: W0916 12:47:41.406060    2580 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 16 12:47:41.416330 ip-10-0-133-218 kubenswrapper[2580]: W0916 12:47:41.406063    2580 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 16 12:47:41.416330 ip-10-0-133-218 kubenswrapper[2580]: W0916 12:47:41.406065    2580 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 16 12:47:41.416330 ip-10-0-133-218 kubenswrapper[2580]: W0916 12:47:41.406068    2580 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 16 12:47:41.416330 ip-10-0-133-218 kubenswrapper[2580]: W0916 12:47:41.406070    2580 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 16 12:47:41.416330 ip-10-0-133-218 kubenswrapper[2580]: W0916 12:47:41.406073    2580 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 16 12:47:41.416330 ip-10-0-133-218 kubenswrapper[2580]: W0916 12:47:41.406080    2580 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 16 12:47:41.416330 ip-10-0-133-218 kubenswrapper[2580]: W0916 12:47:41.406082    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 16 12:47:41.416330 ip-10-0-133-218 kubenswrapper[2580]: W0916 12:47:41.406085    2580 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 16 12:47:41.416330 ip-10-0-133-218 kubenswrapper[2580]: W0916 12:47:41.406088    2580 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 16 12:47:41.416330 ip-10-0-133-218 kubenswrapper[2580]: W0916 12:47:41.406090    2580 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 16 12:47:41.416330 ip-10-0-133-218 kubenswrapper[2580]: W0916 12:47:41.406094    2580 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 16 12:47:41.416330 ip-10-0-133-218 kubenswrapper[2580]: W0916 12:47:41.406096    2580 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 16 12:47:41.416330 ip-10-0-133-218 kubenswrapper[2580]: W0916 12:47:41.406107    2580 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 16 12:47:41.416330 ip-10-0-133-218 kubenswrapper[2580]: W0916 12:47:41.406110    2580 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 16 12:47:41.416330 ip-10-0-133-218 kubenswrapper[2580]: W0916 12:47:41.406113    2580 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 16 12:47:41.416330 ip-10-0-133-218 kubenswrapper[2580]: W0916 12:47:41.406116    2580 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 16 12:47:41.416330 ip-10-0-133-218 kubenswrapper[2580]: W0916 12:47:41.406118    2580 feature_gate.go:328] unrecognized feature gate: Example
Sep 16 12:47:41.416330 ip-10-0-133-218 kubenswrapper[2580]: W0916 12:47:41.406121    2580 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 16 12:47:41.416330 ip-10-0-133-218 kubenswrapper[2580]: W0916 12:47:41.406124    2580 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 16 12:47:41.416330 ip-10-0-133-218 kubenswrapper[2580]: W0916 12:47:41.406127    2580 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 16 12:47:41.416846 ip-10-0-133-218 kubenswrapper[2580]: W0916 12:47:41.406129    2580 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 16 12:47:41.416846 ip-10-0-133-218 kubenswrapper[2580]: W0916 12:47:41.406132    2580 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 16 12:47:41.416846 ip-10-0-133-218 kubenswrapper[2580]: W0916 12:47:41.406136    2580 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 16 12:47:41.416846 ip-10-0-133-218 kubenswrapper[2580]: W0916 12:47:41.406139    2580 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 16 12:47:41.416846 ip-10-0-133-218 kubenswrapper[2580]: W0916 12:47:41.406142    2580 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 16 12:47:41.416846 ip-10-0-133-218 kubenswrapper[2580]: W0916 12:47:41.406144    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 16 12:47:41.416846 ip-10-0-133-218 kubenswrapper[2580]: W0916 12:47:41.406147    2580 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 16 12:47:41.416846 ip-10-0-133-218 kubenswrapper[2580]: W0916 12:47:41.406149    2580 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 16 12:47:41.416846 ip-10-0-133-218 kubenswrapper[2580]: W0916 12:47:41.406152    2580 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 16 12:47:41.416846 ip-10-0-133-218 kubenswrapper[2580]: W0916 12:47:41.406155    2580 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 16 12:47:41.416846 ip-10-0-133-218 kubenswrapper[2580]: W0916 12:47:41.406157    2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 16 12:47:41.416846 ip-10-0-133-218 kubenswrapper[2580]: W0916 12:47:41.406160    2580 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 16 12:47:41.416846 ip-10-0-133-218 kubenswrapper[2580]: W0916 12:47:41.406163    2580 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 16 12:47:41.416846 ip-10-0-133-218 kubenswrapper[2580]: W0916 12:47:41.406165    2580 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 16 12:47:41.416846 ip-10-0-133-218 kubenswrapper[2580]: W0916 12:47:41.406168    2580 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 16 12:47:41.416846 ip-10-0-133-218 kubenswrapper[2580]: W0916 12:47:41.406171    2580 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 16 12:47:41.416846 ip-10-0-133-218 kubenswrapper[2580]: W0916 12:47:41.406173    2580 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 16 12:47:41.416846 ip-10-0-133-218 kubenswrapper[2580]: W0916 12:47:41.406176    2580 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 16 12:47:41.416846 ip-10-0-133-218 kubenswrapper[2580]: W0916 12:47:41.406179    2580 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 16 12:47:41.416846 ip-10-0-133-218 kubenswrapper[2580]: W0916 12:47:41.406182    2580 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 16 12:47:41.417336 ip-10-0-133-218 kubenswrapper[2580]: W0916 12:47:41.406185    2580 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 16 12:47:41.417336 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:47:41.406859    2580 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 16 12:47:41.417336 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:47:41.414672    2580 server.go:529] "Kubelet version" kubeletVersion="v1.34.9"
Sep 16 12:47:41.417336 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:47:41.414687    2580 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Sep 16 12:47:41.417336 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:47:41.415533    2580 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Sep 16 12:47:41.417336 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:47:41.415541    2580 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Sep 16 12:47:41.417336 ip-10-0-133-218 kubenswrapper[2580]: W0916 12:47:41.415602    2580 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 16 12:47:41.417336 ip-10-0-133-218 kubenswrapper[2580]: W0916 12:47:41.415607    2580 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 16 12:47:41.417336 ip-10-0-133-218 kubenswrapper[2580]: W0916 12:47:41.415611    2580 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 16 12:47:41.417336 ip-10-0-133-218 kubenswrapper[2580]: W0916 12:47:41.415614    2580 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 16 12:47:41.417336 ip-10-0-133-218 kubenswrapper[2580]: W0916 12:47:41.415617    2580 feature_gate.go:328] unrecognized feature gate: Example2
Sep 16 12:47:41.417336 ip-10-0-133-218 kubenswrapper[2580]: W0916 12:47:41.415620    2580 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 16 12:47:41.417336 ip-10-0-133-218 kubenswrapper[2580]: W0916 12:47:41.415623    2580 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 16 12:47:41.417336 ip-10-0-133-218 kubenswrapper[2580]: W0916 12:47:41.415626    2580 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 16 12:47:41.417752 ip-10-0-133-218 kubenswrapper[2580]: W0916 12:47:41.415629    2580 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 16 12:47:41.417752 ip-10-0-133-218 kubenswrapper[2580]: W0916 12:47:41.415631    2580 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 16 12:47:41.417752 ip-10-0-133-218 kubenswrapper[2580]: W0916 12:47:41.415634    2580 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 16 12:47:41.417752 ip-10-0-133-218 kubenswrapper[2580]: W0916 12:47:41.415637    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 16 12:47:41.417752 ip-10-0-133-218 kubenswrapper[2580]: W0916 12:47:41.415640    2580 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 16 12:47:41.417752 ip-10-0-133-218 kubenswrapper[2580]: W0916 12:47:41.415642    2580 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 16 12:47:41.417752 ip-10-0-133-218 kubenswrapper[2580]: W0916 12:47:41.415645    2580 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 16 12:47:41.417752 ip-10-0-133-218 kubenswrapper[2580]: W0916 12:47:41.415648    2580 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 16 12:47:41.417752 ip-10-0-133-218 kubenswrapper[2580]: W0916 12:47:41.415651    2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 16 12:47:41.417752 ip-10-0-133-218 kubenswrapper[2580]: W0916 12:47:41.415654    2580 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 16 12:47:41.417752 ip-10-0-133-218 kubenswrapper[2580]: W0916 12:47:41.415657    2580 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 16 12:47:41.417752 ip-10-0-133-218 kubenswrapper[2580]: W0916 12:47:41.415659    2580 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 16 12:47:41.417752 ip-10-0-133-218 kubenswrapper[2580]: W0916 12:47:41.415662    2580 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 16 12:47:41.417752 ip-10-0-133-218 kubenswrapper[2580]: W0916 12:47:41.415665    2580 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 16 12:47:41.417752 ip-10-0-133-218 kubenswrapper[2580]: W0916 12:47:41.415667    2580 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 16 12:47:41.417752 ip-10-0-133-218 kubenswrapper[2580]: W0916 12:47:41.415670    2580 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 16 12:47:41.417752 ip-10-0-133-218 kubenswrapper[2580]: W0916 12:47:41.415672    2580 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 16 12:47:41.417752 ip-10-0-133-218 kubenswrapper[2580]: W0916 12:47:41.415675    2580 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 16 12:47:41.417752 ip-10-0-133-218 kubenswrapper[2580]: W0916 12:47:41.415678    2580 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 16 12:47:41.417752 ip-10-0-133-218 kubenswrapper[2580]: W0916 12:47:41.415680    2580 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 16 12:47:41.418252 ip-10-0-133-218 kubenswrapper[2580]: W0916 12:47:41.415683    2580 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 16 12:47:41.418252 ip-10-0-133-218 kubenswrapper[2580]: W0916 12:47:41.415687    2580 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 16 12:47:41.418252 ip-10-0-133-218 kubenswrapper[2580]: W0916 12:47:41.415690    2580 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 16 12:47:41.418252 ip-10-0-133-218 kubenswrapper[2580]: W0916 12:47:41.415693    2580 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 16 12:47:41.418252 ip-10-0-133-218 kubenswrapper[2580]: W0916 12:47:41.415695    2580 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 16 12:47:41.418252 ip-10-0-133-218 kubenswrapper[2580]: W0916 12:47:41.415698    2580 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 16 12:47:41.418252 ip-10-0-133-218 kubenswrapper[2580]: W0916 12:47:41.415701    2580 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 16 12:47:41.418252 ip-10-0-133-218 kubenswrapper[2580]: W0916 12:47:41.415703    2580 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 16 12:47:41.418252 ip-10-0-133-218 kubenswrapper[2580]: W0916 12:47:41.415706    2580 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 16 12:47:41.418252 ip-10-0-133-218 kubenswrapper[2580]: W0916 12:47:41.415708    2580 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 16 12:47:41.418252 ip-10-0-133-218 kubenswrapper[2580]: W0916 12:47:41.415712    2580 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 16 12:47:41.418252 ip-10-0-133-218 kubenswrapper[2580]: W0916 12:47:41.415716    2580 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 16 12:47:41.418252 ip-10-0-133-218 kubenswrapper[2580]: W0916 12:47:41.415719    2580 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 16 12:47:41.418252 ip-10-0-133-218 kubenswrapper[2580]: W0916 12:47:41.415722    2580 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 16 12:47:41.418252 ip-10-0-133-218 kubenswrapper[2580]: W0916 12:47:41.415725    2580 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 16 12:47:41.418252 ip-10-0-133-218 kubenswrapper[2580]: W0916 12:47:41.415727    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 16 12:47:41.418252 ip-10-0-133-218 kubenswrapper[2580]: W0916 12:47:41.415730    2580 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 16 12:47:41.418252 ip-10-0-133-218 kubenswrapper[2580]: W0916 12:47:41.415732    2580 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 16 12:47:41.418252 ip-10-0-133-218 kubenswrapper[2580]: W0916 12:47:41.415735    2580 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 16 12:47:41.418731 ip-10-0-133-218 kubenswrapper[2580]: W0916 12:47:41.415739    2580 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 16 12:47:41.418731 ip-10-0-133-218 kubenswrapper[2580]: W0916 12:47:41.415741    2580 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 16 12:47:41.418731 ip-10-0-133-218 kubenswrapper[2580]: W0916 12:47:41.415745    2580 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 16 12:47:41.418731 ip-10-0-133-218 kubenswrapper[2580]: W0916 12:47:41.415748    2580 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 16 12:47:41.418731 ip-10-0-133-218 kubenswrapper[2580]: W0916 12:47:41.415751    2580 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 16 12:47:41.418731 ip-10-0-133-218 kubenswrapper[2580]: W0916 12:47:41.415753    2580 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 16 12:47:41.418731 ip-10-0-133-218 kubenswrapper[2580]: W0916 12:47:41.415756    2580 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 16 12:47:41.418731 ip-10-0-133-218 kubenswrapper[2580]: W0916 12:47:41.415759    2580 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 16 12:47:41.418731 ip-10-0-133-218 kubenswrapper[2580]: W0916 12:47:41.415761    2580 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 16 12:47:41.418731 ip-10-0-133-218 kubenswrapper[2580]: W0916 12:47:41.415764    2580 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 16 12:47:41.418731 ip-10-0-133-218 kubenswrapper[2580]: W0916 12:47:41.415766    2580 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 16 12:47:41.418731 ip-10-0-133-218 kubenswrapper[2580]: W0916 12:47:41.415769    2580 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 16 12:47:41.418731 ip-10-0-133-218 kubenswrapper[2580]: W0916 12:47:41.415772    2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 16 12:47:41.418731 ip-10-0-133-218 kubenswrapper[2580]: W0916 12:47:41.415774    2580 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 16 12:47:41.418731 ip-10-0-133-218 kubenswrapper[2580]: W0916 12:47:41.415779    2580 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 16 12:47:41.418731 ip-10-0-133-218 kubenswrapper[2580]: W0916 12:47:41.415783    2580 feature_gate.go:328] unrecognized feature gate: Example
Sep 16 12:47:41.418731 ip-10-0-133-218 kubenswrapper[2580]: W0916 12:47:41.415786    2580 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 16 12:47:41.418731 ip-10-0-133-218 kubenswrapper[2580]: W0916 12:47:41.415789    2580 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 16 12:47:41.418731 ip-10-0-133-218 kubenswrapper[2580]: W0916 12:47:41.415792    2580 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 16 12:47:41.418731 ip-10-0-133-218 kubenswrapper[2580]: W0916 12:47:41.415795    2580 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 16 12:47:41.419261 ip-10-0-133-218 kubenswrapper[2580]: W0916 12:47:41.415798    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 16 12:47:41.419261 ip-10-0-133-218 kubenswrapper[2580]: W0916 12:47:41.415801    2580 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 16 12:47:41.419261 ip-10-0-133-218 kubenswrapper[2580]: W0916 12:47:41.415803    2580 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 16 12:47:41.419261 ip-10-0-133-218 kubenswrapper[2580]: W0916 12:47:41.415806    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 16 12:47:41.419261 ip-10-0-133-218 kubenswrapper[2580]: W0916 12:47:41.415809    2580 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 16 12:47:41.419261 ip-10-0-133-218 kubenswrapper[2580]: W0916 12:47:41.415812    2580 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 16 12:47:41.419261 ip-10-0-133-218 kubenswrapper[2580]: W0916 12:47:41.415814    2580 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 16 12:47:41.419261 ip-10-0-133-218 kubenswrapper[2580]: W0916 12:47:41.415817    2580 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 16 12:47:41.419261 ip-10-0-133-218 kubenswrapper[2580]: W0916 12:47:41.415820    2580 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 16 12:47:41.419261 ip-10-0-133-218 kubenswrapper[2580]: W0916 12:47:41.415822    2580 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 16 12:47:41.419261 ip-10-0-133-218 kubenswrapper[2580]: W0916 12:47:41.415825    2580 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 16 12:47:41.419261 ip-10-0-133-218 kubenswrapper[2580]: W0916 12:47:41.415828    2580 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 16 12:47:41.419261 ip-10-0-133-218 kubenswrapper[2580]: W0916 12:47:41.415830    2580 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 16 12:47:41.419261 ip-10-0-133-218 kubenswrapper[2580]: W0916 12:47:41.415833    2580 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 16 12:47:41.419261 ip-10-0-133-218 kubenswrapper[2580]: W0916 12:47:41.415836    2580 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 16 12:47:41.419261 ip-10-0-133-218 kubenswrapper[2580]: W0916 12:47:41.415839    2580 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 16 12:47:41.419261 ip-10-0-133-218 kubenswrapper[2580]: W0916 12:47:41.415842    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 16 12:47:41.419261 ip-10-0-133-218 kubenswrapper[2580]: W0916 12:47:41.415844    2580 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 16 12:47:41.419261 ip-10-0-133-218 kubenswrapper[2580]: W0916 12:47:41.415847    2580 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 16 12:47:41.419261 ip-10-0-133-218 kubenswrapper[2580]: W0916 12:47:41.415851    2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 16 12:47:41.419818 ip-10-0-133-218 kubenswrapper[2580]: W0916 12:47:41.415855    2580 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 16 12:47:41.419818 ip-10-0-133-218 kubenswrapper[2580]: W0916 12:47:41.415858    2580 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 16 12:47:41.419818 ip-10-0-133-218 kubenswrapper[2580]: W0916 12:47:41.415861    2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 16 12:47:41.419818 ip-10-0-133-218 kubenswrapper[2580]: W0916 12:47:41.415864    2580 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 16 12:47:41.419818 ip-10-0-133-218 kubenswrapper[2580]: W0916 12:47:41.415867    2580 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 16 12:47:41.419818 ip-10-0-133-218 kubenswrapper[2580]: W0916 12:47:41.415870    2580 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 16 12:47:41.419818 ip-10-0-133-218 kubenswrapper[2580]: W0916 12:47:41.415872    2580 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 16 12:47:41.419818 ip-10-0-133-218 kubenswrapper[2580]: W0916 12:47:41.415875    2580 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 16 12:47:41.419818 ip-10-0-133-218 kubenswrapper[2580]: W0916 12:47:41.415879    2580 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 16 12:47:41.419818 ip-10-0-133-218 kubenswrapper[2580]: W0916 12:47:41.415881    2580 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 16 12:47:41.419818 ip-10-0-133-218 kubenswrapper[2580]: W0916 12:47:41.415884    2580 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 16 12:47:41.419818 ip-10-0-133-218 kubenswrapper[2580]: W0916 12:47:41.415887    2580 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 16 12:47:41.419818 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:47:41.415892    2580 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 16 12:47:41.419818 ip-10-0-133-218 kubenswrapper[2580]: W0916 12:47:41.416029    2580 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 16 12:47:41.420186 ip-10-0-133-218 kubenswrapper[2580]: W0916 12:47:41.416036    2580 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 16 12:47:41.420186 ip-10-0-133-218 kubenswrapper[2580]: W0916 12:47:41.416039    2580 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 16 12:47:41.420186 ip-10-0-133-218 kubenswrapper[2580]: W0916 12:47:41.416042    2580 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 16 12:47:41.420186 ip-10-0-133-218 kubenswrapper[2580]: W0916 12:47:41.416045    2580 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 16 12:47:41.420186 ip-10-0-133-218 kubenswrapper[2580]: W0916 12:47:41.416048    2580 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 16 12:47:41.420186 ip-10-0-133-218 kubenswrapper[2580]: W0916 12:47:41.416050    2580 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 16 12:47:41.420186 ip-10-0-133-218 kubenswrapper[2580]: W0916 12:47:41.416054    2580 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 16 12:47:41.420186 ip-10-0-133-218 kubenswrapper[2580]: W0916 12:47:41.416057    2580 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 16 12:47:41.420186 ip-10-0-133-218 kubenswrapper[2580]: W0916 12:47:41.416059    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 16 12:47:41.420186 ip-10-0-133-218 kubenswrapper[2580]: W0916 12:47:41.416062    2580 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 16 12:47:41.420186 ip-10-0-133-218 kubenswrapper[2580]: W0916 12:47:41.416065    2580 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 16 12:47:41.420186 ip-10-0-133-218 kubenswrapper[2580]: W0916 12:47:41.416068    2580 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 16 12:47:41.420186 ip-10-0-133-218 kubenswrapper[2580]: W0916 12:47:41.416071    2580 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 16 12:47:41.420186 ip-10-0-133-218 kubenswrapper[2580]: W0916 12:47:41.416075    2580 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 16 12:47:41.420186 ip-10-0-133-218 kubenswrapper[2580]: W0916 12:47:41.416079    2580 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 16 12:47:41.420186 ip-10-0-133-218 kubenswrapper[2580]: W0916 12:47:41.416083    2580 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 16 12:47:41.420186 ip-10-0-133-218 kubenswrapper[2580]: W0916 12:47:41.416086    2580 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 16 12:47:41.420186 ip-10-0-133-218 kubenswrapper[2580]: W0916 12:47:41.416089    2580 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 16 12:47:41.420186 ip-10-0-133-218 kubenswrapper[2580]: W0916 12:47:41.416092    2580 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 16 12:47:41.420186 ip-10-0-133-218 kubenswrapper[2580]: W0916 12:47:41.416095    2580 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 16 12:47:41.420689 ip-10-0-133-218 kubenswrapper[2580]: W0916 12:47:41.416098    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 16 12:47:41.420689 ip-10-0-133-218 kubenswrapper[2580]: W0916 12:47:41.416101    2580 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 16 12:47:41.420689 ip-10-0-133-218 kubenswrapper[2580]: W0916 12:47:41.416104    2580 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 16 12:47:41.420689 ip-10-0-133-218 kubenswrapper[2580]: W0916 12:47:41.416106    2580 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 16 12:47:41.420689 ip-10-0-133-218 kubenswrapper[2580]: W0916 12:47:41.416109    2580 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 16 12:47:41.420689 ip-10-0-133-218 kubenswrapper[2580]: W0916 12:47:41.416111    2580 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 16 12:47:41.420689 ip-10-0-133-218 kubenswrapper[2580]: W0916 12:47:41.416114    2580 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 16 12:47:41.420689 ip-10-0-133-218 kubenswrapper[2580]: W0916 12:47:41.416118    2580 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 16 12:47:41.420689 ip-10-0-133-218 kubenswrapper[2580]: W0916 12:47:41.416120    2580 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 16 12:47:41.420689 ip-10-0-133-218 kubenswrapper[2580]: W0916 12:47:41.416124    2580 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 16 12:47:41.420689 ip-10-0-133-218 kubenswrapper[2580]: W0916 12:47:41.416127    2580 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 16 12:47:41.420689 ip-10-0-133-218 kubenswrapper[2580]: W0916 12:47:41.416130    2580 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 16 12:47:41.420689 ip-10-0-133-218 kubenswrapper[2580]: W0916 12:47:41.416133    2580 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 16 12:47:41.420689 ip-10-0-133-218 kubenswrapper[2580]: W0916 12:47:41.416135    2580 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 16 12:47:41.420689 ip-10-0-133-218 kubenswrapper[2580]: W0916 12:47:41.416138    2580 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 16 12:47:41.420689 ip-10-0-133-218 kubenswrapper[2580]: W0916 12:47:41.416140    2580 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 16 12:47:41.420689 ip-10-0-133-218 kubenswrapper[2580]: W0916 12:47:41.416143    2580 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 16 12:47:41.420689 ip-10-0-133-218 kubenswrapper[2580]: W0916 12:47:41.416146    2580 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 16 12:47:41.420689 ip-10-0-133-218 kubenswrapper[2580]: W0916 12:47:41.416148    2580 feature_gate.go:328] unrecognized feature gate: Example2
Sep 16 12:47:41.421146 ip-10-0-133-218 kubenswrapper[2580]: W0916 12:47:41.416151    2580 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 16 12:47:41.421146 ip-10-0-133-218 kubenswrapper[2580]: W0916 12:47:41.416153    2580 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 16 12:47:41.421146 ip-10-0-133-218 kubenswrapper[2580]: W0916 12:47:41.416156    2580 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 16 12:47:41.421146 ip-10-0-133-218 kubenswrapper[2580]: W0916 12:47:41.416158    2580 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 16 12:47:41.421146 ip-10-0-133-218 kubenswrapper[2580]: W0916 12:47:41.416161    2580 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 16 12:47:41.421146 ip-10-0-133-218 kubenswrapper[2580]: W0916 12:47:41.416163    2580 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 16 12:47:41.421146 ip-10-0-133-218 kubenswrapper[2580]: W0916 12:47:41.416166    2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 16 12:47:41.421146 ip-10-0-133-218 kubenswrapper[2580]: W0916 12:47:41.416169    2580 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 16 12:47:41.421146 ip-10-0-133-218 kubenswrapper[2580]: W0916 12:47:41.416171    2580 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 16 12:47:41.421146 ip-10-0-133-218 kubenswrapper[2580]: W0916 12:47:41.416174    2580 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 16 12:47:41.421146 ip-10-0-133-218 kubenswrapper[2580]: W0916 12:47:41.416177    2580 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 16 12:47:41.421146 ip-10-0-133-218 kubenswrapper[2580]: W0916 12:47:41.416180    2580 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 16 12:47:41.421146 ip-10-0-133-218 kubenswrapper[2580]: W0916 12:47:41.416182    2580 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 16 12:47:41.421146 ip-10-0-133-218 kubenswrapper[2580]: W0916 12:47:41.416185    2580 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 16 12:47:41.421146 ip-10-0-133-218 kubenswrapper[2580]: W0916 12:47:41.416187    2580 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 16 12:47:41.421146 ip-10-0-133-218 kubenswrapper[2580]: W0916 12:47:41.416190    2580 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 16 12:47:41.421146 ip-10-0-133-218 kubenswrapper[2580]: W0916 12:47:41.416192    2580 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 16 12:47:41.421146 ip-10-0-133-218 kubenswrapper[2580]: W0916 12:47:41.416195    2580 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 16 12:47:41.421146 ip-10-0-133-218 kubenswrapper[2580]: W0916 12:47:41.416197    2580 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 16 12:47:41.421146 ip-10-0-133-218 kubenswrapper[2580]: W0916 12:47:41.416200    2580 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 16 12:47:41.421646 ip-10-0-133-218 kubenswrapper[2580]: W0916 12:47:41.416202    2580 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 16 12:47:41.421646 ip-10-0-133-218 kubenswrapper[2580]: W0916 12:47:41.416205    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 16 12:47:41.421646 ip-10-0-133-218 kubenswrapper[2580]: W0916 12:47:41.416208    2580 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 16 12:47:41.421646 ip-10-0-133-218 kubenswrapper[2580]: W0916 12:47:41.416211    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 16 12:47:41.421646 ip-10-0-133-218 kubenswrapper[2580]: W0916 12:47:41.416213    2580 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 16 12:47:41.421646 ip-10-0-133-218 kubenswrapper[2580]: W0916 12:47:41.416216    2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 16 12:47:41.421646 ip-10-0-133-218 kubenswrapper[2580]: W0916 12:47:41.416218    2580 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 16 12:47:41.421646 ip-10-0-133-218 kubenswrapper[2580]: W0916 12:47:41.416221    2580 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 16 12:47:41.421646 ip-10-0-133-218 kubenswrapper[2580]: W0916 12:47:41.416223    2580 feature_gate.go:328] unrecognized feature gate: Example
Sep 16 12:47:41.421646 ip-10-0-133-218 kubenswrapper[2580]: W0916 12:47:41.416226    2580 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 16 12:47:41.421646 ip-10-0-133-218 kubenswrapper[2580]: W0916 12:47:41.416228    2580 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 16 12:47:41.421646 ip-10-0-133-218 kubenswrapper[2580]: W0916 12:47:41.416231    2580 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 16 12:47:41.421646 ip-10-0-133-218 kubenswrapper[2580]: W0916 12:47:41.416234    2580 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 16 12:47:41.421646 ip-10-0-133-218 kubenswrapper[2580]: W0916 12:47:41.416236    2580 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 16 12:47:41.421646 ip-10-0-133-218 kubenswrapper[2580]: W0916 12:47:41.416239    2580 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 16 12:47:41.421646 ip-10-0-133-218 kubenswrapper[2580]: W0916 12:47:41.416241    2580 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 16 12:47:41.421646 ip-10-0-133-218 kubenswrapper[2580]: W0916 12:47:41.416243    2580 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 16 12:47:41.421646 ip-10-0-133-218 kubenswrapper[2580]: W0916 12:47:41.416246    2580 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 16 12:47:41.421646 ip-10-0-133-218 kubenswrapper[2580]: W0916 12:47:41.416249    2580 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 16 12:47:41.421646 ip-10-0-133-218 kubenswrapper[2580]: W0916 12:47:41.416252    2580 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 16 12:47:41.422154 ip-10-0-133-218 kubenswrapper[2580]: W0916 12:47:41.416254    2580 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 16 12:47:41.422154 ip-10-0-133-218 kubenswrapper[2580]: W0916 12:47:41.416257    2580 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 16 12:47:41.422154 ip-10-0-133-218 kubenswrapper[2580]: W0916 12:47:41.416260    2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 16 12:47:41.422154 ip-10-0-133-218 kubenswrapper[2580]: W0916 12:47:41.416262    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 16 12:47:41.422154 ip-10-0-133-218 kubenswrapper[2580]: W0916 12:47:41.416265    2580 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 16 12:47:41.422154 ip-10-0-133-218 kubenswrapper[2580]: W0916 12:47:41.416268    2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 16 12:47:41.422154 ip-10-0-133-218 kubenswrapper[2580]: W0916 12:47:41.416270    2580 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 16 12:47:41.422154 ip-10-0-133-218 kubenswrapper[2580]: W0916 12:47:41.416273    2580 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 16 12:47:41.422154 ip-10-0-133-218 kubenswrapper[2580]: W0916 12:47:41.416275    2580 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 16 12:47:41.422154 ip-10-0-133-218 kubenswrapper[2580]: W0916 12:47:41.416278    2580 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 16 12:47:41.422154 ip-10-0-133-218 kubenswrapper[2580]: W0916 12:47:41.416281    2580 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 16 12:47:41.422154 ip-10-0-133-218 kubenswrapper[2580]: W0916 12:47:41.416284    2580 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 16 12:47:41.422154 ip-10-0-133-218 kubenswrapper[2580]: W0916 12:47:41.416286    2580 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 16 12:47:41.422154 ip-10-0-133-218 kubenswrapper[2580]: W0916 12:47:41.416289    2580 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 16 12:47:41.422154 ip-10-0-133-218 kubenswrapper[2580]: W0916 12:47:41.416292    2580 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 16 12:47:41.422154 ip-10-0-133-218 kubenswrapper[2580]: W0916 12:47:41.416294    2580 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 16 12:47:41.422154 ip-10-0-133-218 kubenswrapper[2580]: W0916 12:47:41.416297    2580 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 16 12:47:41.422154 ip-10-0-133-218 kubenswrapper[2580]: W0916 12:47:41.416299    2580 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 16 12:47:41.422154 ip-10-0-133-218 kubenswrapper[2580]: W0916 12:47:41.416303    2580 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 16 12:47:41.422647 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:47:41.416308    2580 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 16 12:47:41.422647 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:47:41.416422    2580 server.go:962] "Client rotation is on, will bootstrap in background"
Sep 16 12:47:41.422647 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:47:41.418833    2580 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir"
Sep 16 12:47:41.422647 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:47:41.419963    2580 server.go:1019] "Starting client certificate rotation"
Sep 16 12:47:41.422647 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:47:41.420064    2580 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Sep 16 12:47:41.422647 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:47:41.420111    2580 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet"
Sep 16 12:47:41.423451 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:47:41.423424    2580 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Sep 16 12:47:41.424683 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:47:41.424664    2580 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Sep 16 12:47:41.435290 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:47:41.435270    2580 log.go:25] "Validated CRI v1 runtime API"
Sep 16 12:47:41.438254 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:47:41.438239    2580 log.go:25] "Validated CRI v1 image API"
Sep 16 12:47:41.440566 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:47:41.440547    2580 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Sep 16 12:47:41.446685 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:47:41.446664    2580 fs.go:135] Filesystem UUIDs: map[5480bdc1-6529-4c4c-8ee1-9c24523975d9:/dev/nvme0n1p4 7B77-95E7:/dev/nvme0n1p2 d3c3a458-d478-4dd3-b34a-bbde41c20692:/dev/nvme0n1p3]
Sep 16 12:47:41.446752 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:47:41.446682    2580 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}]
Sep 16 12:47:41.453588 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:47:41.453456    2580 manager.go:216] Machine: {Timestamp:2026-09-16 12:47:41.450805929 +0000 UTC m=+0.373678395 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3109307 MemoryCapacity:33164476416 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2e5561002a81f58cd44111a65e543a SystemUUID:ec2e5561-002a-81f5-8cd4-4111a65e543a BootID:fdd0d3de-6b44-47f6-95fb-d61861261980 Filesystems:[{Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6632898560 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6111232 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16582238208 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16582238208 Type:vfs Inodes:4048398 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:ff:f5:30:c8:1d Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:ff:f5:30:c8:1d Speed:0 Mtu:9001} {Name:ovs-system MacAddress:1a:af:26:1a:f0:5a Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164476416 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Sep 16 12:47:41.453588 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:47:41.453581    2580 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Sep 16 12:47:41.453764 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:47:41.453695    2580 manager.go:232] Version: {KernelVersion:5.14.0-570.140.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260908-1 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Sep 16 12:47:41.455460 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:47:41.455429    2580 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Sep 16 12:47:41.455648 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:47:41.455465    2580 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-133-218.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Sep 16 12:47:41.456358 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:47:41.456347    2580 topology_manager.go:138] "Creating topology manager with none policy"
Sep 16 12:47:41.456425 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:47:41.456363    2580 container_manager_linux.go:309] "Creating device plugin manager"
Sep 16 12:47:41.456425 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:47:41.456373    2580 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Sep 16 12:47:41.456425 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:47:41.456394    2580 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Sep 16 12:47:41.456539 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:47:41.456528    2580 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Sep 16 12:47:41.457558 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:47:41.457545    2580 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager"
Sep 16 12:47:41.461056 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:47:41.461042    2580 state_mem.go:36] "Initialized new in-memory state store"
Sep 16 12:47:41.461192 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:47:41.461181    2580 server.go:1240] "Using root directory" path="/var/lib/kubelet"
Sep 16 12:47:41.461248 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:47:41.461217    2580 kubelet.go:486] "Attempting to sync node with API server"
Sep 16 12:47:41.461248 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:47:41.461231    2580 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests"
Sep 16 12:47:41.461334 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:47:41.461250    2580 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Sep 16 12:47:41.461334 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:47:41.461264    2580 kubelet.go:398] "Adding apiserver pod source"
Sep 16 12:47:41.461334 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:47:41.461277    2580 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Sep 16 12:47:41.461684 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:47:41.461670    2580 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Sep 16 12:47:41.461756 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:47:41.461694    2580 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Sep 16 12:47:41.464106 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:47:41.464091    2580 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.12-5.rhaos4.21.gitd1ec63c.el9" apiVersion="v1"
Sep 16 12:47:41.467113 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:47:41.467099    2580 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Sep 16 12:47:41.467166 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:47:41.467130    2580 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Sep 16 12:47:41.469437 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:47:41.469414    2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Sep 16 12:47:41.469533 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:47:41.469443    2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Sep 16 12:47:41.469533 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:47:41.469453    2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Sep 16 12:47:41.469533 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:47:41.469463    2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Sep 16 12:47:41.469533 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:47:41.469476    2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Sep 16 12:47:41.469533 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:47:41.469505    2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Sep 16 12:47:41.469533 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:47:41.469528    2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Sep 16 12:47:41.469747 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:47:41.469539    2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Sep 16 12:47:41.469747 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:47:41.469549    2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Sep 16 12:47:41.469747 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:47:41.469558    2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Sep 16 12:47:41.469747 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:47:41.469583    2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Sep 16 12:47:41.469747 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:47:41.469594    2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Sep 16 12:47:41.470001 ip-10-0-133-218 kubenswrapper[2580]: E0916 12:47:41.469981    2580 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-133-218.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node"
Sep 16 12:47:41.470047 ip-10-0-133-218 kubenswrapper[2580]: E0916 12:47:41.469996    2580 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service"
Sep 16 12:47:41.470435 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:47:41.470425    2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Sep 16 12:47:41.470465 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:47:41.470437    2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Sep 16 12:47:41.473249 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:47:41.473232    2580 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-133-218.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Sep 16 12:47:41.474264 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:47:41.474252    2580 server.go:1268] "Started kubelet"
Sep 16 12:47:41.474360 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:47:41.474323    2580 server.go:180] "Starting to listen" address="0.0.0.0" port=10250
Sep 16 12:47:41.474461 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:47:41.474358    2580 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Sep 16 12:47:41.474461 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:47:41.474430    2580 server_v1.go:49] "podresources" method="list" useActivePods=true
Sep 16 12:47:41.475134 ip-10-0-133-218 systemd[1]: Started Kubernetes Kubelet.
Sep 16 12:47:41.479158 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:47:41.479131    2580 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-zqjrq"
Sep 16 12:47:41.479435 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:47:41.479390    2580 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Sep 16 12:47:41.480155 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:47:41.480135    2580 server.go:310] "Adding debug handlers to kubelet server"
Sep 16 12:47:41.480508 ip-10-0-133-218 kubenswrapper[2580]: E0916 12:47:41.480475    2580 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache"
Sep 16 12:47:41.483077 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:47:41.483059    2580 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-zqjrq"
Sep 16 12:47:41.489062 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:47:41.489041    2580 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Sep 16 12:47:41.489763 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:47:41.489716    2580 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer"
Sep 16 12:47:41.490740 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:47:41.490713    2580 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Sep 16 12:47:41.490871 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:47:41.490846    2580 volume_manager.go:311] "The desired_state_of_world populator starts"
Sep 16 12:47:41.490932 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:47:41.490875    2580 volume_manager.go:313] "Starting Kubelet Volume Manager"
Sep 16 12:47:41.490989 ip-10-0-133-218 kubenswrapper[2580]: E0916 12:47:41.490971    2580 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-133-218.ec2.internal\" not found"
Sep 16 12:47:41.491037 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:47:41.490994    2580 reconstruct.go:97] "Volume reconstruction finished"
Sep 16 12:47:41.491037 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:47:41.491001    2580 reconciler.go:29] "Reconciler: start to sync state"
Sep 16 12:47:41.492881 ip-10-0-133-218 kubenswrapper[2580]: E0916 12:47:41.492855    2580 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-133-218.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms"
Sep 16 12:47:41.493000 ip-10-0-133-218 kubenswrapper[2580]: E0916 12:47:41.492887    2580 reflector.go:205] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver"
Sep 16 12:47:41.494031 ip-10-0-133-218 kubenswrapper[2580]: E0916 12:47:41.492856    2580 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-133-218.ec2.internal.18d5cdf0e5670c8f  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-133-218.ec2.internal,UID:ip-10-0-133-218.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-133-218.ec2.internal,},FirstTimestamp:2026-09-16 12:47:41.474221199 +0000 UTC m=+0.397093658,LastTimestamp:2026-09-16 12:47:41.474221199 +0000 UTC m=+0.397093658,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-133-218.ec2.internal,}"
Sep 16 12:47:41.494510 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:47:41.494481    2580 factory.go:55] Registering systemd factory
Sep 16 12:47:41.494510 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:47:41.494510    2580 factory.go:223] Registration of the systemd container factory successfully
Sep 16 12:47:41.494716 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:47:41.494705    2580 factory.go:153] Registering CRI-O factory
Sep 16 12:47:41.494716 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:47:41.494714    2580 factory.go:223] Registration of the crio container factory successfully
Sep 16 12:47:41.494818 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:47:41.494758    2580 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Sep 16 12:47:41.494818 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:47:41.494780    2580 factory.go:103] Registering Raw factory
Sep 16 12:47:41.494818 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:47:41.494792    2580 manager.go:1196] Started watching for new ooms in manager
Sep 16 12:47:41.495124 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:47:41.495112    2580 manager.go:349] Starting recovery of all containers
Sep 16 12:47:41.505625 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:47:41.505458    2580 manager.go:354] Recovery completed
Sep 16 12:47:41.509795 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:47:41.509782    2580 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 16 12:47:41.515028 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:47:41.515010    2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-218.ec2.internal" event="NodeHasSufficientMemory"
Sep 16 12:47:41.515105 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:47:41.515043    2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-218.ec2.internal" event="NodeHasNoDiskPressure"
Sep 16 12:47:41.515105 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:47:41.515056    2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-218.ec2.internal" event="NodeHasSufficientPID"
Sep 16 12:47:41.515696 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:47:41.515677    2580 cpu_manager.go:222] "Starting CPU manager" policy="none"
Sep 16 12:47:41.515696 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:47:41.515692    2580 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 16 12:47:41.515816 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:47:41.515695    2580 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s"
Sep 16 12:47:41.515816 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:47:41.515723    2580 state_mem.go:36] "Initialized new in-memory state store"
Sep 16 12:47:41.516846 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:47:41.516831    2580 policy_none.go:49] "None policy: Start"
Sep 16 12:47:41.516899 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:47:41.516855    2580 memory_manager.go:187] "Starting memorymanager" policy="None"
Sep 16 12:47:41.516899 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:47:41.516869    2580 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Sep 16 12:47:41.517295 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:47:41.517278    2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-218.ec2.internal" event="NodeHasSufficientMemory"
Sep 16 12:47:41.517380 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:47:41.517310    2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-218.ec2.internal" event="NodeHasNoDiskPressure"
Sep 16 12:47:41.517380 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:47:41.517322    2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-218.ec2.internal" event="NodeHasSufficientPID"
Sep 16 12:47:41.517912 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:47:41.517899    2580 policy_none.go:47] "Start"
Sep 16 12:47:41.560969 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:47:41.555037    2580 manager.go:337] "Starting Device Plugin manager"
Sep 16 12:47:41.560969 ip-10-0-133-218 kubenswrapper[2580]: E0916 12:47:41.555077    2580 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Sep 16 12:47:41.560969 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:47:41.555088    2580 server.go:87] "Starting device plugin registration server"
Sep 16 12:47:41.560969 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:47:41.555403    2580 eviction_manager.go:189] "Eviction manager: starting control loop"
Sep 16 12:47:41.560969 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:47:41.555413    2580 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Sep 16 12:47:41.560969 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:47:41.555519    2580 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Sep 16 12:47:41.560969 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:47:41.555650    2580 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts"
Sep 16 12:47:41.560969 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:47:41.555659    2580 plugin_manager.go:118] "Starting Kubelet Plugin Manager"
Sep 16 12:47:41.560969 ip-10-0-133-218 kubenswrapper[2580]: E0916 12:47:41.556229    2580 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Sep 16 12:47:41.560969 ip-10-0-133-218 kubenswrapper[2580]: E0916 12:47:41.556269    2580 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-133-218.ec2.internal\" not found"
Sep 16 12:47:41.592606 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:47:41.592577    2580 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Sep 16 12:47:41.593750 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:47:41.593732    2580 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Sep 16 12:47:41.593859 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:47:41.593785    2580 status_manager.go:244] "Starting to sync pod status with apiserver"
Sep 16 12:47:41.593859 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:47:41.593815    2580 kubelet.go:2448] "Starting kubelet main sync loop"
Sep 16 12:47:41.593955 ip-10-0-133-218 kubenswrapper[2580]: E0916 12:47:41.593859    2580 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful"
Sep 16 12:47:41.596325 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:47:41.596309    2580 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 16 12:47:41.656606 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:47:41.656512    2580 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 16 12:47:41.658634 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:47:41.658613    2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-218.ec2.internal" event="NodeHasSufficientMemory"
Sep 16 12:47:41.658705 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:47:41.658651    2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-218.ec2.internal" event="NodeHasNoDiskPressure"
Sep 16 12:47:41.658705 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:47:41.658662    2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-218.ec2.internal" event="NodeHasSufficientPID"
Sep 16 12:47:41.658705 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:47:41.658687    2580 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-133-218.ec2.internal"
Sep 16 12:47:41.664464 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:47:41.664449    2580 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-133-218.ec2.internal"
Sep 16 12:47:41.664561 ip-10-0-133-218 kubenswrapper[2580]: E0916 12:47:41.664473    2580 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-133-218.ec2.internal\": node \"ip-10-0-133-218.ec2.internal\" not found"
Sep 16 12:47:41.683999 ip-10-0-133-218 kubenswrapper[2580]: E0916 12:47:41.683976    2580 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-133-218.ec2.internal\" not found"
Sep 16 12:47:41.694286 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:47:41.694267    2580 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-218.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-133-218.ec2.internal"]
Sep 16 12:47:41.694359 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:47:41.694330    2580 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 16 12:47:41.695273 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:47:41.695255    2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-218.ec2.internal" event="NodeHasSufficientMemory"
Sep 16 12:47:41.695363 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:47:41.695289    2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-218.ec2.internal" event="NodeHasNoDiskPressure"
Sep 16 12:47:41.695363 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:47:41.695302    2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-218.ec2.internal" event="NodeHasSufficientPID"
Sep 16 12:47:41.696600 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:47:41.696586    2580 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 16 12:47:41.696716 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:47:41.696699    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-218.ec2.internal"
Sep 16 12:47:41.696762 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:47:41.696734    2580 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 16 12:47:41.697310 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:47:41.697296    2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-218.ec2.internal" event="NodeHasSufficientMemory"
Sep 16 12:47:41.697356 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:47:41.697317    2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-218.ec2.internal" event="NodeHasNoDiskPressure"
Sep 16 12:47:41.697356 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:47:41.697326    2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-218.ec2.internal" event="NodeHasSufficientPID"
Sep 16 12:47:41.697428 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:47:41.697375    2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-218.ec2.internal" event="NodeHasSufficientMemory"
Sep 16 12:47:41.697428 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:47:41.697400    2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-218.ec2.internal" event="NodeHasNoDiskPressure"
Sep 16 12:47:41.697428 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:47:41.697414    2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-218.ec2.internal" event="NodeHasSufficientPID"
Sep 16 12:47:41.698573 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:47:41.698551    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-133-218.ec2.internal"
Sep 16 12:47:41.698573 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:47:41.698575    2580 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 16 12:47:41.699270 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:47:41.699252    2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-218.ec2.internal" event="NodeHasSufficientMemory"
Sep 16 12:47:41.699345 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:47:41.699281    2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-218.ec2.internal" event="NodeHasNoDiskPressure"
Sep 16 12:47:41.699345 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:47:41.699290    2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-218.ec2.internal" event="NodeHasSufficientPID"
Sep 16 12:47:41.727087 ip-10-0-133-218 kubenswrapper[2580]: E0916 12:47:41.727067    2580 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-133-218.ec2.internal\" not found" node="ip-10-0-133-218.ec2.internal"
Sep 16 12:47:41.731250 ip-10-0-133-218 kubenswrapper[2580]: E0916 12:47:41.731233    2580 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-133-218.ec2.internal\" not found" node="ip-10-0-133-218.ec2.internal"
Sep 16 12:47:41.784997 ip-10-0-133-218 kubenswrapper[2580]: E0916 12:47:41.784966    2580 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-133-218.ec2.internal\" not found"
Sep 16 12:47:41.792244 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:47:41.792224    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/012c7f135b396ea57901e601e20663e2-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-133-218.ec2.internal\" (UID: \"012c7f135b396ea57901e601e20663e2\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-218.ec2.internal"
Sep 16 12:47:41.792298 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:47:41.792249    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/f3d9df87a80a4a167468bfdb12aa0522-config\") pod \"kube-apiserver-proxy-ip-10-0-133-218.ec2.internal\" (UID: \"f3d9df87a80a4a167468bfdb12aa0522\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-133-218.ec2.internal"
Sep 16 12:47:41.792298 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:47:41.792267    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/012c7f135b396ea57901e601e20663e2-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-133-218.ec2.internal\" (UID: \"012c7f135b396ea57901e601e20663e2\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-218.ec2.internal"
Sep 16 12:47:41.885559 ip-10-0-133-218 kubenswrapper[2580]: E0916 12:47:41.885523    2580 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-133-218.ec2.internal\" not found"
Sep 16 12:47:41.892878 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:47:41.892859    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/012c7f135b396ea57901e601e20663e2-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-133-218.ec2.internal\" (UID: \"012c7f135b396ea57901e601e20663e2\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-218.ec2.internal"
Sep 16 12:47:41.892945 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:47:41.892890    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/012c7f135b396ea57901e601e20663e2-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-133-218.ec2.internal\" (UID: \"012c7f135b396ea57901e601e20663e2\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-218.ec2.internal"
Sep 16 12:47:41.892945 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:47:41.892919    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/f3d9df87a80a4a167468bfdb12aa0522-config\") pod \"kube-apiserver-proxy-ip-10-0-133-218.ec2.internal\" (UID: \"f3d9df87a80a4a167468bfdb12aa0522\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-133-218.ec2.internal"
Sep 16 12:47:41.893035 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:47:41.892955    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/012c7f135b396ea57901e601e20663e2-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-133-218.ec2.internal\" (UID: \"012c7f135b396ea57901e601e20663e2\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-218.ec2.internal"
Sep 16 12:47:41.893035 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:47:41.892966    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/012c7f135b396ea57901e601e20663e2-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-133-218.ec2.internal\" (UID: \"012c7f135b396ea57901e601e20663e2\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-218.ec2.internal"
Sep 16 12:47:41.893035 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:47:41.892966    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/f3d9df87a80a4a167468bfdb12aa0522-config\") pod \"kube-apiserver-proxy-ip-10-0-133-218.ec2.internal\" (UID: \"f3d9df87a80a4a167468bfdb12aa0522\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-133-218.ec2.internal"
Sep 16 12:47:41.986349 ip-10-0-133-218 kubenswrapper[2580]: E0916 12:47:41.986281    2580 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-133-218.ec2.internal\" not found"
Sep 16 12:47:42.029822 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:47:42.029793    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-218.ec2.internal"
Sep 16 12:47:42.033587 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:47:42.033389    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-133-218.ec2.internal"
Sep 16 12:47:42.086953 ip-10-0-133-218 kubenswrapper[2580]: E0916 12:47:42.086919    2580 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-133-218.ec2.internal\" not found"
Sep 16 12:47:42.187448 ip-10-0-133-218 kubenswrapper[2580]: E0916 12:47:42.187424    2580 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-133-218.ec2.internal\" not found"
Sep 16 12:47:42.288261 ip-10-0-133-218 kubenswrapper[2580]: E0916 12:47:42.288202    2580 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-133-218.ec2.internal\" not found"
Sep 16 12:47:42.295715 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:47:42.295690    2580 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 16 12:47:42.395725 ip-10-0-133-218 kubenswrapper[2580]: E0916 12:47:42.395688    2580 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-133-218.ec2.internal\" not found"
Sep 16 12:47:42.420737 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:47:42.420703    2580 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials"
Sep 16 12:47:42.420882 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:47:42.420860    2580 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Sep 16 12:47:42.420928 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:47:42.420907    2580 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Sep 16 12:47:42.494233 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:47:42.494197    2580 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-09-15 12:42:42 +0000 UTC" deadline="2028-03-06 07:13:18.443143836 +0000 UTC"
Sep 16 12:47:42.494233 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:47:42.494223    2580 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="12882h25m35.948923693s"
Sep 16 12:47:42.495965 ip-10-0-133-218 kubenswrapper[2580]: E0916 12:47:42.495948    2580 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-133-218.ec2.internal\" not found"
Sep 16 12:47:42.579604 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:47:42.579527    2580 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 16 12:47:42.590201 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:47:42.590173    2580 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-218.ec2.internal"
Sep 16 12:47:42.602931 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:47:42.602905    2580 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Sep 16 12:47:42.603915 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:47:42.603898    2580 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-133-218.ec2.internal"
Sep 16 12:47:42.613864 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:47:42.613846    2580 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Sep 16 12:47:42.761397 ip-10-0-133-218 kubenswrapper[2580]: W0916 12:47:42.761352    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf3d9df87a80a4a167468bfdb12aa0522.slice/crio-251bd5ccc0cad99d22e462106cb2995abd39d01015acd639b3c36a2f9404c291 WatchSource:0}: Error finding container 251bd5ccc0cad99d22e462106cb2995abd39d01015acd639b3c36a2f9404c291: Status 404 returned error can't find the container with id 251bd5ccc0cad99d22e462106cb2995abd39d01015acd639b3c36a2f9404c291
Sep 16 12:47:42.763436 ip-10-0-133-218 kubenswrapper[2580]: W0916 12:47:42.763416    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod012c7f135b396ea57901e601e20663e2.slice/crio-714f2804d496e318dba0ebb2994a2f305f8bd55a852bba014a58214e3b77c71c WatchSource:0}: Error finding container 714f2804d496e318dba0ebb2994a2f305f8bd55a852bba014a58214e3b77c71c: Status 404 returned error can't find the container with id 714f2804d496e318dba0ebb2994a2f305f8bd55a852bba014a58214e3b77c71c
Sep 16 12:47:42.766920 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:47:42.766905    2580 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 16 12:47:42.976316 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:47:42.976223    2580 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 16 12:47:43.462742 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:47:43.462652    2580 apiserver.go:52] "Watching apiserver"
Sep 16 12:47:43.471083 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:47:43.471051    2580 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Sep 16 12:47:43.475022 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:47:43.474986    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-multus/network-metrics-daemon-62j7x","openshift-network-diagnostics/network-check-target-bpv5d","openshift-network-operator/iptables-alerter-rgxfz","openshift-ovn-kubernetes/ovnkube-node-894fk","kube-system/konnectivity-agent-p6wht","kube-system/kube-apiserver-proxy-ip-10-0-133-218.ec2.internal","openshift-cluster-node-tuning-operator/tuned-9gbm5","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-218.ec2.internal","openshift-multus/multus-544bh","openshift-multus/multus-additional-cni-plugins-hsj4z","kube-system/global-pull-secret-syncer-5jkh4","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tztx9","openshift-dns/node-resolver-cx649","openshift-image-registry/node-ca-d9ndr"]
Sep 16 12:47:43.476622 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:47:43.476599    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-9gbm5"
Sep 16 12:47:43.477945 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:47:43.477899    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-cx649"
Sep 16 12:47:43.479964 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:47:43.479945    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-djt86\""
Sep 16 12:47:43.480221 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:47:43.480205    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Sep 16 12:47:43.480293 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:47:43.480234    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Sep 16 12:47:43.480532 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:47:43.480511    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-rgxfz"
Sep 16 12:47:43.480661 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:47:43.480536    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Sep 16 12:47:43.481032 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:47:43.481016    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-jh7wb\""
Sep 16 12:47:43.481212 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:47:43.481187    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Sep 16 12:47:43.481941 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:47:43.481918    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-894fk"
Sep 16 12:47:43.483176 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:47:43.483156    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Sep 16 12:47:43.483176 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:47:43.483171    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-bt87z\""
Sep 16 12:47:43.483339 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:47:43.483324    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-p6wht"
Sep 16 12:47:43.483397 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:47:43.483350    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Sep 16 12:47:43.483535 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:47:43.483470    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Sep 16 12:47:43.483795 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:47:43.483758    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Sep 16 12:47:43.484065 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:47:43.484046    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Sep 16 12:47:43.484254 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:47:43.484234    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-d9ndr"
Sep 16 12:47:43.484422 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:47:43.484403    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-x4kjb\""
Sep 16 12:47:43.485695 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:47:43.485658    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-544bh"
Sep 16 12:47:43.487842 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:47:43.486631    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Sep 16 12:47:43.487842 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:47:43.486691    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Sep 16 12:47:43.487842 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:47:43.486830    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Sep 16 12:47:43.487842 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:47:43.486913    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Sep 16 12:47:43.487842 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:47:43.487038    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Sep 16 12:47:43.487842 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:47:43.487147    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-94rvv\""
Sep 16 12:47:43.487842 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:47:43.487208    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Sep 16 12:47:43.487842 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:47:43.487040    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Sep 16 12:47:43.487842 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:47:43.487293    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Sep 16 12:47:43.487842 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:47:43.487334    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-44z7w\""
Sep 16 12:47:43.487842 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:47:43.487418    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Sep 16 12:47:43.487842 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:47:43.487531    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-62j7x"
Sep 16 12:47:43.487842 ip-10-0-133-218 kubenswrapper[2580]: E0916 12:47:43.487605    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-62j7x" podUID="7d634420-2900-42b3-9609-3fb64baff8f3"
Sep 16 12:47:43.489321 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:47:43.489079    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-bpv5d"
Sep 16 12:47:43.489321 ip-10-0-133-218 kubenswrapper[2580]: E0916 12:47:43.489151    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-bpv5d" podUID="1b26fbcb-3523-4f7c-9e42-5b3653456abc"
Sep 16 12:47:43.489916 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:47:43.489481    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-vvmw6\""
Sep 16 12:47:43.489916 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:47:43.489658    2580 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving"
Sep 16 12:47:43.490893 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:47:43.490873    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Sep 16 12:47:43.491145 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:47:43.491127    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Sep 16 12:47:43.491478 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:47:43.491461    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-hsj4z"
Sep 16 12:47:43.491630 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:47:43.491613    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Sep 16 12:47:43.493333 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:47:43.493314    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5jkh4"
Sep 16 12:47:43.493426 ip-10-0-133-218 kubenswrapper[2580]: E0916 12:47:43.493384    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-5jkh4" podUID="fea676c8-6e68-4894-99fc-fe8b59886a9a"
Sep 16 12:47:43.493581 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:47:43.493556    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Sep 16 12:47:43.493651 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:47:43.493560    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Sep 16 12:47:43.493951 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:47:43.493920    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Sep 16 12:47:43.494110 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:47:43.493923    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-j5k2j\""
Sep 16 12:47:43.495238 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:47:43.495218    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tztx9"
Sep 16 12:47:43.497284 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:47:43.497140    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Sep 16 12:47:43.499771 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:47:43.497679    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Sep 16 12:47:43.499771 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:47:43.497884    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-zctbx\""
Sep 16 12:47:43.499771 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:47:43.498095    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Sep 16 12:47:43.501176 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:47:43.501154    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/ffd98c6e-f973-4bd8-94ba-bd92a16d1acd-host-kubelet\") pod \"ovnkube-node-894fk\" (UID: \"ffd98c6e-f973-4bd8-94ba-bd92a16d1acd\") " pod="openshift-ovn-kubernetes/ovnkube-node-894fk"
Sep 16 12:47:43.501281 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:47:43.501188    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/ffd98c6e-f973-4bd8-94ba-bd92a16d1acd-systemd-units\") pod \"ovnkube-node-894fk\" (UID: \"ffd98c6e-f973-4bd8-94ba-bd92a16d1acd\") " pod="openshift-ovn-kubernetes/ovnkube-node-894fk"
Sep 16 12:47:43.501281 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:47:43.501213    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/ffd98c6e-f973-4bd8-94ba-bd92a16d1acd-host-run-ovn-kubernetes\") pod \"ovnkube-node-894fk\" (UID: \"ffd98c6e-f973-4bd8-94ba-bd92a16d1acd\") " pod="openshift-ovn-kubernetes/ovnkube-node-894fk"
Sep 16 12:47:43.501281 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:47:43.501238    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/8822b539-bda1-46bd-8a2c-27660032a4e5-sys\") pod \"tuned-9gbm5\" (UID: \"8822b539-bda1-46bd-8a2c-27660032a4e5\") " pod="openshift-cluster-node-tuning-operator/tuned-9gbm5"
Sep 16 12:47:43.501281 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:47:43.501264    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-72dc8\" (UniqueName: \"kubernetes.io/projected/8822b539-bda1-46bd-8a2c-27660032a4e5-kube-api-access-72dc8\") pod \"tuned-9gbm5\" (UID: \"8822b539-bda1-46bd-8a2c-27660032a4e5\") " pod="openshift-cluster-node-tuning-operator/tuned-9gbm5"
Sep 16 12:47:43.501441 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:47:43.501298    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/ffd98c6e-f973-4bd8-94ba-bd92a16d1acd-etc-openvswitch\") pod \"ovnkube-node-894fk\" (UID: \"ffd98c6e-f973-4bd8-94ba-bd92a16d1acd\") " pod="openshift-ovn-kubernetes/ovnkube-node-894fk"
Sep 16 12:47:43.501441 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:47:43.501325    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/fb1fa9b3-dee8-433f-b3f9-29f24eab1e3a-multus-daemon-config\") pod \"multus-544bh\" (UID: \"fb1fa9b3-dee8-433f-b3f9-29f24eab1e3a\") " pod="openshift-multus/multus-544bh"
Sep 16 12:47:43.501441 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:47:43.501366    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vggds\" (UniqueName: \"kubernetes.io/projected/7d634420-2900-42b3-9609-3fb64baff8f3-kube-api-access-vggds\") pod \"network-metrics-daemon-62j7x\" (UID: \"7d634420-2900-42b3-9609-3fb64baff8f3\") " pod="openshift-multus/network-metrics-daemon-62j7x"
Sep 16 12:47:43.501441 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:47:43.501394    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/8822b539-bda1-46bd-8a2c-27660032a4e5-etc-sysctl-conf\") pod \"tuned-9gbm5\" (UID: \"8822b539-bda1-46bd-8a2c-27660032a4e5\") " pod="openshift-cluster-node-tuning-operator/tuned-9gbm5"
Sep 16 12:47:43.501441 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:47:43.501420    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0b719a93-a161-4199-9aae-5aa99cdf7165-tmp-dir\") pod \"node-resolver-cx649\" (UID: \"0b719a93-a161-4199-9aae-5aa99cdf7165\") " pod="openshift-dns/node-resolver-cx649"
Sep 16 12:47:43.501677 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:47:43.501445    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vdlc7\" (UniqueName: \"kubernetes.io/projected/0b719a93-a161-4199-9aae-5aa99cdf7165-kube-api-access-vdlc7\") pod \"node-resolver-cx649\" (UID: \"0b719a93-a161-4199-9aae-5aa99cdf7165\") " pod="openshift-dns/node-resolver-cx649"
Sep 16 12:47:43.501677 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:47:43.501470    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/ffd98c6e-f973-4bd8-94ba-bd92a16d1acd-run-openvswitch\") pod \"ovnkube-node-894fk\" (UID: \"ffd98c6e-f973-4bd8-94ba-bd92a16d1acd\") " pod="openshift-ovn-kubernetes/ovnkube-node-894fk"
Sep 16 12:47:43.501677 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:47:43.501530    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/fb1fa9b3-dee8-433f-b3f9-29f24eab1e3a-cnibin\") pod \"multus-544bh\" (UID: \"fb1fa9b3-dee8-433f-b3f9-29f24eab1e3a\") " pod="openshift-multus/multus-544bh"
Sep 16 12:47:43.501677 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:47:43.501558    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/fb1fa9b3-dee8-433f-b3f9-29f24eab1e3a-etc-kubernetes\") pod \"multus-544bh\" (UID: \"fb1fa9b3-dee8-433f-b3f9-29f24eab1e3a\") " pod="openshift-multus/multus-544bh"
Sep 16 12:47:43.501886 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:47:43.501584    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/7b900f8b-a992-44b3-a6ea-7bacb17d2798-cni-binary-copy\") pod \"multus-additional-cni-plugins-hsj4z\" (UID: \"7b900f8b-a992-44b3-a6ea-7bacb17d2798\") " pod="openshift-multus/multus-additional-cni-plugins-hsj4z"
Sep 16 12:47:43.501886 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:47:43.501836    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/8822b539-bda1-46bd-8a2c-27660032a4e5-etc-systemd\") pod \"tuned-9gbm5\" (UID: \"8822b539-bda1-46bd-8a2c-27660032a4e5\") " pod="openshift-cluster-node-tuning-operator/tuned-9gbm5"
Sep 16 12:47:43.501886 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:47:43.501875    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/8822b539-bda1-46bd-8a2c-27660032a4e5-var-lib-kubelet\") pod \"tuned-9gbm5\" (UID: \"8822b539-bda1-46bd-8a2c-27660032a4e5\") " pod="openshift-cluster-node-tuning-operator/tuned-9gbm5"
Sep 16 12:47:43.502041 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:47:43.501909    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/fb1fa9b3-dee8-433f-b3f9-29f24eab1e3a-host-var-lib-kubelet\") pod \"multus-544bh\" (UID: \"fb1fa9b3-dee8-433f-b3f9-29f24eab1e3a\") " pod="openshift-multus/multus-544bh"
Sep 16 12:47:43.502103 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:47:43.501941    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/7b900f8b-a992-44b3-a6ea-7bacb17d2798-cnibin\") pod \"multus-additional-cni-plugins-hsj4z\" (UID: \"7b900f8b-a992-44b3-a6ea-7bacb17d2798\") " pod="openshift-multus/multus-additional-cni-plugins-hsj4z"
Sep 16 12:47:43.502103 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:47:43.502090    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/32fe9109-0026-4609-af0c-923a618db835-iptables-alerter-script\") pod \"iptables-alerter-rgxfz\" (UID: \"32fe9109-0026-4609-af0c-923a618db835\") " pod="openshift-network-operator/iptables-alerter-rgxfz"
Sep 16 12:47:43.502199 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:47:43.502123    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7d634420-2900-42b3-9609-3fb64baff8f3-metrics-certs\") pod \"network-metrics-daemon-62j7x\" (UID: \"7d634420-2900-42b3-9609-3fb64baff8f3\") " pod="openshift-multus/network-metrics-daemon-62j7x"
Sep 16 12:47:43.502199 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:47:43.502154    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/8822b539-bda1-46bd-8a2c-27660032a4e5-tmp\") pod \"tuned-9gbm5\" (UID: \"8822b539-bda1-46bd-8a2c-27660032a4e5\") " pod="openshift-cluster-node-tuning-operator/tuned-9gbm5"
Sep 16 12:47:43.502348 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:47:43.502330    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nwq84\" (UniqueName: \"kubernetes.io/projected/ffd98c6e-f973-4bd8-94ba-bd92a16d1acd-kube-api-access-nwq84\") pod \"ovnkube-node-894fk\" (UID: \"ffd98c6e-f973-4bd8-94ba-bd92a16d1acd\") " pod="openshift-ovn-kubernetes/ovnkube-node-894fk"
Sep 16 12:47:43.502411 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:47:43.502368    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/ffd98c6e-f973-4bd8-94ba-bd92a16d1acd-host-slash\") pod \"ovnkube-node-894fk\" (UID: \"ffd98c6e-f973-4bd8-94ba-bd92a16d1acd\") " pod="openshift-ovn-kubernetes/ovnkube-node-894fk"
Sep 16 12:47:43.502449 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:47:43.502406    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/ffd98c6e-f973-4bd8-94ba-bd92a16d1acd-host-cni-netd\") pod \"ovnkube-node-894fk\" (UID: \"ffd98c6e-f973-4bd8-94ba-bd92a16d1acd\") " pod="openshift-ovn-kubernetes/ovnkube-node-894fk"
Sep 16 12:47:43.502626 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:47:43.502431    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/ffd98c6e-f973-4bd8-94ba-bd92a16d1acd-ovnkube-script-lib\") pod \"ovnkube-node-894fk\" (UID: \"ffd98c6e-f973-4bd8-94ba-bd92a16d1acd\") " pod="openshift-ovn-kubernetes/ovnkube-node-894fk"
Sep 16 12:47:43.502736 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:47:43.502647    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/7b900f8b-a992-44b3-a6ea-7bacb17d2798-system-cni-dir\") pod \"multus-additional-cni-plugins-hsj4z\" (UID: \"7b900f8b-a992-44b3-a6ea-7bacb17d2798\") " pod="openshift-multus/multus-additional-cni-plugins-hsj4z"
Sep 16 12:47:43.502736 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:47:43.502678    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/ffd98c6e-f973-4bd8-94ba-bd92a16d1acd-run-ovn\") pod \"ovnkube-node-894fk\" (UID: \"ffd98c6e-f973-4bd8-94ba-bd92a16d1acd\") " pod="openshift-ovn-kubernetes/ovnkube-node-894fk"
Sep 16 12:47:43.502736 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:47:43.502707    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/5d809960-17f1-4e60-9438-dc50a3b54dab-host\") pod \"node-ca-d9ndr\" (UID: \"5d809960-17f1-4e60-9438-dc50a3b54dab\") " pod="openshift-image-registry/node-ca-d9ndr"
Sep 16 12:47:43.502901 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:47:43.502810    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/5d809960-17f1-4e60-9438-dc50a3b54dab-serviceca\") pod \"node-ca-d9ndr\" (UID: \"5d809960-17f1-4e60-9438-dc50a3b54dab\") " pod="openshift-image-registry/node-ca-d9ndr"
Sep 16 12:47:43.502901 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:47:43.502838    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/fb1fa9b3-dee8-433f-b3f9-29f24eab1e3a-host-run-k8s-cni-cncf-io\") pod \"multus-544bh\" (UID: \"fb1fa9b3-dee8-433f-b3f9-29f24eab1e3a\") " pod="openshift-multus/multus-544bh"
Sep 16 12:47:43.502901 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:47:43.502878    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/7b900f8b-a992-44b3-a6ea-7bacb17d2798-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-hsj4z\" (UID: \"7b900f8b-a992-44b3-a6ea-7bacb17d2798\") " pod="openshift-multus/multus-additional-cni-plugins-hsj4z"
Sep 16 12:47:43.503070 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:47:43.502910    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/32fe9109-0026-4609-af0c-923a618db835-host-slash\") pod \"iptables-alerter-rgxfz\" (UID: \"32fe9109-0026-4609-af0c-923a618db835\") " pod="openshift-network-operator/iptables-alerter-rgxfz"
Sep 16 12:47:43.503070 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:47:43.502944    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6hvc4\" (UniqueName: \"kubernetes.io/projected/32fe9109-0026-4609-af0c-923a618db835-kube-api-access-6hvc4\") pod \"iptables-alerter-rgxfz\" (UID: \"32fe9109-0026-4609-af0c-923a618db835\") " pod="openshift-network-operator/iptables-alerter-rgxfz"
Sep 16 12:47:43.503151 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:47:43.503076    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/8822b539-bda1-46bd-8a2c-27660032a4e5-etc-sysctl-d\") pod \"tuned-9gbm5\" (UID: \"8822b539-bda1-46bd-8a2c-27660032a4e5\") " pod="openshift-cluster-node-tuning-operator/tuned-9gbm5"
Sep 16 12:47:43.503151 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:47:43.503106    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/ffd98c6e-f973-4bd8-94ba-bd92a16d1acd-node-log\") pod \"ovnkube-node-894fk\" (UID: \"ffd98c6e-f973-4bd8-94ba-bd92a16d1acd\") " pod="openshift-ovn-kubernetes/ovnkube-node-894fk"
Sep 16 12:47:43.503151 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:47:43.503136    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/ffd98c6e-f973-4bd8-94ba-bd92a16d1acd-ovnkube-config\") pod \"ovnkube-node-894fk\" (UID: \"ffd98c6e-f973-4bd8-94ba-bd92a16d1acd\") " pod="openshift-ovn-kubernetes/ovnkube-node-894fk"
Sep 16 12:47:43.503354 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:47:43.503170    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/ffd98c6e-f973-4bd8-94ba-bd92a16d1acd-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-894fk\" (UID: \"ffd98c6e-f973-4bd8-94ba-bd92a16d1acd\") " pod="openshift-ovn-kubernetes/ovnkube-node-894fk"
Sep 16 12:47:43.503396 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:47:43.503378    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/fb1fa9b3-dee8-433f-b3f9-29f24eab1e3a-multus-socket-dir-parent\") pod \"multus-544bh\" (UID: \"fb1fa9b3-dee8-433f-b3f9-29f24eab1e3a\") " pod="openshift-multus/multus-544bh"
Sep 16 12:47:43.503430 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:47:43.503407    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/fb1fa9b3-dee8-433f-b3f9-29f24eab1e3a-hostroot\") pod \"multus-544bh\" (UID: \"fb1fa9b3-dee8-433f-b3f9-29f24eab1e3a\") " pod="openshift-multus/multus-544bh"
Sep 16 12:47:43.503463 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:47:43.503433    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bnqhv\" (UniqueName: \"kubernetes.io/projected/7b900f8b-a992-44b3-a6ea-7bacb17d2798-kube-api-access-bnqhv\") pod \"multus-additional-cni-plugins-hsj4z\" (UID: \"7b900f8b-a992-44b3-a6ea-7bacb17d2798\") " pod="openshift-multus/multus-additional-cni-plugins-hsj4z"
Sep 16 12:47:43.504455 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:47:43.503464    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/0b719a93-a161-4199-9aae-5aa99cdf7165-hosts-file\") pod \"node-resolver-cx649\" (UID: \"0b719a93-a161-4199-9aae-5aa99cdf7165\") " pod="openshift-dns/node-resolver-cx649"
Sep 16 12:47:43.504455 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:47:43.503666    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/ffd98c6e-f973-4bd8-94ba-bd92a16d1acd-run-systemd\") pod \"ovnkube-node-894fk\" (UID: \"ffd98c6e-f973-4bd8-94ba-bd92a16d1acd\") " pod="openshift-ovn-kubernetes/ovnkube-node-894fk"
Sep 16 12:47:43.504455 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:47:43.503698    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/fb1fa9b3-dee8-433f-b3f9-29f24eab1e3a-system-cni-dir\") pod \"multus-544bh\" (UID: \"fb1fa9b3-dee8-433f-b3f9-29f24eab1e3a\") " pod="openshift-multus/multus-544bh"
Sep 16 12:47:43.504455 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:47:43.503727    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/fb1fa9b3-dee8-433f-b3f9-29f24eab1e3a-multus-cni-dir\") pod \"multus-544bh\" (UID: \"fb1fa9b3-dee8-433f-b3f9-29f24eab1e3a\") " pod="openshift-multus/multus-544bh"
Sep 16 12:47:43.504455 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:47:43.503763    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/fb1fa9b3-dee8-433f-b3f9-29f24eab1e3a-multus-conf-dir\") pod \"multus-544bh\" (UID: \"fb1fa9b3-dee8-433f-b3f9-29f24eab1e3a\") " pod="openshift-multus/multus-544bh"
Sep 16 12:47:43.504455 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:47:43.503796    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/8822b539-bda1-46bd-8a2c-27660032a4e5-lib-modules\") pod \"tuned-9gbm5\" (UID: \"8822b539-bda1-46bd-8a2c-27660032a4e5\") " pod="openshift-cluster-node-tuning-operator/tuned-9gbm5"
Sep 16 12:47:43.504999 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:47:43.504871    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kmndm\" (UniqueName: \"kubernetes.io/projected/5d809960-17f1-4e60-9438-dc50a3b54dab-kube-api-access-kmndm\") pod \"node-ca-d9ndr\" (UID: \"5d809960-17f1-4e60-9438-dc50a3b54dab\") " pod="openshift-image-registry/node-ca-d9ndr"
Sep 16 12:47:43.505103 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:47:43.505029    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/fb1fa9b3-dee8-433f-b3f9-29f24eab1e3a-host-run-netns\") pod \"multus-544bh\" (UID: \"fb1fa9b3-dee8-433f-b3f9-29f24eab1e3a\") " pod="openshift-multus/multus-544bh"
Sep 16 12:47:43.505103 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:47:43.505076    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/8822b539-bda1-46bd-8a2c-27660032a4e5-etc-kubernetes\") pod \"tuned-9gbm5\" (UID: \"8822b539-bda1-46bd-8a2c-27660032a4e5\") " pod="openshift-cluster-node-tuning-operator/tuned-9gbm5"
Sep 16 12:47:43.505236 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:47:43.505119    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/fb1fa9b3-dee8-433f-b3f9-29f24eab1e3a-host-var-lib-cni-multus\") pod \"multus-544bh\" (UID: \"fb1fa9b3-dee8-433f-b3f9-29f24eab1e3a\") " pod="openshift-multus/multus-544bh"
Sep 16 12:47:43.505236 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:47:43.505167    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/7b900f8b-a992-44b3-a6ea-7bacb17d2798-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-hsj4z\" (UID: \"7b900f8b-a992-44b3-a6ea-7bacb17d2798\") " pod="openshift-multus/multus-additional-cni-plugins-hsj4z"
Sep 16 12:47:43.505236 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:47:43.505220    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/decaea8f-b0ad-428a-8880-e1ef3d565796-agent-certs\") pod \"konnectivity-agent-p6wht\" (UID: \"decaea8f-b0ad-428a-8880-e1ef3d565796\") " pod="kube-system/konnectivity-agent-p6wht"
Sep 16 12:47:43.505442 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:47:43.505255    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/8822b539-bda1-46bd-8a2c-27660032a4e5-etc-modprobe-d\") pod \"tuned-9gbm5\" (UID: \"8822b539-bda1-46bd-8a2c-27660032a4e5\") " pod="openshift-cluster-node-tuning-operator/tuned-9gbm5"
Sep 16 12:47:43.506801 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:47:43.505569    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/8822b539-bda1-46bd-8a2c-27660032a4e5-run\") pod \"tuned-9gbm5\" (UID: \"8822b539-bda1-46bd-8a2c-27660032a4e5\") " pod="openshift-cluster-node-tuning-operator/tuned-9gbm5"
Sep 16 12:47:43.506801 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:47:43.505633    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/ffd98c6e-f973-4bd8-94ba-bd92a16d1acd-var-lib-openvswitch\") pod \"ovnkube-node-894fk\" (UID: \"ffd98c6e-f973-4bd8-94ba-bd92a16d1acd\") " pod="openshift-ovn-kubernetes/ovnkube-node-894fk"
Sep 16 12:47:43.506801 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:47:43.505667    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/8822b539-bda1-46bd-8a2c-27660032a4e5-etc-sysconfig\") pod \"tuned-9gbm5\" (UID: \"8822b539-bda1-46bd-8a2c-27660032a4e5\") " pod="openshift-cluster-node-tuning-operator/tuned-9gbm5"
Sep 16 12:47:43.506801 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:47:43.505706    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/8822b539-bda1-46bd-8a2c-27660032a4e5-host\") pod \"tuned-9gbm5\" (UID: \"8822b539-bda1-46bd-8a2c-27660032a4e5\") " pod="openshift-cluster-node-tuning-operator/tuned-9gbm5"
Sep 16 12:47:43.506801 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:47:43.505736    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/ffd98c6e-f973-4bd8-94ba-bd92a16d1acd-host-run-netns\") pod \"ovnkube-node-894fk\" (UID: \"ffd98c6e-f973-4bd8-94ba-bd92a16d1acd\") " pod="openshift-ovn-kubernetes/ovnkube-node-894fk"
Sep 16 12:47:43.506801 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:47:43.505764    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/ffd98c6e-f973-4bd8-94ba-bd92a16d1acd-env-overrides\") pod \"ovnkube-node-894fk\" (UID: \"ffd98c6e-f973-4bd8-94ba-bd92a16d1acd\") " pod="openshift-ovn-kubernetes/ovnkube-node-894fk"
Sep 16 12:47:43.506801 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:47:43.505813    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/7b900f8b-a992-44b3-a6ea-7bacb17d2798-tuning-conf-dir\") pod \"multus-additional-cni-plugins-hsj4z\" (UID: \"7b900f8b-a992-44b3-a6ea-7bacb17d2798\") " pod="openshift-multus/multus-additional-cni-plugins-hsj4z"
Sep 16 12:47:43.506801 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:47:43.505861    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/decaea8f-b0ad-428a-8880-e1ef3d565796-konnectivity-ca\") pod \"konnectivity-agent-p6wht\" (UID: \"decaea8f-b0ad-428a-8880-e1ef3d565796\") " pod="kube-system/konnectivity-agent-p6wht"
Sep 16 12:47:43.506801 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:47:43.505892    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v4r5z\" (UniqueName: \"kubernetes.io/projected/1b26fbcb-3523-4f7c-9e42-5b3653456abc-kube-api-access-v4r5z\") pod \"network-check-target-bpv5d\" (UID: \"1b26fbcb-3523-4f7c-9e42-5b3653456abc\") " pod="openshift-network-diagnostics/network-check-target-bpv5d"
Sep 16 12:47:43.506801 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:47:43.505941    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/ffd98c6e-f973-4bd8-94ba-bd92a16d1acd-host-cni-bin\") pod \"ovnkube-node-894fk\" (UID: \"ffd98c6e-f973-4bd8-94ba-bd92a16d1acd\") " pod="openshift-ovn-kubernetes/ovnkube-node-894fk"
Sep 16 12:47:43.506801 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:47:43.505967    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/fb1fa9b3-dee8-433f-b3f9-29f24eab1e3a-cni-binary-copy\") pod \"multus-544bh\" (UID: \"fb1fa9b3-dee8-433f-b3f9-29f24eab1e3a\") " pod="openshift-multus/multus-544bh"
Sep 16 12:47:43.506801 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:47:43.506000    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/fb1fa9b3-dee8-433f-b3f9-29f24eab1e3a-host-run-multus-certs\") pod \"multus-544bh\" (UID: \"fb1fa9b3-dee8-433f-b3f9-29f24eab1e3a\") " pod="openshift-multus/multus-544bh"
Sep 16 12:47:43.506801 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:47:43.506031    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/8822b539-bda1-46bd-8a2c-27660032a4e5-etc-tuned\") pod \"tuned-9gbm5\" (UID: \"8822b539-bda1-46bd-8a2c-27660032a4e5\") " pod="openshift-cluster-node-tuning-operator/tuned-9gbm5"
Sep 16 12:47:43.506801 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:47:43.506066    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/ffd98c6e-f973-4bd8-94ba-bd92a16d1acd-log-socket\") pod \"ovnkube-node-894fk\" (UID: \"ffd98c6e-f973-4bd8-94ba-bd92a16d1acd\") " pod="openshift-ovn-kubernetes/ovnkube-node-894fk"
Sep 16 12:47:43.506801 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:47:43.506142    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/ffd98c6e-f973-4bd8-94ba-bd92a16d1acd-ovn-node-metrics-cert\") pod \"ovnkube-node-894fk\" (UID: \"ffd98c6e-f973-4bd8-94ba-bd92a16d1acd\") " pod="openshift-ovn-kubernetes/ovnkube-node-894fk"
Sep 16 12:47:43.506801 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:47:43.506174    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/fb1fa9b3-dee8-433f-b3f9-29f24eab1e3a-host-var-lib-cni-bin\") pod \"multus-544bh\" (UID: \"fb1fa9b3-dee8-433f-b3f9-29f24eab1e3a\") " pod="openshift-multus/multus-544bh"
Sep 16 12:47:43.507723 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:47:43.506205    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ndqqd\" (UniqueName: \"kubernetes.io/projected/fb1fa9b3-dee8-433f-b3f9-29f24eab1e3a-kube-api-access-ndqqd\") pod \"multus-544bh\" (UID: \"fb1fa9b3-dee8-433f-b3f9-29f24eab1e3a\") " pod="openshift-multus/multus-544bh"
Sep 16 12:47:43.509475 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:47:43.509416    2580 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Sep 16 12:47:43.537335 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:47:43.537304    2580 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-tcwbx"
Sep 16 12:47:43.549884 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:47:43.549863    2580 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-tcwbx"
Sep 16 12:47:43.558015 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:47:43.557990    2580 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 16 12:47:43.591634 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:47:43.591602    2580 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Sep 16 12:47:43.600134 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:47:43.600081    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-218.ec2.internal" event={"ID":"012c7f135b396ea57901e601e20663e2","Type":"ContainerStarted","Data":"714f2804d496e318dba0ebb2994a2f305f8bd55a852bba014a58214e3b77c71c"}
Sep 16 12:47:43.601938 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:47:43.601910    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-133-218.ec2.internal" event={"ID":"f3d9df87a80a4a167468bfdb12aa0522","Type":"ContainerStarted","Data":"251bd5ccc0cad99d22e462106cb2995abd39d01015acd639b3c36a2f9404c291"}
Sep 16 12:47:43.607244 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:47:43.607156    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/ffd98c6e-f973-4bd8-94ba-bd92a16d1acd-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-894fk\" (UID: \"ffd98c6e-f973-4bd8-94ba-bd92a16d1acd\") " pod="openshift-ovn-kubernetes/ovnkube-node-894fk"
Sep 16 12:47:43.607244 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:47:43.607193    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/fb1fa9b3-dee8-433f-b3f9-29f24eab1e3a-multus-socket-dir-parent\") pod \"multus-544bh\" (UID: \"fb1fa9b3-dee8-433f-b3f9-29f24eab1e3a\") " pod="openshift-multus/multus-544bh"
Sep 16 12:47:43.607244 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:47:43.607217    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/fb1fa9b3-dee8-433f-b3f9-29f24eab1e3a-hostroot\") pod \"multus-544bh\" (UID: \"fb1fa9b3-dee8-433f-b3f9-29f24eab1e3a\") " pod="openshift-multus/multus-544bh"
Sep 16 12:47:43.607486 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:47:43.607265    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/ffd98c6e-f973-4bd8-94ba-bd92a16d1acd-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-894fk\" (UID: \"ffd98c6e-f973-4bd8-94ba-bd92a16d1acd\") " pod="openshift-ovn-kubernetes/ovnkube-node-894fk"
Sep 16 12:47:43.607486 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:47:43.607317    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/fb1fa9b3-dee8-433f-b3f9-29f24eab1e3a-multus-socket-dir-parent\") pod \"multus-544bh\" (UID: \"fb1fa9b3-dee8-433f-b3f9-29f24eab1e3a\") " pod="openshift-multus/multus-544bh"
Sep 16 12:47:43.607486 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:47:43.607340    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bnqhv\" (UniqueName: \"kubernetes.io/projected/7b900f8b-a992-44b3-a6ea-7bacb17d2798-kube-api-access-bnqhv\") pod \"multus-additional-cni-plugins-hsj4z\" (UID: \"7b900f8b-a992-44b3-a6ea-7bacb17d2798\") " pod="openshift-multus/multus-additional-cni-plugins-hsj4z"
Sep 16 12:47:43.607486 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:47:43.607384    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/145070ab-abd8-4e8c-8cdd-99bfbbadbe1c-socket-dir\") pod \"aws-ebs-csi-driver-node-tztx9\" (UID: \"145070ab-abd8-4e8c-8cdd-99bfbbadbe1c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tztx9"
Sep 16 12:47:43.607486 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:47:43.607396    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/fb1fa9b3-dee8-433f-b3f9-29f24eab1e3a-hostroot\") pod \"multus-544bh\" (UID: \"fb1fa9b3-dee8-433f-b3f9-29f24eab1e3a\") " pod="openshift-multus/multus-544bh"
Sep 16 12:47:43.607486 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:47:43.607417    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/0b719a93-a161-4199-9aae-5aa99cdf7165-hosts-file\") pod \"node-resolver-cx649\" (UID: \"0b719a93-a161-4199-9aae-5aa99cdf7165\") " pod="openshift-dns/node-resolver-cx649"
Sep 16 12:47:43.607486 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:47:43.607446    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/ffd98c6e-f973-4bd8-94ba-bd92a16d1acd-run-systemd\") pod \"ovnkube-node-894fk\" (UID: \"ffd98c6e-f973-4bd8-94ba-bd92a16d1acd\") " pod="openshift-ovn-kubernetes/ovnkube-node-894fk"
Sep 16 12:47:43.607486 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:47:43.607473    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/fb1fa9b3-dee8-433f-b3f9-29f24eab1e3a-system-cni-dir\") pod \"multus-544bh\" (UID: \"fb1fa9b3-dee8-433f-b3f9-29f24eab1e3a\") " pod="openshift-multus/multus-544bh"
Sep 16 12:47:43.607486 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:47:43.607482    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/0b719a93-a161-4199-9aae-5aa99cdf7165-hosts-file\") pod \"node-resolver-cx649\" (UID: \"0b719a93-a161-4199-9aae-5aa99cdf7165\") " pod="openshift-dns/node-resolver-cx649"
Sep 16 12:47:43.607960 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:47:43.607515    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/fb1fa9b3-dee8-433f-b3f9-29f24eab1e3a-multus-cni-dir\") pod \"multus-544bh\" (UID: \"fb1fa9b3-dee8-433f-b3f9-29f24eab1e3a\") " pod="openshift-multus/multus-544bh"
Sep 16 12:47:43.607960 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:47:43.607525    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/ffd98c6e-f973-4bd8-94ba-bd92a16d1acd-run-systemd\") pod \"ovnkube-node-894fk\" (UID: \"ffd98c6e-f973-4bd8-94ba-bd92a16d1acd\") " pod="openshift-ovn-kubernetes/ovnkube-node-894fk"
Sep 16 12:47:43.607960 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:47:43.607543    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/fb1fa9b3-dee8-433f-b3f9-29f24eab1e3a-multus-conf-dir\") pod \"multus-544bh\" (UID: \"fb1fa9b3-dee8-433f-b3f9-29f24eab1e3a\") " pod="openshift-multus/multus-544bh"
Sep 16 12:47:43.607960 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:47:43.607571    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/8822b539-bda1-46bd-8a2c-27660032a4e5-lib-modules\") pod \"tuned-9gbm5\" (UID: \"8822b539-bda1-46bd-8a2c-27660032a4e5\") " pod="openshift-cluster-node-tuning-operator/tuned-9gbm5"
Sep 16 12:47:43.607960 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:47:43.607611    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/145070ab-abd8-4e8c-8cdd-99bfbbadbe1c-device-dir\") pod \"aws-ebs-csi-driver-node-tztx9\" (UID: \"145070ab-abd8-4e8c-8cdd-99bfbbadbe1c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tztx9"
Sep 16 12:47:43.607960 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:47:43.607648    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/fb1fa9b3-dee8-433f-b3f9-29f24eab1e3a-system-cni-dir\") pod \"multus-544bh\" (UID: \"fb1fa9b3-dee8-433f-b3f9-29f24eab1e3a\") " pod="openshift-multus/multus-544bh"
Sep 16 12:47:43.607960 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:47:43.607649    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kmndm\" (UniqueName: \"kubernetes.io/projected/5d809960-17f1-4e60-9438-dc50a3b54dab-kube-api-access-kmndm\") pod \"node-ca-d9ndr\" (UID: \"5d809960-17f1-4e60-9438-dc50a3b54dab\") " pod="openshift-image-registry/node-ca-d9ndr"
Sep 16 12:47:43.607960 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:47:43.607680    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/fb1fa9b3-dee8-433f-b3f9-29f24eab1e3a-multus-conf-dir\") pod \"multus-544bh\" (UID: \"fb1fa9b3-dee8-433f-b3f9-29f24eab1e3a\") " pod="openshift-multus/multus-544bh"
Sep 16 12:47:43.607960 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:47:43.607692    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/fb1fa9b3-dee8-433f-b3f9-29f24eab1e3a-host-run-netns\") pod \"multus-544bh\" (UID: \"fb1fa9b3-dee8-433f-b3f9-29f24eab1e3a\") " pod="openshift-multus/multus-544bh"
Sep 16 12:47:43.607960 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:47:43.607728    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/fb1fa9b3-dee8-433f-b3f9-29f24eab1e3a-host-run-netns\") pod \"multus-544bh\" (UID: \"fb1fa9b3-dee8-433f-b3f9-29f24eab1e3a\") " pod="openshift-multus/multus-544bh"
Sep 16 12:47:43.607960 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:47:43.607730    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/8822b539-bda1-46bd-8a2c-27660032a4e5-etc-kubernetes\") pod \"tuned-9gbm5\" (UID: \"8822b539-bda1-46bd-8a2c-27660032a4e5\") " pod="openshift-cluster-node-tuning-operator/tuned-9gbm5"
Sep 16 12:47:43.607960 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:47:43.607761    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/145070ab-abd8-4e8c-8cdd-99bfbbadbe1c-etc-selinux\") pod \"aws-ebs-csi-driver-node-tztx9\" (UID: \"145070ab-abd8-4e8c-8cdd-99bfbbadbe1c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tztx9"
Sep 16 12:47:43.607960 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:47:43.607776    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/8822b539-bda1-46bd-8a2c-27660032a4e5-etc-kubernetes\") pod \"tuned-9gbm5\" (UID: \"8822b539-bda1-46bd-8a2c-27660032a4e5\") " pod="openshift-cluster-node-tuning-operator/tuned-9gbm5"
Sep 16 12:47:43.607960 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:47:43.607790    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/fb1fa9b3-dee8-433f-b3f9-29f24eab1e3a-host-var-lib-cni-multus\") pod \"multus-544bh\" (UID: \"fb1fa9b3-dee8-433f-b3f9-29f24eab1e3a\") " pod="openshift-multus/multus-544bh"
Sep 16 12:47:43.607960 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:47:43.607820    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/7b900f8b-a992-44b3-a6ea-7bacb17d2798-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-hsj4z\" (UID: \"7b900f8b-a992-44b3-a6ea-7bacb17d2798\") " pod="openshift-multus/multus-additional-cni-plugins-hsj4z"
Sep 16 12:47:43.607960 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:47:43.607845    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/decaea8f-b0ad-428a-8880-e1ef3d565796-agent-certs\") pod \"konnectivity-agent-p6wht\" (UID: \"decaea8f-b0ad-428a-8880-e1ef3d565796\") " pod="kube-system/konnectivity-agent-p6wht"
Sep 16 12:47:43.607960 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:47:43.607874    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/8822b539-bda1-46bd-8a2c-27660032a4e5-lib-modules\") pod \"tuned-9gbm5\" (UID: \"8822b539-bda1-46bd-8a2c-27660032a4e5\") " pod="openshift-cluster-node-tuning-operator/tuned-9gbm5"
Sep 16 12:47:43.607960 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:47:43.607875    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/8822b539-bda1-46bd-8a2c-27660032a4e5-etc-modprobe-d\") pod \"tuned-9gbm5\" (UID: \"8822b539-bda1-46bd-8a2c-27660032a4e5\") " pod="openshift-cluster-node-tuning-operator/tuned-9gbm5"
Sep 16 12:47:43.608792 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:47:43.607918    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/8822b539-bda1-46bd-8a2c-27660032a4e5-run\") pod \"tuned-9gbm5\" (UID: \"8822b539-bda1-46bd-8a2c-27660032a4e5\") " pod="openshift-cluster-node-tuning-operator/tuned-9gbm5"
Sep 16 12:47:43.608792 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:47:43.607940    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/8822b539-bda1-46bd-8a2c-27660032a4e5-etc-modprobe-d\") pod \"tuned-9gbm5\" (UID: \"8822b539-bda1-46bd-8a2c-27660032a4e5\") " pod="openshift-cluster-node-tuning-operator/tuned-9gbm5"
Sep 16 12:47:43.608792 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:47:43.607960    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/ffd98c6e-f973-4bd8-94ba-bd92a16d1acd-var-lib-openvswitch\") pod \"ovnkube-node-894fk\" (UID: \"ffd98c6e-f973-4bd8-94ba-bd92a16d1acd\") " pod="openshift-ovn-kubernetes/ovnkube-node-894fk"
Sep 16 12:47:43.608792 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:47:43.607986    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/fb1fa9b3-dee8-433f-b3f9-29f24eab1e3a-host-var-lib-cni-multus\") pod \"multus-544bh\" (UID: \"fb1fa9b3-dee8-433f-b3f9-29f24eab1e3a\") " pod="openshift-multus/multus-544bh"
Sep 16 12:47:43.608792 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:47:43.607987    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/8822b539-bda1-46bd-8a2c-27660032a4e5-etc-sysconfig\") pod \"tuned-9gbm5\" (UID: \"8822b539-bda1-46bd-8a2c-27660032a4e5\") " pod="openshift-cluster-node-tuning-operator/tuned-9gbm5"
Sep 16 12:47:43.608792 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:47:43.608035    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/8822b539-bda1-46bd-8a2c-27660032a4e5-host\") pod \"tuned-9gbm5\" (UID: \"8822b539-bda1-46bd-8a2c-27660032a4e5\") " pod="openshift-cluster-node-tuning-operator/tuned-9gbm5"
Sep 16 12:47:43.608792 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:47:43.608045    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/8822b539-bda1-46bd-8a2c-27660032a4e5-etc-sysconfig\") pod \"tuned-9gbm5\" (UID: \"8822b539-bda1-46bd-8a2c-27660032a4e5\") " pod="openshift-cluster-node-tuning-operator/tuned-9gbm5"
Sep 16 12:47:43.608792 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:47:43.608060    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/ffd98c6e-f973-4bd8-94ba-bd92a16d1acd-host-run-netns\") pod \"ovnkube-node-894fk\" (UID: \"ffd98c6e-f973-4bd8-94ba-bd92a16d1acd\") " pod="openshift-ovn-kubernetes/ovnkube-node-894fk"
Sep 16 12:47:43.608792 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:47:43.608086    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/ffd98c6e-f973-4bd8-94ba-bd92a16d1acd-env-overrides\") pod \"ovnkube-node-894fk\" (UID: \"ffd98c6e-f973-4bd8-94ba-bd92a16d1acd\") " pod="openshift-ovn-kubernetes/ovnkube-node-894fk"
Sep 16 12:47:43.608792 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:47:43.608105    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/8822b539-bda1-46bd-8a2c-27660032a4e5-run\") pod \"tuned-9gbm5\" (UID: \"8822b539-bda1-46bd-8a2c-27660032a4e5\") " pod="openshift-cluster-node-tuning-operator/tuned-9gbm5"
Sep 16 12:47:43.608792 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:47:43.608111    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/7b900f8b-a992-44b3-a6ea-7bacb17d2798-tuning-conf-dir\") pod \"multus-additional-cni-plugins-hsj4z\" (UID: \"7b900f8b-a992-44b3-a6ea-7bacb17d2798\") " pod="openshift-multus/multus-additional-cni-plugins-hsj4z"
Sep 16 12:47:43.608792 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:47:43.608135    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/ffd98c6e-f973-4bd8-94ba-bd92a16d1acd-var-lib-openvswitch\") pod \"ovnkube-node-894fk\" (UID: \"ffd98c6e-f973-4bd8-94ba-bd92a16d1acd\") " pod="openshift-ovn-kubernetes/ovnkube-node-894fk"
Sep 16 12:47:43.608792 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:47:43.608164    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/8822b539-bda1-46bd-8a2c-27660032a4e5-host\") pod \"tuned-9gbm5\" (UID: \"8822b539-bda1-46bd-8a2c-27660032a4e5\") " pod="openshift-cluster-node-tuning-operator/tuned-9gbm5"
Sep 16 12:47:43.608792 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:47:43.607846    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/fb1fa9b3-dee8-433f-b3f9-29f24eab1e3a-multus-cni-dir\") pod \"multus-544bh\" (UID: \"fb1fa9b3-dee8-433f-b3f9-29f24eab1e3a\") " pod="openshift-multus/multus-544bh"
Sep 16 12:47:43.608792 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:47:43.608169    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/decaea8f-b0ad-428a-8880-e1ef3d565796-konnectivity-ca\") pod \"konnectivity-agent-p6wht\" (UID: \"decaea8f-b0ad-428a-8880-e1ef3d565796\") " pod="kube-system/konnectivity-agent-p6wht"
Sep 16 12:47:43.608792 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:47:43.608196    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v4r5z\" (UniqueName: \"kubernetes.io/projected/1b26fbcb-3523-4f7c-9e42-5b3653456abc-kube-api-access-v4r5z\") pod \"network-check-target-bpv5d\" (UID: \"1b26fbcb-3523-4f7c-9e42-5b3653456abc\") " pod="openshift-network-diagnostics/network-check-target-bpv5d"
Sep 16 12:47:43.608792 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:47:43.608212    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/ffd98c6e-f973-4bd8-94ba-bd92a16d1acd-host-cni-bin\") pod \"ovnkube-node-894fk\" (UID: \"ffd98c6e-f973-4bd8-94ba-bd92a16d1acd\") " pod="openshift-ovn-kubernetes/ovnkube-node-894fk"
Sep 16 12:47:43.608792 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:47:43.608230    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/fb1fa9b3-dee8-433f-b3f9-29f24eab1e3a-cni-binary-copy\") pod \"multus-544bh\" (UID: \"fb1fa9b3-dee8-433f-b3f9-29f24eab1e3a\") " pod="openshift-multus/multus-544bh"
Sep 16 12:47:43.609759 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:47:43.608245    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/fb1fa9b3-dee8-433f-b3f9-29f24eab1e3a-host-run-multus-certs\") pod \"multus-544bh\" (UID: \"fb1fa9b3-dee8-433f-b3f9-29f24eab1e3a\") " pod="openshift-multus/multus-544bh"
Sep 16 12:47:43.609759 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:47:43.608259    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/8822b539-bda1-46bd-8a2c-27660032a4e5-etc-tuned\") pod \"tuned-9gbm5\" (UID: \"8822b539-bda1-46bd-8a2c-27660032a4e5\") " pod="openshift-cluster-node-tuning-operator/tuned-9gbm5"
Sep 16 12:47:43.609759 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:47:43.608275    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/fea676c8-6e68-4894-99fc-fe8b59886a9a-dbus\") pod \"global-pull-secret-syncer-5jkh4\" (UID: \"fea676c8-6e68-4894-99fc-fe8b59886a9a\") " pod="kube-system/global-pull-secret-syncer-5jkh4"
Sep 16 12:47:43.609759 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:47:43.608292    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/ffd98c6e-f973-4bd8-94ba-bd92a16d1acd-log-socket\") pod \"ovnkube-node-894fk\" (UID: \"ffd98c6e-f973-4bd8-94ba-bd92a16d1acd\") " pod="openshift-ovn-kubernetes/ovnkube-node-894fk"
Sep 16 12:47:43.609759 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:47:43.608309    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/ffd98c6e-f973-4bd8-94ba-bd92a16d1acd-ovn-node-metrics-cert\") pod \"ovnkube-node-894fk\" (UID: \"ffd98c6e-f973-4bd8-94ba-bd92a16d1acd\") " pod="openshift-ovn-kubernetes/ovnkube-node-894fk"
Sep 16 12:47:43.609759 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:47:43.608325    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/fb1fa9b3-dee8-433f-b3f9-29f24eab1e3a-host-var-lib-cni-bin\") pod \"multus-544bh\" (UID: \"fb1fa9b3-dee8-433f-b3f9-29f24eab1e3a\") " pod="openshift-multus/multus-544bh"
Sep 16 12:47:43.609759 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:47:43.608341    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ndqqd\" (UniqueName: \"kubernetes.io/projected/fb1fa9b3-dee8-433f-b3f9-29f24eab1e3a-kube-api-access-ndqqd\") pod \"multus-544bh\" (UID: \"fb1fa9b3-dee8-433f-b3f9-29f24eab1e3a\") " pod="openshift-multus/multus-544bh"
Sep 16 12:47:43.609759 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:47:43.608358    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/ffd98c6e-f973-4bd8-94ba-bd92a16d1acd-host-kubelet\") pod \"ovnkube-node-894fk\" (UID: \"ffd98c6e-f973-4bd8-94ba-bd92a16d1acd\") " pod="openshift-ovn-kubernetes/ovnkube-node-894fk"
Sep 16 12:47:43.609759 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:47:43.608374    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/ffd98c6e-f973-4bd8-94ba-bd92a16d1acd-systemd-units\") pod \"ovnkube-node-894fk\" (UID: \"ffd98c6e-f973-4bd8-94ba-bd92a16d1acd\") " pod="openshift-ovn-kubernetes/ovnkube-node-894fk"
Sep 16 12:47:43.609759 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:47:43.608389    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/ffd98c6e-f973-4bd8-94ba-bd92a16d1acd-host-run-ovn-kubernetes\") pod \"ovnkube-node-894fk\" (UID: \"ffd98c6e-f973-4bd8-94ba-bd92a16d1acd\") " pod="openshift-ovn-kubernetes/ovnkube-node-894fk"
Sep 16 12:47:43.609759 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:47:43.608403    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/8822b539-bda1-46bd-8a2c-27660032a4e5-sys\") pod \"tuned-9gbm5\" (UID: \"8822b539-bda1-46bd-8a2c-27660032a4e5\") " pod="openshift-cluster-node-tuning-operator/tuned-9gbm5"
Sep 16 12:47:43.609759 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:47:43.608426    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-72dc8\" (UniqueName: \"kubernetes.io/projected/8822b539-bda1-46bd-8a2c-27660032a4e5-kube-api-access-72dc8\") pod \"tuned-9gbm5\" (UID: \"8822b539-bda1-46bd-8a2c-27660032a4e5\") " pod="openshift-cluster-node-tuning-operator/tuned-9gbm5"
Sep 16 12:47:43.609759 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:47:43.608441    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/145070ab-abd8-4e8c-8cdd-99bfbbadbe1c-sys-fs\") pod \"aws-ebs-csi-driver-node-tztx9\" (UID: \"145070ab-abd8-4e8c-8cdd-99bfbbadbe1c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tztx9"
Sep 16 12:47:43.609759 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:47:43.608472    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/ffd98c6e-f973-4bd8-94ba-bd92a16d1acd-etc-openvswitch\") pod \"ovnkube-node-894fk\" (UID: \"ffd98c6e-f973-4bd8-94ba-bd92a16d1acd\") " pod="openshift-ovn-kubernetes/ovnkube-node-894fk"
Sep 16 12:47:43.609759 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:47:43.608490    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/fb1fa9b3-dee8-433f-b3f9-29f24eab1e3a-multus-daemon-config\") pod \"multus-544bh\" (UID: \"fb1fa9b3-dee8-433f-b3f9-29f24eab1e3a\") " pod="openshift-multus/multus-544bh"
Sep 16 12:47:43.609759 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:47:43.608511    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vggds\" (UniqueName: \"kubernetes.io/projected/7d634420-2900-42b3-9609-3fb64baff8f3-kube-api-access-vggds\") pod \"network-metrics-daemon-62j7x\" (UID: \"7d634420-2900-42b3-9609-3fb64baff8f3\") " pod="openshift-multus/network-metrics-daemon-62j7x"
Sep 16 12:47:43.609759 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:47:43.608528    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/8822b539-bda1-46bd-8a2c-27660032a4e5-etc-sysctl-conf\") pod \"tuned-9gbm5\" (UID: \"8822b539-bda1-46bd-8a2c-27660032a4e5\") " pod="openshift-cluster-node-tuning-operator/tuned-9gbm5"
Sep 16 12:47:43.610655 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:47:43.608545    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/145070ab-abd8-4e8c-8cdd-99bfbbadbe1c-kubelet-dir\") pod \"aws-ebs-csi-driver-node-tztx9\" (UID: \"145070ab-abd8-4e8c-8cdd-99bfbbadbe1c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tztx9"
Sep 16 12:47:43.610655 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:47:43.608562    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0b719a93-a161-4199-9aae-5aa99cdf7165-tmp-dir\") pod \"node-resolver-cx649\" (UID: \"0b719a93-a161-4199-9aae-5aa99cdf7165\") " pod="openshift-dns/node-resolver-cx649"
Sep 16 12:47:43.610655 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:47:43.608603    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vdlc7\" (UniqueName: \"kubernetes.io/projected/0b719a93-a161-4199-9aae-5aa99cdf7165-kube-api-access-vdlc7\") pod \"node-resolver-cx649\" (UID: \"0b719a93-a161-4199-9aae-5aa99cdf7165\") " pod="openshift-dns/node-resolver-cx649"
Sep 16 12:47:43.610655 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:47:43.608631    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/ffd98c6e-f973-4bd8-94ba-bd92a16d1acd-run-openvswitch\") pod \"ovnkube-node-894fk\" (UID: \"ffd98c6e-f973-4bd8-94ba-bd92a16d1acd\") " pod="openshift-ovn-kubernetes/ovnkube-node-894fk"
Sep 16 12:47:43.610655 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:47:43.608655    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/fb1fa9b3-dee8-433f-b3f9-29f24eab1e3a-cnibin\") pod \"multus-544bh\" (UID: \"fb1fa9b3-dee8-433f-b3f9-29f24eab1e3a\") " pod="openshift-multus/multus-544bh"
Sep 16 12:47:43.610655 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:47:43.608698    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/fb1fa9b3-dee8-433f-b3f9-29f24eab1e3a-etc-kubernetes\") pod \"multus-544bh\" (UID: \"fb1fa9b3-dee8-433f-b3f9-29f24eab1e3a\") " pod="openshift-multus/multus-544bh"
Sep 16 12:47:43.610655 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:47:43.608724    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/7b900f8b-a992-44b3-a6ea-7bacb17d2798-cni-binary-copy\") pod \"multus-additional-cni-plugins-hsj4z\" (UID: \"7b900f8b-a992-44b3-a6ea-7bacb17d2798\") " pod="openshift-multus/multus-additional-cni-plugins-hsj4z"
Sep 16 12:47:43.610655 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:47:43.608748    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/8822b539-bda1-46bd-8a2c-27660032a4e5-etc-systemd\") pod \"tuned-9gbm5\" (UID: \"8822b539-bda1-46bd-8a2c-27660032a4e5\") " pod="openshift-cluster-node-tuning-operator/tuned-9gbm5"
Sep 16 12:47:43.610655 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:47:43.608747    2580 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Sep 16 12:47:43.610655 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:47:43.608777    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/8822b539-bda1-46bd-8a2c-27660032a4e5-var-lib-kubelet\") pod \"tuned-9gbm5\" (UID: \"8822b539-bda1-46bd-8a2c-27660032a4e5\") " pod="openshift-cluster-node-tuning-operator/tuned-9gbm5"
Sep 16 12:47:43.610655 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:47:43.608807    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/fb1fa9b3-dee8-433f-b3f9-29f24eab1e3a-host-var-lib-kubelet\") pod \"multus-544bh\" (UID: \"fb1fa9b3-dee8-433f-b3f9-29f24eab1e3a\") " pod="openshift-multus/multus-544bh"
Sep 16 12:47:43.610655 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:47:43.608789    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/decaea8f-b0ad-428a-8880-e1ef3d565796-konnectivity-ca\") pod \"konnectivity-agent-p6wht\" (UID: \"decaea8f-b0ad-428a-8880-e1ef3d565796\") " pod="kube-system/konnectivity-agent-p6wht"
Sep 16 12:47:43.610655 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:47:43.608835    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/7b900f8b-a992-44b3-a6ea-7bacb17d2798-cnibin\") pod \"multus-additional-cni-plugins-hsj4z\" (UID: \"7b900f8b-a992-44b3-a6ea-7bacb17d2798\") " pod="openshift-multus/multus-additional-cni-plugins-hsj4z"
Sep 16 12:47:43.610655 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:47:43.608865    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/32fe9109-0026-4609-af0c-923a618db835-iptables-alerter-script\") pod \"iptables-alerter-rgxfz\" (UID: \"32fe9109-0026-4609-af0c-923a618db835\") " pod="openshift-network-operator/iptables-alerter-rgxfz"
Sep 16 12:47:43.610655 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:47:43.608896    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7d634420-2900-42b3-9609-3fb64baff8f3-metrics-certs\") pod \"network-metrics-daemon-62j7x\" (UID: \"7d634420-2900-42b3-9609-3fb64baff8f3\") " pod="openshift-multus/network-metrics-daemon-62j7x"
Sep 16 12:47:43.610655 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:47:43.608900    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/ffd98c6e-f973-4bd8-94ba-bd92a16d1acd-host-run-netns\") pod \"ovnkube-node-894fk\" (UID: \"ffd98c6e-f973-4bd8-94ba-bd92a16d1acd\") " pod="openshift-ovn-kubernetes/ovnkube-node-894fk"
Sep 16 12:47:43.610655 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:47:43.608922    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/8822b539-bda1-46bd-8a2c-27660032a4e5-tmp\") pod \"tuned-9gbm5\" (UID: \"8822b539-bda1-46bd-8a2c-27660032a4e5\") " pod="openshift-cluster-node-tuning-operator/tuned-9gbm5"
Sep 16 12:47:43.610655 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:47:43.608948    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/fea676c8-6e68-4894-99fc-fe8b59886a9a-kubelet-config\") pod \"global-pull-secret-syncer-5jkh4\" (UID: \"fea676c8-6e68-4894-99fc-fe8b59886a9a\") " pod="kube-system/global-pull-secret-syncer-5jkh4"
Sep 16 12:47:43.611529 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:47:43.608980    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/145070ab-abd8-4e8c-8cdd-99bfbbadbe1c-registration-dir\") pod \"aws-ebs-csi-driver-node-tztx9\" (UID: \"145070ab-abd8-4e8c-8cdd-99bfbbadbe1c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tztx9"
Sep 16 12:47:43.611529 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:47:43.609021    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/ffd98c6e-f973-4bd8-94ba-bd92a16d1acd-etc-openvswitch\") pod \"ovnkube-node-894fk\" (UID: \"ffd98c6e-f973-4bd8-94ba-bd92a16d1acd\") " pod="openshift-ovn-kubernetes/ovnkube-node-894fk"
Sep 16 12:47:43.611529 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:47:43.609010    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nwq84\" (UniqueName: \"kubernetes.io/projected/ffd98c6e-f973-4bd8-94ba-bd92a16d1acd-kube-api-access-nwq84\") pod \"ovnkube-node-894fk\" (UID: \"ffd98c6e-f973-4bd8-94ba-bd92a16d1acd\") " pod="openshift-ovn-kubernetes/ovnkube-node-894fk"
Sep 16 12:47:43.611529 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:47:43.609084    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/ffd98c6e-f973-4bd8-94ba-bd92a16d1acd-host-slash\") pod \"ovnkube-node-894fk\" (UID: \"ffd98c6e-f973-4bd8-94ba-bd92a16d1acd\") " pod="openshift-ovn-kubernetes/ovnkube-node-894fk"
Sep 16 12:47:43.611529 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:47:43.609113    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/ffd98c6e-f973-4bd8-94ba-bd92a16d1acd-host-cni-netd\") pod \"ovnkube-node-894fk\" (UID: \"ffd98c6e-f973-4bd8-94ba-bd92a16d1acd\") " pod="openshift-ovn-kubernetes/ovnkube-node-894fk"
Sep 16 12:47:43.611529 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:47:43.609141    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/ffd98c6e-f973-4bd8-94ba-bd92a16d1acd-ovnkube-script-lib\") pod \"ovnkube-node-894fk\" (UID: \"ffd98c6e-f973-4bd8-94ba-bd92a16d1acd\") " pod="openshift-ovn-kubernetes/ovnkube-node-894fk"
Sep 16 12:47:43.611529 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:47:43.609169    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/7b900f8b-a992-44b3-a6ea-7bacb17d2798-system-cni-dir\") pod \"multus-additional-cni-plugins-hsj4z\" (UID: \"7b900f8b-a992-44b3-a6ea-7bacb17d2798\") " pod="openshift-multus/multus-additional-cni-plugins-hsj4z"
Sep 16 12:47:43.611529 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:47:43.609200    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/fea676c8-6e68-4894-99fc-fe8b59886a9a-original-pull-secret\") pod \"global-pull-secret-syncer-5jkh4\" (UID: \"fea676c8-6e68-4894-99fc-fe8b59886a9a\") " pod="kube-system/global-pull-secret-syncer-5jkh4"
Sep 16 12:47:43.611529 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:47:43.609231    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wnq5t\" (UniqueName: \"kubernetes.io/projected/145070ab-abd8-4e8c-8cdd-99bfbbadbe1c-kube-api-access-wnq5t\") pod \"aws-ebs-csi-driver-node-tztx9\" (UID: \"145070ab-abd8-4e8c-8cdd-99bfbbadbe1c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tztx9"
Sep 16 12:47:43.611529 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:47:43.609238    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/7b900f8b-a992-44b3-a6ea-7bacb17d2798-tuning-conf-dir\") pod \"multus-additional-cni-plugins-hsj4z\" (UID: \"7b900f8b-a992-44b3-a6ea-7bacb17d2798\") " pod="openshift-multus/multus-additional-cni-plugins-hsj4z"
Sep 16 12:47:43.611529 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:47:43.609260    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/ffd98c6e-f973-4bd8-94ba-bd92a16d1acd-run-ovn\") pod \"ovnkube-node-894fk\" (UID: \"ffd98c6e-f973-4bd8-94ba-bd92a16d1acd\") " pod="openshift-ovn-kubernetes/ovnkube-node-894fk"
Sep 16 12:47:43.611529 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:47:43.609310    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/ffd98c6e-f973-4bd8-94ba-bd92a16d1acd-run-ovn\") pod \"ovnkube-node-894fk\" (UID: \"ffd98c6e-f973-4bd8-94ba-bd92a16d1acd\") " pod="openshift-ovn-kubernetes/ovnkube-node-894fk"
Sep 16 12:47:43.611529 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:47:43.609311    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/5d809960-17f1-4e60-9438-dc50a3b54dab-host\") pod \"node-ca-d9ndr\" (UID: \"5d809960-17f1-4e60-9438-dc50a3b54dab\") " pod="openshift-image-registry/node-ca-d9ndr"
Sep 16 12:47:43.611529 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:47:43.609339    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/8822b539-bda1-46bd-8a2c-27660032a4e5-etc-systemd\") pod \"tuned-9gbm5\" (UID: \"8822b539-bda1-46bd-8a2c-27660032a4e5\") " pod="openshift-cluster-node-tuning-operator/tuned-9gbm5"
Sep 16 12:47:43.611529 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:47:43.609347    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/5d809960-17f1-4e60-9438-dc50a3b54dab-serviceca\") pod \"node-ca-d9ndr\" (UID: \"5d809960-17f1-4e60-9438-dc50a3b54dab\") " pod="openshift-image-registry/node-ca-d9ndr"
Sep 16 12:47:43.611529 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:47:43.609390    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/fb1fa9b3-dee8-433f-b3f9-29f24eab1e3a-host-run-k8s-cni-cncf-io\") pod \"multus-544bh\" (UID: \"fb1fa9b3-dee8-433f-b3f9-29f24eab1e3a\") " pod="openshift-multus/multus-544bh"
Sep 16 12:47:43.611529 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:47:43.609405    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/5d809960-17f1-4e60-9438-dc50a3b54dab-host\") pod \"node-ca-d9ndr\" (UID: \"5d809960-17f1-4e60-9438-dc50a3b54dab\") " pod="openshift-image-registry/node-ca-d9ndr"
Sep 16 12:47:43.612080 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:47:43.609414    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/ffd98c6e-f973-4bd8-94ba-bd92a16d1acd-env-overrides\") pod \"ovnkube-node-894fk\" (UID: \"ffd98c6e-f973-4bd8-94ba-bd92a16d1acd\") " pod="openshift-ovn-kubernetes/ovnkube-node-894fk"
Sep 16 12:47:43.612080 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:47:43.609423    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/7b900f8b-a992-44b3-a6ea-7bacb17d2798-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-hsj4z\" (UID: \"7b900f8b-a992-44b3-a6ea-7bacb17d2798\") " pod="openshift-multus/multus-additional-cni-plugins-hsj4z"
Sep 16 12:47:43.612080 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:47:43.609453    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/32fe9109-0026-4609-af0c-923a618db835-host-slash\") pod \"iptables-alerter-rgxfz\" (UID: \"32fe9109-0026-4609-af0c-923a618db835\") " pod="openshift-network-operator/iptables-alerter-rgxfz"
Sep 16 12:47:43.612080 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:47:43.609479    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/ffd98c6e-f973-4bd8-94ba-bd92a16d1acd-host-cni-bin\") pod \"ovnkube-node-894fk\" (UID: \"ffd98c6e-f973-4bd8-94ba-bd92a16d1acd\") " pod="openshift-ovn-kubernetes/ovnkube-node-894fk"
Sep 16 12:47:43.612080 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:47:43.609481    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6hvc4\" (UniqueName: \"kubernetes.io/projected/32fe9109-0026-4609-af0c-923a618db835-kube-api-access-6hvc4\") pod \"iptables-alerter-rgxfz\" (UID: \"32fe9109-0026-4609-af0c-923a618db835\") " pod="openshift-network-operator/iptables-alerter-rgxfz"
Sep 16 12:47:43.612080 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:47:43.609522    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/8822b539-bda1-46bd-8a2c-27660032a4e5-etc-sysctl-d\") pod \"tuned-9gbm5\" (UID: \"8822b539-bda1-46bd-8a2c-27660032a4e5\") " pod="openshift-cluster-node-tuning-operator/tuned-9gbm5"
Sep 16 12:47:43.612080 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:47:43.609572    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/ffd98c6e-f973-4bd8-94ba-bd92a16d1acd-node-log\") pod \"ovnkube-node-894fk\" (UID: \"ffd98c6e-f973-4bd8-94ba-bd92a16d1acd\") " pod="openshift-ovn-kubernetes/ovnkube-node-894fk"
Sep 16 12:47:43.612080 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:47:43.609602    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/ffd98c6e-f973-4bd8-94ba-bd92a16d1acd-ovnkube-config\") pod \"ovnkube-node-894fk\" (UID: \"ffd98c6e-f973-4bd8-94ba-bd92a16d1acd\") " pod="openshift-ovn-kubernetes/ovnkube-node-894fk"
Sep 16 12:47:43.612080 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:47:43.609688    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/fb1fa9b3-dee8-433f-b3f9-29f24eab1e3a-multus-daemon-config\") pod \"multus-544bh\" (UID: \"fb1fa9b3-dee8-433f-b3f9-29f24eab1e3a\") " pod="openshift-multus/multus-544bh"
Sep 16 12:47:43.612080 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:47:43.609766    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/fb1fa9b3-dee8-433f-b3f9-29f24eab1e3a-etc-kubernetes\") pod \"multus-544bh\" (UID: \"fb1fa9b3-dee8-433f-b3f9-29f24eab1e3a\") " pod="openshift-multus/multus-544bh"
Sep 16 12:47:43.612080 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:47:43.609907    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/7b900f8b-a992-44b3-a6ea-7bacb17d2798-cni-binary-copy\") pod \"multus-additional-cni-plugins-hsj4z\" (UID: \"7b900f8b-a992-44b3-a6ea-7bacb17d2798\") " pod="openshift-multus/multus-additional-cni-plugins-hsj4z"
Sep 16 12:47:43.612080 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:47:43.610023    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/8822b539-bda1-46bd-8a2c-27660032a4e5-etc-sysctl-conf\") pod \"tuned-9gbm5\" (UID: \"8822b539-bda1-46bd-8a2c-27660032a4e5\") " pod="openshift-cluster-node-tuning-operator/tuned-9gbm5"
Sep 16 12:47:43.612080 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:47:43.610036    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/ffd98c6e-f973-4bd8-94ba-bd92a16d1acd-systemd-units\") pod \"ovnkube-node-894fk\" (UID: \"ffd98c6e-f973-4bd8-94ba-bd92a16d1acd\") " pod="openshift-ovn-kubernetes/ovnkube-node-894fk"
Sep 16 12:47:43.612080 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:47:43.610113    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/8822b539-bda1-46bd-8a2c-27660032a4e5-sys\") pod \"tuned-9gbm5\" (UID: \"8822b539-bda1-46bd-8a2c-27660032a4e5\") " pod="openshift-cluster-node-tuning-operator/tuned-9gbm5"
Sep 16 12:47:43.612080 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:47:43.610362    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/ffd98c6e-f973-4bd8-94ba-bd92a16d1acd-host-slash\") pod \"ovnkube-node-894fk\" (UID: \"ffd98c6e-f973-4bd8-94ba-bd92a16d1acd\") " pod="openshift-ovn-kubernetes/ovnkube-node-894fk"
Sep 16 12:47:43.612080 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:47:43.610409    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/ffd98c6e-f973-4bd8-94ba-bd92a16d1acd-host-cni-netd\") pod \"ovnkube-node-894fk\" (UID: \"ffd98c6e-f973-4bd8-94ba-bd92a16d1acd\") " pod="openshift-ovn-kubernetes/ovnkube-node-894fk"
Sep 16 12:47:43.612080 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:47:43.610416    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0b719a93-a161-4199-9aae-5aa99cdf7165-tmp-dir\") pod \"node-resolver-cx649\" (UID: \"0b719a93-a161-4199-9aae-5aa99cdf7165\") " pod="openshift-dns/node-resolver-cx649"
Sep 16 12:47:43.612080 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:47:43.610464    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/7b900f8b-a992-44b3-a6ea-7bacb17d2798-system-cni-dir\") pod \"multus-additional-cni-plugins-hsj4z\" (UID: \"7b900f8b-a992-44b3-a6ea-7bacb17d2798\") " pod="openshift-multus/multus-additional-cni-plugins-hsj4z"
Sep 16 12:47:43.614490 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:47:43.610468    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/ffd98c6e-f973-4bd8-94ba-bd92a16d1acd-ovnkube-config\") pod \"ovnkube-node-894fk\" (UID: \"ffd98c6e-f973-4bd8-94ba-bd92a16d1acd\") " pod="openshift-ovn-kubernetes/ovnkube-node-894fk"
Sep 16 12:47:43.614490 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:47:43.610556    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/ffd98c6e-f973-4bd8-94ba-bd92a16d1acd-host-run-ovn-kubernetes\") pod \"ovnkube-node-894fk\" (UID: \"ffd98c6e-f973-4bd8-94ba-bd92a16d1acd\") " pod="openshift-ovn-kubernetes/ovnkube-node-894fk"
Sep 16 12:47:43.614490 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:47:43.610607    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/5d809960-17f1-4e60-9438-dc50a3b54dab-serviceca\") pod \"node-ca-d9ndr\" (UID: \"5d809960-17f1-4e60-9438-dc50a3b54dab\") " pod="openshift-image-registry/node-ca-d9ndr"
Sep 16 12:47:43.614490 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:47:43.610733    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/ffd98c6e-f973-4bd8-94ba-bd92a16d1acd-log-socket\") pod \"ovnkube-node-894fk\" (UID: \"ffd98c6e-f973-4bd8-94ba-bd92a16d1acd\") " pod="openshift-ovn-kubernetes/ovnkube-node-894fk"
Sep 16 12:47:43.614490 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:47:43.610784    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/ffd98c6e-f973-4bd8-94ba-bd92a16d1acd-host-kubelet\") pod \"ovnkube-node-894fk\" (UID: \"ffd98c6e-f973-4bd8-94ba-bd92a16d1acd\") " pod="openshift-ovn-kubernetes/ovnkube-node-894fk"
Sep 16 12:47:43.614490 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:47:43.610809    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/fb1fa9b3-dee8-433f-b3f9-29f24eab1e3a-cnibin\") pod \"multus-544bh\" (UID: \"fb1fa9b3-dee8-433f-b3f9-29f24eab1e3a\") " pod="openshift-multus/multus-544bh"
Sep 16 12:47:43.614490 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:47:43.610841    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/8822b539-bda1-46bd-8a2c-27660032a4e5-var-lib-kubelet\") pod \"tuned-9gbm5\" (UID: \"8822b539-bda1-46bd-8a2c-27660032a4e5\") " pod="openshift-cluster-node-tuning-operator/tuned-9gbm5"
Sep 16 12:47:43.614490 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:47:43.610854    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/fb1fa9b3-dee8-433f-b3f9-29f24eab1e3a-host-var-lib-kubelet\") pod \"multus-544bh\" (UID: \"fb1fa9b3-dee8-433f-b3f9-29f24eab1e3a\") " pod="openshift-multus/multus-544bh"
Sep 16 12:47:43.614490 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:47:43.610882    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/ffd98c6e-f973-4bd8-94ba-bd92a16d1acd-run-openvswitch\") pod \"ovnkube-node-894fk\" (UID: \"ffd98c6e-f973-4bd8-94ba-bd92a16d1acd\") " pod="openshift-ovn-kubernetes/ovnkube-node-894fk"
Sep 16 12:47:43.614490 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:47:43.610987    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/ffd98c6e-f973-4bd8-94ba-bd92a16d1acd-ovnkube-script-lib\") pod \"ovnkube-node-894fk\" (UID: \"ffd98c6e-f973-4bd8-94ba-bd92a16d1acd\") " pod="openshift-ovn-kubernetes/ovnkube-node-894fk"
Sep 16 12:47:43.614490 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:47:43.611302    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/7b900f8b-a992-44b3-a6ea-7bacb17d2798-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-hsj4z\" (UID: \"7b900f8b-a992-44b3-a6ea-7bacb17d2798\") " pod="openshift-multus/multus-additional-cni-plugins-hsj4z"
Sep 16 12:47:43.614490 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:47:43.611345    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/fb1fa9b3-dee8-433f-b3f9-29f24eab1e3a-cni-binary-copy\") pod \"multus-544bh\" (UID: \"fb1fa9b3-dee8-433f-b3f9-29f24eab1e3a\") " pod="openshift-multus/multus-544bh"
Sep 16 12:47:43.614490 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:47:43.611352    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/32fe9109-0026-4609-af0c-923a618db835-host-slash\") pod \"iptables-alerter-rgxfz\" (UID: \"32fe9109-0026-4609-af0c-923a618db835\") " pod="openshift-network-operator/iptables-alerter-rgxfz"
Sep 16 12:47:43.614490 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:47:43.611399    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/fb1fa9b3-dee8-433f-b3f9-29f24eab1e3a-host-var-lib-cni-bin\") pod \"multus-544bh\" (UID: \"fb1fa9b3-dee8-433f-b3f9-29f24eab1e3a\") " pod="openshift-multus/multus-544bh"
Sep 16 12:47:43.614490 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:47:43.611401    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/fb1fa9b3-dee8-433f-b3f9-29f24eab1e3a-host-run-multus-certs\") pod \"multus-544bh\" (UID: \"fb1fa9b3-dee8-433f-b3f9-29f24eab1e3a\") " pod="openshift-multus/multus-544bh"
Sep 16 12:47:43.614490 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:47:43.611434    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/7b900f8b-a992-44b3-a6ea-7bacb17d2798-cnibin\") pod \"multus-additional-cni-plugins-hsj4z\" (UID: \"7b900f8b-a992-44b3-a6ea-7bacb17d2798\") " pod="openshift-multus/multus-additional-cni-plugins-hsj4z"
Sep 16 12:47:43.614490 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:47:43.611449    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/7b900f8b-a992-44b3-a6ea-7bacb17d2798-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-hsj4z\" (UID: \"7b900f8b-a992-44b3-a6ea-7bacb17d2798\") " pod="openshift-multus/multus-additional-cni-plugins-hsj4z"
Sep 16 12:47:43.614490 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:47:43.611495    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/fb1fa9b3-dee8-433f-b3f9-29f24eab1e3a-host-run-k8s-cni-cncf-io\") pod \"multus-544bh\" (UID: \"fb1fa9b3-dee8-433f-b3f9-29f24eab1e3a\") " pod="openshift-multus/multus-544bh"
Sep 16 12:47:43.615424 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:47:43.611497    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/8822b539-bda1-46bd-8a2c-27660032a4e5-etc-sysctl-d\") pod \"tuned-9gbm5\" (UID: \"8822b539-bda1-46bd-8a2c-27660032a4e5\") " pod="openshift-cluster-node-tuning-operator/tuned-9gbm5"
Sep 16 12:47:43.615424 ip-10-0-133-218 kubenswrapper[2580]: E0916 12:47:43.611598    2580 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 16 12:47:43.615424 ip-10-0-133-218 kubenswrapper[2580]: E0916 12:47:43.611669    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7d634420-2900-42b3-9609-3fb64baff8f3-metrics-certs podName:7d634420-2900-42b3-9609-3fb64baff8f3 nodeName:}" failed. No retries permitted until 2026-09-16 12:47:44.111645729 +0000 UTC m=+3.034518717 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/7d634420-2900-42b3-9609-3fb64baff8f3-metrics-certs") pod "network-metrics-daemon-62j7x" (UID: "7d634420-2900-42b3-9609-3fb64baff8f3") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 16 12:47:43.615424 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:47:43.611874    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/ffd98c6e-f973-4bd8-94ba-bd92a16d1acd-node-log\") pod \"ovnkube-node-894fk\" (UID: \"ffd98c6e-f973-4bd8-94ba-bd92a16d1acd\") " pod="openshift-ovn-kubernetes/ovnkube-node-894fk"
Sep 16 12:47:43.615424 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:47:43.611986    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/32fe9109-0026-4609-af0c-923a618db835-iptables-alerter-script\") pod \"iptables-alerter-rgxfz\" (UID: \"32fe9109-0026-4609-af0c-923a618db835\") " pod="openshift-network-operator/iptables-alerter-rgxfz"
Sep 16 12:47:43.615424 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:47:43.614978    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/8822b539-bda1-46bd-8a2c-27660032a4e5-etc-tuned\") pod \"tuned-9gbm5\" (UID: \"8822b539-bda1-46bd-8a2c-27660032a4e5\") " pod="openshift-cluster-node-tuning-operator/tuned-9gbm5"
Sep 16 12:47:43.615424 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:47:43.615157    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/ffd98c6e-f973-4bd8-94ba-bd92a16d1acd-ovn-node-metrics-cert\") pod \"ovnkube-node-894fk\" (UID: \"ffd98c6e-f973-4bd8-94ba-bd92a16d1acd\") " pod="openshift-ovn-kubernetes/ovnkube-node-894fk"
Sep 16 12:47:43.615805 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:47:43.615616    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/8822b539-bda1-46bd-8a2c-27660032a4e5-tmp\") pod \"tuned-9gbm5\" (UID: \"8822b539-bda1-46bd-8a2c-27660032a4e5\") " pod="openshift-cluster-node-tuning-operator/tuned-9gbm5"
Sep 16 12:47:43.616423 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:47:43.615905    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/decaea8f-b0ad-428a-8880-e1ef3d565796-agent-certs\") pod \"konnectivity-agent-p6wht\" (UID: \"decaea8f-b0ad-428a-8880-e1ef3d565796\") " pod="kube-system/konnectivity-agent-p6wht"
Sep 16 12:47:43.616423 ip-10-0-133-218 kubenswrapper[2580]: E0916 12:47:43.616275    2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 16 12:47:43.616423 ip-10-0-133-218 kubenswrapper[2580]: E0916 12:47:43.616296    2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 16 12:47:43.616423 ip-10-0-133-218 kubenswrapper[2580]: E0916 12:47:43.616310    2580 projected.go:196] Error preparing data for projected volume kube-api-access-v4r5z for pod openshift-network-diagnostics/network-check-target-bpv5d: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 16 12:47:43.616423 ip-10-0-133-218 kubenswrapper[2580]: E0916 12:47:43.616373    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/1b26fbcb-3523-4f7c-9e42-5b3653456abc-kube-api-access-v4r5z podName:1b26fbcb-3523-4f7c-9e42-5b3653456abc nodeName:}" failed. No retries permitted until 2026-09-16 12:47:44.116352557 +0000 UTC m=+3.039225005 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-v4r5z" (UniqueName: "kubernetes.io/projected/1b26fbcb-3523-4f7c-9e42-5b3653456abc-kube-api-access-v4r5z") pod "network-check-target-bpv5d" (UID: "1b26fbcb-3523-4f7c-9e42-5b3653456abc") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 16 12:47:43.617987 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:47:43.617925    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kmndm\" (UniqueName: \"kubernetes.io/projected/5d809960-17f1-4e60-9438-dc50a3b54dab-kube-api-access-kmndm\") pod \"node-ca-d9ndr\" (UID: \"5d809960-17f1-4e60-9438-dc50a3b54dab\") " pod="openshift-image-registry/node-ca-d9ndr"
Sep 16 12:47:43.618771 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:47:43.618679    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nwq84\" (UniqueName: \"kubernetes.io/projected/ffd98c6e-f973-4bd8-94ba-bd92a16d1acd-kube-api-access-nwq84\") pod \"ovnkube-node-894fk\" (UID: \"ffd98c6e-f973-4bd8-94ba-bd92a16d1acd\") " pod="openshift-ovn-kubernetes/ovnkube-node-894fk"
Sep 16 12:47:43.620014 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:47:43.619986    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vggds\" (UniqueName: \"kubernetes.io/projected/7d634420-2900-42b3-9609-3fb64baff8f3-kube-api-access-vggds\") pod \"network-metrics-daemon-62j7x\" (UID: \"7d634420-2900-42b3-9609-3fb64baff8f3\") " pod="openshift-multus/network-metrics-daemon-62j7x"
Sep 16 12:47:43.622308 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:47:43.622245    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-72dc8\" (UniqueName: \"kubernetes.io/projected/8822b539-bda1-46bd-8a2c-27660032a4e5-kube-api-access-72dc8\") pod \"tuned-9gbm5\" (UID: \"8822b539-bda1-46bd-8a2c-27660032a4e5\") " pod="openshift-cluster-node-tuning-operator/tuned-9gbm5"
Sep 16 12:47:43.623386 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:47:43.623269    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vdlc7\" (UniqueName: \"kubernetes.io/projected/0b719a93-a161-4199-9aae-5aa99cdf7165-kube-api-access-vdlc7\") pod \"node-resolver-cx649\" (UID: \"0b719a93-a161-4199-9aae-5aa99cdf7165\") " pod="openshift-dns/node-resolver-cx649"
Sep 16 12:47:43.623386 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:47:43.623310    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bnqhv\" (UniqueName: \"kubernetes.io/projected/7b900f8b-a992-44b3-a6ea-7bacb17d2798-kube-api-access-bnqhv\") pod \"multus-additional-cni-plugins-hsj4z\" (UID: \"7b900f8b-a992-44b3-a6ea-7bacb17d2798\") " pod="openshift-multus/multus-additional-cni-plugins-hsj4z"
Sep 16 12:47:43.623711 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:47:43.623664    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6hvc4\" (UniqueName: \"kubernetes.io/projected/32fe9109-0026-4609-af0c-923a618db835-kube-api-access-6hvc4\") pod \"iptables-alerter-rgxfz\" (UID: \"32fe9109-0026-4609-af0c-923a618db835\") " pod="openshift-network-operator/iptables-alerter-rgxfz"
Sep 16 12:47:43.624307 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:47:43.624231    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ndqqd\" (UniqueName: \"kubernetes.io/projected/fb1fa9b3-dee8-433f-b3f9-29f24eab1e3a-kube-api-access-ndqqd\") pod \"multus-544bh\" (UID: \"fb1fa9b3-dee8-433f-b3f9-29f24eab1e3a\") " pod="openshift-multus/multus-544bh"
Sep 16 12:47:43.710598 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:47:43.710551    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/fea676c8-6e68-4894-99fc-fe8b59886a9a-kubelet-config\") pod \"global-pull-secret-syncer-5jkh4\" (UID: \"fea676c8-6e68-4894-99fc-fe8b59886a9a\") " pod="kube-system/global-pull-secret-syncer-5jkh4"
Sep 16 12:47:43.710598 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:47:43.710596    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/145070ab-abd8-4e8c-8cdd-99bfbbadbe1c-registration-dir\") pod \"aws-ebs-csi-driver-node-tztx9\" (UID: \"145070ab-abd8-4e8c-8cdd-99bfbbadbe1c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tztx9"
Sep 16 12:47:43.710841 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:47:43.710627    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/fea676c8-6e68-4894-99fc-fe8b59886a9a-original-pull-secret\") pod \"global-pull-secret-syncer-5jkh4\" (UID: \"fea676c8-6e68-4894-99fc-fe8b59886a9a\") " pod="kube-system/global-pull-secret-syncer-5jkh4"
Sep 16 12:47:43.710841 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:47:43.710655    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wnq5t\" (UniqueName: \"kubernetes.io/projected/145070ab-abd8-4e8c-8cdd-99bfbbadbe1c-kube-api-access-wnq5t\") pod \"aws-ebs-csi-driver-node-tztx9\" (UID: \"145070ab-abd8-4e8c-8cdd-99bfbbadbe1c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tztx9"
Sep 16 12:47:43.710841 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:47:43.710690    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/145070ab-abd8-4e8c-8cdd-99bfbbadbe1c-socket-dir\") pod \"aws-ebs-csi-driver-node-tztx9\" (UID: \"145070ab-abd8-4e8c-8cdd-99bfbbadbe1c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tztx9"
Sep 16 12:47:43.710841 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:47:43.710721    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/145070ab-abd8-4e8c-8cdd-99bfbbadbe1c-device-dir\") pod \"aws-ebs-csi-driver-node-tztx9\" (UID: \"145070ab-abd8-4e8c-8cdd-99bfbbadbe1c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tztx9"
Sep 16 12:47:43.710841 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:47:43.710750    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/145070ab-abd8-4e8c-8cdd-99bfbbadbe1c-etc-selinux\") pod \"aws-ebs-csi-driver-node-tztx9\" (UID: \"145070ab-abd8-4e8c-8cdd-99bfbbadbe1c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tztx9"
Sep 16 12:47:43.710841 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:47:43.710798    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/fea676c8-6e68-4894-99fc-fe8b59886a9a-dbus\") pod \"global-pull-secret-syncer-5jkh4\" (UID: \"fea676c8-6e68-4894-99fc-fe8b59886a9a\") " pod="kube-system/global-pull-secret-syncer-5jkh4"
Sep 16 12:47:43.710841 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:47:43.710832    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/145070ab-abd8-4e8c-8cdd-99bfbbadbe1c-sys-fs\") pod \"aws-ebs-csi-driver-node-tztx9\" (UID: \"145070ab-abd8-4e8c-8cdd-99bfbbadbe1c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tztx9"
Sep 16 12:47:43.711199 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:47:43.710861    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/145070ab-abd8-4e8c-8cdd-99bfbbadbe1c-kubelet-dir\") pod \"aws-ebs-csi-driver-node-tztx9\" (UID: \"145070ab-abd8-4e8c-8cdd-99bfbbadbe1c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tztx9"
Sep 16 12:47:43.711199 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:47:43.710957    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/145070ab-abd8-4e8c-8cdd-99bfbbadbe1c-kubelet-dir\") pod \"aws-ebs-csi-driver-node-tztx9\" (UID: \"145070ab-abd8-4e8c-8cdd-99bfbbadbe1c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tztx9"
Sep 16 12:47:43.711199 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:47:43.711016    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/fea676c8-6e68-4894-99fc-fe8b59886a9a-kubelet-config\") pod \"global-pull-secret-syncer-5jkh4\" (UID: \"fea676c8-6e68-4894-99fc-fe8b59886a9a\") " pod="kube-system/global-pull-secret-syncer-5jkh4"
Sep 16 12:47:43.711199 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:47:43.711062    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/145070ab-abd8-4e8c-8cdd-99bfbbadbe1c-registration-dir\") pod \"aws-ebs-csi-driver-node-tztx9\" (UID: \"145070ab-abd8-4e8c-8cdd-99bfbbadbe1c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tztx9"
Sep 16 12:47:43.711199 ip-10-0-133-218 kubenswrapper[2580]: E0916 12:47:43.711156    2580 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 16 12:47:43.711454 ip-10-0-133-218 kubenswrapper[2580]: E0916 12:47:43.711222    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fea676c8-6e68-4894-99fc-fe8b59886a9a-original-pull-secret podName:fea676c8-6e68-4894-99fc-fe8b59886a9a nodeName:}" failed. No retries permitted until 2026-09-16 12:47:44.211203275 +0000 UTC m=+3.134075729 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/fea676c8-6e68-4894-99fc-fe8b59886a9a-original-pull-secret") pod "global-pull-secret-syncer-5jkh4" (UID: "fea676c8-6e68-4894-99fc-fe8b59886a9a") : object "kube-system"/"original-pull-secret" not registered
Sep 16 12:47:43.711848 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:47:43.711659    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/145070ab-abd8-4e8c-8cdd-99bfbbadbe1c-etc-selinux\") pod \"aws-ebs-csi-driver-node-tztx9\" (UID: \"145070ab-abd8-4e8c-8cdd-99bfbbadbe1c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tztx9"
Sep 16 12:47:43.711848 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:47:43.711709    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/fea676c8-6e68-4894-99fc-fe8b59886a9a-dbus\") pod \"global-pull-secret-syncer-5jkh4\" (UID: \"fea676c8-6e68-4894-99fc-fe8b59886a9a\") " pod="kube-system/global-pull-secret-syncer-5jkh4"
Sep 16 12:47:43.711848 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:47:43.711776    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/145070ab-abd8-4e8c-8cdd-99bfbbadbe1c-sys-fs\") pod \"aws-ebs-csi-driver-node-tztx9\" (UID: \"145070ab-abd8-4e8c-8cdd-99bfbbadbe1c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tztx9"
Sep 16 12:47:43.711848 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:47:43.711817    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/145070ab-abd8-4e8c-8cdd-99bfbbadbe1c-socket-dir\") pod \"aws-ebs-csi-driver-node-tztx9\" (UID: \"145070ab-abd8-4e8c-8cdd-99bfbbadbe1c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tztx9"
Sep 16 12:47:43.711848 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:47:43.711827    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/145070ab-abd8-4e8c-8cdd-99bfbbadbe1c-device-dir\") pod \"aws-ebs-csi-driver-node-tztx9\" (UID: \"145070ab-abd8-4e8c-8cdd-99bfbbadbe1c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tztx9"
Sep 16 12:47:43.722336 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:47:43.722274    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wnq5t\" (UniqueName: \"kubernetes.io/projected/145070ab-abd8-4e8c-8cdd-99bfbbadbe1c-kube-api-access-wnq5t\") pod \"aws-ebs-csi-driver-node-tztx9\" (UID: \"145070ab-abd8-4e8c-8cdd-99bfbbadbe1c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tztx9"
Sep 16 12:47:43.794441 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:47:43.794407    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-9gbm5"
Sep 16 12:47:43.806758 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:47:43.804917    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-cx649"
Sep 16 12:47:43.815906 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:47:43.815715    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-rgxfz"
Sep 16 12:47:43.817404 ip-10-0-133-218 kubenswrapper[2580]: W0916 12:47:43.817378    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0b719a93_a161_4199_9aae_5aa99cdf7165.slice/crio-a23330c01247d4f5adf515aaadfda4bf4370747d43b8b0bc62dc6867e75dff13 WatchSource:0}: Error finding container a23330c01247d4f5adf515aaadfda4bf4370747d43b8b0bc62dc6867e75dff13: Status 404 returned error can't find the container with id a23330c01247d4f5adf515aaadfda4bf4370747d43b8b0bc62dc6867e75dff13
Sep 16 12:47:43.824057 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:47:43.823870    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-894fk"
Sep 16 12:47:43.826591 ip-10-0-133-218 kubenswrapper[2580]: W0916 12:47:43.826543    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod32fe9109_0026_4609_af0c_923a618db835.slice/crio-8571113284ad939ea09120b9a9c6d579265b06db34937120b825ca1dc94c2ee3 WatchSource:0}: Error finding container 8571113284ad939ea09120b9a9c6d579265b06db34937120b825ca1dc94c2ee3: Status 404 returned error can't find the container with id 8571113284ad939ea09120b9a9c6d579265b06db34937120b825ca1dc94c2ee3
Sep 16 12:47:43.831943 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:47:43.831920    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-p6wht"
Sep 16 12:47:43.833866 ip-10-0-133-218 kubenswrapper[2580]: W0916 12:47:43.833837    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podffd98c6e_f973_4bd8_94ba_bd92a16d1acd.slice/crio-b93489dde39401ab6822c57eead73516321dea94891c7e54421552df5247303f WatchSource:0}: Error finding container b93489dde39401ab6822c57eead73516321dea94891c7e54421552df5247303f: Status 404 returned error can't find the container with id b93489dde39401ab6822c57eead73516321dea94891c7e54421552df5247303f
Sep 16 12:47:43.841146 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:47:43.840920    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-d9ndr"
Sep 16 12:47:43.845920 ip-10-0-133-218 kubenswrapper[2580]: W0916 12:47:43.845898    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddecaea8f_b0ad_428a_8880_e1ef3d565796.slice/crio-8d1ed36ecb6e6138332be50c240ef8e90d0f65e03ae7545dbdc72cc7fac438bc WatchSource:0}: Error finding container 8d1ed36ecb6e6138332be50c240ef8e90d0f65e03ae7545dbdc72cc7fac438bc: Status 404 returned error can't find the container with id 8d1ed36ecb6e6138332be50c240ef8e90d0f65e03ae7545dbdc72cc7fac438bc
Sep 16 12:47:43.850400 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:47:43.850379    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-544bh"
Sep 16 12:47:43.850637 ip-10-0-133-218 kubenswrapper[2580]: W0916 12:47:43.850610    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5d809960_17f1_4e60_9438_dc50a3b54dab.slice/crio-2623925e9d650a0d9eff57557088bf293779b6653077188b389b3b8d7df56952 WatchSource:0}: Error finding container 2623925e9d650a0d9eff57557088bf293779b6653077188b389b3b8d7df56952: Status 404 returned error can't find the container with id 2623925e9d650a0d9eff57557088bf293779b6653077188b389b3b8d7df56952
Sep 16 12:47:43.858806 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:47:43.858784    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-hsj4z"
Sep 16 12:47:43.865738 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:47:43.865697    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tztx9"
Sep 16 12:47:44.113128 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:47:44.113044    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7d634420-2900-42b3-9609-3fb64baff8f3-metrics-certs\") pod \"network-metrics-daemon-62j7x\" (UID: \"7d634420-2900-42b3-9609-3fb64baff8f3\") " pod="openshift-multus/network-metrics-daemon-62j7x"
Sep 16 12:47:44.113284 ip-10-0-133-218 kubenswrapper[2580]: E0916 12:47:44.113199    2580 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 16 12:47:44.113284 ip-10-0-133-218 kubenswrapper[2580]: E0916 12:47:44.113279    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7d634420-2900-42b3-9609-3fb64baff8f3-metrics-certs podName:7d634420-2900-42b3-9609-3fb64baff8f3 nodeName:}" failed. No retries permitted until 2026-09-16 12:47:45.113258 +0000 UTC m=+4.036130464 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/7d634420-2900-42b3-9609-3fb64baff8f3-metrics-certs") pod "network-metrics-daemon-62j7x" (UID: "7d634420-2900-42b3-9609-3fb64baff8f3") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 16 12:47:44.214107 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:47:44.214072    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/fea676c8-6e68-4894-99fc-fe8b59886a9a-original-pull-secret\") pod \"global-pull-secret-syncer-5jkh4\" (UID: \"fea676c8-6e68-4894-99fc-fe8b59886a9a\") " pod="kube-system/global-pull-secret-syncer-5jkh4"
Sep 16 12:47:44.214295 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:47:44.214126    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v4r5z\" (UniqueName: \"kubernetes.io/projected/1b26fbcb-3523-4f7c-9e42-5b3653456abc-kube-api-access-v4r5z\") pod \"network-check-target-bpv5d\" (UID: \"1b26fbcb-3523-4f7c-9e42-5b3653456abc\") " pod="openshift-network-diagnostics/network-check-target-bpv5d"
Sep 16 12:47:44.214295 ip-10-0-133-218 kubenswrapper[2580]: E0916 12:47:44.214250    2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 16 12:47:44.214295 ip-10-0-133-218 kubenswrapper[2580]: E0916 12:47:44.214253    2580 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 16 12:47:44.214433 ip-10-0-133-218 kubenswrapper[2580]: E0916 12:47:44.214265    2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 16 12:47:44.214433 ip-10-0-133-218 kubenswrapper[2580]: E0916 12:47:44.214371    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fea676c8-6e68-4894-99fc-fe8b59886a9a-original-pull-secret podName:fea676c8-6e68-4894-99fc-fe8b59886a9a nodeName:}" failed. No retries permitted until 2026-09-16 12:47:45.214346728 +0000 UTC m=+4.137219194 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/fea676c8-6e68-4894-99fc-fe8b59886a9a-original-pull-secret") pod "global-pull-secret-syncer-5jkh4" (UID: "fea676c8-6e68-4894-99fc-fe8b59886a9a") : object "kube-system"/"original-pull-secret" not registered
Sep 16 12:47:44.214433 ip-10-0-133-218 kubenswrapper[2580]: E0916 12:47:44.214372    2580 projected.go:196] Error preparing data for projected volume kube-api-access-v4r5z for pod openshift-network-diagnostics/network-check-target-bpv5d: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 16 12:47:44.214433 ip-10-0-133-218 kubenswrapper[2580]: E0916 12:47:44.214420    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/1b26fbcb-3523-4f7c-9e42-5b3653456abc-kube-api-access-v4r5z podName:1b26fbcb-3523-4f7c-9e42-5b3653456abc nodeName:}" failed. No retries permitted until 2026-09-16 12:47:45.214409577 +0000 UTC m=+4.137282029 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-v4r5z" (UniqueName: "kubernetes.io/projected/1b26fbcb-3523-4f7c-9e42-5b3653456abc-kube-api-access-v4r5z") pod "network-check-target-bpv5d" (UID: "1b26fbcb-3523-4f7c-9e42-5b3653456abc") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 16 12:47:44.553628 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:47:44.553542    2580 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-09-15 12:42:44 +0000 UTC" deadline="2028-03-13 19:03:40.099830524 +0000 UTC"
Sep 16 12:47:44.553628 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:47:44.553573    2580 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13062h15m55.54626118s"
Sep 16 12:47:44.595044 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:47:44.595018    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5jkh4"
Sep 16 12:47:44.595200 ip-10-0-133-218 kubenswrapper[2580]: E0916 12:47:44.595134    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-5jkh4" podUID="fea676c8-6e68-4894-99fc-fe8b59886a9a"
Sep 16 12:47:44.605486 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:47:44.605454    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-p6wht" event={"ID":"decaea8f-b0ad-428a-8880-e1ef3d565796","Type":"ContainerStarted","Data":"8d1ed36ecb6e6138332be50c240ef8e90d0f65e03ae7545dbdc72cc7fac438bc"}
Sep 16 12:47:44.606755 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:47:44.606732    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-894fk" event={"ID":"ffd98c6e-f973-4bd8-94ba-bd92a16d1acd","Type":"ContainerStarted","Data":"b93489dde39401ab6822c57eead73516321dea94891c7e54421552df5247303f"}
Sep 16 12:47:44.607965 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:47:44.607941    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-rgxfz" event={"ID":"32fe9109-0026-4609-af0c-923a618db835","Type":"ContainerStarted","Data":"8571113284ad939ea09120b9a9c6d579265b06db34937120b825ca1dc94c2ee3"}
Sep 16 12:47:44.609090 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:47:44.609065    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-cx649" event={"ID":"0b719a93-a161-4199-9aae-5aa99cdf7165","Type":"ContainerStarted","Data":"a23330c01247d4f5adf515aaadfda4bf4370747d43b8b0bc62dc6867e75dff13"}
Sep 16 12:47:44.610187 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:47:44.610166    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-9gbm5" event={"ID":"8822b539-bda1-46bd-8a2c-27660032a4e5","Type":"ContainerStarted","Data":"aa33c93f32dfc031871d9fb3d956001160a766c25d5f280865f9364566e5fc1b"}
Sep 16 12:47:44.611340 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:47:44.611316    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-d9ndr" event={"ID":"5d809960-17f1-4e60-9438-dc50a3b54dab","Type":"ContainerStarted","Data":"2623925e9d650a0d9eff57557088bf293779b6653077188b389b3b8d7df56952"}
Sep 16 12:47:44.885055 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:47:44.884631    2580 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 16 12:47:45.120051 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:47:45.120017    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7d634420-2900-42b3-9609-3fb64baff8f3-metrics-certs\") pod \"network-metrics-daemon-62j7x\" (UID: \"7d634420-2900-42b3-9609-3fb64baff8f3\") " pod="openshift-multus/network-metrics-daemon-62j7x"
Sep 16 12:47:45.120219 ip-10-0-133-218 kubenswrapper[2580]: E0916 12:47:45.120203    2580 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 16 12:47:45.120300 ip-10-0-133-218 kubenswrapper[2580]: E0916 12:47:45.120284    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7d634420-2900-42b3-9609-3fb64baff8f3-metrics-certs podName:7d634420-2900-42b3-9609-3fb64baff8f3 nodeName:}" failed. No retries permitted until 2026-09-16 12:47:47.120260945 +0000 UTC m=+6.043133398 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/7d634420-2900-42b3-9609-3fb64baff8f3-metrics-certs") pod "network-metrics-daemon-62j7x" (UID: "7d634420-2900-42b3-9609-3fb64baff8f3") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 16 12:47:45.183182 ip-10-0-133-218 kubenswrapper[2580]: W0916 12:47:45.183150    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod145070ab_abd8_4e8c_8cdd_99bfbbadbe1c.slice/crio-ee85e3ab75e988442924e7103913a115ab4038d9513cc01b247c0d43120580f3 WatchSource:0}: Error finding container ee85e3ab75e988442924e7103913a115ab4038d9513cc01b247c0d43120580f3: Status 404 returned error can't find the container with id ee85e3ab75e988442924e7103913a115ab4038d9513cc01b247c0d43120580f3
Sep 16 12:47:45.185146 ip-10-0-133-218 kubenswrapper[2580]: W0916 12:47:45.184937    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfb1fa9b3_dee8_433f_b3f9_29f24eab1e3a.slice/crio-f2c70aa496c6f022ad1e869bb5016a01d09e9411176cadc1534aed9837e39bfc WatchSource:0}: Error finding container f2c70aa496c6f022ad1e869bb5016a01d09e9411176cadc1534aed9837e39bfc: Status 404 returned error can't find the container with id f2c70aa496c6f022ad1e869bb5016a01d09e9411176cadc1534aed9837e39bfc
Sep 16 12:47:45.191660 ip-10-0-133-218 kubenswrapper[2580]: W0916 12:47:45.191635    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7b900f8b_a992_44b3_a6ea_7bacb17d2798.slice/crio-699ae22a48de6e96518f1ee8061e7ba8a9ac1e2f678f7ff13a0f768157b0f02b WatchSource:0}: Error finding container 699ae22a48de6e96518f1ee8061e7ba8a9ac1e2f678f7ff13a0f768157b0f02b: Status 404 returned error can't find the container with id 699ae22a48de6e96518f1ee8061e7ba8a9ac1e2f678f7ff13a0f768157b0f02b
Sep 16 12:47:45.221303 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:47:45.221265    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/fea676c8-6e68-4894-99fc-fe8b59886a9a-original-pull-secret\") pod \"global-pull-secret-syncer-5jkh4\" (UID: \"fea676c8-6e68-4894-99fc-fe8b59886a9a\") " pod="kube-system/global-pull-secret-syncer-5jkh4"
Sep 16 12:47:45.221445 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:47:45.221351    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v4r5z\" (UniqueName: \"kubernetes.io/projected/1b26fbcb-3523-4f7c-9e42-5b3653456abc-kube-api-access-v4r5z\") pod \"network-check-target-bpv5d\" (UID: \"1b26fbcb-3523-4f7c-9e42-5b3653456abc\") " pod="openshift-network-diagnostics/network-check-target-bpv5d"
Sep 16 12:47:45.221445 ip-10-0-133-218 kubenswrapper[2580]: E0916 12:47:45.221396    2580 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 16 12:47:45.221585 ip-10-0-133-218 kubenswrapper[2580]: E0916 12:47:45.221473    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fea676c8-6e68-4894-99fc-fe8b59886a9a-original-pull-secret podName:fea676c8-6e68-4894-99fc-fe8b59886a9a nodeName:}" failed. No retries permitted until 2026-09-16 12:47:47.221452795 +0000 UTC m=+6.144325248 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/fea676c8-6e68-4894-99fc-fe8b59886a9a-original-pull-secret") pod "global-pull-secret-syncer-5jkh4" (UID: "fea676c8-6e68-4894-99fc-fe8b59886a9a") : object "kube-system"/"original-pull-secret" not registered
Sep 16 12:47:45.221585 ip-10-0-133-218 kubenswrapper[2580]: E0916 12:47:45.221492    2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 16 12:47:45.221585 ip-10-0-133-218 kubenswrapper[2580]: E0916 12:47:45.221510    2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 16 12:47:45.221585 ip-10-0-133-218 kubenswrapper[2580]: E0916 12:47:45.221523    2580 projected.go:196] Error preparing data for projected volume kube-api-access-v4r5z for pod openshift-network-diagnostics/network-check-target-bpv5d: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 16 12:47:45.221811 ip-10-0-133-218 kubenswrapper[2580]: E0916 12:47:45.221599    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/1b26fbcb-3523-4f7c-9e42-5b3653456abc-kube-api-access-v4r5z podName:1b26fbcb-3523-4f7c-9e42-5b3653456abc nodeName:}" failed. No retries permitted until 2026-09-16 12:47:47.221582376 +0000 UTC m=+6.144454830 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-v4r5z" (UniqueName: "kubernetes.io/projected/1b26fbcb-3523-4f7c-9e42-5b3653456abc-kube-api-access-v4r5z") pod "network-check-target-bpv5d" (UID: "1b26fbcb-3523-4f7c-9e42-5b3653456abc") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 16 12:47:45.555887 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:47:45.555842    2580 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-09-15 12:42:44 +0000 UTC" deadline="2028-06-02 04:23:31.377098192 +0000 UTC"
Sep 16 12:47:45.555887 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:47:45.555881    2580 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14991h35m45.821221486s"
Sep 16 12:47:45.603338 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:47:45.603303    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-62j7x"
Sep 16 12:47:45.603561 ip-10-0-133-218 kubenswrapper[2580]: E0916 12:47:45.603506    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-62j7x" podUID="7d634420-2900-42b3-9609-3fb64baff8f3"
Sep 16 12:47:45.604421 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:47:45.604384    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-bpv5d"
Sep 16 12:47:45.604616 ip-10-0-133-218 kubenswrapper[2580]: E0916 12:47:45.604593    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-bpv5d" podUID="1b26fbcb-3523-4f7c-9e42-5b3653456abc"
Sep 16 12:47:45.630932 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:47:45.630718    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-133-218.ec2.internal" event={"ID":"f3d9df87a80a4a167468bfdb12aa0522","Type":"ContainerStarted","Data":"7f7c4ef5d3cd028e8939d4bf00de1a033f47f501720ccf3987f9da7cbb226923"}
Sep 16 12:47:45.643683 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:47:45.643621    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-hsj4z" event={"ID":"7b900f8b-a992-44b3-a6ea-7bacb17d2798","Type":"ContainerStarted","Data":"699ae22a48de6e96518f1ee8061e7ba8a9ac1e2f678f7ff13a0f768157b0f02b"}
Sep 16 12:47:45.652579 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:47:45.649494    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-544bh" event={"ID":"fb1fa9b3-dee8-433f-b3f9-29f24eab1e3a","Type":"ContainerStarted","Data":"f2c70aa496c6f022ad1e869bb5016a01d09e9411176cadc1534aed9837e39bfc"}
Sep 16 12:47:45.660302 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:47:45.660229    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tztx9" event={"ID":"145070ab-abd8-4e8c-8cdd-99bfbbadbe1c","Type":"ContainerStarted","Data":"ee85e3ab75e988442924e7103913a115ab4038d9513cc01b247c0d43120580f3"}
Sep 16 12:47:46.595412 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:47:46.594902    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5jkh4"
Sep 16 12:47:46.595412 ip-10-0-133-218 kubenswrapper[2580]: E0916 12:47:46.595046    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-5jkh4" podUID="fea676c8-6e68-4894-99fc-fe8b59886a9a"
Sep 16 12:47:46.669600 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:47:46.669492    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-218.ec2.internal" event={"ID":"012c7f135b396ea57901e601e20663e2","Type":"ContainerStarted","Data":"1d855144f251c5d574c0c14ebfc0b6b215fb48812cccda98d1644219cb8fff96"}
Sep 16 12:47:46.691474 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:47:46.691412    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-133-218.ec2.internal" podStartSLOduration=3.6913901940000002 podStartE2EDuration="3.691390194s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-16 12:47:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 12:47:45.652106101 +0000 UTC m=+4.574978576" watchObservedRunningTime="2026-09-16 12:47:46.691390194 +0000 UTC m=+5.614262667"
Sep 16 12:47:47.139995 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:47:47.139872    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7d634420-2900-42b3-9609-3fb64baff8f3-metrics-certs\") pod \"network-metrics-daemon-62j7x\" (UID: \"7d634420-2900-42b3-9609-3fb64baff8f3\") " pod="openshift-multus/network-metrics-daemon-62j7x"
Sep 16 12:47:47.140198 ip-10-0-133-218 kubenswrapper[2580]: E0916 12:47:47.140076    2580 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 16 12:47:47.140198 ip-10-0-133-218 kubenswrapper[2580]: E0916 12:47:47.140157    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7d634420-2900-42b3-9609-3fb64baff8f3-metrics-certs podName:7d634420-2900-42b3-9609-3fb64baff8f3 nodeName:}" failed. No retries permitted until 2026-09-16 12:47:51.14013504 +0000 UTC m=+10.063007493 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/7d634420-2900-42b3-9609-3fb64baff8f3-metrics-certs") pod "network-metrics-daemon-62j7x" (UID: "7d634420-2900-42b3-9609-3fb64baff8f3") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 16 12:47:47.240788 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:47:47.240325    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v4r5z\" (UniqueName: \"kubernetes.io/projected/1b26fbcb-3523-4f7c-9e42-5b3653456abc-kube-api-access-v4r5z\") pod \"network-check-target-bpv5d\" (UID: \"1b26fbcb-3523-4f7c-9e42-5b3653456abc\") " pod="openshift-network-diagnostics/network-check-target-bpv5d"
Sep 16 12:47:47.240788 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:47:47.240412    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/fea676c8-6e68-4894-99fc-fe8b59886a9a-original-pull-secret\") pod \"global-pull-secret-syncer-5jkh4\" (UID: \"fea676c8-6e68-4894-99fc-fe8b59886a9a\") " pod="kube-system/global-pull-secret-syncer-5jkh4"
Sep 16 12:47:47.240788 ip-10-0-133-218 kubenswrapper[2580]: E0916 12:47:47.240560    2580 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 16 12:47:47.240788 ip-10-0-133-218 kubenswrapper[2580]: E0916 12:47:47.240630    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fea676c8-6e68-4894-99fc-fe8b59886a9a-original-pull-secret podName:fea676c8-6e68-4894-99fc-fe8b59886a9a nodeName:}" failed. No retries permitted until 2026-09-16 12:47:51.240610544 +0000 UTC m=+10.163483018 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/fea676c8-6e68-4894-99fc-fe8b59886a9a-original-pull-secret") pod "global-pull-secret-syncer-5jkh4" (UID: "fea676c8-6e68-4894-99fc-fe8b59886a9a") : object "kube-system"/"original-pull-secret" not registered
Sep 16 12:47:47.241251 ip-10-0-133-218 kubenswrapper[2580]: E0916 12:47:47.241058    2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 16 12:47:47.241251 ip-10-0-133-218 kubenswrapper[2580]: E0916 12:47:47.241077    2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 16 12:47:47.241251 ip-10-0-133-218 kubenswrapper[2580]: E0916 12:47:47.241090    2580 projected.go:196] Error preparing data for projected volume kube-api-access-v4r5z for pod openshift-network-diagnostics/network-check-target-bpv5d: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 16 12:47:47.241251 ip-10-0-133-218 kubenswrapper[2580]: E0916 12:47:47.241139    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/1b26fbcb-3523-4f7c-9e42-5b3653456abc-kube-api-access-v4r5z podName:1b26fbcb-3523-4f7c-9e42-5b3653456abc nodeName:}" failed. No retries permitted until 2026-09-16 12:47:51.241122473 +0000 UTC m=+10.163994925 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-v4r5z" (UniqueName: "kubernetes.io/projected/1b26fbcb-3523-4f7c-9e42-5b3653456abc-kube-api-access-v4r5z") pod "network-check-target-bpv5d" (UID: "1b26fbcb-3523-4f7c-9e42-5b3653456abc") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 16 12:47:47.595682 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:47:47.595646    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-62j7x"
Sep 16 12:47:47.595682 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:47:47.595672    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-bpv5d"
Sep 16 12:47:47.596236 ip-10-0-133-218 kubenswrapper[2580]: E0916 12:47:47.595829    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-62j7x" podUID="7d634420-2900-42b3-9609-3fb64baff8f3"
Sep 16 12:47:47.596312 ip-10-0-133-218 kubenswrapper[2580]: E0916 12:47:47.596261    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-bpv5d" podUID="1b26fbcb-3523-4f7c-9e42-5b3653456abc"
Sep 16 12:47:48.594396 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:47:48.594360    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5jkh4"
Sep 16 12:47:48.594620 ip-10-0-133-218 kubenswrapper[2580]: E0916 12:47:48.594501    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-5jkh4" podUID="fea676c8-6e68-4894-99fc-fe8b59886a9a"
Sep 16 12:47:49.597422 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:47:49.596742    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-62j7x"
Sep 16 12:47:49.597422 ip-10-0-133-218 kubenswrapper[2580]: E0916 12:47:49.596890    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-62j7x" podUID="7d634420-2900-42b3-9609-3fb64baff8f3"
Sep 16 12:47:49.597422 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:47:49.597266    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-bpv5d"
Sep 16 12:47:49.597422 ip-10-0-133-218 kubenswrapper[2580]: E0916 12:47:49.597361    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-bpv5d" podUID="1b26fbcb-3523-4f7c-9e42-5b3653456abc"
Sep 16 12:47:50.594060 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:47:50.594022    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5jkh4"
Sep 16 12:47:50.594270 ip-10-0-133-218 kubenswrapper[2580]: E0916 12:47:50.594178    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-5jkh4" podUID="fea676c8-6e68-4894-99fc-fe8b59886a9a"
Sep 16 12:47:51.176010 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:47:51.175356    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7d634420-2900-42b3-9609-3fb64baff8f3-metrics-certs\") pod \"network-metrics-daemon-62j7x\" (UID: \"7d634420-2900-42b3-9609-3fb64baff8f3\") " pod="openshift-multus/network-metrics-daemon-62j7x"
Sep 16 12:47:51.176010 ip-10-0-133-218 kubenswrapper[2580]: E0916 12:47:51.175568    2580 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 16 12:47:51.176010 ip-10-0-133-218 kubenswrapper[2580]: E0916 12:47:51.175641    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7d634420-2900-42b3-9609-3fb64baff8f3-metrics-certs podName:7d634420-2900-42b3-9609-3fb64baff8f3 nodeName:}" failed. No retries permitted until 2026-09-16 12:47:59.175621178 +0000 UTC m=+18.098493633 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/7d634420-2900-42b3-9609-3fb64baff8f3-metrics-certs") pod "network-metrics-daemon-62j7x" (UID: "7d634420-2900-42b3-9609-3fb64baff8f3") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 16 12:47:51.277291 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:47:51.276511    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v4r5z\" (UniqueName: \"kubernetes.io/projected/1b26fbcb-3523-4f7c-9e42-5b3653456abc-kube-api-access-v4r5z\") pod \"network-check-target-bpv5d\" (UID: \"1b26fbcb-3523-4f7c-9e42-5b3653456abc\") " pod="openshift-network-diagnostics/network-check-target-bpv5d"
Sep 16 12:47:51.277291 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:47:51.276590    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/fea676c8-6e68-4894-99fc-fe8b59886a9a-original-pull-secret\") pod \"global-pull-secret-syncer-5jkh4\" (UID: \"fea676c8-6e68-4894-99fc-fe8b59886a9a\") " pod="kube-system/global-pull-secret-syncer-5jkh4"
Sep 16 12:47:51.277291 ip-10-0-133-218 kubenswrapper[2580]: E0916 12:47:51.276702    2580 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 16 12:47:51.277291 ip-10-0-133-218 kubenswrapper[2580]: E0916 12:47:51.276761    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fea676c8-6e68-4894-99fc-fe8b59886a9a-original-pull-secret podName:fea676c8-6e68-4894-99fc-fe8b59886a9a nodeName:}" failed. No retries permitted until 2026-09-16 12:47:59.276742859 +0000 UTC m=+18.199615321 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/fea676c8-6e68-4894-99fc-fe8b59886a9a-original-pull-secret") pod "global-pull-secret-syncer-5jkh4" (UID: "fea676c8-6e68-4894-99fc-fe8b59886a9a") : object "kube-system"/"original-pull-secret" not registered
Sep 16 12:47:51.277291 ip-10-0-133-218 kubenswrapper[2580]: E0916 12:47:51.277143    2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 16 12:47:51.277291 ip-10-0-133-218 kubenswrapper[2580]: E0916 12:47:51.277159    2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 16 12:47:51.277291 ip-10-0-133-218 kubenswrapper[2580]: E0916 12:47:51.277171    2580 projected.go:196] Error preparing data for projected volume kube-api-access-v4r5z for pod openshift-network-diagnostics/network-check-target-bpv5d: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 16 12:47:51.277291 ip-10-0-133-218 kubenswrapper[2580]: E0916 12:47:51.277213    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/1b26fbcb-3523-4f7c-9e42-5b3653456abc-kube-api-access-v4r5z podName:1b26fbcb-3523-4f7c-9e42-5b3653456abc nodeName:}" failed. No retries permitted until 2026-09-16 12:47:59.277200659 +0000 UTC m=+18.200073127 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-v4r5z" (UniqueName: "kubernetes.io/projected/1b26fbcb-3523-4f7c-9e42-5b3653456abc-kube-api-access-v4r5z") pod "network-check-target-bpv5d" (UID: "1b26fbcb-3523-4f7c-9e42-5b3653456abc") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 16 12:47:51.597463 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:47:51.597428    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-62j7x"
Sep 16 12:47:51.597656 ip-10-0-133-218 kubenswrapper[2580]: E0916 12:47:51.597601    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-62j7x" podUID="7d634420-2900-42b3-9609-3fb64baff8f3"
Sep 16 12:47:51.597968 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:47:51.597951    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-bpv5d"
Sep 16 12:47:51.598040 ip-10-0-133-218 kubenswrapper[2580]: E0916 12:47:51.598026    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-bpv5d" podUID="1b26fbcb-3523-4f7c-9e42-5b3653456abc"
Sep 16 12:47:51.683280 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:47:51.683242    2580 generic.go:356] "Generic (PLEG): container finished" podID="012c7f135b396ea57901e601e20663e2" containerID="1d855144f251c5d574c0c14ebfc0b6b215fb48812cccda98d1644219cb8fff96" exitCode=0
Sep 16 12:47:51.683425 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:47:51.683296    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-218.ec2.internal" event={"ID":"012c7f135b396ea57901e601e20663e2","Type":"ContainerDied","Data":"1d855144f251c5d574c0c14ebfc0b6b215fb48812cccda98d1644219cb8fff96"}
Sep 16 12:47:52.594379 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:47:52.594342    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5jkh4"
Sep 16 12:47:52.594966 ip-10-0-133-218 kubenswrapper[2580]: E0916 12:47:52.594451    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-5jkh4" podUID="fea676c8-6e68-4894-99fc-fe8b59886a9a"
Sep 16 12:47:53.594668 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:47:53.594625    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-62j7x"
Sep 16 12:47:53.595040 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:47:53.594683    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-bpv5d"
Sep 16 12:47:53.595040 ip-10-0-133-218 kubenswrapper[2580]: E0916 12:47:53.594772    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-62j7x" podUID="7d634420-2900-42b3-9609-3fb64baff8f3"
Sep 16 12:47:53.595040 ip-10-0-133-218 kubenswrapper[2580]: E0916 12:47:53.594879    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-bpv5d" podUID="1b26fbcb-3523-4f7c-9e42-5b3653456abc"
Sep 16 12:47:54.594417 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:47:54.594380    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5jkh4"
Sep 16 12:47:54.594615 ip-10-0-133-218 kubenswrapper[2580]: E0916 12:47:54.594590    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-5jkh4" podUID="fea676c8-6e68-4894-99fc-fe8b59886a9a"
Sep 16 12:47:55.594850 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:47:55.594813    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-62j7x"
Sep 16 12:47:55.594850 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:47:55.594861    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-bpv5d"
Sep 16 12:47:55.595349 ip-10-0-133-218 kubenswrapper[2580]: E0916 12:47:55.594974    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-62j7x" podUID="7d634420-2900-42b3-9609-3fb64baff8f3"
Sep 16 12:47:55.595349 ip-10-0-133-218 kubenswrapper[2580]: E0916 12:47:55.595105    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-bpv5d" podUID="1b26fbcb-3523-4f7c-9e42-5b3653456abc"
Sep 16 12:47:56.595017 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:47:56.594982    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5jkh4"
Sep 16 12:47:56.595389 ip-10-0-133-218 kubenswrapper[2580]: E0916 12:47:56.595118    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-5jkh4" podUID="fea676c8-6e68-4894-99fc-fe8b59886a9a"
Sep 16 12:47:57.594382 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:47:57.594178    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-62j7x"
Sep 16 12:47:57.594547 ip-10-0-133-218 kubenswrapper[2580]: E0916 12:47:57.594505    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-62j7x" podUID="7d634420-2900-42b3-9609-3fb64baff8f3"
Sep 16 12:47:57.594831 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:47:57.594244    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-bpv5d"
Sep 16 12:47:57.594991 ip-10-0-133-218 kubenswrapper[2580]: E0916 12:47:57.594922    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-bpv5d" podUID="1b26fbcb-3523-4f7c-9e42-5b3653456abc"
Sep 16 12:47:58.594337 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:47:58.594306    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5jkh4"
Sep 16 12:47:58.594820 ip-10-0-133-218 kubenswrapper[2580]: E0916 12:47:58.594433    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-5jkh4" podUID="fea676c8-6e68-4894-99fc-fe8b59886a9a"
Sep 16 12:47:59.236134 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:47:59.236037    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7d634420-2900-42b3-9609-3fb64baff8f3-metrics-certs\") pod \"network-metrics-daemon-62j7x\" (UID: \"7d634420-2900-42b3-9609-3fb64baff8f3\") " pod="openshift-multus/network-metrics-daemon-62j7x"
Sep 16 12:47:59.236299 ip-10-0-133-218 kubenswrapper[2580]: E0916 12:47:59.236242    2580 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 16 12:47:59.236358 ip-10-0-133-218 kubenswrapper[2580]: E0916 12:47:59.236314    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7d634420-2900-42b3-9609-3fb64baff8f3-metrics-certs podName:7d634420-2900-42b3-9609-3fb64baff8f3 nodeName:}" failed. No retries permitted until 2026-09-16 12:48:15.236293899 +0000 UTC m=+34.159166351 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/7d634420-2900-42b3-9609-3fb64baff8f3-metrics-certs") pod "network-metrics-daemon-62j7x" (UID: "7d634420-2900-42b3-9609-3fb64baff8f3") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 16 12:47:59.336846 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:47:59.336808    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v4r5z\" (UniqueName: \"kubernetes.io/projected/1b26fbcb-3523-4f7c-9e42-5b3653456abc-kube-api-access-v4r5z\") pod \"network-check-target-bpv5d\" (UID: \"1b26fbcb-3523-4f7c-9e42-5b3653456abc\") " pod="openshift-network-diagnostics/network-check-target-bpv5d"
Sep 16 12:47:59.337022 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:47:59.336893    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/fea676c8-6e68-4894-99fc-fe8b59886a9a-original-pull-secret\") pod \"global-pull-secret-syncer-5jkh4\" (UID: \"fea676c8-6e68-4894-99fc-fe8b59886a9a\") " pod="kube-system/global-pull-secret-syncer-5jkh4"
Sep 16 12:47:59.337022 ip-10-0-133-218 kubenswrapper[2580]: E0916 12:47:59.336994    2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 16 12:47:59.337022 ip-10-0-133-218 kubenswrapper[2580]: E0916 12:47:59.337008    2580 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 16 12:47:59.337022 ip-10-0-133-218 kubenswrapper[2580]: E0916 12:47:59.337020    2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 16 12:47:59.337192 ip-10-0-133-218 kubenswrapper[2580]: E0916 12:47:59.337031    2580 projected.go:196] Error preparing data for projected volume kube-api-access-v4r5z for pod openshift-network-diagnostics/network-check-target-bpv5d: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 16 12:47:59.337192 ip-10-0-133-218 kubenswrapper[2580]: E0916 12:47:59.337074    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fea676c8-6e68-4894-99fc-fe8b59886a9a-original-pull-secret podName:fea676c8-6e68-4894-99fc-fe8b59886a9a nodeName:}" failed. No retries permitted until 2026-09-16 12:48:15.337054302 +0000 UTC m=+34.259926749 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/fea676c8-6e68-4894-99fc-fe8b59886a9a-original-pull-secret") pod "global-pull-secret-syncer-5jkh4" (UID: "fea676c8-6e68-4894-99fc-fe8b59886a9a") : object "kube-system"/"original-pull-secret" not registered
Sep 16 12:47:59.337192 ip-10-0-133-218 kubenswrapper[2580]: E0916 12:47:59.337094    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/1b26fbcb-3523-4f7c-9e42-5b3653456abc-kube-api-access-v4r5z podName:1b26fbcb-3523-4f7c-9e42-5b3653456abc nodeName:}" failed. No retries permitted until 2026-09-16 12:48:15.337084543 +0000 UTC m=+34.259956996 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-v4r5z" (UniqueName: "kubernetes.io/projected/1b26fbcb-3523-4f7c-9e42-5b3653456abc-kube-api-access-v4r5z") pod "network-check-target-bpv5d" (UID: "1b26fbcb-3523-4f7c-9e42-5b3653456abc") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 16 12:47:59.595134 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:47:59.595090    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-62j7x"
Sep 16 12:47:59.595569 ip-10-0-133-218 kubenswrapper[2580]: E0916 12:47:59.595254    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-62j7x" podUID="7d634420-2900-42b3-9609-3fb64baff8f3"
Sep 16 12:47:59.595569 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:47:59.595302    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-bpv5d"
Sep 16 12:47:59.595569 ip-10-0-133-218 kubenswrapper[2580]: E0916 12:47:59.595427    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-bpv5d" podUID="1b26fbcb-3523-4f7c-9e42-5b3653456abc"
Sep 16 12:48:00.594674 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:48:00.594637    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5jkh4"
Sep 16 12:48:00.594844 ip-10-0-133-218 kubenswrapper[2580]: E0916 12:48:00.594752    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-5jkh4" podUID="fea676c8-6e68-4894-99fc-fe8b59886a9a"
Sep 16 12:48:01.595138 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:48:01.595105    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-62j7x"
Sep 16 12:48:01.595633 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:48:01.595141    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-bpv5d"
Sep 16 12:48:01.595633 ip-10-0-133-218 kubenswrapper[2580]: E0916 12:48:01.595225    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-62j7x" podUID="7d634420-2900-42b3-9609-3fb64baff8f3"
Sep 16 12:48:01.595633 ip-10-0-133-218 kubenswrapper[2580]: E0916 12:48:01.595360    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-bpv5d" podUID="1b26fbcb-3523-4f7c-9e42-5b3653456abc"
Sep 16 12:48:02.594875 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:48:02.594841    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5jkh4"
Sep 16 12:48:02.595040 ip-10-0-133-218 kubenswrapper[2580]: E0916 12:48:02.594953    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-5jkh4" podUID="fea676c8-6e68-4894-99fc-fe8b59886a9a"
Sep 16 12:48:03.594833 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:48:03.594614    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-bpv5d"
Sep 16 12:48:03.595570 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:48:03.594626    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-62j7x"
Sep 16 12:48:03.595570 ip-10-0-133-218 kubenswrapper[2580]: E0916 12:48:03.594947    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-bpv5d" podUID="1b26fbcb-3523-4f7c-9e42-5b3653456abc"
Sep 16 12:48:03.595570 ip-10-0-133-218 kubenswrapper[2580]: E0916 12:48:03.595032    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-62j7x" podUID="7d634420-2900-42b3-9609-3fb64baff8f3"
Sep 16 12:48:03.709647 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:48:03.709565    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-894fk" event={"ID":"ffd98c6e-f973-4bd8-94ba-bd92a16d1acd","Type":"ContainerStarted","Data":"b971c5de874ddb65befe92de9c1027bd7ef25f81a3f99d57631e21b47664c7b8"}
Sep 16 12:48:03.709794 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:48:03.709656    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-894fk" event={"ID":"ffd98c6e-f973-4bd8-94ba-bd92a16d1acd","Type":"ContainerStarted","Data":"e22a0297db9ff46a84bf5f80e4b4ac059a8c8b4168b0189fabf28fd9a5080500"}
Sep 16 12:48:03.709794 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:48:03.709703    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-894fk" event={"ID":"ffd98c6e-f973-4bd8-94ba-bd92a16d1acd","Type":"ContainerStarted","Data":"9c43bab8ab57caae3463e6f03623a1627f2a1ed708d179197fc1aa6cf27139ad"}
Sep 16 12:48:03.709794 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:48:03.709747    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-894fk" event={"ID":"ffd98c6e-f973-4bd8-94ba-bd92a16d1acd","Type":"ContainerStarted","Data":"fa5aab611e46ace7c62aec22fa4a661bb71d9ec11bf70ad4b66d69a9f6bb7540"}
Sep 16 12:48:03.709794 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:48:03.709777    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-894fk" event={"ID":"ffd98c6e-f973-4bd8-94ba-bd92a16d1acd","Type":"ContainerStarted","Data":"5555cd5d613e9a9fb872166032f7f30daf62645869080060678341f83474cc94"}
Sep 16 12:48:03.709988 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:48:03.709822    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-894fk" event={"ID":"ffd98c6e-f973-4bd8-94ba-bd92a16d1acd","Type":"ContainerStarted","Data":"54da5dbcad964410cfebf9a56fdb4cf9aec540756dd0439599e2f915f07da7d3"}
Sep 16 12:48:03.711314 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:48:03.711288    2580 generic.go:356] "Generic (PLEG): container finished" podID="7b900f8b-a992-44b3-a6ea-7bacb17d2798" containerID="2e6e5e0a26554fd99bf68cc94110a8eb1e39fad563343c42b4aa0902b9d9a038" exitCode=0
Sep 16 12:48:03.711420 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:48:03.711359    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-hsj4z" event={"ID":"7b900f8b-a992-44b3-a6ea-7bacb17d2798","Type":"ContainerDied","Data":"2e6e5e0a26554fd99bf68cc94110a8eb1e39fad563343c42b4aa0902b9d9a038"}
Sep 16 12:48:03.713396 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:48:03.713366    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-d9ndr" event={"ID":"5d809960-17f1-4e60-9438-dc50a3b54dab","Type":"ContainerStarted","Data":"0854bc474f7cfbd511a05cb2f94c56a0122701fd9000f11711c414371e988395"}
Sep 16 12:48:03.715053 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:48:03.715028    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-p6wht" event={"ID":"decaea8f-b0ad-428a-8880-e1ef3d565796","Type":"ContainerStarted","Data":"2a2dfd0f8f60ef1e2f93ec63fdfc47190ed9fcbf840683a9d4a0f3fe9c06d8e5"}
Sep 16 12:48:03.717662 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:48:03.717635    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-cx649" event={"ID":"0b719a93-a161-4199-9aae-5aa99cdf7165","Type":"ContainerStarted","Data":"626afa4160bccd3b0415b8bc928ba7f6ea23c7e2059770846bb69fa0a5a3c1ae"}
Sep 16 12:48:03.719198 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:48:03.719179    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-9gbm5" event={"ID":"8822b539-bda1-46bd-8a2c-27660032a4e5","Type":"ContainerStarted","Data":"d5403797e68ae85ffbcbd0d4c28ceb9b3824b9e2e7de149aba88ed25b33d7be1"}
Sep 16 12:48:03.720844 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:48:03.720823    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-544bh" event={"ID":"fb1fa9b3-dee8-433f-b3f9-29f24eab1e3a","Type":"ContainerStarted","Data":"e3790c298ca2e62a7e202d03f567022fe79b69d9294f4db90650d6e5a5cab522"}
Sep 16 12:48:03.722405 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:48:03.722385    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tztx9" event={"ID":"145070ab-abd8-4e8c-8cdd-99bfbbadbe1c","Type":"ContainerStarted","Data":"ab041ef25bf45fe750216dcc43f7432a73c58b4e9c779bfa52ded4210424fdc6"}
Sep 16 12:48:03.724434 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:48:03.724415    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-218.ec2.internal" event={"ID":"012c7f135b396ea57901e601e20663e2","Type":"ContainerStarted","Data":"2902eca49b18233ad603281c787bfb250da361d164903b7801b1f6afd1a0dd80"}
Sep 16 12:48:03.747298 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:48:03.747245    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-218.ec2.internal" podStartSLOduration=20.747226712 podStartE2EDuration="20.747226712s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-16 12:47:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 12:48:03.746912174 +0000 UTC m=+22.669784630" watchObservedRunningTime="2026-09-16 12:48:03.747226712 +0000 UTC m=+22.670099182"
Sep 16 12:48:03.763438 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:48:03.763387    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-9gbm5" podStartSLOduration=2.922543921 podStartE2EDuration="21.763371626s" totalImagesPullingTime="18.840827705s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-16 12:47:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 12:48:03.76328266 +0000 UTC m=+22.686155130" watchObservedRunningTime="2026-09-16 12:48:03.763371626 +0000 UTC m=+22.686244098"
Sep 16 12:48:03.777224 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:48:03.777173    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-p6wht" podStartSLOduration=6.917533016 podStartE2EDuration="21.777155585s" totalImagesPullingTime="14.859622569s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 12:47:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 12:48:03.776632862 +0000 UTC m=+22.699505332" watchObservedRunningTime="2026-09-16 12:48:03.777155585 +0000 UTC m=+22.700028055"
Sep 16 12:48:03.807824 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:48:03.807782    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-544bh" podStartSLOduration=4.182356133 podStartE2EDuration="21.807763177s" totalImagesPullingTime="17.625407044s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-16 12:47:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 12:48:03.793116196 +0000 UTC m=+22.715988755" watchObservedRunningTime="2026-09-16 12:48:03.807763177 +0000 UTC m=+22.730635627"
Sep 16 12:48:03.808104 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:48:03.808077    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-cx649" podStartSLOduration=6.920604006 podStartE2EDuration="21.808069188s" totalImagesPullingTime="14.887465182s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-16 12:47:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 12:48:03.807845144 +0000 UTC m=+22.730717613" watchObservedRunningTime="2026-09-16 12:48:03.808069188 +0000 UTC m=+22.730941656"
Sep 16 12:48:03.828261 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:48:03.828227    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-d9ndr" podStartSLOduration=3.031433569 podStartE2EDuration="21.828213562s" totalImagesPullingTime="18.796779993s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-16 12:47:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 12:48:03.827544613 +0000 UTC m=+22.750417083" watchObservedRunningTime="2026-09-16 12:48:03.828213562 +0000 UTC m=+22.751086032"
Sep 16 12:48:03.916441 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:48:03.916299    2580 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Sep 16 12:48:04.574142 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:48:04.574033    2580 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-09-16T12:48:03.916437745Z","UUID":"20273927-6bc5-4fbe-af15-218d183e4f6b","Handler":null,"Name":"","Endpoint":""}
Sep 16 12:48:04.575622 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:48:04.575601    2580 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Sep 16 12:48:04.575738 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:48:04.575634    2580 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Sep 16 12:48:04.594711 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:48:04.594691    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5jkh4"
Sep 16 12:48:04.594814 ip-10-0-133-218 kubenswrapper[2580]: E0916 12:48:04.594785    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-5jkh4" podUID="fea676c8-6e68-4894-99fc-fe8b59886a9a"
Sep 16 12:48:04.729072 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:48:04.729034    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tztx9" event={"ID":"145070ab-abd8-4e8c-8cdd-99bfbbadbe1c","Type":"ContainerStarted","Data":"fe4578b5a665e071403768305538d1569d6f520f2e8f303cd38e1af1c4a411d9"}
Sep 16 12:48:04.730735 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:48:04.730691    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-rgxfz" event={"ID":"32fe9109-0026-4609-af0c-923a618db835","Type":"ContainerStarted","Data":"6bde14e7e1f09299399c195759078a63a3ca6fe42e910676ac7019532ecaa295"}
Sep 16 12:48:04.749298 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:48:04.749245    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-rgxfz" podStartSLOduration=3.952311997 podStartE2EDuration="22.749227484s" totalImagesPullingTime="18.796915487s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-16 12:47:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 12:48:04.748909726 +0000 UTC m=+23.671782195" watchObservedRunningTime="2026-09-16 12:48:04.749227484 +0000 UTC m=+23.672099955"
Sep 16 12:48:05.594949 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:48:05.594922    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-62j7x"
Sep 16 12:48:05.595081 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:48:05.595065    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-bpv5d"
Sep 16 12:48:05.595142 ip-10-0-133-218 kubenswrapper[2580]: E0916 12:48:05.595094    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-62j7x" podUID="7d634420-2900-42b3-9609-3fb64baff8f3"
Sep 16 12:48:05.595200 ip-10-0-133-218 kubenswrapper[2580]: E0916 12:48:05.595141    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-bpv5d" podUID="1b26fbcb-3523-4f7c-9e42-5b3653456abc"
Sep 16 12:48:05.737536 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:48:05.737482    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-894fk" event={"ID":"ffd98c6e-f973-4bd8-94ba-bd92a16d1acd","Type":"ContainerStarted","Data":"e35e321577cf6514d27a30e9b39bd41e3fd5d4a3cd3429a3b6cfd8cb8e38b7ab"}
Sep 16 12:48:05.740410 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:48:05.740002    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tztx9" event={"ID":"145070ab-abd8-4e8c-8cdd-99bfbbadbe1c","Type":"ContainerStarted","Data":"a042bfe1f3c1caf1930001e7cd27e062b674eafa8b4a04c241f657904fdc56a5"}
Sep 16 12:48:05.761558 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:48:05.761506    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tztx9" podStartSLOduration=4.6058702910000004 podStartE2EDuration="23.761471875s" totalImagesPullingTime="19.155601584s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-16 12:47:42 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 12:48:05.760745702 +0000 UTC m=+24.683618171" watchObservedRunningTime="2026-09-16 12:48:05.761471875 +0000 UTC m=+24.684344344"
Sep 16 12:48:06.594746 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:48:06.594711    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5jkh4"
Sep 16 12:48:06.594963 ip-10-0-133-218 kubenswrapper[2580]: E0916 12:48:06.594845    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-5jkh4" podUID="fea676c8-6e68-4894-99fc-fe8b59886a9a"
Sep 16 12:48:06.600817 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:48:06.600787    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-p6wht"
Sep 16 12:48:06.601415 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:48:06.601399    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-p6wht"
Sep 16 12:48:06.742929 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:48:06.742889    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-p6wht"
Sep 16 12:48:06.743621 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:48:06.743412    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-p6wht"
Sep 16 12:48:07.594304 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:48:07.594270    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-bpv5d"
Sep 16 12:48:07.594503 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:48:07.594270    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-62j7x"
Sep 16 12:48:07.594503 ip-10-0-133-218 kubenswrapper[2580]: E0916 12:48:07.594405    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-bpv5d" podUID="1b26fbcb-3523-4f7c-9e42-5b3653456abc"
Sep 16 12:48:07.594591 ip-10-0-133-218 kubenswrapper[2580]: E0916 12:48:07.594506    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-62j7x" podUID="7d634420-2900-42b3-9609-3fb64baff8f3"
Sep 16 12:48:08.594947 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:48:08.594772    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5jkh4"
Sep 16 12:48:08.595403 ip-10-0-133-218 kubenswrapper[2580]: E0916 12:48:08.595030    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-5jkh4" podUID="fea676c8-6e68-4894-99fc-fe8b59886a9a"
Sep 16 12:48:08.749775 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:48:08.749744    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-894fk" event={"ID":"ffd98c6e-f973-4bd8-94ba-bd92a16d1acd","Type":"ContainerStarted","Data":"07c1fe57397d63d8731b850128db160144b917a45f4939b6f90387777bb1547a"}
Sep 16 12:48:08.749990 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:48:08.749953    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-894fk"
Sep 16 12:48:08.750117 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:48:08.750041    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-894fk"
Sep 16 12:48:08.750117 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:48:08.750082    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-894fk"
Sep 16 12:48:08.751895 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:48:08.751873    2580 generic.go:356] "Generic (PLEG): container finished" podID="7b900f8b-a992-44b3-a6ea-7bacb17d2798" containerID="00e46368e1636804d3f039bd4600ee84b99de1f1413142b076d7ea8f5e7d1b6e" exitCode=0
Sep 16 12:48:08.751988 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:48:08.751951    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-hsj4z" event={"ID":"7b900f8b-a992-44b3-a6ea-7bacb17d2798","Type":"ContainerDied","Data":"00e46368e1636804d3f039bd4600ee84b99de1f1413142b076d7ea8f5e7d1b6e"}
Sep 16 12:48:08.766918 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:48:08.766888    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-894fk"
Sep 16 12:48:08.767735 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:48:08.767716    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-894fk"
Sep 16 12:48:08.779065 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:48:08.778999    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-894fk" podStartSLOduration=7.41963707 podStartE2EDuration="26.778988372s" totalImagesPullingTime="19.359351302s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-16 12:47:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 12:48:08.778481396 +0000 UTC m=+27.701353864" watchObservedRunningTime="2026-09-16 12:48:08.778988372 +0000 UTC m=+27.701860840"
Sep 16 12:48:09.599079 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:48:09.597788    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-bpv5d"
Sep 16 12:48:09.599079 ip-10-0-133-218 kubenswrapper[2580]: E0916 12:48:09.598070    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-bpv5d" podUID="1b26fbcb-3523-4f7c-9e42-5b3653456abc"
Sep 16 12:48:09.599079 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:48:09.598176    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-62j7x"
Sep 16 12:48:09.599079 ip-10-0-133-218 kubenswrapper[2580]: E0916 12:48:09.598277    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-62j7x" podUID="7d634420-2900-42b3-9609-3fb64baff8f3"
Sep 16 12:48:09.756756 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:48:09.756716    2580 generic.go:356] "Generic (PLEG): container finished" podID="7b900f8b-a992-44b3-a6ea-7bacb17d2798" containerID="52c082afb3ae33be217a3013c40be8c98b0fa9751b881b0dd91422b79a6b93ea" exitCode=0
Sep 16 12:48:09.756890 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:48:09.756797    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-hsj4z" event={"ID":"7b900f8b-a992-44b3-a6ea-7bacb17d2798","Type":"ContainerDied","Data":"52c082afb3ae33be217a3013c40be8c98b0fa9751b881b0dd91422b79a6b93ea"}
Sep 16 12:48:10.222749 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:48:10.222577    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-5jkh4"]
Sep 16 12:48:10.222897 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:48:10.222833    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5jkh4"
Sep 16 12:48:10.222979 ip-10-0-133-218 kubenswrapper[2580]: E0916 12:48:10.222929    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-5jkh4" podUID="fea676c8-6e68-4894-99fc-fe8b59886a9a"
Sep 16 12:48:10.224688 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:48:10.224660    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-62j7x"]
Sep 16 12:48:10.224807 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:48:10.224740    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-62j7x"
Sep 16 12:48:10.224852 ip-10-0-133-218 kubenswrapper[2580]: E0916 12:48:10.224818    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-62j7x" podUID="7d634420-2900-42b3-9609-3fb64baff8f3"
Sep 16 12:48:10.240068 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:48:10.239995    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-bpv5d"]
Sep 16 12:48:10.240160 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:48:10.240084    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-bpv5d"
Sep 16 12:48:10.240193 ip-10-0-133-218 kubenswrapper[2580]: E0916 12:48:10.240174    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-bpv5d" podUID="1b26fbcb-3523-4f7c-9e42-5b3653456abc"
Sep 16 12:48:10.761383 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:48:10.761339    2580 generic.go:356] "Generic (PLEG): container finished" podID="7b900f8b-a992-44b3-a6ea-7bacb17d2798" containerID="269213b2f0b7104cde64c2bfb6482ea0808a1006645c620be7f130212f39a008" exitCode=0
Sep 16 12:48:10.761855 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:48:10.761402    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-hsj4z" event={"ID":"7b900f8b-a992-44b3-a6ea-7bacb17d2798","Type":"ContainerDied","Data":"269213b2f0b7104cde64c2bfb6482ea0808a1006645c620be7f130212f39a008"}
Sep 16 12:48:11.594961 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:48:11.594923    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-62j7x"
Sep 16 12:48:11.595124 ip-10-0-133-218 kubenswrapper[2580]: E0916 12:48:11.595049    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-62j7x" podUID="7d634420-2900-42b3-9609-3fb64baff8f3"
Sep 16 12:48:11.595124 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:48:11.595103    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-bpv5d"
Sep 16 12:48:11.595230 ip-10-0-133-218 kubenswrapper[2580]: E0916 12:48:11.595211    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-bpv5d" podUID="1b26fbcb-3523-4f7c-9e42-5b3653456abc"
Sep 16 12:48:12.594926 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:48:12.594841    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5jkh4"
Sep 16 12:48:12.595393 ip-10-0-133-218 kubenswrapper[2580]: E0916 12:48:12.594973    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-5jkh4" podUID="fea676c8-6e68-4894-99fc-fe8b59886a9a"
Sep 16 12:48:13.597933 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:48:13.597905    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-bpv5d"
Sep 16 12:48:13.598339 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:48:13.597915    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-62j7x"
Sep 16 12:48:13.598339 ip-10-0-133-218 kubenswrapper[2580]: E0916 12:48:13.598024    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-bpv5d" podUID="1b26fbcb-3523-4f7c-9e42-5b3653456abc"
Sep 16 12:48:13.598339 ip-10-0-133-218 kubenswrapper[2580]: E0916 12:48:13.598086    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-62j7x" podUID="7d634420-2900-42b3-9609-3fb64baff8f3"
Sep 16 12:48:14.594517 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:48:14.594473    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5jkh4"
Sep 16 12:48:14.594688 ip-10-0-133-218 kubenswrapper[2580]: E0916 12:48:14.594595    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-5jkh4" podUID="fea676c8-6e68-4894-99fc-fe8b59886a9a"
Sep 16 12:48:15.251135 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:48:15.251094    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7d634420-2900-42b3-9609-3fb64baff8f3-metrics-certs\") pod \"network-metrics-daemon-62j7x\" (UID: \"7d634420-2900-42b3-9609-3fb64baff8f3\") " pod="openshift-multus/network-metrics-daemon-62j7x"
Sep 16 12:48:15.251563 ip-10-0-133-218 kubenswrapper[2580]: E0916 12:48:15.251277    2580 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 16 12:48:15.251563 ip-10-0-133-218 kubenswrapper[2580]: E0916 12:48:15.251349    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7d634420-2900-42b3-9609-3fb64baff8f3-metrics-certs podName:7d634420-2900-42b3-9609-3fb64baff8f3 nodeName:}" failed. No retries permitted until 2026-09-16 12:48:47.251328497 +0000 UTC m=+66.174200956 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/7d634420-2900-42b3-9609-3fb64baff8f3-metrics-certs") pod "network-metrics-daemon-62j7x" (UID: "7d634420-2900-42b3-9609-3fb64baff8f3") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 16 12:48:15.352316 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:48:15.352255    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/fea676c8-6e68-4894-99fc-fe8b59886a9a-original-pull-secret\") pod \"global-pull-secret-syncer-5jkh4\" (UID: \"fea676c8-6e68-4894-99fc-fe8b59886a9a\") " pod="kube-system/global-pull-secret-syncer-5jkh4"
Sep 16 12:48:15.352316 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:48:15.352300    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v4r5z\" (UniqueName: \"kubernetes.io/projected/1b26fbcb-3523-4f7c-9e42-5b3653456abc-kube-api-access-v4r5z\") pod \"network-check-target-bpv5d\" (UID: \"1b26fbcb-3523-4f7c-9e42-5b3653456abc\") " pod="openshift-network-diagnostics/network-check-target-bpv5d"
Sep 16 12:48:15.352474 ip-10-0-133-218 kubenswrapper[2580]: E0916 12:48:15.352420    2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 16 12:48:15.352474 ip-10-0-133-218 kubenswrapper[2580]: E0916 12:48:15.352435    2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 16 12:48:15.352474 ip-10-0-133-218 kubenswrapper[2580]: E0916 12:48:15.352445    2580 projected.go:196] Error preparing data for projected volume kube-api-access-v4r5z for pod openshift-network-diagnostics/network-check-target-bpv5d: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 16 12:48:15.352607 ip-10-0-133-218 kubenswrapper[2580]: E0916 12:48:15.352419    2580 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 16 12:48:15.352607 ip-10-0-133-218 kubenswrapper[2580]: E0916 12:48:15.352519    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/1b26fbcb-3523-4f7c-9e42-5b3653456abc-kube-api-access-v4r5z podName:1b26fbcb-3523-4f7c-9e42-5b3653456abc nodeName:}" failed. No retries permitted until 2026-09-16 12:48:47.352480288 +0000 UTC m=+66.275352736 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-v4r5z" (UniqueName: "kubernetes.io/projected/1b26fbcb-3523-4f7c-9e42-5b3653456abc-kube-api-access-v4r5z") pod "network-check-target-bpv5d" (UID: "1b26fbcb-3523-4f7c-9e42-5b3653456abc") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 16 12:48:15.352607 ip-10-0-133-218 kubenswrapper[2580]: E0916 12:48:15.352561    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fea676c8-6e68-4894-99fc-fe8b59886a9a-original-pull-secret podName:fea676c8-6e68-4894-99fc-fe8b59886a9a nodeName:}" failed. No retries permitted until 2026-09-16 12:48:47.352534313 +0000 UTC m=+66.275406768 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/fea676c8-6e68-4894-99fc-fe8b59886a9a-original-pull-secret") pod "global-pull-secret-syncer-5jkh4" (UID: "fea676c8-6e68-4894-99fc-fe8b59886a9a") : object "kube-system"/"original-pull-secret" not registered
Sep 16 12:48:15.594779 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:48:15.594745    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-62j7x"
Sep 16 12:48:15.594969 ip-10-0-133-218 kubenswrapper[2580]: E0916 12:48:15.594867    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-62j7x" podUID="7d634420-2900-42b3-9609-3fb64baff8f3"
Sep 16 12:48:15.594969 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:48:15.594931    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-bpv5d"
Sep 16 12:48:15.595057 ip-10-0-133-218 kubenswrapper[2580]: E0916 12:48:15.595036    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-bpv5d" podUID="1b26fbcb-3523-4f7c-9e42-5b3653456abc"
Sep 16 12:48:15.776087 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:48:15.776059    2580 generic.go:356] "Generic (PLEG): container finished" podID="7b900f8b-a992-44b3-a6ea-7bacb17d2798" containerID="4a714d078cf7da423827907423f6396702e7831f07eba73eb21ff7e4d7a84392" exitCode=0
Sep 16 12:48:15.776226 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:48:15.776098    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-hsj4z" event={"ID":"7b900f8b-a992-44b3-a6ea-7bacb17d2798","Type":"ContainerDied","Data":"4a714d078cf7da423827907423f6396702e7831f07eba73eb21ff7e4d7a84392"}
Sep 16 12:48:15.860153 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:48:15.860131    2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-218.ec2.internal" event="NodeReady"
Sep 16 12:48:15.860299 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:48:15.860248    2580 kubelet_node_status.go:489] "Fast updating node status as it just became ready"
Sep 16 12:48:15.922163 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:48:15.922138    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-hzdmm"]
Sep 16 12:48:15.938366 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:48:15.938347    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-d4swz"]
Sep 16 12:48:15.938502 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:48:15.938471    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-hzdmm"
Sep 16 12:48:15.940782 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:48:15.940762    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Sep 16 12:48:15.940975 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:48:15.940962    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-dfdpm\""
Sep 16 12:48:15.941283 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:48:15.941265    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Sep 16 12:48:15.952728 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:48:15.952711    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-hzdmm"]
Sep 16 12:48:15.952820 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:48:15.952749    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-d4swz"]
Sep 16 12:48:15.952879 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:48:15.952854    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-d4swz"
Sep 16 12:48:15.955312 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:48:15.955286    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Sep 16 12:48:15.955312 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:48:15.955289    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-8lxmf\""
Sep 16 12:48:15.955462 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:48:15.955348    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Sep 16 12:48:15.955813 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:48:15.955797    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Sep 16 12:48:16.058054 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:48:16.058035    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kwjsn\" (UniqueName: \"kubernetes.io/projected/b352b0f0-afd0-48c2-965f-7cc63cdf04fd-kube-api-access-kwjsn\") pod \"dns-default-hzdmm\" (UID: \"b352b0f0-afd0-48c2-965f-7cc63cdf04fd\") " pod="openshift-dns/dns-default-hzdmm"
Sep 16 12:48:16.058162 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:48:16.058066    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/23d2db14-6f1e-4198-8b68-022153a628d5-cert\") pod \"ingress-canary-d4swz\" (UID: \"23d2db14-6f1e-4198-8b68-022153a628d5\") " pod="openshift-ingress-canary/ingress-canary-d4swz"
Sep 16 12:48:16.058162 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:48:16.058109    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b352b0f0-afd0-48c2-965f-7cc63cdf04fd-tmp-dir\") pod \"dns-default-hzdmm\" (UID: \"b352b0f0-afd0-48c2-965f-7cc63cdf04fd\") " pod="openshift-dns/dns-default-hzdmm"
Sep 16 12:48:16.058162 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:48:16.058139    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rq4rl\" (UniqueName: \"kubernetes.io/projected/23d2db14-6f1e-4198-8b68-022153a628d5-kube-api-access-rq4rl\") pod \"ingress-canary-d4swz\" (UID: \"23d2db14-6f1e-4198-8b68-022153a628d5\") " pod="openshift-ingress-canary/ingress-canary-d4swz"
Sep 16 12:48:16.058264 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:48:16.058165    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/b352b0f0-afd0-48c2-965f-7cc63cdf04fd-metrics-tls\") pod \"dns-default-hzdmm\" (UID: \"b352b0f0-afd0-48c2-965f-7cc63cdf04fd\") " pod="openshift-dns/dns-default-hzdmm"
Sep 16 12:48:16.058264 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:48:16.058223    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/b352b0f0-afd0-48c2-965f-7cc63cdf04fd-config-volume\") pod \"dns-default-hzdmm\" (UID: \"b352b0f0-afd0-48c2-965f-7cc63cdf04fd\") " pod="openshift-dns/dns-default-hzdmm"
Sep 16 12:48:16.159384 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:48:16.159353    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/b352b0f0-afd0-48c2-965f-7cc63cdf04fd-metrics-tls\") pod \"dns-default-hzdmm\" (UID: \"b352b0f0-afd0-48c2-965f-7cc63cdf04fd\") " pod="openshift-dns/dns-default-hzdmm"
Sep 16 12:48:16.159551 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:48:16.159401    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/b352b0f0-afd0-48c2-965f-7cc63cdf04fd-config-volume\") pod \"dns-default-hzdmm\" (UID: \"b352b0f0-afd0-48c2-965f-7cc63cdf04fd\") " pod="openshift-dns/dns-default-hzdmm"
Sep 16 12:48:16.159551 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:48:16.159434    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kwjsn\" (UniqueName: \"kubernetes.io/projected/b352b0f0-afd0-48c2-965f-7cc63cdf04fd-kube-api-access-kwjsn\") pod \"dns-default-hzdmm\" (UID: \"b352b0f0-afd0-48c2-965f-7cc63cdf04fd\") " pod="openshift-dns/dns-default-hzdmm"
Sep 16 12:48:16.159551 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:48:16.159462    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/23d2db14-6f1e-4198-8b68-022153a628d5-cert\") pod \"ingress-canary-d4swz\" (UID: \"23d2db14-6f1e-4198-8b68-022153a628d5\") " pod="openshift-ingress-canary/ingress-canary-d4swz"
Sep 16 12:48:16.159551 ip-10-0-133-218 kubenswrapper[2580]: E0916 12:48:16.159520    2580 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 16 12:48:16.159732 ip-10-0-133-218 kubenswrapper[2580]: E0916 12:48:16.159597    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b352b0f0-afd0-48c2-965f-7cc63cdf04fd-metrics-tls podName:b352b0f0-afd0-48c2-965f-7cc63cdf04fd nodeName:}" failed. No retries permitted until 2026-09-16 12:48:16.659575324 +0000 UTC m=+35.582447808 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/b352b0f0-afd0-48c2-965f-7cc63cdf04fd-metrics-tls") pod "dns-default-hzdmm" (UID: "b352b0f0-afd0-48c2-965f-7cc63cdf04fd") : secret "dns-default-metrics-tls" not found
Sep 16 12:48:16.159732 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:48:16.159524    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b352b0f0-afd0-48c2-965f-7cc63cdf04fd-tmp-dir\") pod \"dns-default-hzdmm\" (UID: \"b352b0f0-afd0-48c2-965f-7cc63cdf04fd\") " pod="openshift-dns/dns-default-hzdmm"
Sep 16 12:48:16.159732 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:48:16.159698    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rq4rl\" (UniqueName: \"kubernetes.io/projected/23d2db14-6f1e-4198-8b68-022153a628d5-kube-api-access-rq4rl\") pod \"ingress-canary-d4swz\" (UID: \"23d2db14-6f1e-4198-8b68-022153a628d5\") " pod="openshift-ingress-canary/ingress-canary-d4swz"
Sep 16 12:48:16.160071 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:48:16.160052    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b352b0f0-afd0-48c2-965f-7cc63cdf04fd-tmp-dir\") pod \"dns-default-hzdmm\" (UID: \"b352b0f0-afd0-48c2-965f-7cc63cdf04fd\") " pod="openshift-dns/dns-default-hzdmm"
Sep 16 12:48:16.160328 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:48:16.160189    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/b352b0f0-afd0-48c2-965f-7cc63cdf04fd-config-volume\") pod \"dns-default-hzdmm\" (UID: \"b352b0f0-afd0-48c2-965f-7cc63cdf04fd\") " pod="openshift-dns/dns-default-hzdmm"
Sep 16 12:48:16.163447 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:48:16.163420    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/23d2db14-6f1e-4198-8b68-022153a628d5-cert\") pod \"ingress-canary-d4swz\" (UID: \"23d2db14-6f1e-4198-8b68-022153a628d5\") " pod="openshift-ingress-canary/ingress-canary-d4swz"
Sep 16 12:48:16.172101 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:48:16.172075    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kwjsn\" (UniqueName: \"kubernetes.io/projected/b352b0f0-afd0-48c2-965f-7cc63cdf04fd-kube-api-access-kwjsn\") pod \"dns-default-hzdmm\" (UID: \"b352b0f0-afd0-48c2-965f-7cc63cdf04fd\") " pod="openshift-dns/dns-default-hzdmm"
Sep 16 12:48:16.172244 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:48:16.172227    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rq4rl\" (UniqueName: \"kubernetes.io/projected/23d2db14-6f1e-4198-8b68-022153a628d5-kube-api-access-rq4rl\") pod \"ingress-canary-d4swz\" (UID: \"23d2db14-6f1e-4198-8b68-022153a628d5\") " pod="openshift-ingress-canary/ingress-canary-d4swz"
Sep 16 12:48:16.307244 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:48:16.307204    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-d4swz"
Sep 16 12:48:16.594619 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:48:16.594544    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5jkh4"
Sep 16 12:48:16.597381 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:48:16.597362    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Sep 16 12:48:16.622129 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:48:16.622105    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-d4swz"]
Sep 16 12:48:16.625188 ip-10-0-133-218 kubenswrapper[2580]: W0916 12:48:16.625162    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod23d2db14_6f1e_4198_8b68_022153a628d5.slice/crio-51b525355e0aeb8d1378e77b0ed60192220d5942c5b1adc5fcc68877d38542e8 WatchSource:0}: Error finding container 51b525355e0aeb8d1378e77b0ed60192220d5942c5b1adc5fcc68877d38542e8: Status 404 returned error can't find the container with id 51b525355e0aeb8d1378e77b0ed60192220d5942c5b1adc5fcc68877d38542e8
Sep 16 12:48:16.663788 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:48:16.663758    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/b352b0f0-afd0-48c2-965f-7cc63cdf04fd-metrics-tls\") pod \"dns-default-hzdmm\" (UID: \"b352b0f0-afd0-48c2-965f-7cc63cdf04fd\") " pod="openshift-dns/dns-default-hzdmm"
Sep 16 12:48:16.663918 ip-10-0-133-218 kubenswrapper[2580]: E0916 12:48:16.663893    2580 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 16 12:48:16.663963 ip-10-0-133-218 kubenswrapper[2580]: E0916 12:48:16.663949    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b352b0f0-afd0-48c2-965f-7cc63cdf04fd-metrics-tls podName:b352b0f0-afd0-48c2-965f-7cc63cdf04fd nodeName:}" failed. No retries permitted until 2026-09-16 12:48:17.663933764 +0000 UTC m=+36.586806211 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/b352b0f0-afd0-48c2-965f-7cc63cdf04fd-metrics-tls") pod "dns-default-hzdmm" (UID: "b352b0f0-afd0-48c2-965f-7cc63cdf04fd") : secret "dns-default-metrics-tls" not found
Sep 16 12:48:16.781711 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:48:16.781674    2580 generic.go:356] "Generic (PLEG): container finished" podID="7b900f8b-a992-44b3-a6ea-7bacb17d2798" containerID="25950a06a67856f7b601b498f7d90a66099360bc9ae906c115081e115037514c" exitCode=0
Sep 16 12:48:16.781711 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:48:16.781706    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-hsj4z" event={"ID":"7b900f8b-a992-44b3-a6ea-7bacb17d2798","Type":"ContainerDied","Data":"25950a06a67856f7b601b498f7d90a66099360bc9ae906c115081e115037514c"}
Sep 16 12:48:16.782912 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:48:16.782886    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-d4swz" event={"ID":"23d2db14-6f1e-4198-8b68-022153a628d5","Type":"ContainerStarted","Data":"51b525355e0aeb8d1378e77b0ed60192220d5942c5b1adc5fcc68877d38542e8"}
Sep 16 12:48:17.598416 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:48:17.598384    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-bpv5d"
Sep 16 12:48:17.599382 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:48:17.598510    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-62j7x"
Sep 16 12:48:17.602186 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:48:17.602019    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-m2lxw\""
Sep 16 12:48:17.602186 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:48:17.602027    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Sep 16 12:48:17.602186 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:48:17.602059    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Sep 16 12:48:17.602186 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:48:17.602057    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-9nrdj\""
Sep 16 12:48:17.602186 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:48:17.602027    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Sep 16 12:48:17.670138 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:48:17.670107    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/b352b0f0-afd0-48c2-965f-7cc63cdf04fd-metrics-tls\") pod \"dns-default-hzdmm\" (UID: \"b352b0f0-afd0-48c2-965f-7cc63cdf04fd\") " pod="openshift-dns/dns-default-hzdmm"
Sep 16 12:48:17.670278 ip-10-0-133-218 kubenswrapper[2580]: E0916 12:48:17.670262    2580 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 16 12:48:17.670336 ip-10-0-133-218 kubenswrapper[2580]: E0916 12:48:17.670322    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b352b0f0-afd0-48c2-965f-7cc63cdf04fd-metrics-tls podName:b352b0f0-afd0-48c2-965f-7cc63cdf04fd nodeName:}" failed. No retries permitted until 2026-09-16 12:48:19.670306972 +0000 UTC m=+38.593179419 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/b352b0f0-afd0-48c2-965f-7cc63cdf04fd-metrics-tls") pod "dns-default-hzdmm" (UID: "b352b0f0-afd0-48c2-965f-7cc63cdf04fd") : secret "dns-default-metrics-tls" not found
Sep 16 12:48:17.789543 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:48:17.789507    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-hsj4z" event={"ID":"7b900f8b-a992-44b3-a6ea-7bacb17d2798","Type":"ContainerStarted","Data":"29564f78e1913e91b2b535123ac4b66f3bc3cd316cc45e719482f1ed29e914a2"}
Sep 16 12:48:17.814618 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:48:17.814563    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-hsj4z" podStartSLOduration=7.904986965 podStartE2EDuration="35.814545469s" totalImagesPullingTime="27.909558504s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-16 12:47:42 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 12:48:17.813637342 +0000 UTC m=+36.736509811" watchObservedRunningTime="2026-09-16 12:48:17.814545469 +0000 UTC m=+36.737417937"
Sep 16 12:48:19.687062 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:48:19.686854    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/b352b0f0-afd0-48c2-965f-7cc63cdf04fd-metrics-tls\") pod \"dns-default-hzdmm\" (UID: \"b352b0f0-afd0-48c2-965f-7cc63cdf04fd\") " pod="openshift-dns/dns-default-hzdmm"
Sep 16 12:48:19.687446 ip-10-0-133-218 kubenswrapper[2580]: E0916 12:48:19.687000    2580 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 16 12:48:19.687446 ip-10-0-133-218 kubenswrapper[2580]: E0916 12:48:19.687194    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b352b0f0-afd0-48c2-965f-7cc63cdf04fd-metrics-tls podName:b352b0f0-afd0-48c2-965f-7cc63cdf04fd nodeName:}" failed. No retries permitted until 2026-09-16 12:48:23.687176817 +0000 UTC m=+42.610049278 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/b352b0f0-afd0-48c2-965f-7cc63cdf04fd-metrics-tls") pod "dns-default-hzdmm" (UID: "b352b0f0-afd0-48c2-965f-7cc63cdf04fd") : secret "dns-default-metrics-tls" not found
Sep 16 12:48:19.795536 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:48:19.795502    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-d4swz" event={"ID":"23d2db14-6f1e-4198-8b68-022153a628d5","Type":"ContainerStarted","Data":"ee6291e39bf8e7af8de05dd6d03b156b8f5cef72323df1aa9b66b4d15de4fac6"}
Sep 16 12:48:23.713278 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:48:23.713238    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/b352b0f0-afd0-48c2-965f-7cc63cdf04fd-metrics-tls\") pod \"dns-default-hzdmm\" (UID: \"b352b0f0-afd0-48c2-965f-7cc63cdf04fd\") " pod="openshift-dns/dns-default-hzdmm"
Sep 16 12:48:23.713742 ip-10-0-133-218 kubenswrapper[2580]: E0916 12:48:23.713412    2580 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 16 12:48:23.713742 ip-10-0-133-218 kubenswrapper[2580]: E0916 12:48:23.713515    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b352b0f0-afd0-48c2-965f-7cc63cdf04fd-metrics-tls podName:b352b0f0-afd0-48c2-965f-7cc63cdf04fd nodeName:}" failed. No retries permitted until 2026-09-16 12:48:31.713473796 +0000 UTC m=+50.636346474 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/b352b0f0-afd0-48c2-965f-7cc63cdf04fd-metrics-tls") pod "dns-default-hzdmm" (UID: "b352b0f0-afd0-48c2-965f-7cc63cdf04fd") : secret "dns-default-metrics-tls" not found
Sep 16 12:48:31.767333 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:48:31.767289    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/b352b0f0-afd0-48c2-965f-7cc63cdf04fd-metrics-tls\") pod \"dns-default-hzdmm\" (UID: \"b352b0f0-afd0-48c2-965f-7cc63cdf04fd\") " pod="openshift-dns/dns-default-hzdmm"
Sep 16 12:48:31.767824 ip-10-0-133-218 kubenswrapper[2580]: E0916 12:48:31.767457    2580 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 16 12:48:31.767824 ip-10-0-133-218 kubenswrapper[2580]: E0916 12:48:31.767556    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b352b0f0-afd0-48c2-965f-7cc63cdf04fd-metrics-tls podName:b352b0f0-afd0-48c2-965f-7cc63cdf04fd nodeName:}" failed. No retries permitted until 2026-09-16 12:48:47.767534738 +0000 UTC m=+66.690407185 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/b352b0f0-afd0-48c2-965f-7cc63cdf04fd-metrics-tls") pod "dns-default-hzdmm" (UID: "b352b0f0-afd0-48c2-965f-7cc63cdf04fd") : secret "dns-default-metrics-tls" not found
Sep 16 12:48:40.771462 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:48:40.771434    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-894fk"
Sep 16 12:48:40.805245 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:48:40.805190    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-d4swz" podStartSLOduration=23.338685692 podStartE2EDuration="25.805176515s" totalImagesPullingTime="2.466490823s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 12:48:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 12:48:19.813345136 +0000 UTC m=+38.736217605" watchObservedRunningTime="2026-09-16 12:48:40.805176515 +0000 UTC m=+59.728048984"
Sep 16 12:48:41.541298 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:48:41.541265    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/router-default-5b4dd555d4-jwzh7"]
Sep 16 12:48:41.543774 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:48:41.543759    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-5b4dd555d4-jwzh7"
Sep 16 12:48:41.550398 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:48:41.550381    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-metrics-certs-default\""
Sep 16 12:48:41.550963 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:48:41.550945    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"service-ca-bundle\""
Sep 16 12:48:41.551015 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:48:41.550953    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-dockercfg-qwtlh\""
Sep 16 12:48:41.551638 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:48:41.551622    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\""
Sep 16 12:48:41.551687 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:48:41.551635    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\""
Sep 16 12:48:41.553380 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:48:41.553366    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-stats-default\""
Sep 16 12:48:41.553703 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:48:41.553690    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"default-ingress-cert\""
Sep 16 12:48:41.565892 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:48:41.565872    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/router-default-5b4dd555d4-jwzh7"]
Sep 16 12:48:41.637098 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:48:41.637057    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/3c9aacb3-1142-4610-b990-a28d15998e71-stats-auth\") pod \"router-default-5b4dd555d4-jwzh7\" (UID: \"3c9aacb3-1142-4610-b990-a28d15998e71\") " pod="openshift-ingress/router-default-5b4dd555d4-jwzh7"
Sep 16 12:48:41.637337 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:48:41.637109    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3c9aacb3-1142-4610-b990-a28d15998e71-metrics-certs\") pod \"router-default-5b4dd555d4-jwzh7\" (UID: \"3c9aacb3-1142-4610-b990-a28d15998e71\") " pod="openshift-ingress/router-default-5b4dd555d4-jwzh7"
Sep 16 12:48:41.637337 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:48:41.637183    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cgwq6\" (UniqueName: \"kubernetes.io/projected/3c9aacb3-1142-4610-b990-a28d15998e71-kube-api-access-cgwq6\") pod \"router-default-5b4dd555d4-jwzh7\" (UID: \"3c9aacb3-1142-4610-b990-a28d15998e71\") " pod="openshift-ingress/router-default-5b4dd555d4-jwzh7"
Sep 16 12:48:41.637337 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:48:41.637231    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3c9aacb3-1142-4610-b990-a28d15998e71-service-ca-bundle\") pod \"router-default-5b4dd555d4-jwzh7\" (UID: \"3c9aacb3-1142-4610-b990-a28d15998e71\") " pod="openshift-ingress/router-default-5b4dd555d4-jwzh7"
Sep 16 12:48:41.637337 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:48:41.637266    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/3c9aacb3-1142-4610-b990-a28d15998e71-default-certificate\") pod \"router-default-5b4dd555d4-jwzh7\" (UID: \"3c9aacb3-1142-4610-b990-a28d15998e71\") " pod="openshift-ingress/router-default-5b4dd555d4-jwzh7"
Sep 16 12:48:41.650781 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:48:41.650743    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-86695b7cc7-zf46w"]
Sep 16 12:48:41.652832 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:48:41.652807    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-xvxlf"]
Sep 16 12:48:41.653256 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:48:41.652984    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-86695b7cc7-zf46w"
Sep 16 12:48:41.654766 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:48:41.654748    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-xvxlf"
Sep 16 12:48:41.656337 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:48:41.656317    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"kube-root-ca.crt\""
Sep 16 12:48:41.661808 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:48:41.661792    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"openshift-service-ca.crt\""
Sep 16 12:48:41.662768 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:48:41.662755    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-storage-operator\"/\"volume-data-source-validator-dockercfg-hnh7s\""
Sep 16 12:48:41.683268 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:48:41.683246    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"openshift-service-ca.crt\""
Sep 16 12:48:41.700546 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:48:41.700524    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"samples-operator-tls\""
Sep 16 12:48:41.700641 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:48:41.700566    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"kube-root-ca.crt\""
Sep 16 12:48:41.700805 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:48:41.700790    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"cluster-samples-operator-dockercfg-dh584\""
Sep 16 12:48:41.707157 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:48:41.707137    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-xvxlf"]
Sep 16 12:48:41.708665 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:48:41.708647    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-86695b7cc7-zf46w"]
Sep 16 12:48:41.738501 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:48:41.738461    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3c9aacb3-1142-4610-b990-a28d15998e71-metrics-certs\") pod \"router-default-5b4dd555d4-jwzh7\" (UID: \"3c9aacb3-1142-4610-b990-a28d15998e71\") " pod="openshift-ingress/router-default-5b4dd555d4-jwzh7"
Sep 16 12:48:41.738627 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:48:41.738513    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cgwq6\" (UniqueName: \"kubernetes.io/projected/3c9aacb3-1142-4610-b990-a28d15998e71-kube-api-access-cgwq6\") pod \"router-default-5b4dd555d4-jwzh7\" (UID: \"3c9aacb3-1142-4610-b990-a28d15998e71\") " pod="openshift-ingress/router-default-5b4dd555d4-jwzh7"
Sep 16 12:48:41.738627 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:48:41.738535    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ftt8k\" (UniqueName: \"kubernetes.io/projected/5abef85b-1f8d-4f0e-9091-c0f42acf1af1-kube-api-access-ftt8k\") pod \"cluster-samples-operator-86499c89c4-xvxlf\" (UID: \"5abef85b-1f8d-4f0e-9091-c0f42acf1af1\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-xvxlf"
Sep 16 12:48:41.738627 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:48:41.738587    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/5abef85b-1f8d-4f0e-9091-c0f42acf1af1-samples-operator-tls\") pod \"cluster-samples-operator-86499c89c4-xvxlf\" (UID: \"5abef85b-1f8d-4f0e-9091-c0f42acf1af1\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-xvxlf"
Sep 16 12:48:41.738627 ip-10-0-133-218 kubenswrapper[2580]: E0916 12:48:41.738614    2580 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Sep 16 12:48:41.738812 ip-10-0-133-218 kubenswrapper[2580]: E0916 12:48:41.738679    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3c9aacb3-1142-4610-b990-a28d15998e71-metrics-certs podName:3c9aacb3-1142-4610-b990-a28d15998e71 nodeName:}" failed. No retries permitted until 2026-09-16 12:48:42.23866318 +0000 UTC m=+61.161535626 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/3c9aacb3-1142-4610-b990-a28d15998e71-metrics-certs") pod "router-default-5b4dd555d4-jwzh7" (UID: "3c9aacb3-1142-4610-b990-a28d15998e71") : secret "router-metrics-certs-default" not found
Sep 16 12:48:41.738812 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:48:41.738760    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3c9aacb3-1142-4610-b990-a28d15998e71-service-ca-bundle\") pod \"router-default-5b4dd555d4-jwzh7\" (UID: \"3c9aacb3-1142-4610-b990-a28d15998e71\") " pod="openshift-ingress/router-default-5b4dd555d4-jwzh7"
Sep 16 12:48:41.738812 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:48:41.738807    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/3c9aacb3-1142-4610-b990-a28d15998e71-default-certificate\") pod \"router-default-5b4dd555d4-jwzh7\" (UID: \"3c9aacb3-1142-4610-b990-a28d15998e71\") " pod="openshift-ingress/router-default-5b4dd555d4-jwzh7"
Sep 16 12:48:41.738984 ip-10-0-133-218 kubenswrapper[2580]: E0916 12:48:41.738963    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/3c9aacb3-1142-4610-b990-a28d15998e71-service-ca-bundle podName:3c9aacb3-1142-4610-b990-a28d15998e71 nodeName:}" failed. No retries permitted until 2026-09-16 12:48:42.238949856 +0000 UTC m=+61.161822303 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/3c9aacb3-1142-4610-b990-a28d15998e71-service-ca-bundle") pod "router-default-5b4dd555d4-jwzh7" (UID: "3c9aacb3-1142-4610-b990-a28d15998e71") : configmap references non-existent config key: service-ca.crt
Sep 16 12:48:41.739040 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:48:41.738996    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-82tcz\" (UniqueName: \"kubernetes.io/projected/418cfec8-573d-461e-9a2a-766b421cfbe6-kube-api-access-82tcz\") pod \"volume-data-source-validator-86695b7cc7-zf46w\" (UID: \"418cfec8-573d-461e-9a2a-766b421cfbe6\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-86695b7cc7-zf46w"
Sep 16 12:48:41.739040 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:48:41.739020    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/3c9aacb3-1142-4610-b990-a28d15998e71-stats-auth\") pod \"router-default-5b4dd555d4-jwzh7\" (UID: \"3c9aacb3-1142-4610-b990-a28d15998e71\") " pod="openshift-ingress/router-default-5b4dd555d4-jwzh7"
Sep 16 12:48:41.741991 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:48:41.741971    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/3c9aacb3-1142-4610-b990-a28d15998e71-default-certificate\") pod \"router-default-5b4dd555d4-jwzh7\" (UID: \"3c9aacb3-1142-4610-b990-a28d15998e71\") " pod="openshift-ingress/router-default-5b4dd555d4-jwzh7"
Sep 16 12:48:41.742066 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:48:41.742028    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/3c9aacb3-1142-4610-b990-a28d15998e71-stats-auth\") pod \"router-default-5b4dd555d4-jwzh7\" (UID: \"3c9aacb3-1142-4610-b990-a28d15998e71\") " pod="openshift-ingress/router-default-5b4dd555d4-jwzh7"
Sep 16 12:48:41.771455 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:48:41.771430    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cgwq6\" (UniqueName: \"kubernetes.io/projected/3c9aacb3-1142-4610-b990-a28d15998e71-kube-api-access-cgwq6\") pod \"router-default-5b4dd555d4-jwzh7\" (UID: \"3c9aacb3-1142-4610-b990-a28d15998e71\") " pod="openshift-ingress/router-default-5b4dd555d4-jwzh7"
Sep 16 12:48:41.840177 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:48:41.840103    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/5abef85b-1f8d-4f0e-9091-c0f42acf1af1-samples-operator-tls\") pod \"cluster-samples-operator-86499c89c4-xvxlf\" (UID: \"5abef85b-1f8d-4f0e-9091-c0f42acf1af1\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-xvxlf"
Sep 16 12:48:41.840572 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:48:41.840202    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-82tcz\" (UniqueName: \"kubernetes.io/projected/418cfec8-573d-461e-9a2a-766b421cfbe6-kube-api-access-82tcz\") pod \"volume-data-source-validator-86695b7cc7-zf46w\" (UID: \"418cfec8-573d-461e-9a2a-766b421cfbe6\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-86695b7cc7-zf46w"
Sep 16 12:48:41.840572 ip-10-0-133-218 kubenswrapper[2580]: E0916 12:48:41.840240    2580 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Sep 16 12:48:41.840572 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:48:41.840245    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ftt8k\" (UniqueName: \"kubernetes.io/projected/5abef85b-1f8d-4f0e-9091-c0f42acf1af1-kube-api-access-ftt8k\") pod \"cluster-samples-operator-86499c89c4-xvxlf\" (UID: \"5abef85b-1f8d-4f0e-9091-c0f42acf1af1\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-xvxlf"
Sep 16 12:48:41.840572 ip-10-0-133-218 kubenswrapper[2580]: E0916 12:48:41.840296    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5abef85b-1f8d-4f0e-9091-c0f42acf1af1-samples-operator-tls podName:5abef85b-1f8d-4f0e-9091-c0f42acf1af1 nodeName:}" failed. No retries permitted until 2026-09-16 12:48:42.340281479 +0000 UTC m=+61.263153927 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/5abef85b-1f8d-4f0e-9091-c0f42acf1af1-samples-operator-tls") pod "cluster-samples-operator-86499c89c4-xvxlf" (UID: "5abef85b-1f8d-4f0e-9091-c0f42acf1af1") : secret "samples-operator-tls" not found
Sep 16 12:48:41.858310 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:48:41.858278    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ftt8k\" (UniqueName: \"kubernetes.io/projected/5abef85b-1f8d-4f0e-9091-c0f42acf1af1-kube-api-access-ftt8k\") pod \"cluster-samples-operator-86499c89c4-xvxlf\" (UID: \"5abef85b-1f8d-4f0e-9091-c0f42acf1af1\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-xvxlf"
Sep 16 12:48:41.858451 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:48:41.858429    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-82tcz\" (UniqueName: \"kubernetes.io/projected/418cfec8-573d-461e-9a2a-766b421cfbe6-kube-api-access-82tcz\") pod \"volume-data-source-validator-86695b7cc7-zf46w\" (UID: \"418cfec8-573d-461e-9a2a-766b421cfbe6\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-86695b7cc7-zf46w"
Sep 16 12:48:41.962385 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:48:41.962349    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-86695b7cc7-zf46w"
Sep 16 12:48:42.236590 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:48:42.236515    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-86695b7cc7-zf46w"]
Sep 16 12:48:42.239016 ip-10-0-133-218 kubenswrapper[2580]: W0916 12:48:42.238993    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod418cfec8_573d_461e_9a2a_766b421cfbe6.slice/crio-82ecc4b542825f52ecdf5f39b33a362123b489ec3ade4d89183f8417b7014eec WatchSource:0}: Error finding container 82ecc4b542825f52ecdf5f39b33a362123b489ec3ade4d89183f8417b7014eec: Status 404 returned error can't find the container with id 82ecc4b542825f52ecdf5f39b33a362123b489ec3ade4d89183f8417b7014eec
Sep 16 12:48:42.243584 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:48:42.243560    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3c9aacb3-1142-4610-b990-a28d15998e71-service-ca-bundle\") pod \"router-default-5b4dd555d4-jwzh7\" (UID: \"3c9aacb3-1142-4610-b990-a28d15998e71\") " pod="openshift-ingress/router-default-5b4dd555d4-jwzh7"
Sep 16 12:48:42.243657 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:48:42.243626    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3c9aacb3-1142-4610-b990-a28d15998e71-metrics-certs\") pod \"router-default-5b4dd555d4-jwzh7\" (UID: \"3c9aacb3-1142-4610-b990-a28d15998e71\") " pod="openshift-ingress/router-default-5b4dd555d4-jwzh7"
Sep 16 12:48:42.243745 ip-10-0-133-218 kubenswrapper[2580]: E0916 12:48:42.243731    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/3c9aacb3-1142-4610-b990-a28d15998e71-service-ca-bundle podName:3c9aacb3-1142-4610-b990-a28d15998e71 nodeName:}" failed. No retries permitted until 2026-09-16 12:48:43.243713811 +0000 UTC m=+62.166586257 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/3c9aacb3-1142-4610-b990-a28d15998e71-service-ca-bundle") pod "router-default-5b4dd555d4-jwzh7" (UID: "3c9aacb3-1142-4610-b990-a28d15998e71") : configmap references non-existent config key: service-ca.crt
Sep 16 12:48:42.243745 ip-10-0-133-218 kubenswrapper[2580]: E0916 12:48:42.243734    2580 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Sep 16 12:48:42.243814 ip-10-0-133-218 kubenswrapper[2580]: E0916 12:48:42.243767    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3c9aacb3-1142-4610-b990-a28d15998e71-metrics-certs podName:3c9aacb3-1142-4610-b990-a28d15998e71 nodeName:}" failed. No retries permitted until 2026-09-16 12:48:43.243759997 +0000 UTC m=+62.166632443 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/3c9aacb3-1142-4610-b990-a28d15998e71-metrics-certs") pod "router-default-5b4dd555d4-jwzh7" (UID: "3c9aacb3-1142-4610-b990-a28d15998e71") : secret "router-metrics-certs-default" not found
Sep 16 12:48:42.344166 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:48:42.344132    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/5abef85b-1f8d-4f0e-9091-c0f42acf1af1-samples-operator-tls\") pod \"cluster-samples-operator-86499c89c4-xvxlf\" (UID: \"5abef85b-1f8d-4f0e-9091-c0f42acf1af1\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-xvxlf"
Sep 16 12:48:42.344314 ip-10-0-133-218 kubenswrapper[2580]: E0916 12:48:42.344248    2580 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Sep 16 12:48:42.344314 ip-10-0-133-218 kubenswrapper[2580]: E0916 12:48:42.344297    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5abef85b-1f8d-4f0e-9091-c0f42acf1af1-samples-operator-tls podName:5abef85b-1f8d-4f0e-9091-c0f42acf1af1 nodeName:}" failed. No retries permitted until 2026-09-16 12:48:43.34428381 +0000 UTC m=+62.267156257 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/5abef85b-1f8d-4f0e-9091-c0f42acf1af1-samples-operator-tls") pod "cluster-samples-operator-86499c89c4-xvxlf" (UID: "5abef85b-1f8d-4f0e-9091-c0f42acf1af1") : secret "samples-operator-tls" not found
Sep 16 12:48:42.848646 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:48:42.848610    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-86695b7cc7-zf46w" event={"ID":"418cfec8-573d-461e-9a2a-766b421cfbe6","Type":"ContainerStarted","Data":"82ecc4b542825f52ecdf5f39b33a362123b489ec3ade4d89183f8417b7014eec"}
Sep 16 12:48:43.251816 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:48:43.251734    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3c9aacb3-1142-4610-b990-a28d15998e71-metrics-certs\") pod \"router-default-5b4dd555d4-jwzh7\" (UID: \"3c9aacb3-1142-4610-b990-a28d15998e71\") " pod="openshift-ingress/router-default-5b4dd555d4-jwzh7"
Sep 16 12:48:43.251968 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:48:43.251812    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3c9aacb3-1142-4610-b990-a28d15998e71-service-ca-bundle\") pod \"router-default-5b4dd555d4-jwzh7\" (UID: \"3c9aacb3-1142-4610-b990-a28d15998e71\") " pod="openshift-ingress/router-default-5b4dd555d4-jwzh7"
Sep 16 12:48:43.251968 ip-10-0-133-218 kubenswrapper[2580]: E0916 12:48:43.251871    2580 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Sep 16 12:48:43.251968 ip-10-0-133-218 kubenswrapper[2580]: E0916 12:48:43.251937    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3c9aacb3-1142-4610-b990-a28d15998e71-metrics-certs podName:3c9aacb3-1142-4610-b990-a28d15998e71 nodeName:}" failed. No retries permitted until 2026-09-16 12:48:45.251922136 +0000 UTC m=+64.174794583 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/3c9aacb3-1142-4610-b990-a28d15998e71-metrics-certs") pod "router-default-5b4dd555d4-jwzh7" (UID: "3c9aacb3-1142-4610-b990-a28d15998e71") : secret "router-metrics-certs-default" not found
Sep 16 12:48:43.251968 ip-10-0-133-218 kubenswrapper[2580]: E0916 12:48:43.251949    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/3c9aacb3-1142-4610-b990-a28d15998e71-service-ca-bundle podName:3c9aacb3-1142-4610-b990-a28d15998e71 nodeName:}" failed. No retries permitted until 2026-09-16 12:48:45.251943546 +0000 UTC m=+64.174815993 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/3c9aacb3-1142-4610-b990-a28d15998e71-service-ca-bundle") pod "router-default-5b4dd555d4-jwzh7" (UID: "3c9aacb3-1142-4610-b990-a28d15998e71") : configmap references non-existent config key: service-ca.crt
Sep 16 12:48:43.352726 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:48:43.352688    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/5abef85b-1f8d-4f0e-9091-c0f42acf1af1-samples-operator-tls\") pod \"cluster-samples-operator-86499c89c4-xvxlf\" (UID: \"5abef85b-1f8d-4f0e-9091-c0f42acf1af1\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-xvxlf"
Sep 16 12:48:43.352895 ip-10-0-133-218 kubenswrapper[2580]: E0916 12:48:43.352872    2580 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Sep 16 12:48:43.352964 ip-10-0-133-218 kubenswrapper[2580]: E0916 12:48:43.352953    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5abef85b-1f8d-4f0e-9091-c0f42acf1af1-samples-operator-tls podName:5abef85b-1f8d-4f0e-9091-c0f42acf1af1 nodeName:}" failed. No retries permitted until 2026-09-16 12:48:45.352930483 +0000 UTC m=+64.275802936 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/5abef85b-1f8d-4f0e-9091-c0f42acf1af1-samples-operator-tls") pod "cluster-samples-operator-86499c89c4-xvxlf" (UID: "5abef85b-1f8d-4f0e-9091-c0f42acf1af1") : secret "samples-operator-tls" not found
Sep 16 12:48:43.852697 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:48:43.852663    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-86695b7cc7-zf46w" event={"ID":"418cfec8-573d-461e-9a2a-766b421cfbe6","Type":"ContainerStarted","Data":"1b9a433932bffb83f4584fcf4b633be572bf903278d7615f8c40bbc200212ca2"}
Sep 16 12:48:43.874058 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:48:43.874007    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-storage-operator/volume-data-source-validator-86695b7cc7-zf46w" podStartSLOduration=1.5965659410000002 podStartE2EDuration="2.873991831s" totalImagesPullingTime="1.27742589s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 12:48:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 12:48:43.872838099 +0000 UTC m=+62.795710568" watchObservedRunningTime="2026-09-16 12:48:43.873991831 +0000 UTC m=+62.796864299"
Sep 16 12:48:45.268920 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:48:45.268876    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3c9aacb3-1142-4610-b990-a28d15998e71-service-ca-bundle\") pod \"router-default-5b4dd555d4-jwzh7\" (UID: \"3c9aacb3-1142-4610-b990-a28d15998e71\") " pod="openshift-ingress/router-default-5b4dd555d4-jwzh7"
Sep 16 12:48:45.269357 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:48:45.268955    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3c9aacb3-1142-4610-b990-a28d15998e71-metrics-certs\") pod \"router-default-5b4dd555d4-jwzh7\" (UID: \"3c9aacb3-1142-4610-b990-a28d15998e71\") " pod="openshift-ingress/router-default-5b4dd555d4-jwzh7"
Sep 16 12:48:45.269357 ip-10-0-133-218 kubenswrapper[2580]: E0916 12:48:45.269058    2580 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Sep 16 12:48:45.269357 ip-10-0-133-218 kubenswrapper[2580]: E0916 12:48:45.269081    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/3c9aacb3-1142-4610-b990-a28d15998e71-service-ca-bundle podName:3c9aacb3-1142-4610-b990-a28d15998e71 nodeName:}" failed. No retries permitted until 2026-09-16 12:48:49.269056889 +0000 UTC m=+68.191929353 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/3c9aacb3-1142-4610-b990-a28d15998e71-service-ca-bundle") pod "router-default-5b4dd555d4-jwzh7" (UID: "3c9aacb3-1142-4610-b990-a28d15998e71") : configmap references non-existent config key: service-ca.crt
Sep 16 12:48:45.269357 ip-10-0-133-218 kubenswrapper[2580]: E0916 12:48:45.269110    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3c9aacb3-1142-4610-b990-a28d15998e71-metrics-certs podName:3c9aacb3-1142-4610-b990-a28d15998e71 nodeName:}" failed. No retries permitted until 2026-09-16 12:48:49.269101035 +0000 UTC m=+68.191973487 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/3c9aacb3-1142-4610-b990-a28d15998e71-metrics-certs") pod "router-default-5b4dd555d4-jwzh7" (UID: "3c9aacb3-1142-4610-b990-a28d15998e71") : secret "router-metrics-certs-default" not found
Sep 16 12:48:45.369978 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:48:45.369943    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/5abef85b-1f8d-4f0e-9091-c0f42acf1af1-samples-operator-tls\") pod \"cluster-samples-operator-86499c89c4-xvxlf\" (UID: \"5abef85b-1f8d-4f0e-9091-c0f42acf1af1\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-xvxlf"
Sep 16 12:48:45.370165 ip-10-0-133-218 kubenswrapper[2580]: E0916 12:48:45.370111    2580 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Sep 16 12:48:45.370233 ip-10-0-133-218 kubenswrapper[2580]: E0916 12:48:45.370183    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5abef85b-1f8d-4f0e-9091-c0f42acf1af1-samples-operator-tls podName:5abef85b-1f8d-4f0e-9091-c0f42acf1af1 nodeName:}" failed. No retries permitted until 2026-09-16 12:48:49.370163624 +0000 UTC m=+68.293036085 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/5abef85b-1f8d-4f0e-9091-c0f42acf1af1-samples-operator-tls") pod "cluster-samples-operator-86499c89c4-xvxlf" (UID: "5abef85b-1f8d-4f0e-9091-c0f42acf1af1") : secret "samples-operator-tls" not found
Sep 16 12:48:45.718257 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:48:45.718219    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-network-diagnostics/network-check-source-6f8b768455-jl4wt"]
Sep 16 12:48:45.736787 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:48:45.736755    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-6f8b768455-jl4wt"
Sep 16 12:48:45.745434 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:48:45.745414    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-diagnostics-dockercfg-jjcb2\""
Sep 16 12:48:45.763679 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:48:45.763654    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-6f8b768455-jl4wt"]
Sep 16 12:48:45.874152 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:48:45.874118    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rb6kx\" (UniqueName: \"kubernetes.io/projected/f2a2dd43-5c98-4e27-a59a-8ac33a6d9c6e-kube-api-access-rb6kx\") pod \"network-check-source-6f8b768455-jl4wt\" (UID: \"f2a2dd43-5c98-4e27-a59a-8ac33a6d9c6e\") " pod="openshift-network-diagnostics/network-check-source-6f8b768455-jl4wt"
Sep 16 12:48:45.974798 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:48:45.974728    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rb6kx\" (UniqueName: \"kubernetes.io/projected/f2a2dd43-5c98-4e27-a59a-8ac33a6d9c6e-kube-api-access-rb6kx\") pod \"network-check-source-6f8b768455-jl4wt\" (UID: \"f2a2dd43-5c98-4e27-a59a-8ac33a6d9c6e\") " pod="openshift-network-diagnostics/network-check-source-6f8b768455-jl4wt"
Sep 16 12:48:45.986137 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:48:45.985854    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Sep 16 12:48:45.995850 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:48:45.995824    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Sep 16 12:48:46.005872 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:48:46.005848    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rb6kx\" (UniqueName: \"kubernetes.io/projected/f2a2dd43-5c98-4e27-a59a-8ac33a6d9c6e-kube-api-access-rb6kx\") pod \"network-check-source-6f8b768455-jl4wt\" (UID: \"f2a2dd43-5c98-4e27-a59a-8ac33a6d9c6e\") " pod="openshift-network-diagnostics/network-check-source-6f8b768455-jl4wt"
Sep 16 12:48:46.044695 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:48:46.044668    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-6f8b768455-jl4wt"
Sep 16 12:48:46.370528 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:48:46.370483    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-6f8b768455-jl4wt"]
Sep 16 12:48:46.374012 ip-10-0-133-218 kubenswrapper[2580]: W0916 12:48:46.373989    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf2a2dd43_5c98_4e27_a59a_8ac33a6d9c6e.slice/crio-540548479e14d8244436ccf5f6fe5ab58e6f29912701ec13227e07ce9a361465 WatchSource:0}: Error finding container 540548479e14d8244436ccf5f6fe5ab58e6f29912701ec13227e07ce9a361465: Status 404 returned error can't find the container with id 540548479e14d8244436ccf5f6fe5ab58e6f29912701ec13227e07ce9a361465
Sep 16 12:48:46.861225 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:48:46.861188    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-6f8b768455-jl4wt" event={"ID":"f2a2dd43-5c98-4e27-a59a-8ac33a6d9c6e","Type":"ContainerStarted","Data":"540548479e14d8244436ccf5f6fe5ab58e6f29912701ec13227e07ce9a361465"}
Sep 16 12:48:47.032913 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:48:47.032878    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-2m9hh"]
Sep 16 12:48:47.051293 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:48:47.051262    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-2m9hh"
Sep 16 12:48:47.053622 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:48:47.053598    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Sep 16 12:48:47.054136 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:48:47.054112    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Sep 16 12:48:47.054652 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:48:47.054636    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Sep 16 12:48:47.054750 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:48:47.054678    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-8zmc7\""
Sep 16 12:48:47.057273 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:48:47.057256    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Sep 16 12:48:47.072858 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:48:47.072834    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-2m9hh"]
Sep 16 12:48:47.185907 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:48:47.185764    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/e12762e0-2a1d-4bf3-8fca-d22ad8e1eecf-data-volume\") pod \"insights-runtime-extractor-2m9hh\" (UID: \"e12762e0-2a1d-4bf3-8fca-d22ad8e1eecf\") " pod="openshift-insights/insights-runtime-extractor-2m9hh"
Sep 16 12:48:47.185907 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:48:47.185803    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/e12762e0-2a1d-4bf3-8fca-d22ad8e1eecf-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-2m9hh\" (UID: \"e12762e0-2a1d-4bf3-8fca-d22ad8e1eecf\") " pod="openshift-insights/insights-runtime-extractor-2m9hh"
Sep 16 12:48:47.185907 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:48:47.185838    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/e12762e0-2a1d-4bf3-8fca-d22ad8e1eecf-crio-socket\") pod \"insights-runtime-extractor-2m9hh\" (UID: \"e12762e0-2a1d-4bf3-8fca-d22ad8e1eecf\") " pod="openshift-insights/insights-runtime-extractor-2m9hh"
Sep 16 12:48:47.186175 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:48:47.185951    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-78tw8\" (UniqueName: \"kubernetes.io/projected/e12762e0-2a1d-4bf3-8fca-d22ad8e1eecf-kube-api-access-78tw8\") pod \"insights-runtime-extractor-2m9hh\" (UID: \"e12762e0-2a1d-4bf3-8fca-d22ad8e1eecf\") " pod="openshift-insights/insights-runtime-extractor-2m9hh"
Sep 16 12:48:47.186175 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:48:47.186043    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/e12762e0-2a1d-4bf3-8fca-d22ad8e1eecf-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-2m9hh\" (UID: \"e12762e0-2a1d-4bf3-8fca-d22ad8e1eecf\") " pod="openshift-insights/insights-runtime-extractor-2m9hh"
Sep 16 12:48:47.286538 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:48:47.286499    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/e12762e0-2a1d-4bf3-8fca-d22ad8e1eecf-crio-socket\") pod \"insights-runtime-extractor-2m9hh\" (UID: \"e12762e0-2a1d-4bf3-8fca-d22ad8e1eecf\") " pod="openshift-insights/insights-runtime-extractor-2m9hh"
Sep 16 12:48:47.286707 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:48:47.286627    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/e12762e0-2a1d-4bf3-8fca-d22ad8e1eecf-crio-socket\") pod \"insights-runtime-extractor-2m9hh\" (UID: \"e12762e0-2a1d-4bf3-8fca-d22ad8e1eecf\") " pod="openshift-insights/insights-runtime-extractor-2m9hh"
Sep 16 12:48:47.286707 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:48:47.286665    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-78tw8\" (UniqueName: \"kubernetes.io/projected/e12762e0-2a1d-4bf3-8fca-d22ad8e1eecf-kube-api-access-78tw8\") pod \"insights-runtime-extractor-2m9hh\" (UID: \"e12762e0-2a1d-4bf3-8fca-d22ad8e1eecf\") " pod="openshift-insights/insights-runtime-extractor-2m9hh"
Sep 16 12:48:47.286804 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:48:47.286715    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/e12762e0-2a1d-4bf3-8fca-d22ad8e1eecf-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-2m9hh\" (UID: \"e12762e0-2a1d-4bf3-8fca-d22ad8e1eecf\") " pod="openshift-insights/insights-runtime-extractor-2m9hh"
Sep 16 12:48:47.286804 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:48:47.286770    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7d634420-2900-42b3-9609-3fb64baff8f3-metrics-certs\") pod \"network-metrics-daemon-62j7x\" (UID: \"7d634420-2900-42b3-9609-3fb64baff8f3\") " pod="openshift-multus/network-metrics-daemon-62j7x"
Sep 16 12:48:47.286895 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:48:47.286810    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/e12762e0-2a1d-4bf3-8fca-d22ad8e1eecf-data-volume\") pod \"insights-runtime-extractor-2m9hh\" (UID: \"e12762e0-2a1d-4bf3-8fca-d22ad8e1eecf\") " pod="openshift-insights/insights-runtime-extractor-2m9hh"
Sep 16 12:48:47.286895 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:48:47.286833    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/e12762e0-2a1d-4bf3-8fca-d22ad8e1eecf-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-2m9hh\" (UID: \"e12762e0-2a1d-4bf3-8fca-d22ad8e1eecf\") " pod="openshift-insights/insights-runtime-extractor-2m9hh"
Sep 16 12:48:47.286895 ip-10-0-133-218 kubenswrapper[2580]: E0916 12:48:47.286879    2580 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 16 12:48:47.287039 ip-10-0-133-218 kubenswrapper[2580]: E0916 12:48:47.286901    2580 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 16 12:48:47.287039 ip-10-0-133-218 kubenswrapper[2580]: E0916 12:48:47.286963    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e12762e0-2a1d-4bf3-8fca-d22ad8e1eecf-insights-runtime-extractor-tls podName:e12762e0-2a1d-4bf3-8fca-d22ad8e1eecf nodeName:}" failed. No retries permitted until 2026-09-16 12:48:47.786942357 +0000 UTC m=+66.709814806 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/e12762e0-2a1d-4bf3-8fca-d22ad8e1eecf-insights-runtime-extractor-tls") pod "insights-runtime-extractor-2m9hh" (UID: "e12762e0-2a1d-4bf3-8fca-d22ad8e1eecf") : secret "insights-runtime-extractor-tls" not found
Sep 16 12:48:47.287039 ip-10-0-133-218 kubenswrapper[2580]: E0916 12:48:47.286984    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/e12762e0-2a1d-4bf3-8fca-d22ad8e1eecf-kube-rbac-proxy-cm podName:e12762e0-2a1d-4bf3-8fca-d22ad8e1eecf nodeName:}" failed. No retries permitted until 2026-09-16 12:48:47.786973319 +0000 UTC m=+66.709845772 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/e12762e0-2a1d-4bf3-8fca-d22ad8e1eecf-kube-rbac-proxy-cm") pod "insights-runtime-extractor-2m9hh" (UID: "e12762e0-2a1d-4bf3-8fca-d22ad8e1eecf") : configmap "kube-rbac-proxy" not found
Sep 16 12:48:47.287273 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:48:47.287248    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/e12762e0-2a1d-4bf3-8fca-d22ad8e1eecf-data-volume\") pod \"insights-runtime-extractor-2m9hh\" (UID: \"e12762e0-2a1d-4bf3-8fca-d22ad8e1eecf\") " pod="openshift-insights/insights-runtime-extractor-2m9hh"
Sep 16 12:48:47.289418 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:48:47.289394    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Sep 16 12:48:47.297972 ip-10-0-133-218 kubenswrapper[2580]: E0916 12:48:47.297955    2580 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
Sep 16 12:48:47.298066 ip-10-0-133-218 kubenswrapper[2580]: E0916 12:48:47.298037    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7d634420-2900-42b3-9609-3fb64baff8f3-metrics-certs podName:7d634420-2900-42b3-9609-3fb64baff8f3 nodeName:}" failed. No retries permitted until 2026-09-16 12:49:51.298017629 +0000 UTC m=+130.220890091 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/7d634420-2900-42b3-9609-3fb64baff8f3-metrics-certs") pod "network-metrics-daemon-62j7x" (UID: "7d634420-2900-42b3-9609-3fb64baff8f3") : secret "metrics-daemon-secret" not found
Sep 16 12:48:47.308277 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:48:47.308257    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-78tw8\" (UniqueName: \"kubernetes.io/projected/e12762e0-2a1d-4bf3-8fca-d22ad8e1eecf-kube-api-access-78tw8\") pod \"insights-runtime-extractor-2m9hh\" (UID: \"e12762e0-2a1d-4bf3-8fca-d22ad8e1eecf\") " pod="openshift-insights/insights-runtime-extractor-2m9hh"
Sep 16 12:48:47.387920 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:48:47.387885    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v4r5z\" (UniqueName: \"kubernetes.io/projected/1b26fbcb-3523-4f7c-9e42-5b3653456abc-kube-api-access-v4r5z\") pod \"network-check-target-bpv5d\" (UID: \"1b26fbcb-3523-4f7c-9e42-5b3653456abc\") " pod="openshift-network-diagnostics/network-check-target-bpv5d"
Sep 16 12:48:47.388317 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:48:47.387983    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/fea676c8-6e68-4894-99fc-fe8b59886a9a-original-pull-secret\") pod \"global-pull-secret-syncer-5jkh4\" (UID: \"fea676c8-6e68-4894-99fc-fe8b59886a9a\") " pod="kube-system/global-pull-secret-syncer-5jkh4"
Sep 16 12:48:47.390284 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:48:47.390265    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-v4r5z\" (UniqueName: \"kubernetes.io/projected/1b26fbcb-3523-4f7c-9e42-5b3653456abc-kube-api-access-v4r5z\") pod \"network-check-target-bpv5d\" (UID: \"1b26fbcb-3523-4f7c-9e42-5b3653456abc\") " pod="openshift-network-diagnostics/network-check-target-bpv5d"
Sep 16 12:48:47.393916 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:48:47.393890    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Sep 16 12:48:47.400953 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:48:47.400932    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/fea676c8-6e68-4894-99fc-fe8b59886a9a-original-pull-secret\") pod \"global-pull-secret-syncer-5jkh4\" (UID: \"fea676c8-6e68-4894-99fc-fe8b59886a9a\") " pod="kube-system/global-pull-secret-syncer-5jkh4"
Sep 16 12:48:47.503865 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:48:47.503797    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5jkh4"
Sep 16 12:48:47.611986 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:48:47.611957    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-m2lxw\""
Sep 16 12:48:47.615010 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:48:47.614987    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator/migrator-8b4d697c5-bzww8"]
Sep 16 12:48:47.617116 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:48:47.617095    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-8b4d697c5-bzww8"
Sep 16 12:48:47.619558 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:48:47.619541    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-bpv5d"
Sep 16 12:48:47.623646 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:48:47.623624    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"openshift-service-ca.crt\""
Sep 16 12:48:47.624084 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:48:47.624063    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-storage-version-migrator-sa-dockercfg-5kprw\""
Sep 16 12:48:47.627125 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:48:47.627100    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-root-ca.crt\""
Sep 16 12:48:47.644847 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:48:47.644820    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-8b4d697c5-bzww8"]
Sep 16 12:48:47.789411 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:48:47.789380    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-5jkh4"]
Sep 16 12:48:47.790458 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:48:47.790433    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wmvzn\" (UniqueName: \"kubernetes.io/projected/46e66faf-5912-4f71-8fe8-57c2c2d64bc4-kube-api-access-wmvzn\") pod \"migrator-8b4d697c5-bzww8\" (UID: \"46e66faf-5912-4f71-8fe8-57c2c2d64bc4\") " pod="openshift-kube-storage-version-migrator/migrator-8b4d697c5-bzww8"
Sep 16 12:48:47.790577 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:48:47.790522    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/b352b0f0-afd0-48c2-965f-7cc63cdf04fd-metrics-tls\") pod \"dns-default-hzdmm\" (UID: \"b352b0f0-afd0-48c2-965f-7cc63cdf04fd\") " pod="openshift-dns/dns-default-hzdmm"
Sep 16 12:48:47.790707 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:48:47.790603    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/e12762e0-2a1d-4bf3-8fca-d22ad8e1eecf-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-2m9hh\" (UID: \"e12762e0-2a1d-4bf3-8fca-d22ad8e1eecf\") " pod="openshift-insights/insights-runtime-extractor-2m9hh"
Sep 16 12:48:47.790707 ip-10-0-133-218 kubenswrapper[2580]: E0916 12:48:47.790627    2580 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 16 12:48:47.790707 ip-10-0-133-218 kubenswrapper[2580]: E0916 12:48:47.790687    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b352b0f0-afd0-48c2-965f-7cc63cdf04fd-metrics-tls podName:b352b0f0-afd0-48c2-965f-7cc63cdf04fd nodeName:}" failed. No retries permitted until 2026-09-16 12:49:19.790667909 +0000 UTC m=+98.713540376 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/b352b0f0-afd0-48c2-965f-7cc63cdf04fd-metrics-tls") pod "dns-default-hzdmm" (UID: "b352b0f0-afd0-48c2-965f-7cc63cdf04fd") : secret "dns-default-metrics-tls" not found
Sep 16 12:48:47.791001 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:48:47.790707    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/e12762e0-2a1d-4bf3-8fca-d22ad8e1eecf-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-2m9hh\" (UID: \"e12762e0-2a1d-4bf3-8fca-d22ad8e1eecf\") " pod="openshift-insights/insights-runtime-extractor-2m9hh"
Sep 16 12:48:47.791001 ip-10-0-133-218 kubenswrapper[2580]: E0916 12:48:47.790729    2580 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 16 12:48:47.791001 ip-10-0-133-218 kubenswrapper[2580]: E0916 12:48:47.790776    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e12762e0-2a1d-4bf3-8fca-d22ad8e1eecf-insights-runtime-extractor-tls podName:e12762e0-2a1d-4bf3-8fca-d22ad8e1eecf nodeName:}" failed. No retries permitted until 2026-09-16 12:48:48.790762176 +0000 UTC m=+67.713634639 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/e12762e0-2a1d-4bf3-8fca-d22ad8e1eecf-insights-runtime-extractor-tls") pod "insights-runtime-extractor-2m9hh" (UID: "e12762e0-2a1d-4bf3-8fca-d22ad8e1eecf") : secret "insights-runtime-extractor-tls" not found
Sep 16 12:48:47.791001 ip-10-0-133-218 kubenswrapper[2580]: E0916 12:48:47.790789    2580 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 16 12:48:47.791001 ip-10-0-133-218 kubenswrapper[2580]: E0916 12:48:47.790828    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/e12762e0-2a1d-4bf3-8fca-d22ad8e1eecf-kube-rbac-proxy-cm podName:e12762e0-2a1d-4bf3-8fca-d22ad8e1eecf nodeName:}" failed. No retries permitted until 2026-09-16 12:48:48.790816563 +0000 UTC m=+67.713689028 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/e12762e0-2a1d-4bf3-8fca-d22ad8e1eecf-kube-rbac-proxy-cm") pod "insights-runtime-extractor-2m9hh" (UID: "e12762e0-2a1d-4bf3-8fca-d22ad8e1eecf") : configmap "kube-rbac-proxy" not found
Sep 16 12:48:47.793029 ip-10-0-133-218 kubenswrapper[2580]: W0916 12:48:47.793001    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfea676c8_6e68_4894_99fc_fe8b59886a9a.slice/crio-90f190a05d1f3758fc1d2912471c169fd84331576b0dd1b0330d80803ec34966 WatchSource:0}: Error finding container 90f190a05d1f3758fc1d2912471c169fd84331576b0dd1b0330d80803ec34966: Status 404 returned error can't find the container with id 90f190a05d1f3758fc1d2912471c169fd84331576b0dd1b0330d80803ec34966
Sep 16 12:48:47.865779 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:48:47.865738    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-5jkh4" event={"ID":"fea676c8-6e68-4894-99fc-fe8b59886a9a","Type":"ContainerStarted","Data":"90f190a05d1f3758fc1d2912471c169fd84331576b0dd1b0330d80803ec34966"}
Sep 16 12:48:47.891102 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:48:47.891070    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wmvzn\" (UniqueName: \"kubernetes.io/projected/46e66faf-5912-4f71-8fe8-57c2c2d64bc4-kube-api-access-wmvzn\") pod \"migrator-8b4d697c5-bzww8\" (UID: \"46e66faf-5912-4f71-8fe8-57c2c2d64bc4\") " pod="openshift-kube-storage-version-migrator/migrator-8b4d697c5-bzww8"
Sep 16 12:48:47.900852 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:48:47.900812    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wmvzn\" (UniqueName: \"kubernetes.io/projected/46e66faf-5912-4f71-8fe8-57c2c2d64bc4-kube-api-access-wmvzn\") pod \"migrator-8b4d697c5-bzww8\" (UID: \"46e66faf-5912-4f71-8fe8-57c2c2d64bc4\") " pod="openshift-kube-storage-version-migrator/migrator-8b4d697c5-bzww8"
Sep 16 12:48:47.902878 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:48:47.902850    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-bpv5d"]
Sep 16 12:48:47.905815 ip-10-0-133-218 kubenswrapper[2580]: W0916 12:48:47.905788    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1b26fbcb_3523_4f7c_9e42_5b3653456abc.slice/crio-8d5e3ac1103133909b9be14cb2b61098bc8d214f59b34be02c4473e1e0126e0e WatchSource:0}: Error finding container 8d5e3ac1103133909b9be14cb2b61098bc8d214f59b34be02c4473e1e0126e0e: Status 404 returned error can't find the container with id 8d5e3ac1103133909b9be14cb2b61098bc8d214f59b34be02c4473e1e0126e0e
Sep 16 12:48:47.928228 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:48:47.928207    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-8b4d697c5-bzww8"
Sep 16 12:48:48.221669 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:48:48.221590    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-8b4d697c5-bzww8"]
Sep 16 12:48:48.225802 ip-10-0-133-218 kubenswrapper[2580]: W0916 12:48:48.225774    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod46e66faf_5912_4f71_8fe8_57c2c2d64bc4.slice/crio-e26b24114fa6bcb9cb293950933ef747056cebd7e69d2b9eec36860f9010f11a WatchSource:0}: Error finding container e26b24114fa6bcb9cb293950933ef747056cebd7e69d2b9eec36860f9010f11a: Status 404 returned error can't find the container with id e26b24114fa6bcb9cb293950933ef747056cebd7e69d2b9eec36860f9010f11a
Sep 16 12:48:48.736709 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:48:48.736673    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-cx649_0b719a93-a161-4199-9aae-5aa99cdf7165/dns-node-resolver/0.log"
Sep 16 12:48:48.798161 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:48:48.798130    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/e12762e0-2a1d-4bf3-8fca-d22ad8e1eecf-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-2m9hh\" (UID: \"e12762e0-2a1d-4bf3-8fca-d22ad8e1eecf\") " pod="openshift-insights/insights-runtime-extractor-2m9hh"
Sep 16 12:48:48.798307 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:48:48.798189    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/e12762e0-2a1d-4bf3-8fca-d22ad8e1eecf-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-2m9hh\" (UID: \"e12762e0-2a1d-4bf3-8fca-d22ad8e1eecf\") " pod="openshift-insights/insights-runtime-extractor-2m9hh"
Sep 16 12:48:48.798307 ip-10-0-133-218 kubenswrapper[2580]: E0916 12:48:48.798263    2580 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 16 12:48:48.798371 ip-10-0-133-218 kubenswrapper[2580]: E0916 12:48:48.798310    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/e12762e0-2a1d-4bf3-8fca-d22ad8e1eecf-kube-rbac-proxy-cm podName:e12762e0-2a1d-4bf3-8fca-d22ad8e1eecf nodeName:}" failed. No retries permitted until 2026-09-16 12:48:50.798297407 +0000 UTC m=+69.721169867 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/e12762e0-2a1d-4bf3-8fca-d22ad8e1eecf-kube-rbac-proxy-cm") pod "insights-runtime-extractor-2m9hh" (UID: "e12762e0-2a1d-4bf3-8fca-d22ad8e1eecf") : configmap "kube-rbac-proxy" not found
Sep 16 12:48:48.798733 ip-10-0-133-218 kubenswrapper[2580]: E0916 12:48:48.798709    2580 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 16 12:48:48.798841 ip-10-0-133-218 kubenswrapper[2580]: E0916 12:48:48.798775    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e12762e0-2a1d-4bf3-8fca-d22ad8e1eecf-insights-runtime-extractor-tls podName:e12762e0-2a1d-4bf3-8fca-d22ad8e1eecf nodeName:}" failed. No retries permitted until 2026-09-16 12:48:50.798758864 +0000 UTC m=+69.721631329 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/e12762e0-2a1d-4bf3-8fca-d22ad8e1eecf-insights-runtime-extractor-tls") pod "insights-runtime-extractor-2m9hh" (UID: "e12762e0-2a1d-4bf3-8fca-d22ad8e1eecf") : secret "insights-runtime-extractor-tls" not found
Sep 16 12:48:48.873381 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:48:48.873343    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-8b4d697c5-bzww8" event={"ID":"46e66faf-5912-4f71-8fe8-57c2c2d64bc4","Type":"ContainerStarted","Data":"e26b24114fa6bcb9cb293950933ef747056cebd7e69d2b9eec36860f9010f11a"}
Sep 16 12:48:48.875036 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:48:48.874969    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-bpv5d" event={"ID":"1b26fbcb-3523-4f7c-9e42-5b3653456abc","Type":"ContainerStarted","Data":"8d5e3ac1103133909b9be14cb2b61098bc8d214f59b34be02c4473e1e0126e0e"}
Sep 16 12:48:49.305434 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:48:49.302474    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3c9aacb3-1142-4610-b990-a28d15998e71-metrics-certs\") pod \"router-default-5b4dd555d4-jwzh7\" (UID: \"3c9aacb3-1142-4610-b990-a28d15998e71\") " pod="openshift-ingress/router-default-5b4dd555d4-jwzh7"
Sep 16 12:48:49.305434 ip-10-0-133-218 kubenswrapper[2580]: E0916 12:48:49.302669    2580 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Sep 16 12:48:49.305434 ip-10-0-133-218 kubenswrapper[2580]: E0916 12:48:49.302756    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3c9aacb3-1142-4610-b990-a28d15998e71-metrics-certs podName:3c9aacb3-1142-4610-b990-a28d15998e71 nodeName:}" failed. No retries permitted until 2026-09-16 12:48:57.302734403 +0000 UTC m=+76.225606875 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/3c9aacb3-1142-4610-b990-a28d15998e71-metrics-certs") pod "router-default-5b4dd555d4-jwzh7" (UID: "3c9aacb3-1142-4610-b990-a28d15998e71") : secret "router-metrics-certs-default" not found
Sep 16 12:48:49.305434 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:48:49.302915    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3c9aacb3-1142-4610-b990-a28d15998e71-service-ca-bundle\") pod \"router-default-5b4dd555d4-jwzh7\" (UID: \"3c9aacb3-1142-4610-b990-a28d15998e71\") " pod="openshift-ingress/router-default-5b4dd555d4-jwzh7"
Sep 16 12:48:49.305434 ip-10-0-133-218 kubenswrapper[2580]: E0916 12:48:49.303181    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/3c9aacb3-1142-4610-b990-a28d15998e71-service-ca-bundle podName:3c9aacb3-1142-4610-b990-a28d15998e71 nodeName:}" failed. No retries permitted until 2026-09-16 12:48:57.303157727 +0000 UTC m=+76.226030194 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/3c9aacb3-1142-4610-b990-a28d15998e71-service-ca-bundle") pod "router-default-5b4dd555d4-jwzh7" (UID: "3c9aacb3-1142-4610-b990-a28d15998e71") : configmap references non-existent config key: service-ca.crt
Sep 16 12:48:49.404778 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:48:49.404014    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/5abef85b-1f8d-4f0e-9091-c0f42acf1af1-samples-operator-tls\") pod \"cluster-samples-operator-86499c89c4-xvxlf\" (UID: \"5abef85b-1f8d-4f0e-9091-c0f42acf1af1\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-xvxlf"
Sep 16 12:48:49.404778 ip-10-0-133-218 kubenswrapper[2580]: E0916 12:48:49.404336    2580 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Sep 16 12:48:49.404778 ip-10-0-133-218 kubenswrapper[2580]: E0916 12:48:49.404410    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5abef85b-1f8d-4f0e-9091-c0f42acf1af1-samples-operator-tls podName:5abef85b-1f8d-4f0e-9091-c0f42acf1af1 nodeName:}" failed. No retries permitted until 2026-09-16 12:48:57.404388769 +0000 UTC m=+76.327261215 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/5abef85b-1f8d-4f0e-9091-c0f42acf1af1-samples-operator-tls") pod "cluster-samples-operator-86499c89c4-xvxlf" (UID: "5abef85b-1f8d-4f0e-9091-c0f42acf1af1") : secret "samples-operator-tls" not found
Sep 16 12:48:49.881476 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:48:49.881443    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-6f8b768455-jl4wt" event={"ID":"f2a2dd43-5c98-4e27-a59a-8ac33a6d9c6e","Type":"ContainerStarted","Data":"debdf2222587da9b64b66976a604986176087e2bdf1891063fa628ab8956b279"}
Sep 16 12:48:49.908193 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:48:49.908147    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-source-6f8b768455-jl4wt" podStartSLOduration=1.657787176 podStartE2EDuration="4.908131305s" totalImagesPullingTime="3.250344129s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 12:48:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 12:48:49.907389779 +0000 UTC m=+68.830262248" watchObservedRunningTime="2026-09-16 12:48:49.908131305 +0000 UTC m=+68.831003773"
Sep 16 12:48:50.141502 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:48:50.141405    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-d9ndr_5d809960-17f1-4e60-9438-dc50a3b54dab/node-ca/0.log"
Sep 16 12:48:50.743597 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:48:50.743539    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-d4swz_23d2db14-6f1e-4198-8b68-022153a628d5/serve-healthcheck-canary/0.log"
Sep 16 12:48:50.815475 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:48:50.815437    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/e12762e0-2a1d-4bf3-8fca-d22ad8e1eecf-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-2m9hh\" (UID: \"e12762e0-2a1d-4bf3-8fca-d22ad8e1eecf\") " pod="openshift-insights/insights-runtime-extractor-2m9hh"
Sep 16 12:48:50.815663 ip-10-0-133-218 kubenswrapper[2580]: E0916 12:48:50.815562    2580 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 16 12:48:50.815663 ip-10-0-133-218 kubenswrapper[2580]: E0916 12:48:50.815644    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/e12762e0-2a1d-4bf3-8fca-d22ad8e1eecf-kube-rbac-proxy-cm podName:e12762e0-2a1d-4bf3-8fca-d22ad8e1eecf nodeName:}" failed. No retries permitted until 2026-09-16 12:48:54.815624286 +0000 UTC m=+73.738496756 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/e12762e0-2a1d-4bf3-8fca-d22ad8e1eecf-kube-rbac-proxy-cm") pod "insights-runtime-extractor-2m9hh" (UID: "e12762e0-2a1d-4bf3-8fca-d22ad8e1eecf") : configmap "kube-rbac-proxy" not found
Sep 16 12:48:50.815764 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:48:50.815688    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/e12762e0-2a1d-4bf3-8fca-d22ad8e1eecf-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-2m9hh\" (UID: \"e12762e0-2a1d-4bf3-8fca-d22ad8e1eecf\") " pod="openshift-insights/insights-runtime-extractor-2m9hh"
Sep 16 12:48:50.815860 ip-10-0-133-218 kubenswrapper[2580]: E0916 12:48:50.815824    2580 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 16 12:48:50.815932 ip-10-0-133-218 kubenswrapper[2580]: E0916 12:48:50.815882    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e12762e0-2a1d-4bf3-8fca-d22ad8e1eecf-insights-runtime-extractor-tls podName:e12762e0-2a1d-4bf3-8fca-d22ad8e1eecf nodeName:}" failed. No retries permitted until 2026-09-16 12:48:54.815867291 +0000 UTC m=+73.738739764 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/e12762e0-2a1d-4bf3-8fca-d22ad8e1eecf-insights-runtime-extractor-tls") pod "insights-runtime-extractor-2m9hh" (UID: "e12762e0-2a1d-4bf3-8fca-d22ad8e1eecf") : secret "insights-runtime-extractor-tls" not found
Sep 16 12:48:50.887257 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:48:50.887190    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-bpv5d" event={"ID":"1b26fbcb-3523-4f7c-9e42-5b3653456abc","Type":"ContainerStarted","Data":"35a2dcf62d15c5870efa7f81d5c24a7bd5635cafe78e4c788db1bc2a11b9c532"}
Sep 16 12:48:50.914765 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:48:50.914706    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-bpv5d" podStartSLOduration=66.953534017 podStartE2EDuration="1m8.914684141s" totalImagesPullingTime="1.961150124s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 12:47:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 12:48:50.91236381 +0000 UTC m=+69.835236282" watchObservedRunningTime="2026-09-16 12:48:50.914684141 +0000 UTC m=+69.837556612"
Sep 16 12:48:51.890727 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:48:51.890693    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-bpv5d"
Sep 16 12:48:52.895476 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:48:52.895437    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-5jkh4" event={"ID":"fea676c8-6e68-4894-99fc-fe8b59886a9a","Type":"ContainerStarted","Data":"519fad863c1f89ad4e46f278da447ac9c32f5abcd04fb4d85e31673b567e7a6d"}
Sep 16 12:48:52.897306 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:48:52.897274    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-8b4d697c5-bzww8" event={"ID":"46e66faf-5912-4f71-8fe8-57c2c2d64bc4","Type":"ContainerStarted","Data":"830d32a3d94cdb14569359dee33f99a83615ca6aaa7c4af129fdacc226f0a67f"}
Sep 16 12:48:52.897417 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:48:52.897326    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-8b4d697c5-bzww8" event={"ID":"46e66faf-5912-4f71-8fe8-57c2c2d64bc4","Type":"ContainerStarted","Data":"a63cfd7d3099dd411592d06101f47e4ba9e6db0cd369b514d61829b5863d40e8"}
Sep 16 12:48:52.914863 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:48:52.914814    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-5jkh4" podStartSLOduration=66.607952493 podStartE2EDuration="1m10.914798347s" totalImagesPullingTime="4.306845854s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-16 12:47:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 12:48:52.913888179 +0000 UTC m=+71.836760650" watchObservedRunningTime="2026-09-16 12:48:52.914798347 +0000 UTC m=+71.837670814"
Sep 16 12:48:52.933296 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:48:52.933258    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator/migrator-8b4d697c5-bzww8" podStartSLOduration=2.095132562 podStartE2EDuration="5.933241531s" totalImagesPullingTime="3.838108969s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 12:48:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 12:48:52.932542878 +0000 UTC m=+71.855415346" watchObservedRunningTime="2026-09-16 12:48:52.933241531 +0000 UTC m=+71.856114056"
Sep 16 12:48:54.839421 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:48:54.839375    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/e12762e0-2a1d-4bf3-8fca-d22ad8e1eecf-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-2m9hh\" (UID: \"e12762e0-2a1d-4bf3-8fca-d22ad8e1eecf\") " pod="openshift-insights/insights-runtime-extractor-2m9hh"
Sep 16 12:48:54.839799 ip-10-0-133-218 kubenswrapper[2580]: E0916 12:48:54.839546    2580 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 16 12:48:54.839799 ip-10-0-133-218 kubenswrapper[2580]: E0916 12:48:54.839594    2580 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 16 12:48:54.839799 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:48:54.839551    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/e12762e0-2a1d-4bf3-8fca-d22ad8e1eecf-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-2m9hh\" (UID: \"e12762e0-2a1d-4bf3-8fca-d22ad8e1eecf\") " pod="openshift-insights/insights-runtime-extractor-2m9hh"
Sep 16 12:48:54.839799 ip-10-0-133-218 kubenswrapper[2580]: E0916 12:48:54.839617    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e12762e0-2a1d-4bf3-8fca-d22ad8e1eecf-insights-runtime-extractor-tls podName:e12762e0-2a1d-4bf3-8fca-d22ad8e1eecf nodeName:}" failed. No retries permitted until 2026-09-16 12:49:02.839597729 +0000 UTC m=+81.762470189 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/e12762e0-2a1d-4bf3-8fca-d22ad8e1eecf-insights-runtime-extractor-tls") pod "insights-runtime-extractor-2m9hh" (UID: "e12762e0-2a1d-4bf3-8fca-d22ad8e1eecf") : secret "insights-runtime-extractor-tls" not found
Sep 16 12:48:54.839799 ip-10-0-133-218 kubenswrapper[2580]: E0916 12:48:54.839640    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/e12762e0-2a1d-4bf3-8fca-d22ad8e1eecf-kube-rbac-proxy-cm podName:e12762e0-2a1d-4bf3-8fca-d22ad8e1eecf nodeName:}" failed. No retries permitted until 2026-09-16 12:49:02.839631046 +0000 UTC m=+81.762503499 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/e12762e0-2a1d-4bf3-8fca-d22ad8e1eecf-kube-rbac-proxy-cm") pod "insights-runtime-extractor-2m9hh" (UID: "e12762e0-2a1d-4bf3-8fca-d22ad8e1eecf") : configmap "kube-rbac-proxy" not found
Sep 16 12:48:57.355583 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:48:57.355549    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3c9aacb3-1142-4610-b990-a28d15998e71-metrics-certs\") pod \"router-default-5b4dd555d4-jwzh7\" (UID: \"3c9aacb3-1142-4610-b990-a28d15998e71\") " pod="openshift-ingress/router-default-5b4dd555d4-jwzh7"
Sep 16 12:48:57.356070 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:48:57.355611    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3c9aacb3-1142-4610-b990-a28d15998e71-service-ca-bundle\") pod \"router-default-5b4dd555d4-jwzh7\" (UID: \"3c9aacb3-1142-4610-b990-a28d15998e71\") " pod="openshift-ingress/router-default-5b4dd555d4-jwzh7"
Sep 16 12:48:57.356070 ip-10-0-133-218 kubenswrapper[2580]: E0916 12:48:57.355745    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/3c9aacb3-1142-4610-b990-a28d15998e71-service-ca-bundle podName:3c9aacb3-1142-4610-b990-a28d15998e71 nodeName:}" failed. No retries permitted until 2026-09-16 12:49:13.355729823 +0000 UTC m=+92.278602283 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/3c9aacb3-1142-4610-b990-a28d15998e71-service-ca-bundle") pod "router-default-5b4dd555d4-jwzh7" (UID: "3c9aacb3-1142-4610-b990-a28d15998e71") : configmap references non-existent config key: service-ca.crt
Sep 16 12:48:57.357902 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:48:57.357875    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3c9aacb3-1142-4610-b990-a28d15998e71-metrics-certs\") pod \"router-default-5b4dd555d4-jwzh7\" (UID: \"3c9aacb3-1142-4610-b990-a28d15998e71\") " pod="openshift-ingress/router-default-5b4dd555d4-jwzh7"
Sep 16 12:48:57.456774 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:48:57.456740    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/5abef85b-1f8d-4f0e-9091-c0f42acf1af1-samples-operator-tls\") pod \"cluster-samples-operator-86499c89c4-xvxlf\" (UID: \"5abef85b-1f8d-4f0e-9091-c0f42acf1af1\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-xvxlf"
Sep 16 12:48:57.459012 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:48:57.458993    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/5abef85b-1f8d-4f0e-9091-c0f42acf1af1-samples-operator-tls\") pod \"cluster-samples-operator-86499c89c4-xvxlf\" (UID: \"5abef85b-1f8d-4f0e-9091-c0f42acf1af1\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-xvxlf"
Sep 16 12:48:57.567647 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:48:57.567616    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-xvxlf"
Sep 16 12:48:57.844368 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:48:57.844324    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-xvxlf"]
Sep 16 12:48:57.915080 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:48:57.915048    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-xvxlf" event={"ID":"5abef85b-1f8d-4f0e-9091-c0f42acf1af1","Type":"ContainerStarted","Data":"4c4a7bd7489a5341e5d3f38afe4d3b00087f555d918731b9a35b237926f14ec0"}
Sep 16 12:48:59.923673 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:48:59.923585    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-xvxlf" event={"ID":"5abef85b-1f8d-4f0e-9091-c0f42acf1af1","Type":"ContainerStarted","Data":"19fe0386a2710a5dfd0ebde55a97eb61d1b100d4e5daa659888066a9fc36de85"}
Sep 16 12:48:59.923673 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:48:59.923642    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-xvxlf" event={"ID":"5abef85b-1f8d-4f0e-9091-c0f42acf1af1","Type":"ContainerStarted","Data":"39242a4f3f4c0ddf0674a200457281bdcd687f282eaf0db8c5102387a906b968"}
Sep 16 12:48:59.942137 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:48:59.942090    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-xvxlf" podStartSLOduration=17.275568161 podStartE2EDuration="18.942072795s" totalImagesPullingTime="1.666504634s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 12:48:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 12:48:59.94163424 +0000 UTC m=+78.864506709" watchObservedRunningTime="2026-09-16 12:48:59.942072795 +0000 UTC m=+78.864945264"
Sep 16 12:49:02.897336 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:49:02.897299    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/e12762e0-2a1d-4bf3-8fca-d22ad8e1eecf-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-2m9hh\" (UID: \"e12762e0-2a1d-4bf3-8fca-d22ad8e1eecf\") " pod="openshift-insights/insights-runtime-extractor-2m9hh"
Sep 16 12:49:02.897731 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:49:02.897364    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/e12762e0-2a1d-4bf3-8fca-d22ad8e1eecf-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-2m9hh\" (UID: \"e12762e0-2a1d-4bf3-8fca-d22ad8e1eecf\") " pod="openshift-insights/insights-runtime-extractor-2m9hh"
Sep 16 12:49:02.897857 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:49:02.897837    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/e12762e0-2a1d-4bf3-8fca-d22ad8e1eecf-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-2m9hh\" (UID: \"e12762e0-2a1d-4bf3-8fca-d22ad8e1eecf\") " pod="openshift-insights/insights-runtime-extractor-2m9hh"
Sep 16 12:49:02.899444 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:49:02.899426    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/e12762e0-2a1d-4bf3-8fca-d22ad8e1eecf-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-2m9hh\" (UID: \"e12762e0-2a1d-4bf3-8fca-d22ad8e1eecf\") " pod="openshift-insights/insights-runtime-extractor-2m9hh"
Sep 16 12:49:02.961927 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:49:02.961890    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-2m9hh"
Sep 16 12:49:03.234734 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:49:03.234653    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-2m9hh"]
Sep 16 12:49:03.237937 ip-10-0-133-218 kubenswrapper[2580]: W0916 12:49:03.237912    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode12762e0_2a1d_4bf3_8fca_d22ad8e1eecf.slice/crio-9d6bb34f8f86e012a1f6a29dc8665f5335a746d9cdf6e16df435c35a3538b3d1 WatchSource:0}: Error finding container 9d6bb34f8f86e012a1f6a29dc8665f5335a746d9cdf6e16df435c35a3538b3d1: Status 404 returned error can't find the container with id 9d6bb34f8f86e012a1f6a29dc8665f5335a746d9cdf6e16df435c35a3538b3d1
Sep 16 12:49:03.939655 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:49:03.939620    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-2m9hh" event={"ID":"e12762e0-2a1d-4bf3-8fca-d22ad8e1eecf","Type":"ContainerStarted","Data":"b7222ad45dcffdebdd2e5fa411ae90727b3f572535187fd0e4914e84e2791eb2"}
Sep 16 12:49:03.940060 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:49:03.939676    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-2m9hh" event={"ID":"e12762e0-2a1d-4bf3-8fca-d22ad8e1eecf","Type":"ContainerStarted","Data":"9d6bb34f8f86e012a1f6a29dc8665f5335a746d9cdf6e16df435c35a3538b3d1"}
Sep 16 12:49:04.944525 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:49:04.944437    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-2m9hh" event={"ID":"e12762e0-2a1d-4bf3-8fca-d22ad8e1eecf","Type":"ContainerStarted","Data":"c3e8de95d4aed43da5fd9c0877d223ab4b7c7bdd58c9fcf2d40faac4b473dd2a"}
Sep 16 12:49:06.953002 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:49:06.952966    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-2m9hh" event={"ID":"e12762e0-2a1d-4bf3-8fca-d22ad8e1eecf","Type":"ContainerStarted","Data":"190e65c657c8ad8dcb436abf9d5d993994097c5b8a76c22b4b6367e82bc7fb0f"}
Sep 16 12:49:06.975780 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:49:06.975736    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-2m9hh" podStartSLOduration=18.364031727 podStartE2EDuration="20.975719359s" totalImagesPullingTime="2.611687632s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-16 12:48:46 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 12:49:06.975356095 +0000 UTC m=+85.898228564" watchObservedRunningTime="2026-09-16 12:49:06.975719359 +0000 UTC m=+85.898591827"
Sep 16 12:49:12.965443 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:49:12.965411    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-network-console/networking-console-plugin-5dcbc7df6b-jv8fx"]
Sep 16 12:49:12.969333 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:49:12.969315    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-jv8fx"
Sep 16 12:49:12.972074 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:49:12.972053    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"default-dockercfg-g6cg7\""
Sep 16 12:49:12.972802 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:49:12.972786    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"networking-console-plugin-cert\""
Sep 16 12:49:12.973088 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:49:12.973072    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-console\"/\"networking-console-plugin\""
Sep 16 12:49:12.988414 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:49:12.988394    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-5dcbc7df6b-jv8fx"]
Sep 16 12:49:13.072440 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:49:13.072407    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/14b398fc-37e1-4712-8196-6f1ac6f70967-nginx-conf\") pod \"networking-console-plugin-5dcbc7df6b-jv8fx\" (UID: \"14b398fc-37e1-4712-8196-6f1ac6f70967\") " pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-jv8fx"
Sep 16 12:49:13.072440 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:49:13.072447    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/14b398fc-37e1-4712-8196-6f1ac6f70967-networking-console-plugin-cert\") pod \"networking-console-plugin-5dcbc7df6b-jv8fx\" (UID: \"14b398fc-37e1-4712-8196-6f1ac6f70967\") " pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-jv8fx"
Sep 16 12:49:13.173564 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:49:13.173526    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/14b398fc-37e1-4712-8196-6f1ac6f70967-nginx-conf\") pod \"networking-console-plugin-5dcbc7df6b-jv8fx\" (UID: \"14b398fc-37e1-4712-8196-6f1ac6f70967\") " pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-jv8fx"
Sep 16 12:49:13.174025 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:49:13.173565    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/14b398fc-37e1-4712-8196-6f1ac6f70967-networking-console-plugin-cert\") pod \"networking-console-plugin-5dcbc7df6b-jv8fx\" (UID: \"14b398fc-37e1-4712-8196-6f1ac6f70967\") " pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-jv8fx"
Sep 16 12:49:13.174848 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:49:13.174817    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/14b398fc-37e1-4712-8196-6f1ac6f70967-nginx-conf\") pod \"networking-console-plugin-5dcbc7df6b-jv8fx\" (UID: \"14b398fc-37e1-4712-8196-6f1ac6f70967\") " pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-jv8fx"
Sep 16 12:49:13.177672 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:49:13.176883    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/14b398fc-37e1-4712-8196-6f1ac6f70967-networking-console-plugin-cert\") pod \"networking-console-plugin-5dcbc7df6b-jv8fx\" (UID: \"14b398fc-37e1-4712-8196-6f1ac6f70967\") " pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-jv8fx"
Sep 16 12:49:13.277340 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:49:13.277261    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-jv8fx"
Sep 16 12:49:13.375536 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:49:13.375479    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3c9aacb3-1142-4610-b990-a28d15998e71-service-ca-bundle\") pod \"router-default-5b4dd555d4-jwzh7\" (UID: \"3c9aacb3-1142-4610-b990-a28d15998e71\") " pod="openshift-ingress/router-default-5b4dd555d4-jwzh7"
Sep 16 12:49:13.376073 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:49:13.376054    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3c9aacb3-1142-4610-b990-a28d15998e71-service-ca-bundle\") pod \"router-default-5b4dd555d4-jwzh7\" (UID: \"3c9aacb3-1142-4610-b990-a28d15998e71\") " pod="openshift-ingress/router-default-5b4dd555d4-jwzh7"
Sep 16 12:49:13.549805 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:49:13.549776    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-5dcbc7df6b-jv8fx"]
Sep 16 12:49:13.553070 ip-10-0-133-218 kubenswrapper[2580]: W0916 12:49:13.553043    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod14b398fc_37e1_4712_8196_6f1ac6f70967.slice/crio-b6772aad6c780ff9020cd9df5f1a063613552bda4aa3827d74d3ab8d86279def WatchSource:0}: Error finding container b6772aad6c780ff9020cd9df5f1a063613552bda4aa3827d74d3ab8d86279def: Status 404 returned error can't find the container with id b6772aad6c780ff9020cd9df5f1a063613552bda4aa3827d74d3ab8d86279def
Sep 16 12:49:13.652400 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:49:13.652369    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-5b4dd555d4-jwzh7"
Sep 16 12:49:13.954608 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:49:13.954518    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/router-default-5b4dd555d4-jwzh7"]
Sep 16 12:49:13.959159 ip-10-0-133-218 kubenswrapper[2580]: W0916 12:49:13.959130    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3c9aacb3_1142_4610_b990_a28d15998e71.slice/crio-a7e120225a9053251030f720d466cff6158fc36abe20ab8fb0773377b29df7c5 WatchSource:0}: Error finding container a7e120225a9053251030f720d466cff6158fc36abe20ab8fb0773377b29df7c5: Status 404 returned error can't find the container with id a7e120225a9053251030f720d466cff6158fc36abe20ab8fb0773377b29df7c5
Sep 16 12:49:13.975717 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:49:13.975689    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-jv8fx" event={"ID":"14b398fc-37e1-4712-8196-6f1ac6f70967","Type":"ContainerStarted","Data":"b6772aad6c780ff9020cd9df5f1a063613552bda4aa3827d74d3ab8d86279def"}
Sep 16 12:49:13.977300 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:49:13.977270    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-5b4dd555d4-jwzh7" event={"ID":"3c9aacb3-1142-4610-b990-a28d15998e71","Type":"ContainerStarted","Data":"a7e120225a9053251030f720d466cff6158fc36abe20ab8fb0773377b29df7c5"}
Sep 16 12:49:14.982321 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:49:14.982286    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-jv8fx" event={"ID":"14b398fc-37e1-4712-8196-6f1ac6f70967","Type":"ContainerStarted","Data":"025644a52607293ac4638bb3b916ac5acc2ce814de32a9045caac1ce153affb4"}
Sep 16 12:49:14.983881 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:49:14.983861    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-5b4dd555d4-jwzh7" event={"ID":"3c9aacb3-1142-4610-b990-a28d15998e71","Type":"ContainerStarted","Data":"6537ee9a86db1ac60a1e96e122faddced99e009c07dd3036cf3c3f91ad2de902"}
Sep 16 12:49:14.998266 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:49:14.998221    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-jv8fx" podStartSLOduration=2.061544007 podStartE2EDuration="2.998206099s" totalImagesPullingTime="936.662092ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 12:49:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 12:49:14.997605575 +0000 UTC m=+93.920478048" watchObservedRunningTime="2026-09-16 12:49:14.998206099 +0000 UTC m=+93.921078567"
Sep 16 12:49:15.019540 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:49:15.019473    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/router-default-5b4dd555d4-jwzh7" podStartSLOduration=34.019436001 podStartE2EDuration="34.019436001s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 12:48:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 12:49:15.018637785 +0000 UTC m=+93.941510253" watchObservedRunningTime="2026-09-16 12:49:15.019436001 +0000 UTC m=+93.942308470"
Sep 16 12:49:15.652695 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:49:15.652658    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/router-default-5b4dd555d4-jwzh7"
Sep 16 12:49:15.655065 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:49:15.655042    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/router-default-5b4dd555d4-jwzh7"
Sep 16 12:49:15.987242 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:49:15.987161    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/router-default-5b4dd555d4-jwzh7"
Sep 16 12:49:15.988359 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:49:15.988336    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/router-default-5b4dd555d4-jwzh7"
Sep 16 12:49:16.959615 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:49:16.959568    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-bzpbv"]
Sep 16 12:49:16.964682 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:49:16.964659    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-bzpbv"
Sep 16 12:49:16.968579 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:49:16.968559    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-tls\""
Sep 16 12:49:16.968665 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:49:16.968559    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-dockercfg-cdmxh\""
Sep 16 12:49:16.977373 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:49:16.977351    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-bzpbv"]
Sep 16 12:49:17.000883 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:49:17.000855    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/882dd5ec-93c0-40fb-a3c0-c463add9c9ca-tls-certificates\") pod \"prometheus-operator-admission-webhook-5864cd99d7-bzpbv\" (UID: \"882dd5ec-93c0-40fb-a3c0-c463add9c9ca\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-bzpbv"
Sep 16 12:49:17.101313 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:49:17.101276    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/882dd5ec-93c0-40fb-a3c0-c463add9c9ca-tls-certificates\") pod \"prometheus-operator-admission-webhook-5864cd99d7-bzpbv\" (UID: \"882dd5ec-93c0-40fb-a3c0-c463add9c9ca\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-bzpbv"
Sep 16 12:49:17.101514 ip-10-0-133-218 kubenswrapper[2580]: E0916 12:49:17.101427    2580 secret.go:189] Couldn't get secret openshift-monitoring/prometheus-operator-admission-webhook-tls: secret "prometheus-operator-admission-webhook-tls" not found
Sep 16 12:49:17.101598 ip-10-0-133-218 kubenswrapper[2580]: E0916 12:49:17.101517    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/882dd5ec-93c0-40fb-a3c0-c463add9c9ca-tls-certificates podName:882dd5ec-93c0-40fb-a3c0-c463add9c9ca nodeName:}" failed. No retries permitted until 2026-09-16 12:49:17.601477498 +0000 UTC m=+96.524349963 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "tls-certificates" (UniqueName: "kubernetes.io/secret/882dd5ec-93c0-40fb-a3c0-c463add9c9ca-tls-certificates") pod "prometheus-operator-admission-webhook-5864cd99d7-bzpbv" (UID: "882dd5ec-93c0-40fb-a3c0-c463add9c9ca") : secret "prometheus-operator-admission-webhook-tls" not found
Sep 16 12:49:17.604547 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:49:17.604513    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/882dd5ec-93c0-40fb-a3c0-c463add9c9ca-tls-certificates\") pod \"prometheus-operator-admission-webhook-5864cd99d7-bzpbv\" (UID: \"882dd5ec-93c0-40fb-a3c0-c463add9c9ca\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-bzpbv"
Sep 16 12:49:17.604710 ip-10-0-133-218 kubenswrapper[2580]: E0916 12:49:17.604620    2580 secret.go:189] Couldn't get secret openshift-monitoring/prometheus-operator-admission-webhook-tls: secret "prometheus-operator-admission-webhook-tls" not found
Sep 16 12:49:17.604710 ip-10-0-133-218 kubenswrapper[2580]: E0916 12:49:17.604668    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/882dd5ec-93c0-40fb-a3c0-c463add9c9ca-tls-certificates podName:882dd5ec-93c0-40fb-a3c0-c463add9c9ca nodeName:}" failed. No retries permitted until 2026-09-16 12:49:18.604655347 +0000 UTC m=+97.527527794 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "tls-certificates" (UniqueName: "kubernetes.io/secret/882dd5ec-93c0-40fb-a3c0-c463add9c9ca-tls-certificates") pod "prometheus-operator-admission-webhook-5864cd99d7-bzpbv" (UID: "882dd5ec-93c0-40fb-a3c0-c463add9c9ca") : secret "prometheus-operator-admission-webhook-tls" not found
Sep 16 12:49:18.612153 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:49:18.612109    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/882dd5ec-93c0-40fb-a3c0-c463add9c9ca-tls-certificates\") pod \"prometheus-operator-admission-webhook-5864cd99d7-bzpbv\" (UID: \"882dd5ec-93c0-40fb-a3c0-c463add9c9ca\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-bzpbv"
Sep 16 12:49:18.614359 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:49:18.614340    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/882dd5ec-93c0-40fb-a3c0-c463add9c9ca-tls-certificates\") pod \"prometheus-operator-admission-webhook-5864cd99d7-bzpbv\" (UID: \"882dd5ec-93c0-40fb-a3c0-c463add9c9ca\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-bzpbv"
Sep 16 12:49:18.773015 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:49:18.772989    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-bzpbv"
Sep 16 12:49:19.067284 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:49:19.067247    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-bzpbv"]
Sep 16 12:49:19.072610 ip-10-0-133-218 kubenswrapper[2580]: W0916 12:49:19.072584    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod882dd5ec_93c0_40fb_a3c0_c463add9c9ca.slice/crio-250fd82f2599f67cab4073658fbd8f1abc462025f6b43bf6e942c9d727b7846b WatchSource:0}: Error finding container 250fd82f2599f67cab4073658fbd8f1abc462025f6b43bf6e942c9d727b7846b: Status 404 returned error can't find the container with id 250fd82f2599f67cab4073658fbd8f1abc462025f6b43bf6e942c9d727b7846b
Sep 16 12:49:19.822426 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:49:19.822386    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/b352b0f0-afd0-48c2-965f-7cc63cdf04fd-metrics-tls\") pod \"dns-default-hzdmm\" (UID: \"b352b0f0-afd0-48c2-965f-7cc63cdf04fd\") " pod="openshift-dns/dns-default-hzdmm"
Sep 16 12:49:19.824923 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:49:19.824900    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/b352b0f0-afd0-48c2-965f-7cc63cdf04fd-metrics-tls\") pod \"dns-default-hzdmm\" (UID: \"b352b0f0-afd0-48c2-965f-7cc63cdf04fd\") " pod="openshift-dns/dns-default-hzdmm"
Sep 16 12:49:19.900436 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:49:19.900404    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-dfdpm\""
Sep 16 12:49:19.908538 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:49:19.908514    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-hzdmm"
Sep 16 12:49:20.001604 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:49:20.001523    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-bzpbv" event={"ID":"882dd5ec-93c0-40fb-a3c0-c463add9c9ca","Type":"ContainerStarted","Data":"250fd82f2599f67cab4073658fbd8f1abc462025f6b43bf6e942c9d727b7846b"}
Sep 16 12:49:20.295344 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:49:20.295316    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-hzdmm"]
Sep 16 12:49:20.298707 ip-10-0-133-218 kubenswrapper[2580]: W0916 12:49:20.298684    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb352b0f0_afd0_48c2_965f_7cc63cdf04fd.slice/crio-d18264595bde6268ea389684b710f01288b003f3df8c211a6c7273bed16fc02b WatchSource:0}: Error finding container d18264595bde6268ea389684b710f01288b003f3df8c211a6c7273bed16fc02b: Status 404 returned error can't find the container with id d18264595bde6268ea389684b710f01288b003f3df8c211a6c7273bed16fc02b
Sep 16 12:49:20.594564 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:49:20.594474    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-d55cc74b5-x27t7"]
Sep 16 12:49:20.596524 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:49:20.596507    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-d55cc74b5-x27t7"
Sep 16 12:49:20.599172 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:49:20.599148    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\""
Sep 16 12:49:20.599405 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:49:20.599386    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\""
Sep 16 12:49:20.599529 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:49:20.599513    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\""
Sep 16 12:49:20.600191 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:49:20.600175    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\""
Sep 16 12:49:20.600387 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:49:20.600365    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\""
Sep 16 12:49:20.600518 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:49:20.600392    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-d5xsr\""
Sep 16 12:49:20.600518 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:49:20.600429    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\""
Sep 16 12:49:20.600638 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:49:20.600626    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\""
Sep 16 12:49:20.613416 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:49:20.613394    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-d55cc74b5-x27t7"]
Sep 16 12:49:20.729887 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:49:20.729855    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/3720e939-652a-4ad2-938d-72adb38487ad-oauth-serving-cert\") pod \"console-d55cc74b5-x27t7\" (UID: \"3720e939-652a-4ad2-938d-72adb38487ad\") " pod="openshift-console/console-d55cc74b5-x27t7"
Sep 16 12:49:20.729887 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:49:20.729892    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/3720e939-652a-4ad2-938d-72adb38487ad-service-ca\") pod \"console-d55cc74b5-x27t7\" (UID: \"3720e939-652a-4ad2-938d-72adb38487ad\") " pod="openshift-console/console-d55cc74b5-x27t7"
Sep 16 12:49:20.730108 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:49:20.729911    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-smjrp\" (UniqueName: \"kubernetes.io/projected/3720e939-652a-4ad2-938d-72adb38487ad-kube-api-access-smjrp\") pod \"console-d55cc74b5-x27t7\" (UID: \"3720e939-652a-4ad2-938d-72adb38487ad\") " pod="openshift-console/console-d55cc74b5-x27t7"
Sep 16 12:49:20.730108 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:49:20.729974    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/3720e939-652a-4ad2-938d-72adb38487ad-console-serving-cert\") pod \"console-d55cc74b5-x27t7\" (UID: \"3720e939-652a-4ad2-938d-72adb38487ad\") " pod="openshift-console/console-d55cc74b5-x27t7"
Sep 16 12:49:20.730108 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:49:20.729991    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/3720e939-652a-4ad2-938d-72adb38487ad-console-config\") pod \"console-d55cc74b5-x27t7\" (UID: \"3720e939-652a-4ad2-938d-72adb38487ad\") " pod="openshift-console/console-d55cc74b5-x27t7"
Sep 16 12:49:20.730108 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:49:20.730016    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/3720e939-652a-4ad2-938d-72adb38487ad-console-oauth-config\") pod \"console-d55cc74b5-x27t7\" (UID: \"3720e939-652a-4ad2-938d-72adb38487ad\") " pod="openshift-console/console-d55cc74b5-x27t7"
Sep 16 12:49:20.830949 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:49:20.830918    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/3720e939-652a-4ad2-938d-72adb38487ad-console-serving-cert\") pod \"console-d55cc74b5-x27t7\" (UID: \"3720e939-652a-4ad2-938d-72adb38487ad\") " pod="openshift-console/console-d55cc74b5-x27t7"
Sep 16 12:49:20.830949 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:49:20.830964    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/3720e939-652a-4ad2-938d-72adb38487ad-console-config\") pod \"console-d55cc74b5-x27t7\" (UID: \"3720e939-652a-4ad2-938d-72adb38487ad\") " pod="openshift-console/console-d55cc74b5-x27t7"
Sep 16 12:49:20.831465 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:49:20.831023    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/3720e939-652a-4ad2-938d-72adb38487ad-console-oauth-config\") pod \"console-d55cc74b5-x27t7\" (UID: \"3720e939-652a-4ad2-938d-72adb38487ad\") " pod="openshift-console/console-d55cc74b5-x27t7"
Sep 16 12:49:20.831465 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:49:20.831074    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/3720e939-652a-4ad2-938d-72adb38487ad-oauth-serving-cert\") pod \"console-d55cc74b5-x27t7\" (UID: \"3720e939-652a-4ad2-938d-72adb38487ad\") " pod="openshift-console/console-d55cc74b5-x27t7"
Sep 16 12:49:20.831465 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:49:20.831098    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/3720e939-652a-4ad2-938d-72adb38487ad-service-ca\") pod \"console-d55cc74b5-x27t7\" (UID: \"3720e939-652a-4ad2-938d-72adb38487ad\") " pod="openshift-console/console-d55cc74b5-x27t7"
Sep 16 12:49:20.831465 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:49:20.831116    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-smjrp\" (UniqueName: \"kubernetes.io/projected/3720e939-652a-4ad2-938d-72adb38487ad-kube-api-access-smjrp\") pod \"console-d55cc74b5-x27t7\" (UID: \"3720e939-652a-4ad2-938d-72adb38487ad\") " pod="openshift-console/console-d55cc74b5-x27t7"
Sep 16 12:49:20.831825 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:49:20.831805    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/3720e939-652a-4ad2-938d-72adb38487ad-console-config\") pod \"console-d55cc74b5-x27t7\" (UID: \"3720e939-652a-4ad2-938d-72adb38487ad\") " pod="openshift-console/console-d55cc74b5-x27t7"
Sep 16 12:49:20.831891 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:49:20.831820    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/3720e939-652a-4ad2-938d-72adb38487ad-service-ca\") pod \"console-d55cc74b5-x27t7\" (UID: \"3720e939-652a-4ad2-938d-72adb38487ad\") " pod="openshift-console/console-d55cc74b5-x27t7"
Sep 16 12:49:20.831891 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:49:20.831826    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/3720e939-652a-4ad2-938d-72adb38487ad-oauth-serving-cert\") pod \"console-d55cc74b5-x27t7\" (UID: \"3720e939-652a-4ad2-938d-72adb38487ad\") " pod="openshift-console/console-d55cc74b5-x27t7"
Sep 16 12:49:20.833478 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:49:20.833457    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/3720e939-652a-4ad2-938d-72adb38487ad-console-serving-cert\") pod \"console-d55cc74b5-x27t7\" (UID: \"3720e939-652a-4ad2-938d-72adb38487ad\") " pod="openshift-console/console-d55cc74b5-x27t7"
Sep 16 12:49:20.833478 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:49:20.833466    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/3720e939-652a-4ad2-938d-72adb38487ad-console-oauth-config\") pod \"console-d55cc74b5-x27t7\" (UID: \"3720e939-652a-4ad2-938d-72adb38487ad\") " pod="openshift-console/console-d55cc74b5-x27t7"
Sep 16 12:49:20.839863 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:49:20.839838    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-smjrp\" (UniqueName: \"kubernetes.io/projected/3720e939-652a-4ad2-938d-72adb38487ad-kube-api-access-smjrp\") pod \"console-d55cc74b5-x27t7\" (UID: \"3720e939-652a-4ad2-938d-72adb38487ad\") " pod="openshift-console/console-d55cc74b5-x27t7"
Sep 16 12:49:20.905906 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:49:20.905829    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-d55cc74b5-x27t7"
Sep 16 12:49:21.006434 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:49:21.006400    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-bzpbv" event={"ID":"882dd5ec-93c0-40fb-a3c0-c463add9c9ca","Type":"ContainerStarted","Data":"fa87cdec503457ef28da4d222c5a7c93364bc848010ea2c656f8141eb58ded6d"}
Sep 16 12:49:21.006634 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:49:21.006616    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-bzpbv"
Sep 16 12:49:21.008079 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:49:21.008055    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-hzdmm" event={"ID":"b352b0f0-afd0-48c2-965f-7cc63cdf04fd","Type":"ContainerStarted","Data":"d18264595bde6268ea389684b710f01288b003f3df8c211a6c7273bed16fc02b"}
Sep 16 12:49:21.012972 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:49:21.012924    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-bzpbv"
Sep 16 12:49:21.029373 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:49:21.029321    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-bzpbv" podStartSLOduration=4.054282418 podStartE2EDuration="5.029303214s" totalImagesPullingTime="975.020796ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 12:49:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 12:49:21.028594879 +0000 UTC m=+99.951467350" watchObservedRunningTime="2026-09-16 12:49:21.029303214 +0000 UTC m=+99.952175684"
Sep 16 12:49:21.179109 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:49:21.178999    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-d55cc74b5-x27t7"]
Sep 16 12:49:21.182668 ip-10-0-133-218 kubenswrapper[2580]: W0916 12:49:21.182641    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3720e939_652a_4ad2_938d_72adb38487ad.slice/crio-0daabbeb96f09671190104a791d0a67cc98a11e67432e661673b6e91d95593ff WatchSource:0}: Error finding container 0daabbeb96f09671190104a791d0a67cc98a11e67432e661673b6e91d95593ff: Status 404 returned error can't find the container with id 0daabbeb96f09671190104a791d0a67cc98a11e67432e661673b6e91d95593ff
Sep 16 12:49:22.014064 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:49:22.014014    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-d55cc74b5-x27t7" event={"ID":"3720e939-652a-4ad2-938d-72adb38487ad","Type":"ContainerStarted","Data":"0daabbeb96f09671190104a791d0a67cc98a11e67432e661673b6e91d95593ff"}
Sep 16 12:49:22.899788 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:49:22.899757    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-bpv5d"
Sep 16 12:49:23.021022 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:49:23.020990    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-hzdmm" event={"ID":"b352b0f0-afd0-48c2-965f-7cc63cdf04fd","Type":"ContainerStarted","Data":"d7e4aa4df2ac91fb1edc5d9563e7fac48cc24b494615e1824ebadaa5bb0b3db8"}
Sep 16 12:49:23.021433 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:49:23.021044    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-hzdmm" event={"ID":"b352b0f0-afd0-48c2-965f-7cc63cdf04fd","Type":"ContainerStarted","Data":"c4ba73aa53029936478308da5254558f9e5c2b2327a6e5bdeed8ad7b357dc8b2"}
Sep 16 12:49:23.040499 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:49:23.040444    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-hzdmm" podStartSLOduration=66.019868064 podStartE2EDuration="1m8.040424706s" totalImagesPullingTime="2.020556642s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 12:48:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 12:49:23.039323556 +0000 UTC m=+101.962196055" watchObservedRunningTime="2026-09-16 12:49:23.040424706 +0000 UTC m=+101.963297225"
Sep 16 12:49:24.028113 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:49:24.026755    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-hzdmm"
Sep 16 12:49:25.030630 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:49:25.030588    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-d55cc74b5-x27t7" event={"ID":"3720e939-652a-4ad2-938d-72adb38487ad","Type":"ContainerStarted","Data":"f57210c5691648e52d71f656b0238592e32bd407dd704a5e5e12f90ecc4a8de8"}
Sep 16 12:49:25.051813 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:49:25.051768    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-d55cc74b5-x27t7" podStartSLOduration=2.185582384 podStartE2EDuration="5.051755262s" totalImagesPullingTime="2.866172878s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 12:49:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 12:49:25.050354568 +0000 UTC m=+103.973227047" watchObservedRunningTime="2026-09-16 12:49:25.051755262 +0000 UTC m=+103.974627730"
Sep 16 12:49:27.552837 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:49:27.552805    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-7996cc44cd-p6ljc"]
Sep 16 12:49:27.555260 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:49:27.555244    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7996cc44cd-p6ljc"
Sep 16 12:49:27.580003 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:49:27.579977    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\""
Sep 16 12:49:27.592733 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:49:27.592711    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-7996cc44cd-p6ljc"]
Sep 16 12:49:27.682092 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:49:27.682059    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/c5a49713-de16-4f73-b654-d4ed06ce1946-service-ca\") pod \"console-7996cc44cd-p6ljc\" (UID: \"c5a49713-de16-4f73-b654-d4ed06ce1946\") " pod="openshift-console/console-7996cc44cd-p6ljc"
Sep 16 12:49:27.682092 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:49:27.682091    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c5a49713-de16-4f73-b654-d4ed06ce1946-trusted-ca-bundle\") pod \"console-7996cc44cd-p6ljc\" (UID: \"c5a49713-de16-4f73-b654-d4ed06ce1946\") " pod="openshift-console/console-7996cc44cd-p6ljc"
Sep 16 12:49:27.682299 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:49:27.682120    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/c5a49713-de16-4f73-b654-d4ed06ce1946-console-serving-cert\") pod \"console-7996cc44cd-p6ljc\" (UID: \"c5a49713-de16-4f73-b654-d4ed06ce1946\") " pod="openshift-console/console-7996cc44cd-p6ljc"
Sep 16 12:49:27.682299 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:49:27.682214    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/c5a49713-de16-4f73-b654-d4ed06ce1946-oauth-serving-cert\") pod \"console-7996cc44cd-p6ljc\" (UID: \"c5a49713-de16-4f73-b654-d4ed06ce1946\") " pod="openshift-console/console-7996cc44cd-p6ljc"
Sep 16 12:49:27.682299 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:49:27.682258    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/c5a49713-de16-4f73-b654-d4ed06ce1946-console-oauth-config\") pod \"console-7996cc44cd-p6ljc\" (UID: \"c5a49713-de16-4f73-b654-d4ed06ce1946\") " pod="openshift-console/console-7996cc44cd-p6ljc"
Sep 16 12:49:27.682299 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:49:27.682279    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/c5a49713-de16-4f73-b654-d4ed06ce1946-console-config\") pod \"console-7996cc44cd-p6ljc\" (UID: \"c5a49713-de16-4f73-b654-d4ed06ce1946\") " pod="openshift-console/console-7996cc44cd-p6ljc"
Sep 16 12:49:27.682436 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:49:27.682329    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zd74h\" (UniqueName: \"kubernetes.io/projected/c5a49713-de16-4f73-b654-d4ed06ce1946-kube-api-access-zd74h\") pod \"console-7996cc44cd-p6ljc\" (UID: \"c5a49713-de16-4f73-b654-d4ed06ce1946\") " pod="openshift-console/console-7996cc44cd-p6ljc"
Sep 16 12:49:27.783378 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:49:27.783343    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/c5a49713-de16-4f73-b654-d4ed06ce1946-oauth-serving-cert\") pod \"console-7996cc44cd-p6ljc\" (UID: \"c5a49713-de16-4f73-b654-d4ed06ce1946\") " pod="openshift-console/console-7996cc44cd-p6ljc"
Sep 16 12:49:27.783557 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:49:27.783392    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/c5a49713-de16-4f73-b654-d4ed06ce1946-console-oauth-config\") pod \"console-7996cc44cd-p6ljc\" (UID: \"c5a49713-de16-4f73-b654-d4ed06ce1946\") " pod="openshift-console/console-7996cc44cd-p6ljc"
Sep 16 12:49:27.783557 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:49:27.783435    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/c5a49713-de16-4f73-b654-d4ed06ce1946-console-config\") pod \"console-7996cc44cd-p6ljc\" (UID: \"c5a49713-de16-4f73-b654-d4ed06ce1946\") " pod="openshift-console/console-7996cc44cd-p6ljc"
Sep 16 12:49:27.783557 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:49:27.783476    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zd74h\" (UniqueName: \"kubernetes.io/projected/c5a49713-de16-4f73-b654-d4ed06ce1946-kube-api-access-zd74h\") pod \"console-7996cc44cd-p6ljc\" (UID: \"c5a49713-de16-4f73-b654-d4ed06ce1946\") " pod="openshift-console/console-7996cc44cd-p6ljc"
Sep 16 12:49:27.783705 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:49:27.783618    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/c5a49713-de16-4f73-b654-d4ed06ce1946-service-ca\") pod \"console-7996cc44cd-p6ljc\" (UID: \"c5a49713-de16-4f73-b654-d4ed06ce1946\") " pod="openshift-console/console-7996cc44cd-p6ljc"
Sep 16 12:49:27.783705 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:49:27.783658    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c5a49713-de16-4f73-b654-d4ed06ce1946-trusted-ca-bundle\") pod \"console-7996cc44cd-p6ljc\" (UID: \"c5a49713-de16-4f73-b654-d4ed06ce1946\") " pod="openshift-console/console-7996cc44cd-p6ljc"
Sep 16 12:49:27.783705 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:49:27.783693    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/c5a49713-de16-4f73-b654-d4ed06ce1946-console-serving-cert\") pod \"console-7996cc44cd-p6ljc\" (UID: \"c5a49713-de16-4f73-b654-d4ed06ce1946\") " pod="openshift-console/console-7996cc44cd-p6ljc"
Sep 16 12:49:27.784255 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:49:27.784189    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/c5a49713-de16-4f73-b654-d4ed06ce1946-oauth-serving-cert\") pod \"console-7996cc44cd-p6ljc\" (UID: \"c5a49713-de16-4f73-b654-d4ed06ce1946\") " pod="openshift-console/console-7996cc44cd-p6ljc"
Sep 16 12:49:27.784255 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:49:27.784259    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/c5a49713-de16-4f73-b654-d4ed06ce1946-console-config\") pod \"console-7996cc44cd-p6ljc\" (UID: \"c5a49713-de16-4f73-b654-d4ed06ce1946\") " pod="openshift-console/console-7996cc44cd-p6ljc"
Sep 16 12:49:27.784441 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:49:27.784301    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/c5a49713-de16-4f73-b654-d4ed06ce1946-service-ca\") pod \"console-7996cc44cd-p6ljc\" (UID: \"c5a49713-de16-4f73-b654-d4ed06ce1946\") " pod="openshift-console/console-7996cc44cd-p6ljc"
Sep 16 12:49:27.784664 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:49:27.784642    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c5a49713-de16-4f73-b654-d4ed06ce1946-trusted-ca-bundle\") pod \"console-7996cc44cd-p6ljc\" (UID: \"c5a49713-de16-4f73-b654-d4ed06ce1946\") " pod="openshift-console/console-7996cc44cd-p6ljc"
Sep 16 12:49:27.785962 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:49:27.785940    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/c5a49713-de16-4f73-b654-d4ed06ce1946-console-oauth-config\") pod \"console-7996cc44cd-p6ljc\" (UID: \"c5a49713-de16-4f73-b654-d4ed06ce1946\") " pod="openshift-console/console-7996cc44cd-p6ljc"
Sep 16 12:49:27.786083 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:49:27.786065    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/c5a49713-de16-4f73-b654-d4ed06ce1946-console-serving-cert\") pod \"console-7996cc44cd-p6ljc\" (UID: \"c5a49713-de16-4f73-b654-d4ed06ce1946\") " pod="openshift-console/console-7996cc44cd-p6ljc"
Sep 16 12:49:27.795579 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:49:27.795558    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zd74h\" (UniqueName: \"kubernetes.io/projected/c5a49713-de16-4f73-b654-d4ed06ce1946-kube-api-access-zd74h\") pod \"console-7996cc44cd-p6ljc\" (UID: \"c5a49713-de16-4f73-b654-d4ed06ce1946\") " pod="openshift-console/console-7996cc44cd-p6ljc"
Sep 16 12:49:27.863778 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:49:27.863701    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7996cc44cd-p6ljc"
Sep 16 12:49:28.147344 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:49:28.147276    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-7996cc44cd-p6ljc"]
Sep 16 12:49:28.150502 ip-10-0-133-218 kubenswrapper[2580]: W0916 12:49:28.150458    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc5a49713_de16_4f73_b654_d4ed06ce1946.slice/crio-fc0f1d94d98365283df52e710152c3245d835d8e6edddaeded10ae07ffd9f27d WatchSource:0}: Error finding container fc0f1d94d98365283df52e710152c3245d835d8e6edddaeded10ae07ffd9f27d: Status 404 returned error can't find the container with id fc0f1d94d98365283df52e710152c3245d835d8e6edddaeded10ae07ffd9f27d
Sep 16 12:49:29.045256 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:49:29.045223    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7996cc44cd-p6ljc" event={"ID":"c5a49713-de16-4f73-b654-d4ed06ce1946","Type":"ContainerStarted","Data":"99b285154caf119af4f490e632090411f57cc87e5e3777f5101a2396d4e89eb7"}
Sep 16 12:49:29.045716 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:49:29.045300    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7996cc44cd-p6ljc" event={"ID":"c5a49713-de16-4f73-b654-d4ed06ce1946","Type":"ContainerStarted","Data":"fc0f1d94d98365283df52e710152c3245d835d8e6edddaeded10ae07ffd9f27d"}
Sep 16 12:49:29.068550 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:49:29.068506    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-7996cc44cd-p6ljc" podStartSLOduration=2.068476177 podStartE2EDuration="2.068476177s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 12:49:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 12:49:29.067011128 +0000 UTC m=+107.989883594" watchObservedRunningTime="2026-09-16 12:49:29.068476177 +0000 UTC m=+107.991348646"
Sep 16 12:49:30.906595 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:49:30.906557    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-d55cc74b5-x27t7"
Sep 16 12:49:30.906992 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:49:30.906613    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-d55cc74b5-x27t7"
Sep 16 12:49:30.911270 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:49:30.911253    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-d55cc74b5-x27t7"
Sep 16 12:49:31.055822 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:49:31.055796    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-d55cc74b5-x27t7"
Sep 16 12:49:34.032659 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:49:34.032627    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-hzdmm"
Sep 16 12:49:35.496174 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:49:35.496139    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/openshift-state-metrics-67dcf8c979-22bgc"]
Sep 16 12:49:35.498479 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:49:35.498464    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-22bgc"
Sep 16 12:49:35.500765 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:49:35.500735    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-tls\""
Sep 16 12:49:35.500894 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:49:35.500791    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
Sep 16 12:49:35.500894 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:49:35.500795    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\""
Sep 16 12:49:35.501392 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:49:35.501373    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
Sep 16 12:49:35.501518 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:49:35.501417    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-dockercfg-vnch6\""
Sep 16 12:49:35.501518 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:49:35.501418    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-kube-rbac-proxy-config\""
Sep 16 12:49:35.511865 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:49:35.511843    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-67dcf8c979-22bgc"]
Sep 16 12:49:35.544820 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:49:35.544795    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/kube-state-metrics-59fbdd77d-lbkms"]
Sep 16 12:49:35.547053 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:49:35.547034    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-59fbdd77d-lbkms"
Sep 16 12:49:35.549399 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:49:35.549380    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-kube-rbac-proxy-config\""
Sep 16 12:49:35.549607 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:49:35.549593    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-dockercfg-nshrk\""
Sep 16 12:49:35.549826 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:49:35.549810    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-custom-resource-state-configmap\""
Sep 16 12:49:35.552809 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:49:35.552785    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-tls\""
Sep 16 12:49:35.560622 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:49:35.560605    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-9tn5t"]
Sep 16 12:49:35.562817 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:49:35.562802    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-9tn5t"
Sep 16 12:49:35.565097 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:49:35.565078    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-xr86w\""
Sep 16 12:49:35.565097 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:49:35.565091    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\""
Sep 16 12:49:35.565191 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:49:35.565091    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\""
Sep 16 12:49:35.565460 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:49:35.565445    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\""
Sep 16 12:49:35.569747 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:49:35.569729    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-59fbdd77d-lbkms"]
Sep 16 12:49:35.644012 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:49:35.643980    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/a0c1e96d-918c-423b-b136-d4c3584fd568-node-exporter-accelerators-collector-config\") pod \"node-exporter-9tn5t\" (UID: \"a0c1e96d-918c-423b-b136-d4c3584fd568\") " pod="openshift-monitoring/node-exporter-9tn5t"
Sep 16 12:49:35.644192 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:49:35.644022    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tkjd9\" (UniqueName: \"kubernetes.io/projected/cdef4a92-60a3-42f2-a20b-a15bce25691e-kube-api-access-tkjd9\") pod \"kube-state-metrics-59fbdd77d-lbkms\" (UID: \"cdef4a92-60a3-42f2-a20b-a15bce25691e\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-lbkms"
Sep 16 12:49:35.644192 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:49:35.644050    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/cdef4a92-60a3-42f2-a20b-a15bce25691e-kube-state-metrics-tls\") pod \"kube-state-metrics-59fbdd77d-lbkms\" (UID: \"cdef4a92-60a3-42f2-a20b-a15bce25691e\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-lbkms"
Sep 16 12:49:35.644192 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:49:35.644075    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/726a71b5-e75b-4313-a1f9-3d3d123b729c-metrics-client-ca\") pod \"openshift-state-metrics-67dcf8c979-22bgc\" (UID: \"726a71b5-e75b-4313-a1f9-3d3d123b729c\") " pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-22bgc"
Sep 16 12:49:35.644192 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:49:35.644096    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/a0c1e96d-918c-423b-b136-d4c3584fd568-node-exporter-textfile\") pod \"node-exporter-9tn5t\" (UID: \"a0c1e96d-918c-423b-b136-d4c3584fd568\") " pod="openshift-monitoring/node-exporter-9tn5t"
Sep 16 12:49:35.644192 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:49:35.644139    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/cdef4a92-60a3-42f2-a20b-a15bce25691e-metrics-client-ca\") pod \"kube-state-metrics-59fbdd77d-lbkms\" (UID: \"cdef4a92-60a3-42f2-a20b-a15bce25691e\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-lbkms"
Sep 16 12:49:35.644192 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:49:35.644176    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/cdef4a92-60a3-42f2-a20b-a15bce25691e-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-59fbdd77d-lbkms\" (UID: \"cdef4a92-60a3-42f2-a20b-a15bce25691e\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-lbkms"
Sep 16 12:49:35.644408 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:49:35.644226    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/a0c1e96d-918c-423b-b136-d4c3584fd568-root\") pod \"node-exporter-9tn5t\" (UID: \"a0c1e96d-918c-423b-b136-d4c3584fd568\") " pod="openshift-monitoring/node-exporter-9tn5t"
Sep 16 12:49:35.644408 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:49:35.644256    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/726a71b5-e75b-4313-a1f9-3d3d123b729c-openshift-state-metrics-tls\") pod \"openshift-state-metrics-67dcf8c979-22bgc\" (UID: \"726a71b5-e75b-4313-a1f9-3d3d123b729c\") " pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-22bgc"
Sep 16 12:49:35.644408 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:49:35.644279    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/a0c1e96d-918c-423b-b136-d4c3584fd568-node-exporter-wtmp\") pod \"node-exporter-9tn5t\" (UID: \"a0c1e96d-918c-423b-b136-d4c3584fd568\") " pod="openshift-monitoring/node-exporter-9tn5t"
Sep 16 12:49:35.644408 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:49:35.644320    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/a0c1e96d-918c-423b-b136-d4c3584fd568-metrics-client-ca\") pod \"node-exporter-9tn5t\" (UID: \"a0c1e96d-918c-423b-b136-d4c3584fd568\") " pod="openshift-monitoring/node-exporter-9tn5t"
Sep 16 12:49:35.644408 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:49:35.644355    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/a0c1e96d-918c-423b-b136-d4c3584fd568-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-9tn5t\" (UID: \"a0c1e96d-918c-423b-b136-d4c3584fd568\") " pod="openshift-monitoring/node-exporter-9tn5t"
Sep 16 12:49:35.644408 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:49:35.644380    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/cdef4a92-60a3-42f2-a20b-a15bce25691e-volume-directive-shadow\") pod \"kube-state-metrics-59fbdd77d-lbkms\" (UID: \"cdef4a92-60a3-42f2-a20b-a15bce25691e\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-lbkms"
Sep 16 12:49:35.644657 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:49:35.644407    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/a0c1e96d-918c-423b-b136-d4c3584fd568-sys\") pod \"node-exporter-9tn5t\" (UID: \"a0c1e96d-918c-423b-b136-d4c3584fd568\") " pod="openshift-monitoring/node-exporter-9tn5t"
Sep 16 12:49:35.644657 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:49:35.644434    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z8xxg\" (UniqueName: \"kubernetes.io/projected/726a71b5-e75b-4313-a1f9-3d3d123b729c-kube-api-access-z8xxg\") pod \"openshift-state-metrics-67dcf8c979-22bgc\" (UID: \"726a71b5-e75b-4313-a1f9-3d3d123b729c\") " pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-22bgc"
Sep 16 12:49:35.644657 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:49:35.644465    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/a0c1e96d-918c-423b-b136-d4c3584fd568-node-exporter-tls\") pod \"node-exporter-9tn5t\" (UID: \"a0c1e96d-918c-423b-b136-d4c3584fd568\") " pod="openshift-monitoring/node-exporter-9tn5t"
Sep 16 12:49:35.644657 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:49:35.644555    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/726a71b5-e75b-4313-a1f9-3d3d123b729c-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-67dcf8c979-22bgc\" (UID: \"726a71b5-e75b-4313-a1f9-3d3d123b729c\") " pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-22bgc"
Sep 16 12:49:35.644657 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:49:35.644578    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qz4vf\" (UniqueName: \"kubernetes.io/projected/a0c1e96d-918c-423b-b136-d4c3584fd568-kube-api-access-qz4vf\") pod \"node-exporter-9tn5t\" (UID: \"a0c1e96d-918c-423b-b136-d4c3584fd568\") " pod="openshift-monitoring/node-exporter-9tn5t"
Sep 16 12:49:35.644657 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:49:35.644601    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/cdef4a92-60a3-42f2-a20b-a15bce25691e-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-59fbdd77d-lbkms\" (UID: \"cdef4a92-60a3-42f2-a20b-a15bce25691e\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-lbkms"
Sep 16 12:49:35.745447 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:49:35.745407    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/cdef4a92-60a3-42f2-a20b-a15bce25691e-volume-directive-shadow\") pod \"kube-state-metrics-59fbdd77d-lbkms\" (UID: \"cdef4a92-60a3-42f2-a20b-a15bce25691e\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-lbkms"
Sep 16 12:49:35.745611 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:49:35.745457    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/a0c1e96d-918c-423b-b136-d4c3584fd568-sys\") pod \"node-exporter-9tn5t\" (UID: \"a0c1e96d-918c-423b-b136-d4c3584fd568\") " pod="openshift-monitoring/node-exporter-9tn5t"
Sep 16 12:49:35.745611 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:49:35.745542    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/a0c1e96d-918c-423b-b136-d4c3584fd568-sys\") pod \"node-exporter-9tn5t\" (UID: \"a0c1e96d-918c-423b-b136-d4c3584fd568\") " pod="openshift-monitoring/node-exporter-9tn5t"
Sep 16 12:49:35.745611 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:49:35.745579    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z8xxg\" (UniqueName: \"kubernetes.io/projected/726a71b5-e75b-4313-a1f9-3d3d123b729c-kube-api-access-z8xxg\") pod \"openshift-state-metrics-67dcf8c979-22bgc\" (UID: \"726a71b5-e75b-4313-a1f9-3d3d123b729c\") " pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-22bgc"
Sep 16 12:49:35.745765 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:49:35.745612    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/a0c1e96d-918c-423b-b136-d4c3584fd568-node-exporter-tls\") pod \"node-exporter-9tn5t\" (UID: \"a0c1e96d-918c-423b-b136-d4c3584fd568\") " pod="openshift-monitoring/node-exporter-9tn5t"
Sep 16 12:49:35.745765 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:49:35.745655    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/726a71b5-e75b-4313-a1f9-3d3d123b729c-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-67dcf8c979-22bgc\" (UID: \"726a71b5-e75b-4313-a1f9-3d3d123b729c\") " pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-22bgc"
Sep 16 12:49:35.745765 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:49:35.745681    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qz4vf\" (UniqueName: \"kubernetes.io/projected/a0c1e96d-918c-423b-b136-d4c3584fd568-kube-api-access-qz4vf\") pod \"node-exporter-9tn5t\" (UID: \"a0c1e96d-918c-423b-b136-d4c3584fd568\") " pod="openshift-monitoring/node-exporter-9tn5t"
Sep 16 12:49:35.745765 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:49:35.745718    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/cdef4a92-60a3-42f2-a20b-a15bce25691e-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-59fbdd77d-lbkms\" (UID: \"cdef4a92-60a3-42f2-a20b-a15bce25691e\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-lbkms"
Sep 16 12:49:35.745944 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:49:35.745764    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/a0c1e96d-918c-423b-b136-d4c3584fd568-node-exporter-accelerators-collector-config\") pod \"node-exporter-9tn5t\" (UID: \"a0c1e96d-918c-423b-b136-d4c3584fd568\") " pod="openshift-monitoring/node-exporter-9tn5t"
Sep 16 12:49:35.745944 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:49:35.745786    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/cdef4a92-60a3-42f2-a20b-a15bce25691e-volume-directive-shadow\") pod \"kube-state-metrics-59fbdd77d-lbkms\" (UID: \"cdef4a92-60a3-42f2-a20b-a15bce25691e\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-lbkms"
Sep 16 12:49:35.745944 ip-10-0-133-218 kubenswrapper[2580]: E0916 12:49:35.745796    2580 secret.go:189] Couldn't get secret openshift-monitoring/node-exporter-tls: secret "node-exporter-tls" not found
Sep 16 12:49:35.745944 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:49:35.745792    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tkjd9\" (UniqueName: \"kubernetes.io/projected/cdef4a92-60a3-42f2-a20b-a15bce25691e-kube-api-access-tkjd9\") pod \"kube-state-metrics-59fbdd77d-lbkms\" (UID: \"cdef4a92-60a3-42f2-a20b-a15bce25691e\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-lbkms"
Sep 16 12:49:35.746502 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:49:35.745859    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/cdef4a92-60a3-42f2-a20b-a15bce25691e-kube-state-metrics-tls\") pod \"kube-state-metrics-59fbdd77d-lbkms\" (UID: \"cdef4a92-60a3-42f2-a20b-a15bce25691e\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-lbkms"
Sep 16 12:49:35.746502 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:49:35.746324    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/726a71b5-e75b-4313-a1f9-3d3d123b729c-metrics-client-ca\") pod \"openshift-state-metrics-67dcf8c979-22bgc\" (UID: \"726a71b5-e75b-4313-a1f9-3d3d123b729c\") " pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-22bgc"
Sep 16 12:49:35.746502 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:49:35.746377    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/a0c1e96d-918c-423b-b136-d4c3584fd568-node-exporter-textfile\") pod \"node-exporter-9tn5t\" (UID: \"a0c1e96d-918c-423b-b136-d4c3584fd568\") " pod="openshift-monitoring/node-exporter-9tn5t"
Sep 16 12:49:35.746502 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:49:35.746415    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/cdef4a92-60a3-42f2-a20b-a15bce25691e-metrics-client-ca\") pod \"kube-state-metrics-59fbdd77d-lbkms\" (UID: \"cdef4a92-60a3-42f2-a20b-a15bce25691e\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-lbkms"
Sep 16 12:49:35.746502 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:49:35.746448    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/cdef4a92-60a3-42f2-a20b-a15bce25691e-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-59fbdd77d-lbkms\" (UID: \"cdef4a92-60a3-42f2-a20b-a15bce25691e\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-lbkms"
Sep 16 12:49:35.746502 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:49:35.746473    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/a0c1e96d-918c-423b-b136-d4c3584fd568-root\") pod \"node-exporter-9tn5t\" (UID: \"a0c1e96d-918c-423b-b136-d4c3584fd568\") " pod="openshift-monitoring/node-exporter-9tn5t"
Sep 16 12:49:35.746860 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:49:35.746526    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/726a71b5-e75b-4313-a1f9-3d3d123b729c-openshift-state-metrics-tls\") pod \"openshift-state-metrics-67dcf8c979-22bgc\" (UID: \"726a71b5-e75b-4313-a1f9-3d3d123b729c\") " pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-22bgc"
Sep 16 12:49:35.746860 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:49:35.746558    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/a0c1e96d-918c-423b-b136-d4c3584fd568-node-exporter-wtmp\") pod \"node-exporter-9tn5t\" (UID: \"a0c1e96d-918c-423b-b136-d4c3584fd568\") " pod="openshift-monitoring/node-exporter-9tn5t"
Sep 16 12:49:35.746860 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:49:35.746606    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/a0c1e96d-918c-423b-b136-d4c3584fd568-metrics-client-ca\") pod \"node-exporter-9tn5t\" (UID: \"a0c1e96d-918c-423b-b136-d4c3584fd568\") " pod="openshift-monitoring/node-exporter-9tn5t"
Sep 16 12:49:35.747008 ip-10-0-133-218 kubenswrapper[2580]: E0916 12:49:35.746897    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a0c1e96d-918c-423b-b136-d4c3584fd568-node-exporter-tls podName:a0c1e96d-918c-423b-b136-d4c3584fd568 nodeName:}" failed. No retries permitted until 2026-09-16 12:49:36.246871023 +0000 UTC m=+115.169743490 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "node-exporter-tls" (UniqueName: "kubernetes.io/secret/a0c1e96d-918c-423b-b136-d4c3584fd568-node-exporter-tls") pod "node-exporter-9tn5t" (UID: "a0c1e96d-918c-423b-b136-d4c3584fd568") : secret "node-exporter-tls" not found
Sep 16 12:49:35.747008 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:49:35.746861    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/cdef4a92-60a3-42f2-a20b-a15bce25691e-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-59fbdd77d-lbkms\" (UID: \"cdef4a92-60a3-42f2-a20b-a15bce25691e\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-lbkms"
Sep 16 12:49:35.747539 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:49:35.747517    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/a0c1e96d-918c-423b-b136-d4c3584fd568-metrics-client-ca\") pod \"node-exporter-9tn5t\" (UID: \"a0c1e96d-918c-423b-b136-d4c3584fd568\") " pod="openshift-monitoring/node-exporter-9tn5t"
Sep 16 12:49:35.748110 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:49:35.747977    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/a0c1e96d-918c-423b-b136-d4c3584fd568-node-exporter-accelerators-collector-config\") pod \"node-exporter-9tn5t\" (UID: \"a0c1e96d-918c-423b-b136-d4c3584fd568\") " pod="openshift-monitoring/node-exporter-9tn5t"
Sep 16 12:49:35.748570 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:49:35.748457    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/726a71b5-e75b-4313-a1f9-3d3d123b729c-metrics-client-ca\") pod \"openshift-state-metrics-67dcf8c979-22bgc\" (UID: \"726a71b5-e75b-4313-a1f9-3d3d123b729c\") " pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-22bgc"
Sep 16 12:49:35.748669 ip-10-0-133-218 kubenswrapper[2580]: E0916 12:49:35.748595    2580 secret.go:189] Couldn't get secret openshift-monitoring/kube-state-metrics-tls: secret "kube-state-metrics-tls" not found
Sep 16 12:49:35.748669 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:49:35.748626    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/a0c1e96d-918c-423b-b136-d4c3584fd568-root\") pod \"node-exporter-9tn5t\" (UID: \"a0c1e96d-918c-423b-b136-d4c3584fd568\") " pod="openshift-monitoring/node-exporter-9tn5t"
Sep 16 12:49:35.748772 ip-10-0-133-218 kubenswrapper[2580]: E0916 12:49:35.748739    2580 secret.go:189] Couldn't get secret openshift-monitoring/openshift-state-metrics-tls: secret "openshift-state-metrics-tls" not found
Sep 16 12:49:35.749606 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:49:35.749034    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/a0c1e96d-918c-423b-b136-d4c3584fd568-node-exporter-textfile\") pod \"node-exporter-9tn5t\" (UID: \"a0c1e96d-918c-423b-b136-d4c3584fd568\") " pod="openshift-monitoring/node-exporter-9tn5t"
Sep 16 12:49:35.749606 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:49:35.749106    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/a0c1e96d-918c-423b-b136-d4c3584fd568-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-9tn5t\" (UID: \"a0c1e96d-918c-423b-b136-d4c3584fd568\") " pod="openshift-monitoring/node-exporter-9tn5t"
Sep 16 12:49:35.749606 ip-10-0-133-218 kubenswrapper[2580]: E0916 12:49:35.749138    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/726a71b5-e75b-4313-a1f9-3d3d123b729c-openshift-state-metrics-tls podName:726a71b5-e75b-4313-a1f9-3d3d123b729c nodeName:}" failed. No retries permitted until 2026-09-16 12:49:36.249119086 +0000 UTC m=+115.171991536 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "openshift-state-metrics-tls" (UniqueName: "kubernetes.io/secret/726a71b5-e75b-4313-a1f9-3d3d123b729c-openshift-state-metrics-tls") pod "openshift-state-metrics-67dcf8c979-22bgc" (UID: "726a71b5-e75b-4313-a1f9-3d3d123b729c") : secret "openshift-state-metrics-tls" not found
Sep 16 12:49:35.749606 ip-10-0-133-218 kubenswrapper[2580]: E0916 12:49:35.749400    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cdef4a92-60a3-42f2-a20b-a15bce25691e-kube-state-metrics-tls podName:cdef4a92-60a3-42f2-a20b-a15bce25691e nodeName:}" failed. No retries permitted until 2026-09-16 12:49:36.249365452 +0000 UTC m=+115.172237906 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-state-metrics-tls" (UniqueName: "kubernetes.io/secret/cdef4a92-60a3-42f2-a20b-a15bce25691e-kube-state-metrics-tls") pod "kube-state-metrics-59fbdd77d-lbkms" (UID: "cdef4a92-60a3-42f2-a20b-a15bce25691e") : secret "kube-state-metrics-tls" not found
Sep 16 12:49:35.749606 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:49:35.749409    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/a0c1e96d-918c-423b-b136-d4c3584fd568-node-exporter-wtmp\") pod \"node-exporter-9tn5t\" (UID: \"a0c1e96d-918c-423b-b136-d4c3584fd568\") " pod="openshift-monitoring/node-exporter-9tn5t"
Sep 16 12:49:35.749606 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:49:35.749603    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/cdef4a92-60a3-42f2-a20b-a15bce25691e-metrics-client-ca\") pod \"kube-state-metrics-59fbdd77d-lbkms\" (UID: \"cdef4a92-60a3-42f2-a20b-a15bce25691e\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-lbkms"
Sep 16 12:49:35.749975 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:49:35.749947    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/726a71b5-e75b-4313-a1f9-3d3d123b729c-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-67dcf8c979-22bgc\" (UID: \"726a71b5-e75b-4313-a1f9-3d3d123b729c\") " pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-22bgc"
Sep 16 12:49:35.751968 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:49:35.751947    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/cdef4a92-60a3-42f2-a20b-a15bce25691e-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-59fbdd77d-lbkms\" (UID: \"cdef4a92-60a3-42f2-a20b-a15bce25691e\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-lbkms"
Sep 16 12:49:35.752228 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:49:35.752201    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/a0c1e96d-918c-423b-b136-d4c3584fd568-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-9tn5t\" (UID: \"a0c1e96d-918c-423b-b136-d4c3584fd568\") " pod="openshift-monitoring/node-exporter-9tn5t"
Sep 16 12:49:35.756832 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:49:35.756812    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-z8xxg\" (UniqueName: \"kubernetes.io/projected/726a71b5-e75b-4313-a1f9-3d3d123b729c-kube-api-access-z8xxg\") pod \"openshift-state-metrics-67dcf8c979-22bgc\" (UID: \"726a71b5-e75b-4313-a1f9-3d3d123b729c\") " pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-22bgc"
Sep 16 12:49:35.759761 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:49:35.759740    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qz4vf\" (UniqueName: \"kubernetes.io/projected/a0c1e96d-918c-423b-b136-d4c3584fd568-kube-api-access-qz4vf\") pod \"node-exporter-9tn5t\" (UID: \"a0c1e96d-918c-423b-b136-d4c3584fd568\") " pod="openshift-monitoring/node-exporter-9tn5t"
Sep 16 12:49:35.760882 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:49:35.760856    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tkjd9\" (UniqueName: \"kubernetes.io/projected/cdef4a92-60a3-42f2-a20b-a15bce25691e-kube-api-access-tkjd9\") pod \"kube-state-metrics-59fbdd77d-lbkms\" (UID: \"cdef4a92-60a3-42f2-a20b-a15bce25691e\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-lbkms"
Sep 16 12:49:36.253749 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:49:36.253716    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/cdef4a92-60a3-42f2-a20b-a15bce25691e-kube-state-metrics-tls\") pod \"kube-state-metrics-59fbdd77d-lbkms\" (UID: \"cdef4a92-60a3-42f2-a20b-a15bce25691e\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-lbkms"
Sep 16 12:49:36.253972 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:49:36.253771    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/726a71b5-e75b-4313-a1f9-3d3d123b729c-openshift-state-metrics-tls\") pod \"openshift-state-metrics-67dcf8c979-22bgc\" (UID: \"726a71b5-e75b-4313-a1f9-3d3d123b729c\") " pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-22bgc"
Sep 16 12:49:36.253972 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:49:36.253844    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/a0c1e96d-918c-423b-b136-d4c3584fd568-node-exporter-tls\") pod \"node-exporter-9tn5t\" (UID: \"a0c1e96d-918c-423b-b136-d4c3584fd568\") " pod="openshift-monitoring/node-exporter-9tn5t"
Sep 16 12:49:36.253972 ip-10-0-133-218 kubenswrapper[2580]: E0916 12:49:36.253961    2580 secret.go:189] Couldn't get secret openshift-monitoring/node-exporter-tls: secret "node-exporter-tls" not found
Sep 16 12:49:36.254148 ip-10-0-133-218 kubenswrapper[2580]: E0916 12:49:36.254044    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a0c1e96d-918c-423b-b136-d4c3584fd568-node-exporter-tls podName:a0c1e96d-918c-423b-b136-d4c3584fd568 nodeName:}" failed. No retries permitted until 2026-09-16 12:49:37.254024687 +0000 UTC m=+116.176897138 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "node-exporter-tls" (UniqueName: "kubernetes.io/secret/a0c1e96d-918c-423b-b136-d4c3584fd568-node-exporter-tls") pod "node-exporter-9tn5t" (UID: "a0c1e96d-918c-423b-b136-d4c3584fd568") : secret "node-exporter-tls" not found
Sep 16 12:49:36.256111 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:49:36.256081    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/726a71b5-e75b-4313-a1f9-3d3d123b729c-openshift-state-metrics-tls\") pod \"openshift-state-metrics-67dcf8c979-22bgc\" (UID: \"726a71b5-e75b-4313-a1f9-3d3d123b729c\") " pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-22bgc"
Sep 16 12:49:36.256223 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:49:36.256196    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/cdef4a92-60a3-42f2-a20b-a15bce25691e-kube-state-metrics-tls\") pod \"kube-state-metrics-59fbdd77d-lbkms\" (UID: \"cdef4a92-60a3-42f2-a20b-a15bce25691e\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-lbkms"
Sep 16 12:49:36.407269 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:49:36.407235    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-22bgc"
Sep 16 12:49:36.455820 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:49:36.455561    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-59fbdd77d-lbkms"
Sep 16 12:49:36.707013 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:49:36.706973    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-67dcf8c979-22bgc"]
Sep 16 12:49:36.710792 ip-10-0-133-218 kubenswrapper[2580]: W0916 12:49:36.710764    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod726a71b5_e75b_4313_a1f9_3d3d123b729c.slice/crio-fc165e235af611da6c9b7a70a173ce0cb0a63489b651071f8c8ab52746be31fe WatchSource:0}: Error finding container fc165e235af611da6c9b7a70a173ce0cb0a63489b651071f8c8ab52746be31fe: Status 404 returned error can't find the container with id fc165e235af611da6c9b7a70a173ce0cb0a63489b651071f8c8ab52746be31fe
Sep 16 12:49:36.748565 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:49:36.748540    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Sep 16 12:49:36.751622 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:49:36.751603    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Sep 16 12:49:36.757511 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:49:36.757376    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\""
Sep 16 12:49:36.757624 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:49:36.757573    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\""
Sep 16 12:49:36.757708 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:49:36.757633    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\""
Sep 16 12:49:36.757793 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:49:36.757777    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-jk8nq\""
Sep 16 12:49:36.758617 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:49:36.758598    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\""
Sep 16 12:49:36.758945 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:49:36.758926    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\""
Sep 16 12:49:36.759097 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:49:36.759035    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\""
Sep 16 12:49:36.759214 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:49:36.759199    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\""
Sep 16 12:49:36.759696 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:49:36.759679    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\""
Sep 16 12:49:36.759832 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:49:36.759814    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\""
Sep 16 12:49:36.828587 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:49:36.828558    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-59fbdd77d-lbkms"]
Sep 16 12:49:36.835004 ip-10-0-133-218 kubenswrapper[2580]: W0916 12:49:36.834968    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcdef4a92_60a3_42f2_a20b_a15bce25691e.slice/crio-d4a546f382ee4e00003440c6eafa1be2009e02fa24e4e97deabcb07aef2b728c WatchSource:0}: Error finding container d4a546f382ee4e00003440c6eafa1be2009e02fa24e4e97deabcb07aef2b728c: Status 404 returned error can't find the container with id d4a546f382ee4e00003440c6eafa1be2009e02fa24e4e97deabcb07aef2b728c
Sep 16 12:49:36.847375 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:49:36.847355    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Sep 16 12:49:36.858793 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:49:36.858768    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/1273107e-4c61-4b7c-983d-1dbf0dfc99ad-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"1273107e-4c61-4b7c-983d-1dbf0dfc99ad\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 12:49:36.858908 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:49:36.858806    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/1273107e-4c61-4b7c-983d-1dbf0dfc99ad-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"1273107e-4c61-4b7c-983d-1dbf0dfc99ad\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 12:49:36.858908 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:49:36.858842    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/1273107e-4c61-4b7c-983d-1dbf0dfc99ad-tls-assets\") pod \"alertmanager-main-0\" (UID: \"1273107e-4c61-4b7c-983d-1dbf0dfc99ad\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 12:49:36.858999 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:49:36.858929    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/1273107e-4c61-4b7c-983d-1dbf0dfc99ad-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"1273107e-4c61-4b7c-983d-1dbf0dfc99ad\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 12:49:36.858999 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:49:36.858954    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/1273107e-4c61-4b7c-983d-1dbf0dfc99ad-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"1273107e-4c61-4b7c-983d-1dbf0dfc99ad\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 12:49:36.858999 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:49:36.858973    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1273107e-4c61-4b7c-983d-1dbf0dfc99ad-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"1273107e-4c61-4b7c-983d-1dbf0dfc99ad\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 12:49:36.859105 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:49:36.858995    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-982lr\" (UniqueName: \"kubernetes.io/projected/1273107e-4c61-4b7c-983d-1dbf0dfc99ad-kube-api-access-982lr\") pod \"alertmanager-main-0\" (UID: \"1273107e-4c61-4b7c-983d-1dbf0dfc99ad\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 12:49:36.859105 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:49:36.859018    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/1273107e-4c61-4b7c-983d-1dbf0dfc99ad-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"1273107e-4c61-4b7c-983d-1dbf0dfc99ad\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 12:49:36.859105 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:49:36.859047    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/1273107e-4c61-4b7c-983d-1dbf0dfc99ad-config-volume\") pod \"alertmanager-main-0\" (UID: \"1273107e-4c61-4b7c-983d-1dbf0dfc99ad\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 12:49:36.859105 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:49:36.859068    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/1273107e-4c61-4b7c-983d-1dbf0dfc99ad-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"1273107e-4c61-4b7c-983d-1dbf0dfc99ad\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 12:49:36.859105 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:49:36.859099    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/1273107e-4c61-4b7c-983d-1dbf0dfc99ad-web-config\") pod \"alertmanager-main-0\" (UID: \"1273107e-4c61-4b7c-983d-1dbf0dfc99ad\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 12:49:36.859329 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:49:36.859116    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/1273107e-4c61-4b7c-983d-1dbf0dfc99ad-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"1273107e-4c61-4b7c-983d-1dbf0dfc99ad\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 12:49:36.859329 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:49:36.859132    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/1273107e-4c61-4b7c-983d-1dbf0dfc99ad-config-out\") pod \"alertmanager-main-0\" (UID: \"1273107e-4c61-4b7c-983d-1dbf0dfc99ad\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 12:49:36.959761 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:49:36.959725    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/1273107e-4c61-4b7c-983d-1dbf0dfc99ad-config-volume\") pod \"alertmanager-main-0\" (UID: \"1273107e-4c61-4b7c-983d-1dbf0dfc99ad\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 12:49:36.959761 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:49:36.959761    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/1273107e-4c61-4b7c-983d-1dbf0dfc99ad-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"1273107e-4c61-4b7c-983d-1dbf0dfc99ad\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 12:49:36.959978 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:49:36.959799    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/1273107e-4c61-4b7c-983d-1dbf0dfc99ad-web-config\") pod \"alertmanager-main-0\" (UID: \"1273107e-4c61-4b7c-983d-1dbf0dfc99ad\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 12:49:36.959978 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:49:36.959817    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/1273107e-4c61-4b7c-983d-1dbf0dfc99ad-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"1273107e-4c61-4b7c-983d-1dbf0dfc99ad\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 12:49:36.959978 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:49:36.959837    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/1273107e-4c61-4b7c-983d-1dbf0dfc99ad-config-out\") pod \"alertmanager-main-0\" (UID: \"1273107e-4c61-4b7c-983d-1dbf0dfc99ad\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 12:49:36.959978 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:49:36.959856    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/1273107e-4c61-4b7c-983d-1dbf0dfc99ad-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"1273107e-4c61-4b7c-983d-1dbf0dfc99ad\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 12:49:36.959978 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:49:36.959875    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/1273107e-4c61-4b7c-983d-1dbf0dfc99ad-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"1273107e-4c61-4b7c-983d-1dbf0dfc99ad\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 12:49:36.959978 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:49:36.959897    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/1273107e-4c61-4b7c-983d-1dbf0dfc99ad-tls-assets\") pod \"alertmanager-main-0\" (UID: \"1273107e-4c61-4b7c-983d-1dbf0dfc99ad\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 12:49:36.960248 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:49:36.960000    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/1273107e-4c61-4b7c-983d-1dbf0dfc99ad-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"1273107e-4c61-4b7c-983d-1dbf0dfc99ad\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 12:49:36.960248 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:49:36.960025    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/1273107e-4c61-4b7c-983d-1dbf0dfc99ad-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"1273107e-4c61-4b7c-983d-1dbf0dfc99ad\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 12:49:36.960248 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:49:36.960053    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1273107e-4c61-4b7c-983d-1dbf0dfc99ad-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"1273107e-4c61-4b7c-983d-1dbf0dfc99ad\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 12:49:36.960248 ip-10-0-133-218 kubenswrapper[2580]: E0916 12:49:36.960069    2580 secret.go:189] Couldn't get secret openshift-monitoring/alertmanager-main-tls: secret "alertmanager-main-tls" not found
Sep 16 12:49:36.960248 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:49:36.960092    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-982lr\" (UniqueName: \"kubernetes.io/projected/1273107e-4c61-4b7c-983d-1dbf0dfc99ad-kube-api-access-982lr\") pod \"alertmanager-main-0\" (UID: \"1273107e-4c61-4b7c-983d-1dbf0dfc99ad\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 12:49:36.960248 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:49:36.960118    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/1273107e-4c61-4b7c-983d-1dbf0dfc99ad-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"1273107e-4c61-4b7c-983d-1dbf0dfc99ad\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 12:49:36.960248 ip-10-0-133-218 kubenswrapper[2580]: E0916 12:49:36.960152    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1273107e-4c61-4b7c-983d-1dbf0dfc99ad-secret-alertmanager-main-tls podName:1273107e-4c61-4b7c-983d-1dbf0dfc99ad nodeName:}" failed. No retries permitted until 2026-09-16 12:49:37.460129343 +0000 UTC m=+116.383001805 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "secret-alertmanager-main-tls" (UniqueName: "kubernetes.io/secret/1273107e-4c61-4b7c-983d-1dbf0dfc99ad-secret-alertmanager-main-tls") pod "alertmanager-main-0" (UID: "1273107e-4c61-4b7c-983d-1dbf0dfc99ad") : secret "alertmanager-main-tls" not found
Sep 16 12:49:36.960648 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:49:36.960630    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/1273107e-4c61-4b7c-983d-1dbf0dfc99ad-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"1273107e-4c61-4b7c-983d-1dbf0dfc99ad\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 12:49:36.960708 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:49:36.960642    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/1273107e-4c61-4b7c-983d-1dbf0dfc99ad-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"1273107e-4c61-4b7c-983d-1dbf0dfc99ad\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 12:49:36.961330 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:49:36.961302    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1273107e-4c61-4b7c-983d-1dbf0dfc99ad-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"1273107e-4c61-4b7c-983d-1dbf0dfc99ad\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 12:49:36.963121 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:49:36.963095    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/1273107e-4c61-4b7c-983d-1dbf0dfc99ad-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"1273107e-4c61-4b7c-983d-1dbf0dfc99ad\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 12:49:36.963234 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:49:36.963122    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/1273107e-4c61-4b7c-983d-1dbf0dfc99ad-tls-assets\") pod \"alertmanager-main-0\" (UID: \"1273107e-4c61-4b7c-983d-1dbf0dfc99ad\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 12:49:36.963403 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:49:36.963378    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/1273107e-4c61-4b7c-983d-1dbf0dfc99ad-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"1273107e-4c61-4b7c-983d-1dbf0dfc99ad\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 12:49:36.963559 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:49:36.963538    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/1273107e-4c61-4b7c-983d-1dbf0dfc99ad-config-volume\") pod \"alertmanager-main-0\" (UID: \"1273107e-4c61-4b7c-983d-1dbf0dfc99ad\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 12:49:36.963654 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:49:36.963639    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/1273107e-4c61-4b7c-983d-1dbf0dfc99ad-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"1273107e-4c61-4b7c-983d-1dbf0dfc99ad\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 12:49:36.963711 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:49:36.963668    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/1273107e-4c61-4b7c-983d-1dbf0dfc99ad-web-config\") pod \"alertmanager-main-0\" (UID: \"1273107e-4c61-4b7c-983d-1dbf0dfc99ad\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 12:49:36.963768 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:49:36.963732    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/1273107e-4c61-4b7c-983d-1dbf0dfc99ad-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"1273107e-4c61-4b7c-983d-1dbf0dfc99ad\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 12:49:36.963768 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:49:36.963741    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/1273107e-4c61-4b7c-983d-1dbf0dfc99ad-config-out\") pod \"alertmanager-main-0\" (UID: \"1273107e-4c61-4b7c-983d-1dbf0dfc99ad\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 12:49:36.982586 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:49:36.982560    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-982lr\" (UniqueName: \"kubernetes.io/projected/1273107e-4c61-4b7c-983d-1dbf0dfc99ad-kube-api-access-982lr\") pod \"alertmanager-main-0\" (UID: \"1273107e-4c61-4b7c-983d-1dbf0dfc99ad\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 12:49:37.073956 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:49:37.073924    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-59fbdd77d-lbkms" event={"ID":"cdef4a92-60a3-42f2-a20b-a15bce25691e","Type":"ContainerStarted","Data":"d4a546f382ee4e00003440c6eafa1be2009e02fa24e4e97deabcb07aef2b728c"}
Sep 16 12:49:37.075641 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:49:37.075620    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-22bgc" event={"ID":"726a71b5-e75b-4313-a1f9-3d3d123b729c","Type":"ContainerStarted","Data":"8c4df635e3b544f0805ee543a19cdfb5a9fb69d46907586b98fd6e29f4c07594"}
Sep 16 12:49:37.075729 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:49:37.075664    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-22bgc" event={"ID":"726a71b5-e75b-4313-a1f9-3d3d123b729c","Type":"ContainerStarted","Data":"f23e24b387044e9d7b79523c809454855d8a8a194abcb4bf1f9500e7cb9c0668"}
Sep 16 12:49:37.075729 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:49:37.075683    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-22bgc" event={"ID":"726a71b5-e75b-4313-a1f9-3d3d123b729c","Type":"ContainerStarted","Data":"fc165e235af611da6c9b7a70a173ce0cb0a63489b651071f8c8ab52746be31fe"}
Sep 16 12:49:37.263606 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:49:37.263570    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/a0c1e96d-918c-423b-b136-d4c3584fd568-node-exporter-tls\") pod \"node-exporter-9tn5t\" (UID: \"a0c1e96d-918c-423b-b136-d4c3584fd568\") " pod="openshift-monitoring/node-exporter-9tn5t"
Sep 16 12:49:37.265796 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:49:37.265779    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/a0c1e96d-918c-423b-b136-d4c3584fd568-node-exporter-tls\") pod \"node-exporter-9tn5t\" (UID: \"a0c1e96d-918c-423b-b136-d4c3584fd568\") " pod="openshift-monitoring/node-exporter-9tn5t"
Sep 16 12:49:37.371076 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:49:37.370999    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-9tn5t"
Sep 16 12:49:37.380597 ip-10-0-133-218 kubenswrapper[2580]: W0916 12:49:37.380569    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda0c1e96d_918c_423b_b136_d4c3584fd568.slice/crio-62a6d1ed3a22556a103a5c0d1946a3105883a75b2dcce0a73bdeb108d9b44953 WatchSource:0}: Error finding container 62a6d1ed3a22556a103a5c0d1946a3105883a75b2dcce0a73bdeb108d9b44953: Status 404 returned error can't find the container with id 62a6d1ed3a22556a103a5c0d1946a3105883a75b2dcce0a73bdeb108d9b44953
Sep 16 12:49:37.464762 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:49:37.464731    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/1273107e-4c61-4b7c-983d-1dbf0dfc99ad-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"1273107e-4c61-4b7c-983d-1dbf0dfc99ad\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 12:49:37.464930 ip-10-0-133-218 kubenswrapper[2580]: E0916 12:49:37.464909    2580 secret.go:189] Couldn't get secret openshift-monitoring/alertmanager-main-tls: secret "alertmanager-main-tls" not found
Sep 16 12:49:37.465012 ip-10-0-133-218 kubenswrapper[2580]: E0916 12:49:37.464985    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1273107e-4c61-4b7c-983d-1dbf0dfc99ad-secret-alertmanager-main-tls podName:1273107e-4c61-4b7c-983d-1dbf0dfc99ad nodeName:}" failed. No retries permitted until 2026-09-16 12:49:38.464965577 +0000 UTC m=+117.387838025 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "secret-alertmanager-main-tls" (UniqueName: "kubernetes.io/secret/1273107e-4c61-4b7c-983d-1dbf0dfc99ad-secret-alertmanager-main-tls") pod "alertmanager-main-0" (UID: "1273107e-4c61-4b7c-983d-1dbf0dfc99ad") : secret "alertmanager-main-tls" not found
Sep 16 12:49:37.864877 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:49:37.864843    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-7996cc44cd-p6ljc"
Sep 16 12:49:37.865280 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:49:37.864908    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-7996cc44cd-p6ljc"
Sep 16 12:49:37.870143 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:49:37.870119    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-7996cc44cd-p6ljc"
Sep 16 12:49:38.088380 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:49:38.088344    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-9tn5t" event={"ID":"a0c1e96d-918c-423b-b136-d4c3584fd568","Type":"ContainerStarted","Data":"62a6d1ed3a22556a103a5c0d1946a3105883a75b2dcce0a73bdeb108d9b44953"}
Sep 16 12:49:38.093889 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:49:38.093868    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-7996cc44cd-p6ljc"
Sep 16 12:49:38.145270 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:49:38.145196    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-d55cc74b5-x27t7"]
Sep 16 12:49:38.474397 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:49:38.474246    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/1273107e-4c61-4b7c-983d-1dbf0dfc99ad-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"1273107e-4c61-4b7c-983d-1dbf0dfc99ad\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 12:49:38.477935 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:49:38.477157    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/1273107e-4c61-4b7c-983d-1dbf0dfc99ad-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"1273107e-4c61-4b7c-983d-1dbf0dfc99ad\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 12:49:38.568316 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:49:38.568084    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Sep 16 12:49:38.884520 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:49:38.884468    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Sep 16 12:49:38.963743 ip-10-0-133-218 kubenswrapper[2580]: W0916 12:49:38.963708    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1273107e_4c61_4b7c_983d_1dbf0dfc99ad.slice/crio-33a9acd679857671e1db56231a967f3202a4d85fa354ec5c498c7e1b6d48770c WatchSource:0}: Error finding container 33a9acd679857671e1db56231a967f3202a4d85fa354ec5c498c7e1b6d48770c: Status 404 returned error can't find the container with id 33a9acd679857671e1db56231a967f3202a4d85fa354ec5c498c7e1b6d48770c
Sep 16 12:49:39.094406 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:49:39.094372    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-59fbdd77d-lbkms" event={"ID":"cdef4a92-60a3-42f2-a20b-a15bce25691e","Type":"ContainerStarted","Data":"dfcc0bd4081cb634c1f7b79d947145e8f3134dd58f542de0c79051482fe1782b"}
Sep 16 12:49:39.094551 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:49:39.094448    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-59fbdd77d-lbkms" event={"ID":"cdef4a92-60a3-42f2-a20b-a15bce25691e","Type":"ContainerStarted","Data":"fc29a8c1ef27b8b64e21151412e15de5b39199ab3db050a3046a001aa76685ad"}
Sep 16 12:49:39.094551 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:49:39.094482    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-59fbdd77d-lbkms" event={"ID":"cdef4a92-60a3-42f2-a20b-a15bce25691e","Type":"ContainerStarted","Data":"be52b7166ad61e75f467ed368f9344bab8a376f3fdbadd82720647d5eef0c656"}
Sep 16 12:49:39.095771 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:49:39.095745    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"1273107e-4c61-4b7c-983d-1dbf0dfc99ad","Type":"ContainerStarted","Data":"33a9acd679857671e1db56231a967f3202a4d85fa354ec5c498c7e1b6d48770c"}
Sep 16 12:49:39.097928 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:49:39.097906    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-22bgc" event={"ID":"726a71b5-e75b-4313-a1f9-3d3d123b729c","Type":"ContainerStarted","Data":"3df0eb2e74dd86719a6a5c45202e8d2d3cd7347b0966edd95040aa58f50cd508"}
Sep 16 12:49:39.099685 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:49:39.099664    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-9tn5t" event={"ID":"a0c1e96d-918c-423b-b136-d4c3584fd568","Type":"ContainerStarted","Data":"8b905fa0a5d0f8883020881accf78f1bc1bbbb9eeb5ad947d90f5eea5e8168d9"}
Sep 16 12:49:39.120136 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:49:39.120093    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/kube-state-metrics-59fbdd77d-lbkms" podStartSLOduration=2.582981169 podStartE2EDuration="4.120079549s" totalImagesPullingTime="1.53709838s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 12:49:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 12:49:39.118813617 +0000 UTC m=+118.041686098" watchObservedRunningTime="2026-09-16 12:49:39.120079549 +0000 UTC m=+118.042952035"
Sep 16 12:49:39.178666 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:49:39.178612    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-22bgc" podStartSLOduration=2.621383067 podStartE2EDuration="4.178593302s" totalImagesPullingTime="1.557210235s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 12:49:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 12:49:39.178547802 +0000 UTC m=+118.101420271" watchObservedRunningTime="2026-09-16 12:49:39.178593302 +0000 UTC m=+118.101465771"
Sep 16 12:49:40.105117 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:49:40.105038    2580 generic.go:356] "Generic (PLEG): container finished" podID="1273107e-4c61-4b7c-983d-1dbf0dfc99ad" containerID="58e291fb7794e85c73e709c0bee46fe2b765faee850abb671b04805bc70b33de" exitCode=0
Sep 16 12:49:40.105500 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:49:40.105125    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"1273107e-4c61-4b7c-983d-1dbf0dfc99ad","Type":"ContainerDied","Data":"58e291fb7794e85c73e709c0bee46fe2b765faee850abb671b04805bc70b33de"}
Sep 16 12:49:40.106690 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:49:40.106625    2580 generic.go:356] "Generic (PLEG): container finished" podID="a0c1e96d-918c-423b-b136-d4c3584fd568" containerID="8b905fa0a5d0f8883020881accf78f1bc1bbbb9eeb5ad947d90f5eea5e8168d9" exitCode=0
Sep 16 12:49:40.106741 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:49:40.106708    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-9tn5t" event={"ID":"a0c1e96d-918c-423b-b136-d4c3584fd568","Type":"ContainerDied","Data":"8b905fa0a5d0f8883020881accf78f1bc1bbbb9eeb5ad947d90f5eea5e8168d9"}
Sep 16 12:49:41.113093 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:49:41.113058    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-9tn5t" event={"ID":"a0c1e96d-918c-423b-b136-d4c3584fd568","Type":"ContainerStarted","Data":"8facc5a42f4c56b3ebc36319d92debc9b3ecc02e17eb419394a1eebdc11ebcf7"}
Sep 16 12:49:41.113562 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:49:41.113154    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-9tn5t" event={"ID":"a0c1e96d-918c-423b-b136-d4c3584fd568","Type":"ContainerStarted","Data":"41118ed28648fec285beeadb5cc68f385bac1e2b22273a3eb3401f3265e7872e"}
Sep 16 12:49:41.133438 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:49:41.133386    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-9tn5t" podStartSLOduration=4.4450162429999995 podStartE2EDuration="6.133369209s" totalImagesPullingTime="1.625210471s" totalInitContainerRuntime="63.142495ms" isStatefulPod=true podCreationTimestamp="2026-09-16 12:49:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 12:49:41.131801586 +0000 UTC m=+120.054674056" watchObservedRunningTime="2026-09-16 12:49:41.133369209 +0000 UTC m=+120.056241677"
Sep 16 12:49:41.848909 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:49:41.848882    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 16 12:49:41.852614 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:49:41.852595    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 12:49:41.856027 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:49:41.856008    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\""
Sep 16 12:49:41.856120 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:49:41.856064    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\""
Sep 16 12:49:41.856179 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:49:41.856113    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\""
Sep 16 12:49:41.856265 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:49:41.856245    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\""
Sep 16 12:49:41.857270 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:49:41.856975    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\""
Sep 16 12:49:41.857270 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:49:41.857031    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\""
Sep 16 12:49:41.857270 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:49:41.857039    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-9bkct\""
Sep 16 12:49:41.857270 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:49:41.857162    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\""
Sep 16 12:49:41.857270 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:49:41.857172    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\""
Sep 16 12:49:41.857270 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:49:41.857162    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-cf5t6v2n01b5c\""
Sep 16 12:49:41.857270 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:49:41.857189    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\""
Sep 16 12:49:41.857640 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:49:41.857374    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\""
Sep 16 12:49:41.857640 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:49:41.857387    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\""
Sep 16 12:49:41.857640 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:49:41.857414    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\""
Sep 16 12:49:41.858641 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:49:41.858598    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\""
Sep 16 12:49:41.858836 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:49:41.858818    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\""
Sep 16 12:49:41.864402 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:49:41.864385    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\""
Sep 16 12:49:41.876712 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:49:41.876693    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 16 12:49:42.007088 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:49:42.007005    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/84b4b343-d43c-4639-a768-3f52806bd8c7-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"84b4b343-d43c-4639-a768-3f52806bd8c7\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 12:49:42.007088 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:49:42.007042    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/84b4b343-d43c-4639-a768-3f52806bd8c7-web-config\") pod \"prometheus-k8s-0\" (UID: \"84b4b343-d43c-4639-a768-3f52806bd8c7\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 12:49:42.007088 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:49:42.007062    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/84b4b343-d43c-4639-a768-3f52806bd8c7-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"84b4b343-d43c-4639-a768-3f52806bd8c7\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 12:49:42.007088 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:49:42.007079    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/84b4b343-d43c-4639-a768-3f52806bd8c7-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"84b4b343-d43c-4639-a768-3f52806bd8c7\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 12:49:42.007376 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:49:42.007126    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/84b4b343-d43c-4639-a768-3f52806bd8c7-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"84b4b343-d43c-4639-a768-3f52806bd8c7\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 12:49:42.007376 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:49:42.007159    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/84b4b343-d43c-4639-a768-3f52806bd8c7-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"84b4b343-d43c-4639-a768-3f52806bd8c7\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 12:49:42.007376 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:49:42.007176    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/84b4b343-d43c-4639-a768-3f52806bd8c7-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"84b4b343-d43c-4639-a768-3f52806bd8c7\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 12:49:42.007376 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:49:42.007196    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/84b4b343-d43c-4639-a768-3f52806bd8c7-config\") pod \"prometheus-k8s-0\" (UID: \"84b4b343-d43c-4639-a768-3f52806bd8c7\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 12:49:42.007376 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:49:42.007219    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/84b4b343-d43c-4639-a768-3f52806bd8c7-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"84b4b343-d43c-4639-a768-3f52806bd8c7\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 12:49:42.007376 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:49:42.007234    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/84b4b343-d43c-4639-a768-3f52806bd8c7-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"84b4b343-d43c-4639-a768-3f52806bd8c7\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 12:49:42.007376 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:49:42.007321    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/84b4b343-d43c-4639-a768-3f52806bd8c7-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"84b4b343-d43c-4639-a768-3f52806bd8c7\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 12:49:42.007741 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:49:42.007384    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/84b4b343-d43c-4639-a768-3f52806bd8c7-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"84b4b343-d43c-4639-a768-3f52806bd8c7\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 12:49:42.007741 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:49:42.007411    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/84b4b343-d43c-4639-a768-3f52806bd8c7-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"84b4b343-d43c-4639-a768-3f52806bd8c7\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 12:49:42.007741 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:49:42.007441    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/84b4b343-d43c-4639-a768-3f52806bd8c7-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"84b4b343-d43c-4639-a768-3f52806bd8c7\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 12:49:42.007741 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:49:42.007467    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/84b4b343-d43c-4639-a768-3f52806bd8c7-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"84b4b343-d43c-4639-a768-3f52806bd8c7\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 12:49:42.007741 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:49:42.007525    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/84b4b343-d43c-4639-a768-3f52806bd8c7-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"84b4b343-d43c-4639-a768-3f52806bd8c7\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 12:49:42.007741 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:49:42.007566    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/84b4b343-d43c-4639-a768-3f52806bd8c7-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"84b4b343-d43c-4639-a768-3f52806bd8c7\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 12:49:42.007741 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:49:42.007587    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hxkpn\" (UniqueName: \"kubernetes.io/projected/84b4b343-d43c-4639-a768-3f52806bd8c7-kube-api-access-hxkpn\") pod \"prometheus-k8s-0\" (UID: \"84b4b343-d43c-4639-a768-3f52806bd8c7\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 12:49:42.007741 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:49:42.007636    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/84b4b343-d43c-4639-a768-3f52806bd8c7-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"84b4b343-d43c-4639-a768-3f52806bd8c7\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 12:49:42.007741 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:49:42.007668    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/84b4b343-d43c-4639-a768-3f52806bd8c7-config-out\") pod \"prometheus-k8s-0\" (UID: \"84b4b343-d43c-4639-a768-3f52806bd8c7\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 12:49:42.108370 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:49:42.108323    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/84b4b343-d43c-4639-a768-3f52806bd8c7-config\") pod \"prometheus-k8s-0\" (UID: \"84b4b343-d43c-4639-a768-3f52806bd8c7\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 12:49:42.108586 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:49:42.108383    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/84b4b343-d43c-4639-a768-3f52806bd8c7-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"84b4b343-d43c-4639-a768-3f52806bd8c7\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 12:49:42.108586 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:49:42.108410    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/84b4b343-d43c-4639-a768-3f52806bd8c7-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"84b4b343-d43c-4639-a768-3f52806bd8c7\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 12:49:42.108586 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:49:42.108461    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/84b4b343-d43c-4639-a768-3f52806bd8c7-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"84b4b343-d43c-4639-a768-3f52806bd8c7\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 12:49:42.108586 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:49:42.108532    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/84b4b343-d43c-4639-a768-3f52806bd8c7-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"84b4b343-d43c-4639-a768-3f52806bd8c7\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 12:49:42.108586 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:49:42.108558    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/84b4b343-d43c-4639-a768-3f52806bd8c7-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"84b4b343-d43c-4639-a768-3f52806bd8c7\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 12:49:42.108586 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:49:42.108581    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/84b4b343-d43c-4639-a768-3f52806bd8c7-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"84b4b343-d43c-4639-a768-3f52806bd8c7\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 12:49:42.108899 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:49:42.108605    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/84b4b343-d43c-4639-a768-3f52806bd8c7-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"84b4b343-d43c-4639-a768-3f52806bd8c7\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 12:49:42.108899 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:49:42.108634    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/84b4b343-d43c-4639-a768-3f52806bd8c7-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"84b4b343-d43c-4639-a768-3f52806bd8c7\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 12:49:42.108899 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:49:42.108664    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/84b4b343-d43c-4639-a768-3f52806bd8c7-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"84b4b343-d43c-4639-a768-3f52806bd8c7\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 12:49:42.108899 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:49:42.108687    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hxkpn\" (UniqueName: \"kubernetes.io/projected/84b4b343-d43c-4639-a768-3f52806bd8c7-kube-api-access-hxkpn\") pod \"prometheus-k8s-0\" (UID: \"84b4b343-d43c-4639-a768-3f52806bd8c7\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 12:49:42.108899 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:49:42.108722    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/84b4b343-d43c-4639-a768-3f52806bd8c7-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"84b4b343-d43c-4639-a768-3f52806bd8c7\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 12:49:42.108899 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:49:42.108750    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/84b4b343-d43c-4639-a768-3f52806bd8c7-config-out\") pod \"prometheus-k8s-0\" (UID: \"84b4b343-d43c-4639-a768-3f52806bd8c7\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 12:49:42.108899 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:49:42.108788    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/84b4b343-d43c-4639-a768-3f52806bd8c7-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"84b4b343-d43c-4639-a768-3f52806bd8c7\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 12:49:42.108899 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:49:42.108817    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/84b4b343-d43c-4639-a768-3f52806bd8c7-web-config\") pod \"prometheus-k8s-0\" (UID: \"84b4b343-d43c-4639-a768-3f52806bd8c7\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 12:49:42.108899 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:49:42.108842    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/84b4b343-d43c-4639-a768-3f52806bd8c7-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"84b4b343-d43c-4639-a768-3f52806bd8c7\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 12:49:42.108899 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:49:42.108866    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/84b4b343-d43c-4639-a768-3f52806bd8c7-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"84b4b343-d43c-4639-a768-3f52806bd8c7\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 12:49:42.109448 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:49:42.108910    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/84b4b343-d43c-4639-a768-3f52806bd8c7-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"84b4b343-d43c-4639-a768-3f52806bd8c7\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 12:49:42.109448 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:49:42.108935    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/84b4b343-d43c-4639-a768-3f52806bd8c7-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"84b4b343-d43c-4639-a768-3f52806bd8c7\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 12:49:42.109448 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:49:42.108957    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/84b4b343-d43c-4639-a768-3f52806bd8c7-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"84b4b343-d43c-4639-a768-3f52806bd8c7\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 12:49:42.109448 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:49:42.109274    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/84b4b343-d43c-4639-a768-3f52806bd8c7-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"84b4b343-d43c-4639-a768-3f52806bd8c7\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 12:49:42.109680 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:49:42.109645    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/84b4b343-d43c-4639-a768-3f52806bd8c7-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"84b4b343-d43c-4639-a768-3f52806bd8c7\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 12:49:42.110298 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:49:42.109966    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/84b4b343-d43c-4639-a768-3f52806bd8c7-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"84b4b343-d43c-4639-a768-3f52806bd8c7\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 12:49:42.111703 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:49:42.111673    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/84b4b343-d43c-4639-a768-3f52806bd8c7-config\") pod \"prometheus-k8s-0\" (UID: \"84b4b343-d43c-4639-a768-3f52806bd8c7\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 12:49:42.111817 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:49:42.111796    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/84b4b343-d43c-4639-a768-3f52806bd8c7-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"84b4b343-d43c-4639-a768-3f52806bd8c7\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 12:49:42.111884 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:49:42.111796    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/84b4b343-d43c-4639-a768-3f52806bd8c7-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"84b4b343-d43c-4639-a768-3f52806bd8c7\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 12:49:42.112730 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:49:42.112144    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/84b4b343-d43c-4639-a768-3f52806bd8c7-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"84b4b343-d43c-4639-a768-3f52806bd8c7\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 12:49:42.112730 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:49:42.112305    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/84b4b343-d43c-4639-a768-3f52806bd8c7-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"84b4b343-d43c-4639-a768-3f52806bd8c7\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 12:49:42.112730 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:49:42.112412    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/84b4b343-d43c-4639-a768-3f52806bd8c7-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"84b4b343-d43c-4639-a768-3f52806bd8c7\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 12:49:42.112730 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:49:42.112430    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/84b4b343-d43c-4639-a768-3f52806bd8c7-config-out\") pod \"prometheus-k8s-0\" (UID: \"84b4b343-d43c-4639-a768-3f52806bd8c7\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 12:49:42.112730 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:49:42.112414    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/84b4b343-d43c-4639-a768-3f52806bd8c7-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"84b4b343-d43c-4639-a768-3f52806bd8c7\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 12:49:42.112730 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:49:42.112718    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/84b4b343-d43c-4639-a768-3f52806bd8c7-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"84b4b343-d43c-4639-a768-3f52806bd8c7\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 12:49:42.113071 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:49:42.112988    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/84b4b343-d43c-4639-a768-3f52806bd8c7-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"84b4b343-d43c-4639-a768-3f52806bd8c7\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 12:49:42.113354 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:49:42.113330    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/84b4b343-d43c-4639-a768-3f52806bd8c7-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"84b4b343-d43c-4639-a768-3f52806bd8c7\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 12:49:42.113796 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:49:42.113773    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/84b4b343-d43c-4639-a768-3f52806bd8c7-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"84b4b343-d43c-4639-a768-3f52806bd8c7\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 12:49:42.114200 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:49:42.114157    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/84b4b343-d43c-4639-a768-3f52806bd8c7-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"84b4b343-d43c-4639-a768-3f52806bd8c7\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 12:49:42.114442 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:49:42.114423    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/84b4b343-d43c-4639-a768-3f52806bd8c7-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"84b4b343-d43c-4639-a768-3f52806bd8c7\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 12:49:42.115015 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:49:42.114983    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/84b4b343-d43c-4639-a768-3f52806bd8c7-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"84b4b343-d43c-4639-a768-3f52806bd8c7\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 12:49:42.115015 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:49:42.115011    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/84b4b343-d43c-4639-a768-3f52806bd8c7-web-config\") pod \"prometheus-k8s-0\" (UID: \"84b4b343-d43c-4639-a768-3f52806bd8c7\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 12:49:42.121011 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:49:42.120986    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"1273107e-4c61-4b7c-983d-1dbf0dfc99ad","Type":"ContainerStarted","Data":"ee0cb37034a38c293b34ca754ffa15d42f3aa71d10bd2ae1174dd331fa1c8f91"}
Sep 16 12:49:42.121092 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:49:42.121045    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"1273107e-4c61-4b7c-983d-1dbf0dfc99ad","Type":"ContainerStarted","Data":"1f551efb68c802a59f189321abd2ea45682bc3486d4e1657f48da9d943b999be"}
Sep 16 12:49:42.121092 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:49:42.121070    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"1273107e-4c61-4b7c-983d-1dbf0dfc99ad","Type":"ContainerStarted","Data":"a54c7e46af1ebc257e0d9b958162957016e258e3208097d7668218fcff7fd53b"}
Sep 16 12:49:42.121176 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:49:42.121095    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"1273107e-4c61-4b7c-983d-1dbf0dfc99ad","Type":"ContainerStarted","Data":"a05cd1d38e33163d352bc753f901b862bf717eda0caf953613c789d74c67b89e"}
Sep 16 12:49:42.121176 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:49:42.121118    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"1273107e-4c61-4b7c-983d-1dbf0dfc99ad","Type":"ContainerStarted","Data":"3475af9ba00b872b5ebd574e0ccca4eeb9719e9746fd3e75c899d04a19f56cfc"}
Sep 16 12:49:42.121730 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:49:42.121713    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hxkpn\" (UniqueName: \"kubernetes.io/projected/84b4b343-d43c-4639-a768-3f52806bd8c7-kube-api-access-hxkpn\") pod \"prometheus-k8s-0\" (UID: \"84b4b343-d43c-4639-a768-3f52806bd8c7\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 12:49:42.163790 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:49:42.163765    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 12:49:42.458157 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:49:42.457998    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 16 12:49:42.471057 ip-10-0-133-218 kubenswrapper[2580]: W0916 12:49:42.471030    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod84b4b343_d43c_4639_a768_3f52806bd8c7.slice/crio-a2eb6c46fdd0ae7ed87a9f95dcfc290e7549cbb0ebc32fe675393a2e2773b8ab WatchSource:0}: Error finding container a2eb6c46fdd0ae7ed87a9f95dcfc290e7549cbb0ebc32fe675393a2e2773b8ab: Status 404 returned error can't find the container with id a2eb6c46fdd0ae7ed87a9f95dcfc290e7549cbb0ebc32fe675393a2e2773b8ab
Sep 16 12:49:43.128088 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:49:43.128048    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"1273107e-4c61-4b7c-983d-1dbf0dfc99ad","Type":"ContainerStarted","Data":"2d3e6c0437a661a0e7d886892ed3bc6a9578309126c70f34735a72bee8457976"}
Sep 16 12:49:43.129675 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:49:43.129650    2580 generic.go:356] "Generic (PLEG): container finished" podID="84b4b343-d43c-4639-a768-3f52806bd8c7" containerID="e4c8b98a3e29dc5927fd6e25c39c6f6e8e5a52bc45b8d99817ec406280a1c6a0" exitCode=0
Sep 16 12:49:43.129780 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:49:43.129683    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"84b4b343-d43c-4639-a768-3f52806bd8c7","Type":"ContainerDied","Data":"e4c8b98a3e29dc5927fd6e25c39c6f6e8e5a52bc45b8d99817ec406280a1c6a0"}
Sep 16 12:49:43.129780 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:49:43.129725    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"84b4b343-d43c-4639-a768-3f52806bd8c7","Type":"ContainerStarted","Data":"a2eb6c46fdd0ae7ed87a9f95dcfc290e7549cbb0ebc32fe675393a2e2773b8ab"}
Sep 16 12:49:43.160655 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:49:43.160599    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/alertmanager-main-0" podStartSLOduration=4.230556111 podStartE2EDuration="7.1605787s" totalImagesPullingTime="2.914485693s" totalInitContainerRuntime="15.536896ms" isStatefulPod=false podCreationTimestamp="2026-09-16 12:49:36 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 12:49:43.15937025 +0000 UTC m=+122.082242756" watchObservedRunningTime="2026-09-16 12:49:43.1605787 +0000 UTC m=+122.083451170"
Sep 16 12:49:47.150368 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:49:47.150333    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"84b4b343-d43c-4639-a768-3f52806bd8c7","Type":"ContainerStarted","Data":"935230d89b4c3cbbbdea303993cac3eb841c6b2fedfaf8ce16f5f9142794b6a8"}
Sep 16 12:49:47.150871 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:49:47.150400    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"84b4b343-d43c-4639-a768-3f52806bd8c7","Type":"ContainerStarted","Data":"a90db64ac3c00006d0d70fb1a4756654bf3c583bc4468edeff701d86e7c05c31"}
Sep 16 12:49:48.497249 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:49:48.497211    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-c9f8d47dd-z85ph"]
Sep 16 12:49:48.500931 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:49:48.500899    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-c9f8d47dd-z85ph"
Sep 16 12:49:48.517368 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:49:48.517336    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-c9f8d47dd-z85ph"]
Sep 16 12:49:48.674588 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:49:48.674562    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hccqg\" (UniqueName: \"kubernetes.io/projected/34893d6e-7a42-47d3-9ad8-cd1845c6ccd8-kube-api-access-hccqg\") pod \"console-c9f8d47dd-z85ph\" (UID: \"34893d6e-7a42-47d3-9ad8-cd1845c6ccd8\") " pod="openshift-console/console-c9f8d47dd-z85ph"
Sep 16 12:49:48.674684 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:49:48.674618    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/34893d6e-7a42-47d3-9ad8-cd1845c6ccd8-console-config\") pod \"console-c9f8d47dd-z85ph\" (UID: \"34893d6e-7a42-47d3-9ad8-cd1845c6ccd8\") " pod="openshift-console/console-c9f8d47dd-z85ph"
Sep 16 12:49:48.674684 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:49:48.674657    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/34893d6e-7a42-47d3-9ad8-cd1845c6ccd8-trusted-ca-bundle\") pod \"console-c9f8d47dd-z85ph\" (UID: \"34893d6e-7a42-47d3-9ad8-cd1845c6ccd8\") " pod="openshift-console/console-c9f8d47dd-z85ph"
Sep 16 12:49:48.674684 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:49:48.674679    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/34893d6e-7a42-47d3-9ad8-cd1845c6ccd8-oauth-serving-cert\") pod \"console-c9f8d47dd-z85ph\" (UID: \"34893d6e-7a42-47d3-9ad8-cd1845c6ccd8\") " pod="openshift-console/console-c9f8d47dd-z85ph"
Sep 16 12:49:48.674790 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:49:48.674739    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/34893d6e-7a42-47d3-9ad8-cd1845c6ccd8-console-serving-cert\") pod \"console-c9f8d47dd-z85ph\" (UID: \"34893d6e-7a42-47d3-9ad8-cd1845c6ccd8\") " pod="openshift-console/console-c9f8d47dd-z85ph"
Sep 16 12:49:48.674790 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:49:48.674760    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/34893d6e-7a42-47d3-9ad8-cd1845c6ccd8-service-ca\") pod \"console-c9f8d47dd-z85ph\" (UID: \"34893d6e-7a42-47d3-9ad8-cd1845c6ccd8\") " pod="openshift-console/console-c9f8d47dd-z85ph"
Sep 16 12:49:48.674857 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:49:48.674813    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/34893d6e-7a42-47d3-9ad8-cd1845c6ccd8-console-oauth-config\") pod \"console-c9f8d47dd-z85ph\" (UID: \"34893d6e-7a42-47d3-9ad8-cd1845c6ccd8\") " pod="openshift-console/console-c9f8d47dd-z85ph"
Sep 16 12:49:48.776223 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:49:48.776199    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hccqg\" (UniqueName: \"kubernetes.io/projected/34893d6e-7a42-47d3-9ad8-cd1845c6ccd8-kube-api-access-hccqg\") pod \"console-c9f8d47dd-z85ph\" (UID: \"34893d6e-7a42-47d3-9ad8-cd1845c6ccd8\") " pod="openshift-console/console-c9f8d47dd-z85ph"
Sep 16 12:49:48.776333 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:49:48.776243    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/34893d6e-7a42-47d3-9ad8-cd1845c6ccd8-console-config\") pod \"console-c9f8d47dd-z85ph\" (UID: \"34893d6e-7a42-47d3-9ad8-cd1845c6ccd8\") " pod="openshift-console/console-c9f8d47dd-z85ph"
Sep 16 12:49:48.776333 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:49:48.776268    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/34893d6e-7a42-47d3-9ad8-cd1845c6ccd8-trusted-ca-bundle\") pod \"console-c9f8d47dd-z85ph\" (UID: \"34893d6e-7a42-47d3-9ad8-cd1845c6ccd8\") " pod="openshift-console/console-c9f8d47dd-z85ph"
Sep 16 12:49:48.776333 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:49:48.776293    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/34893d6e-7a42-47d3-9ad8-cd1845c6ccd8-oauth-serving-cert\") pod \"console-c9f8d47dd-z85ph\" (UID: \"34893d6e-7a42-47d3-9ad8-cd1845c6ccd8\") " pod="openshift-console/console-c9f8d47dd-z85ph"
Sep 16 12:49:48.776509 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:49:48.776346    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/34893d6e-7a42-47d3-9ad8-cd1845c6ccd8-console-serving-cert\") pod \"console-c9f8d47dd-z85ph\" (UID: \"34893d6e-7a42-47d3-9ad8-cd1845c6ccd8\") " pod="openshift-console/console-c9f8d47dd-z85ph"
Sep 16 12:49:48.776509 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:49:48.776379    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/34893d6e-7a42-47d3-9ad8-cd1845c6ccd8-service-ca\") pod \"console-c9f8d47dd-z85ph\" (UID: \"34893d6e-7a42-47d3-9ad8-cd1845c6ccd8\") " pod="openshift-console/console-c9f8d47dd-z85ph"
Sep 16 12:49:48.776509 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:49:48.776461    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/34893d6e-7a42-47d3-9ad8-cd1845c6ccd8-console-oauth-config\") pod \"console-c9f8d47dd-z85ph\" (UID: \"34893d6e-7a42-47d3-9ad8-cd1845c6ccd8\") " pod="openshift-console/console-c9f8d47dd-z85ph"
Sep 16 12:49:48.777020 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:49:48.776998    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/34893d6e-7a42-47d3-9ad8-cd1845c6ccd8-console-config\") pod \"console-c9f8d47dd-z85ph\" (UID: \"34893d6e-7a42-47d3-9ad8-cd1845c6ccd8\") " pod="openshift-console/console-c9f8d47dd-z85ph"
Sep 16 12:49:48.777238 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:49:48.777209    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/34893d6e-7a42-47d3-9ad8-cd1845c6ccd8-service-ca\") pod \"console-c9f8d47dd-z85ph\" (UID: \"34893d6e-7a42-47d3-9ad8-cd1845c6ccd8\") " pod="openshift-console/console-c9f8d47dd-z85ph"
Sep 16 12:49:48.777843 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:49:48.777569    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/34893d6e-7a42-47d3-9ad8-cd1845c6ccd8-trusted-ca-bundle\") pod \"console-c9f8d47dd-z85ph\" (UID: \"34893d6e-7a42-47d3-9ad8-cd1845c6ccd8\") " pod="openshift-console/console-c9f8d47dd-z85ph"
Sep 16 12:49:48.777843 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:49:48.777778    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/34893d6e-7a42-47d3-9ad8-cd1845c6ccd8-oauth-serving-cert\") pod \"console-c9f8d47dd-z85ph\" (UID: \"34893d6e-7a42-47d3-9ad8-cd1845c6ccd8\") " pod="openshift-console/console-c9f8d47dd-z85ph"
Sep 16 12:49:48.779711 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:49:48.779669    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/34893d6e-7a42-47d3-9ad8-cd1845c6ccd8-console-serving-cert\") pod \"console-c9f8d47dd-z85ph\" (UID: \"34893d6e-7a42-47d3-9ad8-cd1845c6ccd8\") " pod="openshift-console/console-c9f8d47dd-z85ph"
Sep 16 12:49:48.779922 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:49:48.779901    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/34893d6e-7a42-47d3-9ad8-cd1845c6ccd8-console-oauth-config\") pod \"console-c9f8d47dd-z85ph\" (UID: \"34893d6e-7a42-47d3-9ad8-cd1845c6ccd8\") " pod="openshift-console/console-c9f8d47dd-z85ph"
Sep 16 12:49:48.786914 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:49:48.786887    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hccqg\" (UniqueName: \"kubernetes.io/projected/34893d6e-7a42-47d3-9ad8-cd1845c6ccd8-kube-api-access-hccqg\") pod \"console-c9f8d47dd-z85ph\" (UID: \"34893d6e-7a42-47d3-9ad8-cd1845c6ccd8\") " pod="openshift-console/console-c9f8d47dd-z85ph"
Sep 16 12:49:48.816320 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:49:48.816295    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-c9f8d47dd-z85ph"
Sep 16 12:49:49.090580 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:49:49.089922    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-c9f8d47dd-z85ph"]
Sep 16 12:49:49.095432 ip-10-0-133-218 kubenswrapper[2580]: W0916 12:49:49.095406    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod34893d6e_7a42_47d3_9ad8_cd1845c6ccd8.slice/crio-09af190660de48385938b802a9ef300f5cdb3ca8e20031fd0fab37ec32c3efba WatchSource:0}: Error finding container 09af190660de48385938b802a9ef300f5cdb3ca8e20031fd0fab37ec32c3efba: Status 404 returned error can't find the container with id 09af190660de48385938b802a9ef300f5cdb3ca8e20031fd0fab37ec32c3efba
Sep 16 12:49:49.159779 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:49:49.159753    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-c9f8d47dd-z85ph" event={"ID":"34893d6e-7a42-47d3-9ad8-cd1845c6ccd8","Type":"ContainerStarted","Data":"d2f72e49ac444d32872f2cc454acc1a994309f2af2a91e85d0249fb78ab4a984"}
Sep 16 12:49:49.159900 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:49:49.159808    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-c9f8d47dd-z85ph" event={"ID":"34893d6e-7a42-47d3-9ad8-cd1845c6ccd8","Type":"ContainerStarted","Data":"09af190660de48385938b802a9ef300f5cdb3ca8e20031fd0fab37ec32c3efba"}
Sep 16 12:49:49.163138 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:49:49.163111    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"84b4b343-d43c-4639-a768-3f52806bd8c7","Type":"ContainerStarted","Data":"84be625f3f386f8baeda5297f5efad680c1f6b97a7f91fa03f3523b58c9415ef"}
Sep 16 12:49:49.163230 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:49:49.163184    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"84b4b343-d43c-4639-a768-3f52806bd8c7","Type":"ContainerStarted","Data":"d25934a737be5b16f1c782d611e02f6bbbd06974fd86a136b102a4d77e134e92"}
Sep 16 12:49:49.163230 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:49:49.163224    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"84b4b343-d43c-4639-a768-3f52806bd8c7","Type":"ContainerStarted","Data":"d1f754c7c06351c83ed6e00e5bc4f31c69ca8c2e4fb704760304995862185e2a"}
Sep 16 12:49:49.163340 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:49:49.163270    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"84b4b343-d43c-4639-a768-3f52806bd8c7","Type":"ContainerStarted","Data":"1418326e938a5682f35998260a697824f95c6fee3512f8fd106f74caadec39c8"}
Sep 16 12:49:49.181591 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:49:49.181548    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-c9f8d47dd-z85ph" podStartSLOduration=1.181533494 podStartE2EDuration="1.181533494s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 12:49:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 12:49:49.181253345 +0000 UTC m=+128.104125813" watchObservedRunningTime="2026-09-16 12:49:49.181533494 +0000 UTC m=+128.104405964"
Sep 16 12:49:49.217055 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:49:49.217002    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-k8s-0" podStartSLOduration=2.873174719 podStartE2EDuration="8.216982709s" totalImagesPullingTime="5.325906704s" totalInitContainerRuntime="17.901286ms" isStatefulPod=false podCreationTimestamp="2026-09-16 12:49:41 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 12:49:49.21528758 +0000 UTC m=+128.138160050" watchObservedRunningTime="2026-09-16 12:49:49.216982709 +0000 UTC m=+128.139855177"
Sep 16 12:49:51.305831 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:49:51.305795    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7d634420-2900-42b3-9609-3fb64baff8f3-metrics-certs\") pod \"network-metrics-daemon-62j7x\" (UID: \"7d634420-2900-42b3-9609-3fb64baff8f3\") " pod="openshift-multus/network-metrics-daemon-62j7x"
Sep 16 12:49:51.308077 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:49:51.308056    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7d634420-2900-42b3-9609-3fb64baff8f3-metrics-certs\") pod \"network-metrics-daemon-62j7x\" (UID: \"7d634420-2900-42b3-9609-3fb64baff8f3\") " pod="openshift-multus/network-metrics-daemon-62j7x"
Sep 16 12:49:51.517361 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:49:51.517328    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-9nrdj\""
Sep 16 12:49:51.525287 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:49:51.525257    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-62j7x"
Sep 16 12:49:51.796458 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:49:51.796433    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-62j7x"]
Sep 16 12:49:51.798718 ip-10-0-133-218 kubenswrapper[2580]: W0916 12:49:51.798692    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7d634420_2900_42b3_9609_3fb64baff8f3.slice/crio-0e7e3eb419f99d64d7a564a3516798ae370db35686ee48d5768095afbee70f3e WatchSource:0}: Error finding container 0e7e3eb419f99d64d7a564a3516798ae370db35686ee48d5768095afbee70f3e: Status 404 returned error can't find the container with id 0e7e3eb419f99d64d7a564a3516798ae370db35686ee48d5768095afbee70f3e
Sep 16 12:49:52.164535 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:49:52.164504    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 12:49:52.175762 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:49:52.175734    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-62j7x" event={"ID":"7d634420-2900-42b3-9609-3fb64baff8f3","Type":"ContainerStarted","Data":"0e7e3eb419f99d64d7a564a3516798ae370db35686ee48d5768095afbee70f3e"}
Sep 16 12:49:53.181542 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:49:53.181480    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-62j7x" event={"ID":"7d634420-2900-42b3-9609-3fb64baff8f3","Type":"ContainerStarted","Data":"9551ad54b1d37ad257a401eba4047a6030e7b7d75541cc04ad0ffa2008e06c8c"}
Sep 16 12:49:53.181886 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:49:53.181553    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-62j7x" event={"ID":"7d634420-2900-42b3-9609-3fb64baff8f3","Type":"ContainerStarted","Data":"cc818874befc24159a145f0529947cbbcb01484da8b2b6857a4abf9e8721b142"}
Sep 16 12:49:53.204015 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:49:53.203967    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-62j7x" podStartSLOduration=130.324545653 podStartE2EDuration="2m11.203952113s" totalImagesPullingTime="879.40646ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 12:47:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 12:49:53.197528341 +0000 UTC m=+132.120400810" watchObservedRunningTime="2026-09-16 12:49:53.203952113 +0000 UTC m=+132.126824581"
Sep 16 12:49:58.817246 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:49:58.817204    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-c9f8d47dd-z85ph"
Sep 16 12:49:58.817779 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:49:58.817324    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-c9f8d47dd-z85ph"
Sep 16 12:49:58.822046 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:49:58.822026    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-c9f8d47dd-z85ph"
Sep 16 12:49:59.210270 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:49:59.210189    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-c9f8d47dd-z85ph"
Sep 16 12:49:59.264141 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:49:59.264102    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-7996cc44cd-p6ljc"]
Sep 16 12:50:03.165806 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:50:03.165754    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-d55cc74b5-x27t7" podUID="3720e939-652a-4ad2-938d-72adb38487ad" containerName="console" containerID="cri-o://f57210c5691648e52d71f656b0238592e32bd407dd704a5e5e12f90ecc4a8de8" gracePeriod=15
Sep 16 12:50:03.418932 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:50:03.418875    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-d55cc74b5-x27t7"
Sep 16 12:50:03.510036 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:50:03.509996    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/3720e939-652a-4ad2-938d-72adb38487ad-service-ca\") pod \"3720e939-652a-4ad2-938d-72adb38487ad\" (UID: \"3720e939-652a-4ad2-938d-72adb38487ad\") "
Sep 16 12:50:03.510205 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:50:03.510046    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/3720e939-652a-4ad2-938d-72adb38487ad-oauth-serving-cert\") pod \"3720e939-652a-4ad2-938d-72adb38487ad\" (UID: \"3720e939-652a-4ad2-938d-72adb38487ad\") "
Sep 16 12:50:03.510205 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:50:03.510074    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-smjrp\" (UniqueName: \"kubernetes.io/projected/3720e939-652a-4ad2-938d-72adb38487ad-kube-api-access-smjrp\") pod \"3720e939-652a-4ad2-938d-72adb38487ad\" (UID: \"3720e939-652a-4ad2-938d-72adb38487ad\") "
Sep 16 12:50:03.510205 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:50:03.510090    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/3720e939-652a-4ad2-938d-72adb38487ad-console-oauth-config\") pod \"3720e939-652a-4ad2-938d-72adb38487ad\" (UID: \"3720e939-652a-4ad2-938d-72adb38487ad\") "
Sep 16 12:50:03.510205 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:50:03.510159    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/3720e939-652a-4ad2-938d-72adb38487ad-console-config\") pod \"3720e939-652a-4ad2-938d-72adb38487ad\" (UID: \"3720e939-652a-4ad2-938d-72adb38487ad\") "
Sep 16 12:50:03.510205 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:50:03.510190    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/3720e939-652a-4ad2-938d-72adb38487ad-console-serving-cert\") pod \"3720e939-652a-4ad2-938d-72adb38487ad\" (UID: \"3720e939-652a-4ad2-938d-72adb38487ad\") "
Sep 16 12:50:03.510522 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:50:03.510460    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3720e939-652a-4ad2-938d-72adb38487ad-service-ca" (OuterVolumeSpecName: "service-ca") pod "3720e939-652a-4ad2-938d-72adb38487ad" (UID: "3720e939-652a-4ad2-938d-72adb38487ad"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 12:50:03.510676 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:50:03.510540    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3720e939-652a-4ad2-938d-72adb38487ad-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "3720e939-652a-4ad2-938d-72adb38487ad" (UID: "3720e939-652a-4ad2-938d-72adb38487ad"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 12:50:03.510743 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:50:03.510678    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3720e939-652a-4ad2-938d-72adb38487ad-console-config" (OuterVolumeSpecName: "console-config") pod "3720e939-652a-4ad2-938d-72adb38487ad" (UID: "3720e939-652a-4ad2-938d-72adb38487ad"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 12:50:03.512477 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:50:03.512453    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3720e939-652a-4ad2-938d-72adb38487ad-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "3720e939-652a-4ad2-938d-72adb38487ad" (UID: "3720e939-652a-4ad2-938d-72adb38487ad"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 12:50:03.512590 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:50:03.512483    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3720e939-652a-4ad2-938d-72adb38487ad-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "3720e939-652a-4ad2-938d-72adb38487ad" (UID: "3720e939-652a-4ad2-938d-72adb38487ad"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 12:50:03.512590 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:50:03.512515    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3720e939-652a-4ad2-938d-72adb38487ad-kube-api-access-smjrp" (OuterVolumeSpecName: "kube-api-access-smjrp") pod "3720e939-652a-4ad2-938d-72adb38487ad" (UID: "3720e939-652a-4ad2-938d-72adb38487ad"). InnerVolumeSpecName "kube-api-access-smjrp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 12:50:03.611692 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:50:03.611666    2580 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/3720e939-652a-4ad2-938d-72adb38487ad-console-config\") on node \"ip-10-0-133-218.ec2.internal\" DevicePath \"\""
Sep 16 12:50:03.611692 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:50:03.611691    2580 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/3720e939-652a-4ad2-938d-72adb38487ad-console-serving-cert\") on node \"ip-10-0-133-218.ec2.internal\" DevicePath \"\""
Sep 16 12:50:03.611847 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:50:03.611701    2580 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/3720e939-652a-4ad2-938d-72adb38487ad-service-ca\") on node \"ip-10-0-133-218.ec2.internal\" DevicePath \"\""
Sep 16 12:50:03.611847 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:50:03.611710    2580 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/3720e939-652a-4ad2-938d-72adb38487ad-oauth-serving-cert\") on node \"ip-10-0-133-218.ec2.internal\" DevicePath \"\""
Sep 16 12:50:03.611847 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:50:03.611718    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-smjrp\" (UniqueName: \"kubernetes.io/projected/3720e939-652a-4ad2-938d-72adb38487ad-kube-api-access-smjrp\") on node \"ip-10-0-133-218.ec2.internal\" DevicePath \"\""
Sep 16 12:50:03.611847 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:50:03.611727    2580 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/3720e939-652a-4ad2-938d-72adb38487ad-console-oauth-config\") on node \"ip-10-0-133-218.ec2.internal\" DevicePath \"\""
Sep 16 12:50:04.228396 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:50:04.228364    2580 generic.go:356] "Generic (PLEG): container finished" podID="3720e939-652a-4ad2-938d-72adb38487ad" containerID="f57210c5691648e52d71f656b0238592e32bd407dd704a5e5e12f90ecc4a8de8" exitCode=0
Sep 16 12:50:04.228804 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:50:04.228405    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-d55cc74b5-x27t7"
Sep 16 12:50:04.228804 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:50:04.228451    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-d55cc74b5-x27t7" event={"ID":"3720e939-652a-4ad2-938d-72adb38487ad","Type":"ContainerDied","Data":"f57210c5691648e52d71f656b0238592e32bd407dd704a5e5e12f90ecc4a8de8"}
Sep 16 12:50:04.228804 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:50:04.228529    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-d55cc74b5-x27t7" event={"ID":"3720e939-652a-4ad2-938d-72adb38487ad","Type":"ContainerDied","Data":"0daabbeb96f09671190104a791d0a67cc98a11e67432e661673b6e91d95593ff"}
Sep 16 12:50:04.228804 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:50:04.228551    2580 scope.go:117] "RemoveContainer" containerID="f57210c5691648e52d71f656b0238592e32bd407dd704a5e5e12f90ecc4a8de8"
Sep 16 12:50:04.236972 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:50:04.236955    2580 scope.go:117] "RemoveContainer" containerID="f57210c5691648e52d71f656b0238592e32bd407dd704a5e5e12f90ecc4a8de8"
Sep 16 12:50:04.237281 ip-10-0-133-218 kubenswrapper[2580]: E0916 12:50:04.237250    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f57210c5691648e52d71f656b0238592e32bd407dd704a5e5e12f90ecc4a8de8\": container with ID starting with f57210c5691648e52d71f656b0238592e32bd407dd704a5e5e12f90ecc4a8de8 not found: ID does not exist" containerID="f57210c5691648e52d71f656b0238592e32bd407dd704a5e5e12f90ecc4a8de8"
Sep 16 12:50:04.237329 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:50:04.237291    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f57210c5691648e52d71f656b0238592e32bd407dd704a5e5e12f90ecc4a8de8"} err="failed to get container status \"f57210c5691648e52d71f656b0238592e32bd407dd704a5e5e12f90ecc4a8de8\": rpc error: code = NotFound desc = could not find container \"f57210c5691648e52d71f656b0238592e32bd407dd704a5e5e12f90ecc4a8de8\": container with ID starting with f57210c5691648e52d71f656b0238592e32bd407dd704a5e5e12f90ecc4a8de8 not found: ID does not exist"
Sep 16 12:50:04.249610 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:50:04.249588    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-d55cc74b5-x27t7"]
Sep 16 12:50:04.258733 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:50:04.258710    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-d55cc74b5-x27t7"]
Sep 16 12:50:05.599106 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:50:05.599070    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3720e939-652a-4ad2-938d-72adb38487ad" path="/var/lib/kubelet/pods/3720e939-652a-4ad2-938d-72adb38487ad/volumes"
Sep 16 12:50:24.286226 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:50:24.286169    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-7996cc44cd-p6ljc" podUID="c5a49713-de16-4f73-b654-d4ed06ce1946" containerName="console" containerID="cri-o://99b285154caf119af4f490e632090411f57cc87e5e3777f5101a2396d4e89eb7" gracePeriod=15
Sep 16 12:50:24.522935 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:50:24.522912    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7996cc44cd-p6ljc"
Sep 16 12:50:24.691639 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:50:24.691557    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/c5a49713-de16-4f73-b654-d4ed06ce1946-console-oauth-config\") pod \"c5a49713-de16-4f73-b654-d4ed06ce1946\" (UID: \"c5a49713-de16-4f73-b654-d4ed06ce1946\") "
Sep 16 12:50:24.691639 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:50:24.691607    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c5a49713-de16-4f73-b654-d4ed06ce1946-trusted-ca-bundle\") pod \"c5a49713-de16-4f73-b654-d4ed06ce1946\" (UID: \"c5a49713-de16-4f73-b654-d4ed06ce1946\") "
Sep 16 12:50:24.691864 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:50:24.691645    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/c5a49713-de16-4f73-b654-d4ed06ce1946-console-serving-cert\") pod \"c5a49713-de16-4f73-b654-d4ed06ce1946\" (UID: \"c5a49713-de16-4f73-b654-d4ed06ce1946\") "
Sep 16 12:50:24.691864 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:50:24.691670    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/c5a49713-de16-4f73-b654-d4ed06ce1946-service-ca\") pod \"c5a49713-de16-4f73-b654-d4ed06ce1946\" (UID: \"c5a49713-de16-4f73-b654-d4ed06ce1946\") "
Sep 16 12:50:24.691864 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:50:24.691699    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zd74h\" (UniqueName: \"kubernetes.io/projected/c5a49713-de16-4f73-b654-d4ed06ce1946-kube-api-access-zd74h\") pod \"c5a49713-de16-4f73-b654-d4ed06ce1946\" (UID: \"c5a49713-de16-4f73-b654-d4ed06ce1946\") "
Sep 16 12:50:24.691864 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:50:24.691770    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/c5a49713-de16-4f73-b654-d4ed06ce1946-console-config\") pod \"c5a49713-de16-4f73-b654-d4ed06ce1946\" (UID: \"c5a49713-de16-4f73-b654-d4ed06ce1946\") "
Sep 16 12:50:24.691864 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:50:24.691800    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/c5a49713-de16-4f73-b654-d4ed06ce1946-oauth-serving-cert\") pod \"c5a49713-de16-4f73-b654-d4ed06ce1946\" (UID: \"c5a49713-de16-4f73-b654-d4ed06ce1946\") "
Sep 16 12:50:24.692124 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:50:24.692096    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c5a49713-de16-4f73-b654-d4ed06ce1946-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "c5a49713-de16-4f73-b654-d4ed06ce1946" (UID: "c5a49713-de16-4f73-b654-d4ed06ce1946"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 12:50:24.692179 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:50:24.692108    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c5a49713-de16-4f73-b654-d4ed06ce1946-service-ca" (OuterVolumeSpecName: "service-ca") pod "c5a49713-de16-4f73-b654-d4ed06ce1946" (UID: "c5a49713-de16-4f73-b654-d4ed06ce1946"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 12:50:24.692234 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:50:24.692165    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c5a49713-de16-4f73-b654-d4ed06ce1946-console-config" (OuterVolumeSpecName: "console-config") pod "c5a49713-de16-4f73-b654-d4ed06ce1946" (UID: "c5a49713-de16-4f73-b654-d4ed06ce1946"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 12:50:24.692234 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:50:24.692210    2580 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c5a49713-de16-4f73-b654-d4ed06ce1946-trusted-ca-bundle\") on node \"ip-10-0-133-218.ec2.internal\" DevicePath \"\""
Sep 16 12:50:24.692234 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:50:24.692231    2580 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/c5a49713-de16-4f73-b654-d4ed06ce1946-service-ca\") on node \"ip-10-0-133-218.ec2.internal\" DevicePath \"\""
Sep 16 12:50:24.692439 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:50:24.692417    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c5a49713-de16-4f73-b654-d4ed06ce1946-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "c5a49713-de16-4f73-b654-d4ed06ce1946" (UID: "c5a49713-de16-4f73-b654-d4ed06ce1946"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 12:50:24.693891 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:50:24.693870    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c5a49713-de16-4f73-b654-d4ed06ce1946-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "c5a49713-de16-4f73-b654-d4ed06ce1946" (UID: "c5a49713-de16-4f73-b654-d4ed06ce1946"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 12:50:24.694200 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:50:24.694176    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c5a49713-de16-4f73-b654-d4ed06ce1946-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "c5a49713-de16-4f73-b654-d4ed06ce1946" (UID: "c5a49713-de16-4f73-b654-d4ed06ce1946"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 12:50:24.694274 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:50:24.694254    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c5a49713-de16-4f73-b654-d4ed06ce1946-kube-api-access-zd74h" (OuterVolumeSpecName: "kube-api-access-zd74h") pod "c5a49713-de16-4f73-b654-d4ed06ce1946" (UID: "c5a49713-de16-4f73-b654-d4ed06ce1946"). InnerVolumeSpecName "kube-api-access-zd74h". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 12:50:24.793425 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:50:24.793395    2580 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/c5a49713-de16-4f73-b654-d4ed06ce1946-console-config\") on node \"ip-10-0-133-218.ec2.internal\" DevicePath \"\""
Sep 16 12:50:24.793425 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:50:24.793421    2580 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/c5a49713-de16-4f73-b654-d4ed06ce1946-oauth-serving-cert\") on node \"ip-10-0-133-218.ec2.internal\" DevicePath \"\""
Sep 16 12:50:24.793425 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:50:24.793430    2580 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/c5a49713-de16-4f73-b654-d4ed06ce1946-console-oauth-config\") on node \"ip-10-0-133-218.ec2.internal\" DevicePath \"\""
Sep 16 12:50:24.793652 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:50:24.793439    2580 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/c5a49713-de16-4f73-b654-d4ed06ce1946-console-serving-cert\") on node \"ip-10-0-133-218.ec2.internal\" DevicePath \"\""
Sep 16 12:50:24.793652 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:50:24.793448    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zd74h\" (UniqueName: \"kubernetes.io/projected/c5a49713-de16-4f73-b654-d4ed06ce1946-kube-api-access-zd74h\") on node \"ip-10-0-133-218.ec2.internal\" DevicePath \"\""
Sep 16 12:50:25.316504 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:50:25.316456    2580 generic.go:356] "Generic (PLEG): container finished" podID="c5a49713-de16-4f73-b654-d4ed06ce1946" containerID="99b285154caf119af4f490e632090411f57cc87e5e3777f5101a2396d4e89eb7" exitCode=0
Sep 16 12:50:25.316947 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:50:25.316523    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7996cc44cd-p6ljc"
Sep 16 12:50:25.316947 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:50:25.316525    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7996cc44cd-p6ljc" event={"ID":"c5a49713-de16-4f73-b654-d4ed06ce1946","Type":"ContainerDied","Data":"99b285154caf119af4f490e632090411f57cc87e5e3777f5101a2396d4e89eb7"}
Sep 16 12:50:25.316947 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:50:25.316583    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7996cc44cd-p6ljc" event={"ID":"c5a49713-de16-4f73-b654-d4ed06ce1946","Type":"ContainerDied","Data":"fc0f1d94d98365283df52e710152c3245d835d8e6edddaeded10ae07ffd9f27d"}
Sep 16 12:50:25.316947 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:50:25.316612    2580 scope.go:117] "RemoveContainer" containerID="99b285154caf119af4f490e632090411f57cc87e5e3777f5101a2396d4e89eb7"
Sep 16 12:50:25.325209 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:50:25.325195    2580 scope.go:117] "RemoveContainer" containerID="99b285154caf119af4f490e632090411f57cc87e5e3777f5101a2396d4e89eb7"
Sep 16 12:50:25.325456 ip-10-0-133-218 kubenswrapper[2580]: E0916 12:50:25.325437    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"99b285154caf119af4f490e632090411f57cc87e5e3777f5101a2396d4e89eb7\": container with ID starting with 99b285154caf119af4f490e632090411f57cc87e5e3777f5101a2396d4e89eb7 not found: ID does not exist" containerID="99b285154caf119af4f490e632090411f57cc87e5e3777f5101a2396d4e89eb7"
Sep 16 12:50:25.325548 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:50:25.325463    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"99b285154caf119af4f490e632090411f57cc87e5e3777f5101a2396d4e89eb7"} err="failed to get container status \"99b285154caf119af4f490e632090411f57cc87e5e3777f5101a2396d4e89eb7\": rpc error: code = NotFound desc = could not find container \"99b285154caf119af4f490e632090411f57cc87e5e3777f5101a2396d4e89eb7\": container with ID starting with 99b285154caf119af4f490e632090411f57cc87e5e3777f5101a2396d4e89eb7 not found: ID does not exist"
Sep 16 12:50:25.336700 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:50:25.336676    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-7996cc44cd-p6ljc"]
Sep 16 12:50:25.342764 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:50:25.342745    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-7996cc44cd-p6ljc"]
Sep 16 12:50:25.599465 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:50:25.599386    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c5a49713-de16-4f73-b654-d4ed06ce1946" path="/var/lib/kubelet/pods/c5a49713-de16-4f73-b654-d4ed06ce1946/volumes"
Sep 16 12:50:42.164669 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:50:42.164631    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 12:50:42.180337 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:50:42.180313    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 12:50:42.400878 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:50:42.400850    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 12:50:55.872170 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:50:55.872135    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Sep 16 12:50:55.872771 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:50:55.872505    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="1273107e-4c61-4b7c-983d-1dbf0dfc99ad" containerName="alertmanager" containerID="cri-o://3475af9ba00b872b5ebd574e0ccca4eeb9719e9746fd3e75c899d04a19f56cfc" gracePeriod=120
Sep 16 12:50:55.872771 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:50:55.872556    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="1273107e-4c61-4b7c-983d-1dbf0dfc99ad" containerName="kube-rbac-proxy-metric" containerID="cri-o://ee0cb37034a38c293b34ca754ffa15d42f3aa71d10bd2ae1174dd331fa1c8f91" gracePeriod=120
Sep 16 12:50:55.872771 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:50:55.872584    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="1273107e-4c61-4b7c-983d-1dbf0dfc99ad" containerName="prom-label-proxy" containerID="cri-o://2d3e6c0437a661a0e7d886892ed3bc6a9578309126c70f34735a72bee8457976" gracePeriod=120
Sep 16 12:50:55.872771 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:50:55.872624    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="1273107e-4c61-4b7c-983d-1dbf0dfc99ad" containerName="config-reloader" containerID="cri-o://a05cd1d38e33163d352bc753f901b862bf717eda0caf953613c789d74c67b89e" gracePeriod=120
Sep 16 12:50:55.872771 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:50:55.872661    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="1273107e-4c61-4b7c-983d-1dbf0dfc99ad" containerName="kube-rbac-proxy-web" containerID="cri-o://a54c7e46af1ebc257e0d9b958162957016e258e3208097d7668218fcff7fd53b" gracePeriod=120
Sep 16 12:50:55.872771 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:50:55.872703    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="1273107e-4c61-4b7c-983d-1dbf0dfc99ad" containerName="kube-rbac-proxy" containerID="cri-o://1f551efb68c802a59f189321abd2ea45682bc3486d4e1657f48da9d943b999be" gracePeriod=120
Sep 16 12:50:56.442063 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:50:56.442030    2580 generic.go:356] "Generic (PLEG): container finished" podID="1273107e-4c61-4b7c-983d-1dbf0dfc99ad" containerID="2d3e6c0437a661a0e7d886892ed3bc6a9578309126c70f34735a72bee8457976" exitCode=0
Sep 16 12:50:56.442063 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:50:56.442056    2580 generic.go:356] "Generic (PLEG): container finished" podID="1273107e-4c61-4b7c-983d-1dbf0dfc99ad" containerID="1f551efb68c802a59f189321abd2ea45682bc3486d4e1657f48da9d943b999be" exitCode=0
Sep 16 12:50:56.442063 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:50:56.442064    2580 generic.go:356] "Generic (PLEG): container finished" podID="1273107e-4c61-4b7c-983d-1dbf0dfc99ad" containerID="a05cd1d38e33163d352bc753f901b862bf717eda0caf953613c789d74c67b89e" exitCode=0
Sep 16 12:50:56.442063 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:50:56.442070    2580 generic.go:356] "Generic (PLEG): container finished" podID="1273107e-4c61-4b7c-983d-1dbf0dfc99ad" containerID="3475af9ba00b872b5ebd574e0ccca4eeb9719e9746fd3e75c899d04a19f56cfc" exitCode=0
Sep 16 12:50:56.442318 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:50:56.442103    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"1273107e-4c61-4b7c-983d-1dbf0dfc99ad","Type":"ContainerDied","Data":"2d3e6c0437a661a0e7d886892ed3bc6a9578309126c70f34735a72bee8457976"}
Sep 16 12:50:56.442318 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:50:56.442164    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"1273107e-4c61-4b7c-983d-1dbf0dfc99ad","Type":"ContainerDied","Data":"1f551efb68c802a59f189321abd2ea45682bc3486d4e1657f48da9d943b999be"}
Sep 16 12:50:56.442318 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:50:56.442193    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"1273107e-4c61-4b7c-983d-1dbf0dfc99ad","Type":"ContainerDied","Data":"a05cd1d38e33163d352bc753f901b862bf717eda0caf953613c789d74c67b89e"}
Sep 16 12:50:56.442318 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:50:56.442217    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"1273107e-4c61-4b7c-983d-1dbf0dfc99ad","Type":"ContainerDied","Data":"3475af9ba00b872b5ebd574e0ccca4eeb9719e9746fd3e75c899d04a19f56cfc"}
Sep 16 12:50:57.116551 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:50:57.116526    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Sep 16 12:50:57.162411 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:50:57.162377    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/1273107e-4c61-4b7c-983d-1dbf0dfc99ad-tls-assets\") pod \"1273107e-4c61-4b7c-983d-1dbf0dfc99ad\" (UID: \"1273107e-4c61-4b7c-983d-1dbf0dfc99ad\") "
Sep 16 12:50:57.162594 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:50:57.162426    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/1273107e-4c61-4b7c-983d-1dbf0dfc99ad-cluster-tls-config\") pod \"1273107e-4c61-4b7c-983d-1dbf0dfc99ad\" (UID: \"1273107e-4c61-4b7c-983d-1dbf0dfc99ad\") "
Sep 16 12:50:57.162594 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:50:57.162449    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/1273107e-4c61-4b7c-983d-1dbf0dfc99ad-config-volume\") pod \"1273107e-4c61-4b7c-983d-1dbf0dfc99ad\" (UID: \"1273107e-4c61-4b7c-983d-1dbf0dfc99ad\") "
Sep 16 12:50:57.162594 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:50:57.162471    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/1273107e-4c61-4b7c-983d-1dbf0dfc99ad-config-out\") pod \"1273107e-4c61-4b7c-983d-1dbf0dfc99ad\" (UID: \"1273107e-4c61-4b7c-983d-1dbf0dfc99ad\") "
Sep 16 12:50:57.162594 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:50:57.162518    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/1273107e-4c61-4b7c-983d-1dbf0dfc99ad-secret-alertmanager-kube-rbac-proxy-metric\") pod \"1273107e-4c61-4b7c-983d-1dbf0dfc99ad\" (UID: \"1273107e-4c61-4b7c-983d-1dbf0dfc99ad\") "
Sep 16 12:50:57.162594 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:50:57.162550    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/1273107e-4c61-4b7c-983d-1dbf0dfc99ad-alertmanager-main-db\") pod \"1273107e-4c61-4b7c-983d-1dbf0dfc99ad\" (UID: \"1273107e-4c61-4b7c-983d-1dbf0dfc99ad\") "
Sep 16 12:50:57.162594 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:50:57.162591    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/1273107e-4c61-4b7c-983d-1dbf0dfc99ad-secret-alertmanager-main-tls\") pod \"1273107e-4c61-4b7c-983d-1dbf0dfc99ad\" (UID: \"1273107e-4c61-4b7c-983d-1dbf0dfc99ad\") "
Sep 16 12:50:57.162970 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:50:57.162627    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/1273107e-4c61-4b7c-983d-1dbf0dfc99ad-secret-alertmanager-kube-rbac-proxy\") pod \"1273107e-4c61-4b7c-983d-1dbf0dfc99ad\" (UID: \"1273107e-4c61-4b7c-983d-1dbf0dfc99ad\") "
Sep 16 12:50:57.162970 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:50:57.162651    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/1273107e-4c61-4b7c-983d-1dbf0dfc99ad-web-config\") pod \"1273107e-4c61-4b7c-983d-1dbf0dfc99ad\" (UID: \"1273107e-4c61-4b7c-983d-1dbf0dfc99ad\") "
Sep 16 12:50:57.162970 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:50:57.162682    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-982lr\" (UniqueName: \"kubernetes.io/projected/1273107e-4c61-4b7c-983d-1dbf0dfc99ad-kube-api-access-982lr\") pod \"1273107e-4c61-4b7c-983d-1dbf0dfc99ad\" (UID: \"1273107e-4c61-4b7c-983d-1dbf0dfc99ad\") "
Sep 16 12:50:57.162970 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:50:57.162712    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1273107e-4c61-4b7c-983d-1dbf0dfc99ad-alertmanager-trusted-ca-bundle\") pod \"1273107e-4c61-4b7c-983d-1dbf0dfc99ad\" (UID: \"1273107e-4c61-4b7c-983d-1dbf0dfc99ad\") "
Sep 16 12:50:57.162970 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:50:57.162752    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/1273107e-4c61-4b7c-983d-1dbf0dfc99ad-metrics-client-ca\") pod \"1273107e-4c61-4b7c-983d-1dbf0dfc99ad\" (UID: \"1273107e-4c61-4b7c-983d-1dbf0dfc99ad\") "
Sep 16 12:50:57.162970 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:50:57.162840    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/1273107e-4c61-4b7c-983d-1dbf0dfc99ad-secret-alertmanager-kube-rbac-proxy-web\") pod \"1273107e-4c61-4b7c-983d-1dbf0dfc99ad\" (UID: \"1273107e-4c61-4b7c-983d-1dbf0dfc99ad\") "
Sep 16 12:50:57.163637 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:50:57.163469    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1273107e-4c61-4b7c-983d-1dbf0dfc99ad-alertmanager-trusted-ca-bundle" (OuterVolumeSpecName: "alertmanager-trusted-ca-bundle") pod "1273107e-4c61-4b7c-983d-1dbf0dfc99ad" (UID: "1273107e-4c61-4b7c-983d-1dbf0dfc99ad"). InnerVolumeSpecName "alertmanager-trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 12:50:57.163767 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:50:57.163698    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1273107e-4c61-4b7c-983d-1dbf0dfc99ad-alertmanager-main-db" (OuterVolumeSpecName: "alertmanager-main-db") pod "1273107e-4c61-4b7c-983d-1dbf0dfc99ad" (UID: "1273107e-4c61-4b7c-983d-1dbf0dfc99ad"). InnerVolumeSpecName "alertmanager-main-db". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 12:50:57.164074 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:50:57.163947    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1273107e-4c61-4b7c-983d-1dbf0dfc99ad-metrics-client-ca" (OuterVolumeSpecName: "metrics-client-ca") pod "1273107e-4c61-4b7c-983d-1dbf0dfc99ad" (UID: "1273107e-4c61-4b7c-983d-1dbf0dfc99ad"). InnerVolumeSpecName "metrics-client-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 12:50:57.168068 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:50:57.167864    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1273107e-4c61-4b7c-983d-1dbf0dfc99ad-secret-alertmanager-kube-rbac-proxy" (OuterVolumeSpecName: "secret-alertmanager-kube-rbac-proxy") pod "1273107e-4c61-4b7c-983d-1dbf0dfc99ad" (UID: "1273107e-4c61-4b7c-983d-1dbf0dfc99ad"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 12:50:57.168068 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:50:57.167953    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1273107e-4c61-4b7c-983d-1dbf0dfc99ad-config-out" (OuterVolumeSpecName: "config-out") pod "1273107e-4c61-4b7c-983d-1dbf0dfc99ad" (UID: "1273107e-4c61-4b7c-983d-1dbf0dfc99ad"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 12:50:57.168068 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:50:57.168041    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1273107e-4c61-4b7c-983d-1dbf0dfc99ad-secret-alertmanager-kube-rbac-proxy-web" (OuterVolumeSpecName: "secret-alertmanager-kube-rbac-proxy-web") pod "1273107e-4c61-4b7c-983d-1dbf0dfc99ad" (UID: "1273107e-4c61-4b7c-983d-1dbf0dfc99ad"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy-web". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 12:50:57.168478 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:50:57.168381    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1273107e-4c61-4b7c-983d-1dbf0dfc99ad-kube-api-access-982lr" (OuterVolumeSpecName: "kube-api-access-982lr") pod "1273107e-4c61-4b7c-983d-1dbf0dfc99ad" (UID: "1273107e-4c61-4b7c-983d-1dbf0dfc99ad"). InnerVolumeSpecName "kube-api-access-982lr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 12:50:57.168478 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:50:57.168445    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1273107e-4c61-4b7c-983d-1dbf0dfc99ad-config-volume" (OuterVolumeSpecName: "config-volume") pod "1273107e-4c61-4b7c-983d-1dbf0dfc99ad" (UID: "1273107e-4c61-4b7c-983d-1dbf0dfc99ad"). InnerVolumeSpecName "config-volume". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 12:50:57.168953 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:50:57.168926    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1273107e-4c61-4b7c-983d-1dbf0dfc99ad-secret-alertmanager-kube-rbac-proxy-metric" (OuterVolumeSpecName: "secret-alertmanager-kube-rbac-proxy-metric") pod "1273107e-4c61-4b7c-983d-1dbf0dfc99ad" (UID: "1273107e-4c61-4b7c-983d-1dbf0dfc99ad"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy-metric". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 12:50:57.169257 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:50:57.169231    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1273107e-4c61-4b7c-983d-1dbf0dfc99ad-tls-assets" (OuterVolumeSpecName: "tls-assets") pod "1273107e-4c61-4b7c-983d-1dbf0dfc99ad" (UID: "1273107e-4c61-4b7c-983d-1dbf0dfc99ad"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 12:50:57.169771 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:50:57.169746    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1273107e-4c61-4b7c-983d-1dbf0dfc99ad-secret-alertmanager-main-tls" (OuterVolumeSpecName: "secret-alertmanager-main-tls") pod "1273107e-4c61-4b7c-983d-1dbf0dfc99ad" (UID: "1273107e-4c61-4b7c-983d-1dbf0dfc99ad"). InnerVolumeSpecName "secret-alertmanager-main-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 12:50:57.171747 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:50:57.171613    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1273107e-4c61-4b7c-983d-1dbf0dfc99ad-cluster-tls-config" (OuterVolumeSpecName: "cluster-tls-config") pod "1273107e-4c61-4b7c-983d-1dbf0dfc99ad" (UID: "1273107e-4c61-4b7c-983d-1dbf0dfc99ad"). InnerVolumeSpecName "cluster-tls-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 12:50:57.179592 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:50:57.179564    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1273107e-4c61-4b7c-983d-1dbf0dfc99ad-web-config" (OuterVolumeSpecName: "web-config") pod "1273107e-4c61-4b7c-983d-1dbf0dfc99ad" (UID: "1273107e-4c61-4b7c-983d-1dbf0dfc99ad"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 12:50:57.263621 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:50:57.263543    2580 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/1273107e-4c61-4b7c-983d-1dbf0dfc99ad-secret-alertmanager-kube-rbac-proxy-web\") on node \"ip-10-0-133-218.ec2.internal\" DevicePath \"\""
Sep 16 12:50:57.263621 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:50:57.263569    2580 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/1273107e-4c61-4b7c-983d-1dbf0dfc99ad-tls-assets\") on node \"ip-10-0-133-218.ec2.internal\" DevicePath \"\""
Sep 16 12:50:57.263621 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:50:57.263581    2580 reconciler_common.go:299] "Volume detached for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/1273107e-4c61-4b7c-983d-1dbf0dfc99ad-cluster-tls-config\") on node \"ip-10-0-133-218.ec2.internal\" DevicePath \"\""
Sep 16 12:50:57.263621 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:50:57.263590    2580 reconciler_common.go:299] "Volume detached for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/1273107e-4c61-4b7c-983d-1dbf0dfc99ad-config-volume\") on node \"ip-10-0-133-218.ec2.internal\" DevicePath \"\""
Sep 16 12:50:57.263621 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:50:57.263598    2580 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/1273107e-4c61-4b7c-983d-1dbf0dfc99ad-config-out\") on node \"ip-10-0-133-218.ec2.internal\" DevicePath \"\""
Sep 16 12:50:57.263621 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:50:57.263607    2580 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/1273107e-4c61-4b7c-983d-1dbf0dfc99ad-secret-alertmanager-kube-rbac-proxy-metric\") on node \"ip-10-0-133-218.ec2.internal\" DevicePath \"\""
Sep 16 12:50:57.263621 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:50:57.263617    2580 reconciler_common.go:299] "Volume detached for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/1273107e-4c61-4b7c-983d-1dbf0dfc99ad-alertmanager-main-db\") on node \"ip-10-0-133-218.ec2.internal\" DevicePath \"\""
Sep 16 12:50:57.263621 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:50:57.263626    2580 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/1273107e-4c61-4b7c-983d-1dbf0dfc99ad-secret-alertmanager-main-tls\") on node \"ip-10-0-133-218.ec2.internal\" DevicePath \"\""
Sep 16 12:50:57.263961 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:50:57.263635    2580 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/1273107e-4c61-4b7c-983d-1dbf0dfc99ad-secret-alertmanager-kube-rbac-proxy\") on node \"ip-10-0-133-218.ec2.internal\" DevicePath \"\""
Sep 16 12:50:57.263961 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:50:57.263645    2580 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/1273107e-4c61-4b7c-983d-1dbf0dfc99ad-web-config\") on node \"ip-10-0-133-218.ec2.internal\" DevicePath \"\""
Sep 16 12:50:57.263961 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:50:57.263652    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-982lr\" (UniqueName: \"kubernetes.io/projected/1273107e-4c61-4b7c-983d-1dbf0dfc99ad-kube-api-access-982lr\") on node \"ip-10-0-133-218.ec2.internal\" DevicePath \"\""
Sep 16 12:50:57.263961 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:50:57.263660    2580 reconciler_common.go:299] "Volume detached for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1273107e-4c61-4b7c-983d-1dbf0dfc99ad-alertmanager-trusted-ca-bundle\") on node \"ip-10-0-133-218.ec2.internal\" DevicePath \"\""
Sep 16 12:50:57.263961 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:50:57.263669    2580 reconciler_common.go:299] "Volume detached for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/1273107e-4c61-4b7c-983d-1dbf0dfc99ad-metrics-client-ca\") on node \"ip-10-0-133-218.ec2.internal\" DevicePath \"\""
Sep 16 12:50:57.449139 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:50:57.449102    2580 generic.go:356] "Generic (PLEG): container finished" podID="1273107e-4c61-4b7c-983d-1dbf0dfc99ad" containerID="ee0cb37034a38c293b34ca754ffa15d42f3aa71d10bd2ae1174dd331fa1c8f91" exitCode=0
Sep 16 12:50:57.449139 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:50:57.449127    2580 generic.go:356] "Generic (PLEG): container finished" podID="1273107e-4c61-4b7c-983d-1dbf0dfc99ad" containerID="a54c7e46af1ebc257e0d9b958162957016e258e3208097d7668218fcff7fd53b" exitCode=0
Sep 16 12:50:57.449376 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:50:57.449146    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"1273107e-4c61-4b7c-983d-1dbf0dfc99ad","Type":"ContainerDied","Data":"ee0cb37034a38c293b34ca754ffa15d42f3aa71d10bd2ae1174dd331fa1c8f91"}
Sep 16 12:50:57.449376 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:50:57.449177    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Sep 16 12:50:57.449376 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:50:57.449235    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"1273107e-4c61-4b7c-983d-1dbf0dfc99ad","Type":"ContainerDied","Data":"a54c7e46af1ebc257e0d9b958162957016e258e3208097d7668218fcff7fd53b"}
Sep 16 12:50:57.449376 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:50:57.449278    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"1273107e-4c61-4b7c-983d-1dbf0dfc99ad","Type":"ContainerDied","Data":"33a9acd679857671e1db56231a967f3202a4d85fa354ec5c498c7e1b6d48770c"}
Sep 16 12:50:57.449376 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:50:57.449297    2580 scope.go:117] "RemoveContainer" containerID="2d3e6c0437a661a0e7d886892ed3bc6a9578309126c70f34735a72bee8457976"
Sep 16 12:50:57.457091 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:50:57.457074    2580 scope.go:117] "RemoveContainer" containerID="ee0cb37034a38c293b34ca754ffa15d42f3aa71d10bd2ae1174dd331fa1c8f91"
Sep 16 12:50:57.463990 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:50:57.463973    2580 scope.go:117] "RemoveContainer" containerID="1f551efb68c802a59f189321abd2ea45682bc3486d4e1657f48da9d943b999be"
Sep 16 12:50:57.470355 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:50:57.470337    2580 scope.go:117] "RemoveContainer" containerID="a54c7e46af1ebc257e0d9b958162957016e258e3208097d7668218fcff7fd53b"
Sep 16 12:50:57.473635 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:50:57.473615    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Sep 16 12:50:57.478072 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:50:57.478050    2580 scope.go:117] "RemoveContainer" containerID="a05cd1d38e33163d352bc753f901b862bf717eda0caf953613c789d74c67b89e"
Sep 16 12:50:57.478455 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:50:57.478434    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Sep 16 12:50:57.485087 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:50:57.485071    2580 scope.go:117] "RemoveContainer" containerID="3475af9ba00b872b5ebd574e0ccca4eeb9719e9746fd3e75c899d04a19f56cfc"
Sep 16 12:50:57.491901 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:50:57.491886    2580 scope.go:117] "RemoveContainer" containerID="58e291fb7794e85c73e709c0bee46fe2b765faee850abb671b04805bc70b33de"
Sep 16 12:50:57.498689 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:50:57.498668    2580 scope.go:117] "RemoveContainer" containerID="2d3e6c0437a661a0e7d886892ed3bc6a9578309126c70f34735a72bee8457976"
Sep 16 12:50:57.498959 ip-10-0-133-218 kubenswrapper[2580]: E0916 12:50:57.498942    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2d3e6c0437a661a0e7d886892ed3bc6a9578309126c70f34735a72bee8457976\": container with ID starting with 2d3e6c0437a661a0e7d886892ed3bc6a9578309126c70f34735a72bee8457976 not found: ID does not exist" containerID="2d3e6c0437a661a0e7d886892ed3bc6a9578309126c70f34735a72bee8457976"
Sep 16 12:50:57.499034 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:50:57.498966    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2d3e6c0437a661a0e7d886892ed3bc6a9578309126c70f34735a72bee8457976"} err="failed to get container status \"2d3e6c0437a661a0e7d886892ed3bc6a9578309126c70f34735a72bee8457976\": rpc error: code = NotFound desc = could not find container \"2d3e6c0437a661a0e7d886892ed3bc6a9578309126c70f34735a72bee8457976\": container with ID starting with 2d3e6c0437a661a0e7d886892ed3bc6a9578309126c70f34735a72bee8457976 not found: ID does not exist"
Sep 16 12:50:57.499034 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:50:57.498983    2580 scope.go:117] "RemoveContainer" containerID="ee0cb37034a38c293b34ca754ffa15d42f3aa71d10bd2ae1174dd331fa1c8f91"
Sep 16 12:50:57.499228 ip-10-0-133-218 kubenswrapper[2580]: E0916 12:50:57.499210    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ee0cb37034a38c293b34ca754ffa15d42f3aa71d10bd2ae1174dd331fa1c8f91\": container with ID starting with ee0cb37034a38c293b34ca754ffa15d42f3aa71d10bd2ae1174dd331fa1c8f91 not found: ID does not exist" containerID="ee0cb37034a38c293b34ca754ffa15d42f3aa71d10bd2ae1174dd331fa1c8f91"
Sep 16 12:50:57.499274 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:50:57.499234    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ee0cb37034a38c293b34ca754ffa15d42f3aa71d10bd2ae1174dd331fa1c8f91"} err="failed to get container status \"ee0cb37034a38c293b34ca754ffa15d42f3aa71d10bd2ae1174dd331fa1c8f91\": rpc error: code = NotFound desc = could not find container \"ee0cb37034a38c293b34ca754ffa15d42f3aa71d10bd2ae1174dd331fa1c8f91\": container with ID starting with ee0cb37034a38c293b34ca754ffa15d42f3aa71d10bd2ae1174dd331fa1c8f91 not found: ID does not exist"
Sep 16 12:50:57.499274 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:50:57.499249    2580 scope.go:117] "RemoveContainer" containerID="1f551efb68c802a59f189321abd2ea45682bc3486d4e1657f48da9d943b999be"
Sep 16 12:50:57.499477 ip-10-0-133-218 kubenswrapper[2580]: E0916 12:50:57.499458    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1f551efb68c802a59f189321abd2ea45682bc3486d4e1657f48da9d943b999be\": container with ID starting with 1f551efb68c802a59f189321abd2ea45682bc3486d4e1657f48da9d943b999be not found: ID does not exist" containerID="1f551efb68c802a59f189321abd2ea45682bc3486d4e1657f48da9d943b999be"
Sep 16 12:50:57.499536 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:50:57.499481    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1f551efb68c802a59f189321abd2ea45682bc3486d4e1657f48da9d943b999be"} err="failed to get container status \"1f551efb68c802a59f189321abd2ea45682bc3486d4e1657f48da9d943b999be\": rpc error: code = NotFound desc = could not find container \"1f551efb68c802a59f189321abd2ea45682bc3486d4e1657f48da9d943b999be\": container with ID starting with 1f551efb68c802a59f189321abd2ea45682bc3486d4e1657f48da9d943b999be not found: ID does not exist"
Sep 16 12:50:57.499536 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:50:57.499511    2580 scope.go:117] "RemoveContainer" containerID="a54c7e46af1ebc257e0d9b958162957016e258e3208097d7668218fcff7fd53b"
Sep 16 12:50:57.499773 ip-10-0-133-218 kubenswrapper[2580]: E0916 12:50:57.499753    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a54c7e46af1ebc257e0d9b958162957016e258e3208097d7668218fcff7fd53b\": container with ID starting with a54c7e46af1ebc257e0d9b958162957016e258e3208097d7668218fcff7fd53b not found: ID does not exist" containerID="a54c7e46af1ebc257e0d9b958162957016e258e3208097d7668218fcff7fd53b"
Sep 16 12:50:57.499830 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:50:57.499778    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a54c7e46af1ebc257e0d9b958162957016e258e3208097d7668218fcff7fd53b"} err="failed to get container status \"a54c7e46af1ebc257e0d9b958162957016e258e3208097d7668218fcff7fd53b\": rpc error: code = NotFound desc = could not find container \"a54c7e46af1ebc257e0d9b958162957016e258e3208097d7668218fcff7fd53b\": container with ID starting with a54c7e46af1ebc257e0d9b958162957016e258e3208097d7668218fcff7fd53b not found: ID does not exist"
Sep 16 12:50:57.499830 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:50:57.499793    2580 scope.go:117] "RemoveContainer" containerID="a05cd1d38e33163d352bc753f901b862bf717eda0caf953613c789d74c67b89e"
Sep 16 12:50:57.500021 ip-10-0-133-218 kubenswrapper[2580]: E0916 12:50:57.500007    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a05cd1d38e33163d352bc753f901b862bf717eda0caf953613c789d74c67b89e\": container with ID starting with a05cd1d38e33163d352bc753f901b862bf717eda0caf953613c789d74c67b89e not found: ID does not exist" containerID="a05cd1d38e33163d352bc753f901b862bf717eda0caf953613c789d74c67b89e"
Sep 16 12:50:57.500060 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:50:57.500023    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a05cd1d38e33163d352bc753f901b862bf717eda0caf953613c789d74c67b89e"} err="failed to get container status \"a05cd1d38e33163d352bc753f901b862bf717eda0caf953613c789d74c67b89e\": rpc error: code = NotFound desc = could not find container \"a05cd1d38e33163d352bc753f901b862bf717eda0caf953613c789d74c67b89e\": container with ID starting with a05cd1d38e33163d352bc753f901b862bf717eda0caf953613c789d74c67b89e not found: ID does not exist"
Sep 16 12:50:57.500060 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:50:57.500035    2580 scope.go:117] "RemoveContainer" containerID="3475af9ba00b872b5ebd574e0ccca4eeb9719e9746fd3e75c899d04a19f56cfc"
Sep 16 12:50:57.500238 ip-10-0-133-218 kubenswrapper[2580]: E0916 12:50:57.500224    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3475af9ba00b872b5ebd574e0ccca4eeb9719e9746fd3e75c899d04a19f56cfc\": container with ID starting with 3475af9ba00b872b5ebd574e0ccca4eeb9719e9746fd3e75c899d04a19f56cfc not found: ID does not exist" containerID="3475af9ba00b872b5ebd574e0ccca4eeb9719e9746fd3e75c899d04a19f56cfc"
Sep 16 12:50:57.500281 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:50:57.500240    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3475af9ba00b872b5ebd574e0ccca4eeb9719e9746fd3e75c899d04a19f56cfc"} err="failed to get container status \"3475af9ba00b872b5ebd574e0ccca4eeb9719e9746fd3e75c899d04a19f56cfc\": rpc error: code = NotFound desc = could not find container \"3475af9ba00b872b5ebd574e0ccca4eeb9719e9746fd3e75c899d04a19f56cfc\": container with ID starting with 3475af9ba00b872b5ebd574e0ccca4eeb9719e9746fd3e75c899d04a19f56cfc not found: ID does not exist"
Sep 16 12:50:57.500281 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:50:57.500251    2580 scope.go:117] "RemoveContainer" containerID="58e291fb7794e85c73e709c0bee46fe2b765faee850abb671b04805bc70b33de"
Sep 16 12:50:57.500500 ip-10-0-133-218 kubenswrapper[2580]: E0916 12:50:57.500470    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"58e291fb7794e85c73e709c0bee46fe2b765faee850abb671b04805bc70b33de\": container with ID starting with 58e291fb7794e85c73e709c0bee46fe2b765faee850abb671b04805bc70b33de not found: ID does not exist" containerID="58e291fb7794e85c73e709c0bee46fe2b765faee850abb671b04805bc70b33de"
Sep 16 12:50:57.500548 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:50:57.500515    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"58e291fb7794e85c73e709c0bee46fe2b765faee850abb671b04805bc70b33de"} err="failed to get container status \"58e291fb7794e85c73e709c0bee46fe2b765faee850abb671b04805bc70b33de\": rpc error: code = NotFound desc = could not find container \"58e291fb7794e85c73e709c0bee46fe2b765faee850abb671b04805bc70b33de\": container with ID starting with 58e291fb7794e85c73e709c0bee46fe2b765faee850abb671b04805bc70b33de not found: ID does not exist"
Sep 16 12:50:57.500548 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:50:57.500531    2580 scope.go:117] "RemoveContainer" containerID="2d3e6c0437a661a0e7d886892ed3bc6a9578309126c70f34735a72bee8457976"
Sep 16 12:50:57.500744 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:50:57.500728    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2d3e6c0437a661a0e7d886892ed3bc6a9578309126c70f34735a72bee8457976"} err="failed to get container status \"2d3e6c0437a661a0e7d886892ed3bc6a9578309126c70f34735a72bee8457976\": rpc error: code = NotFound desc = could not find container \"2d3e6c0437a661a0e7d886892ed3bc6a9578309126c70f34735a72bee8457976\": container with ID starting with 2d3e6c0437a661a0e7d886892ed3bc6a9578309126c70f34735a72bee8457976 not found: ID does not exist"
Sep 16 12:50:57.500782 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:50:57.500746    2580 scope.go:117] "RemoveContainer" containerID="ee0cb37034a38c293b34ca754ffa15d42f3aa71d10bd2ae1174dd331fa1c8f91"
Sep 16 12:50:57.500962 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:50:57.500945    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ee0cb37034a38c293b34ca754ffa15d42f3aa71d10bd2ae1174dd331fa1c8f91"} err="failed to get container status \"ee0cb37034a38c293b34ca754ffa15d42f3aa71d10bd2ae1174dd331fa1c8f91\": rpc error: code = NotFound desc = could not find container \"ee0cb37034a38c293b34ca754ffa15d42f3aa71d10bd2ae1174dd331fa1c8f91\": container with ID starting with ee0cb37034a38c293b34ca754ffa15d42f3aa71d10bd2ae1174dd331fa1c8f91 not found: ID does not exist"
Sep 16 12:50:57.501003 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:50:57.500962    2580 scope.go:117] "RemoveContainer" containerID="1f551efb68c802a59f189321abd2ea45682bc3486d4e1657f48da9d943b999be"
Sep 16 12:50:57.501130 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:50:57.501116    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1f551efb68c802a59f189321abd2ea45682bc3486d4e1657f48da9d943b999be"} err="failed to get container status \"1f551efb68c802a59f189321abd2ea45682bc3486d4e1657f48da9d943b999be\": rpc error: code = NotFound desc = could not find container \"1f551efb68c802a59f189321abd2ea45682bc3486d4e1657f48da9d943b999be\": container with ID starting with 1f551efb68c802a59f189321abd2ea45682bc3486d4e1657f48da9d943b999be not found: ID does not exist"
Sep 16 12:50:57.501178 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:50:57.501129    2580 scope.go:117] "RemoveContainer" containerID="a54c7e46af1ebc257e0d9b958162957016e258e3208097d7668218fcff7fd53b"
Sep 16 12:50:57.501346 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:50:57.501330    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a54c7e46af1ebc257e0d9b958162957016e258e3208097d7668218fcff7fd53b"} err="failed to get container status \"a54c7e46af1ebc257e0d9b958162957016e258e3208097d7668218fcff7fd53b\": rpc error: code = NotFound desc = could not find container \"a54c7e46af1ebc257e0d9b958162957016e258e3208097d7668218fcff7fd53b\": container with ID starting with a54c7e46af1ebc257e0d9b958162957016e258e3208097d7668218fcff7fd53b not found: ID does not exist"
Sep 16 12:50:57.501346 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:50:57.501344    2580 scope.go:117] "RemoveContainer" containerID="a05cd1d38e33163d352bc753f901b862bf717eda0caf953613c789d74c67b89e"
Sep 16 12:50:57.501569 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:50:57.501553    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a05cd1d38e33163d352bc753f901b862bf717eda0caf953613c789d74c67b89e"} err="failed to get container status \"a05cd1d38e33163d352bc753f901b862bf717eda0caf953613c789d74c67b89e\": rpc error: code = NotFound desc = could not find container \"a05cd1d38e33163d352bc753f901b862bf717eda0caf953613c789d74c67b89e\": container with ID starting with a05cd1d38e33163d352bc753f901b862bf717eda0caf953613c789d74c67b89e not found: ID does not exist"
Sep 16 12:50:57.501615 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:50:57.501569    2580 scope.go:117] "RemoveContainer" containerID="3475af9ba00b872b5ebd574e0ccca4eeb9719e9746fd3e75c899d04a19f56cfc"
Sep 16 12:50:57.501776 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:50:57.501760    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3475af9ba00b872b5ebd574e0ccca4eeb9719e9746fd3e75c899d04a19f56cfc"} err="failed to get container status \"3475af9ba00b872b5ebd574e0ccca4eeb9719e9746fd3e75c899d04a19f56cfc\": rpc error: code = NotFound desc = could not find container \"3475af9ba00b872b5ebd574e0ccca4eeb9719e9746fd3e75c899d04a19f56cfc\": container with ID starting with 3475af9ba00b872b5ebd574e0ccca4eeb9719e9746fd3e75c899d04a19f56cfc not found: ID does not exist"
Sep 16 12:50:57.501776 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:50:57.501775    2580 scope.go:117] "RemoveContainer" containerID="58e291fb7794e85c73e709c0bee46fe2b765faee850abb671b04805bc70b33de"
Sep 16 12:50:57.501954 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:50:57.501938    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"58e291fb7794e85c73e709c0bee46fe2b765faee850abb671b04805bc70b33de"} err="failed to get container status \"58e291fb7794e85c73e709c0bee46fe2b765faee850abb671b04805bc70b33de\": rpc error: code = NotFound desc = could not find container \"58e291fb7794e85c73e709c0bee46fe2b765faee850abb671b04805bc70b33de\": container with ID starting with 58e291fb7794e85c73e709c0bee46fe2b765faee850abb671b04805bc70b33de not found: ID does not exist"
Sep 16 12:50:57.598922 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:50:57.598891    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1273107e-4c61-4b7c-983d-1dbf0dfc99ad" path="/var/lib/kubelet/pods/1273107e-4c61-4b7c-983d-1dbf0dfc99ad/volumes"
Sep 16 12:51:00.106320 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:51:00.106286    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 16 12:51:00.106785 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:51:00.106577    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="84b4b343-d43c-4639-a768-3f52806bd8c7" containerName="prometheus" containerID="cri-o://a90db64ac3c00006d0d70fb1a4756654bf3c583bc4468edeff701d86e7c05c31" gracePeriod=600
Sep 16 12:51:00.106785 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:51:00.106620    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="84b4b343-d43c-4639-a768-3f52806bd8c7" containerName="thanos-sidecar" containerID="cri-o://1418326e938a5682f35998260a697824f95c6fee3512f8fd106f74caadec39c8" gracePeriod=600
Sep 16 12:51:00.106785 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:51:00.106656    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="84b4b343-d43c-4639-a768-3f52806bd8c7" containerName="kube-rbac-proxy-thanos" containerID="cri-o://84be625f3f386f8baeda5297f5efad680c1f6b97a7f91fa03f3523b58c9415ef" gracePeriod=600
Sep 16 12:51:00.106785 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:51:00.106651    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="84b4b343-d43c-4639-a768-3f52806bd8c7" containerName="config-reloader" containerID="cri-o://935230d89b4c3cbbbdea303993cac3eb841c6b2fedfaf8ce16f5f9142794b6a8" gracePeriod=600
Sep 16 12:51:00.106785 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:51:00.106609    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="84b4b343-d43c-4639-a768-3f52806bd8c7" containerName="kube-rbac-proxy" containerID="cri-o://d25934a737be5b16f1c782d611e02f6bbbd06974fd86a136b102a4d77e134e92" gracePeriod=600
Sep 16 12:51:00.106785 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:51:00.106689    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="84b4b343-d43c-4639-a768-3f52806bd8c7" containerName="kube-rbac-proxy-web" containerID="cri-o://d1f754c7c06351c83ed6e00e5bc4f31c69ca8c2e4fb704760304995862185e2a" gracePeriod=600
Sep 16 12:51:00.345771 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:51:00.345748    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 12:51:00.465665 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:51:00.465572    2580 generic.go:356] "Generic (PLEG): container finished" podID="84b4b343-d43c-4639-a768-3f52806bd8c7" containerID="84be625f3f386f8baeda5297f5efad680c1f6b97a7f91fa03f3523b58c9415ef" exitCode=0
Sep 16 12:51:00.465665 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:51:00.465603    2580 generic.go:356] "Generic (PLEG): container finished" podID="84b4b343-d43c-4639-a768-3f52806bd8c7" containerID="d25934a737be5b16f1c782d611e02f6bbbd06974fd86a136b102a4d77e134e92" exitCode=0
Sep 16 12:51:00.465665 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:51:00.465611    2580 generic.go:356] "Generic (PLEG): container finished" podID="84b4b343-d43c-4639-a768-3f52806bd8c7" containerID="d1f754c7c06351c83ed6e00e5bc4f31c69ca8c2e4fb704760304995862185e2a" exitCode=0
Sep 16 12:51:00.465665 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:51:00.465621    2580 generic.go:356] "Generic (PLEG): container finished" podID="84b4b343-d43c-4639-a768-3f52806bd8c7" containerID="1418326e938a5682f35998260a697824f95c6fee3512f8fd106f74caadec39c8" exitCode=0
Sep 16 12:51:00.465665 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:51:00.465630    2580 generic.go:356] "Generic (PLEG): container finished" podID="84b4b343-d43c-4639-a768-3f52806bd8c7" containerID="935230d89b4c3cbbbdea303993cac3eb841c6b2fedfaf8ce16f5f9142794b6a8" exitCode=0
Sep 16 12:51:00.465665 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:51:00.465637    2580 generic.go:356] "Generic (PLEG): container finished" podID="84b4b343-d43c-4639-a768-3f52806bd8c7" containerID="a90db64ac3c00006d0d70fb1a4756654bf3c583bc4468edeff701d86e7c05c31" exitCode=0
Sep 16 12:51:00.465665 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:51:00.465668    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 12:51:00.466096 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:51:00.465658    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"84b4b343-d43c-4639-a768-3f52806bd8c7","Type":"ContainerDied","Data":"84be625f3f386f8baeda5297f5efad680c1f6b97a7f91fa03f3523b58c9415ef"}
Sep 16 12:51:00.466096 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:51:00.465814    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"84b4b343-d43c-4639-a768-3f52806bd8c7","Type":"ContainerDied","Data":"d25934a737be5b16f1c782d611e02f6bbbd06974fd86a136b102a4d77e134e92"}
Sep 16 12:51:00.466096 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:51:00.465861    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"84b4b343-d43c-4639-a768-3f52806bd8c7","Type":"ContainerDied","Data":"d1f754c7c06351c83ed6e00e5bc4f31c69ca8c2e4fb704760304995862185e2a"}
Sep 16 12:51:00.466096 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:51:00.465882    2580 scope.go:117] "RemoveContainer" containerID="84be625f3f386f8baeda5297f5efad680c1f6b97a7f91fa03f3523b58c9415ef"
Sep 16 12:51:00.466096 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:51:00.465905    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"84b4b343-d43c-4639-a768-3f52806bd8c7","Type":"ContainerDied","Data":"1418326e938a5682f35998260a697824f95c6fee3512f8fd106f74caadec39c8"}
Sep 16 12:51:00.466096 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:51:00.465951    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"84b4b343-d43c-4639-a768-3f52806bd8c7","Type":"ContainerDied","Data":"935230d89b4c3cbbbdea303993cac3eb841c6b2fedfaf8ce16f5f9142794b6a8"}
Sep 16 12:51:00.466096 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:51:00.465992    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"84b4b343-d43c-4639-a768-3f52806bd8c7","Type":"ContainerDied","Data":"a90db64ac3c00006d0d70fb1a4756654bf3c583bc4468edeff701d86e7c05c31"}
Sep 16 12:51:00.466096 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:51:00.466033    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"84b4b343-d43c-4639-a768-3f52806bd8c7","Type":"ContainerDied","Data":"a2eb6c46fdd0ae7ed87a9f95dcfc290e7549cbb0ebc32fe675393a2e2773b8ab"}
Sep 16 12:51:00.473592 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:51:00.473576    2580 scope.go:117] "RemoveContainer" containerID="d25934a737be5b16f1c782d611e02f6bbbd06974fd86a136b102a4d77e134e92"
Sep 16 12:51:00.480201 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:51:00.480186    2580 scope.go:117] "RemoveContainer" containerID="d1f754c7c06351c83ed6e00e5bc4f31c69ca8c2e4fb704760304995862185e2a"
Sep 16 12:51:00.486557 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:51:00.486539    2580 scope.go:117] "RemoveContainer" containerID="1418326e938a5682f35998260a697824f95c6fee3512f8fd106f74caadec39c8"
Sep 16 12:51:00.491654 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:51:00.491635    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/84b4b343-d43c-4639-a768-3f52806bd8c7-tls-assets\") pod \"84b4b343-d43c-4639-a768-3f52806bd8c7\" (UID: \"84b4b343-d43c-4639-a768-3f52806bd8c7\") "
Sep 16 12:51:00.491748 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:51:00.491677    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/84b4b343-d43c-4639-a768-3f52806bd8c7-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"84b4b343-d43c-4639-a768-3f52806bd8c7\" (UID: \"84b4b343-d43c-4639-a768-3f52806bd8c7\") "
Sep 16 12:51:00.491748 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:51:00.491715    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/84b4b343-d43c-4639-a768-3f52806bd8c7-web-config\") pod \"84b4b343-d43c-4639-a768-3f52806bd8c7\" (UID: \"84b4b343-d43c-4639-a768-3f52806bd8c7\") "
Sep 16 12:51:00.491748 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:51:00.491742    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/84b4b343-d43c-4639-a768-3f52806bd8c7-configmap-kubelet-serving-ca-bundle\") pod \"84b4b343-d43c-4639-a768-3f52806bd8c7\" (UID: \"84b4b343-d43c-4639-a768-3f52806bd8c7\") "
Sep 16 12:51:00.491898 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:51:00.491768    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/84b4b343-d43c-4639-a768-3f52806bd8c7-prometheus-k8s-db\") pod \"84b4b343-d43c-4639-a768-3f52806bd8c7\" (UID: \"84b4b343-d43c-4639-a768-3f52806bd8c7\") "
Sep 16 12:51:00.491898 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:51:00.491809    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/84b4b343-d43c-4639-a768-3f52806bd8c7-thanos-prometheus-http-client-file\") pod \"84b4b343-d43c-4639-a768-3f52806bd8c7\" (UID: \"84b4b343-d43c-4639-a768-3f52806bd8c7\") "
Sep 16 12:51:00.491898 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:51:00.491843    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/84b4b343-d43c-4639-a768-3f52806bd8c7-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"84b4b343-d43c-4639-a768-3f52806bd8c7\" (UID: \"84b4b343-d43c-4639-a768-3f52806bd8c7\") "
Sep 16 12:51:00.491898 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:51:00.491874    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/84b4b343-d43c-4639-a768-3f52806bd8c7-prometheus-k8s-rulefiles-1\") pod \"84b4b343-d43c-4639-a768-3f52806bd8c7\" (UID: \"84b4b343-d43c-4639-a768-3f52806bd8c7\") "
Sep 16 12:51:00.492087 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:51:00.491901    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/84b4b343-d43c-4639-a768-3f52806bd8c7-prometheus-trusted-ca-bundle\") pod \"84b4b343-d43c-4639-a768-3f52806bd8c7\" (UID: \"84b4b343-d43c-4639-a768-3f52806bd8c7\") "
Sep 16 12:51:00.492087 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:51:00.491938    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hxkpn\" (UniqueName: \"kubernetes.io/projected/84b4b343-d43c-4639-a768-3f52806bd8c7-kube-api-access-hxkpn\") pod \"84b4b343-d43c-4639-a768-3f52806bd8c7\" (UID: \"84b4b343-d43c-4639-a768-3f52806bd8c7\") "
Sep 16 12:51:00.492087 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:51:00.491969    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/84b4b343-d43c-4639-a768-3f52806bd8c7-prometheus-k8s-rulefiles-2\") pod \"84b4b343-d43c-4639-a768-3f52806bd8c7\" (UID: \"84b4b343-d43c-4639-a768-3f52806bd8c7\") "
Sep 16 12:51:00.492087 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:51:00.492020    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/84b4b343-d43c-4639-a768-3f52806bd8c7-secret-grpc-tls\") pod \"84b4b343-d43c-4639-a768-3f52806bd8c7\" (UID: \"84b4b343-d43c-4639-a768-3f52806bd8c7\") "
Sep 16 12:51:00.492087 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:51:00.492053    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/84b4b343-d43c-4639-a768-3f52806bd8c7-configmap-serving-certs-ca-bundle\") pod \"84b4b343-d43c-4639-a768-3f52806bd8c7\" (UID: \"84b4b343-d43c-4639-a768-3f52806bd8c7\") "
Sep 16 12:51:00.492087 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:51:00.492080    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/84b4b343-d43c-4639-a768-3f52806bd8c7-secret-kube-rbac-proxy\") pod \"84b4b343-d43c-4639-a768-3f52806bd8c7\" (UID: \"84b4b343-d43c-4639-a768-3f52806bd8c7\") "
Sep 16 12:51:00.492365 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:51:00.492111    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/84b4b343-d43c-4639-a768-3f52806bd8c7-config-out\") pod \"84b4b343-d43c-4639-a768-3f52806bd8c7\" (UID: \"84b4b343-d43c-4639-a768-3f52806bd8c7\") "
Sep 16 12:51:00.492365 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:51:00.492144    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/84b4b343-d43c-4639-a768-3f52806bd8c7-configmap-metrics-client-ca\") pod \"84b4b343-d43c-4639-a768-3f52806bd8c7\" (UID: \"84b4b343-d43c-4639-a768-3f52806bd8c7\") "
Sep 16 12:51:00.492365 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:51:00.492180    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/84b4b343-d43c-4639-a768-3f52806bd8c7-secret-prometheus-k8s-tls\") pod \"84b4b343-d43c-4639-a768-3f52806bd8c7\" (UID: \"84b4b343-d43c-4639-a768-3f52806bd8c7\") "
Sep 16 12:51:00.492365 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:51:00.492209    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/84b4b343-d43c-4639-a768-3f52806bd8c7-secret-metrics-client-certs\") pod \"84b4b343-d43c-4639-a768-3f52806bd8c7\" (UID: \"84b4b343-d43c-4639-a768-3f52806bd8c7\") "
Sep 16 12:51:00.492365 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:51:00.492263    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/84b4b343-d43c-4639-a768-3f52806bd8c7-prometheus-k8s-rulefiles-0\") pod \"84b4b343-d43c-4639-a768-3f52806bd8c7\" (UID: \"84b4b343-d43c-4639-a768-3f52806bd8c7\") "
Sep 16 12:51:00.492365 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:51:00.492287    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/84b4b343-d43c-4639-a768-3f52806bd8c7-config\") pod \"84b4b343-d43c-4639-a768-3f52806bd8c7\" (UID: \"84b4b343-d43c-4639-a768-3f52806bd8c7\") "
Sep 16 12:51:00.493195 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:51:00.492931    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/84b4b343-d43c-4639-a768-3f52806bd8c7-configmap-serving-certs-ca-bundle" (OuterVolumeSpecName: "configmap-serving-certs-ca-bundle") pod "84b4b343-d43c-4639-a768-3f52806bd8c7" (UID: "84b4b343-d43c-4639-a768-3f52806bd8c7"). InnerVolumeSpecName "configmap-serving-certs-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 12:51:00.493195 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:51:00.493041    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/84b4b343-d43c-4639-a768-3f52806bd8c7-prometheus-k8s-rulefiles-1" (OuterVolumeSpecName: "prometheus-k8s-rulefiles-1") pod "84b4b343-d43c-4639-a768-3f52806bd8c7" (UID: "84b4b343-d43c-4639-a768-3f52806bd8c7"). InnerVolumeSpecName "prometheus-k8s-rulefiles-1". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 12:51:00.494556 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:51:00.493380    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/84b4b343-d43c-4639-a768-3f52806bd8c7-configmap-kubelet-serving-ca-bundle" (OuterVolumeSpecName: "configmap-kubelet-serving-ca-bundle") pod "84b4b343-d43c-4639-a768-3f52806bd8c7" (UID: "84b4b343-d43c-4639-a768-3f52806bd8c7"). InnerVolumeSpecName "configmap-kubelet-serving-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 12:51:00.494556 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:51:00.493992    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/84b4b343-d43c-4639-a768-3f52806bd8c7-prometheus-k8s-rulefiles-2" (OuterVolumeSpecName: "prometheus-k8s-rulefiles-2") pod "84b4b343-d43c-4639-a768-3f52806bd8c7" (UID: "84b4b343-d43c-4639-a768-3f52806bd8c7"). InnerVolumeSpecName "prometheus-k8s-rulefiles-2". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 12:51:00.494556 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:51:00.494085    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/84b4b343-d43c-4639-a768-3f52806bd8c7-prometheus-k8s-db" (OuterVolumeSpecName: "prometheus-k8s-db") pod "84b4b343-d43c-4639-a768-3f52806bd8c7" (UID: "84b4b343-d43c-4639-a768-3f52806bd8c7"). InnerVolumeSpecName "prometheus-k8s-db". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 12:51:00.494556 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:51:00.494157    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/84b4b343-d43c-4639-a768-3f52806bd8c7-tls-assets" (OuterVolumeSpecName: "tls-assets") pod "84b4b343-d43c-4639-a768-3f52806bd8c7" (UID: "84b4b343-d43c-4639-a768-3f52806bd8c7"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 12:51:00.494556 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:51:00.494243    2580 scope.go:117] "RemoveContainer" containerID="935230d89b4c3cbbbdea303993cac3eb841c6b2fedfaf8ce16f5f9142794b6a8"
Sep 16 12:51:00.495269 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:51:00.495026    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/84b4b343-d43c-4639-a768-3f52806bd8c7-prometheus-trusted-ca-bundle" (OuterVolumeSpecName: "prometheus-trusted-ca-bundle") pod "84b4b343-d43c-4639-a768-3f52806bd8c7" (UID: "84b4b343-d43c-4639-a768-3f52806bd8c7"). InnerVolumeSpecName "prometheus-trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 12:51:00.495449 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:51:00.495413    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/84b4b343-d43c-4639-a768-3f52806bd8c7-configmap-metrics-client-ca" (OuterVolumeSpecName: "configmap-metrics-client-ca") pod "84b4b343-d43c-4639-a768-3f52806bd8c7" (UID: "84b4b343-d43c-4639-a768-3f52806bd8c7"). InnerVolumeSpecName "configmap-metrics-client-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 12:51:00.495983 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:51:00.495850    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/84b4b343-d43c-4639-a768-3f52806bd8c7-secret-grpc-tls" (OuterVolumeSpecName: "secret-grpc-tls") pod "84b4b343-d43c-4639-a768-3f52806bd8c7" (UID: "84b4b343-d43c-4639-a768-3f52806bd8c7"). InnerVolumeSpecName "secret-grpc-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 12:51:00.496212 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:51:00.496155    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/84b4b343-d43c-4639-a768-3f52806bd8c7-prometheus-k8s-rulefiles-0" (OuterVolumeSpecName: "prometheus-k8s-rulefiles-0") pod "84b4b343-d43c-4639-a768-3f52806bd8c7" (UID: "84b4b343-d43c-4639-a768-3f52806bd8c7"). InnerVolumeSpecName "prometheus-k8s-rulefiles-0". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 12:51:00.496367 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:51:00.496345    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/84b4b343-d43c-4639-a768-3f52806bd8c7-config" (OuterVolumeSpecName: "config") pod "84b4b343-d43c-4639-a768-3f52806bd8c7" (UID: "84b4b343-d43c-4639-a768-3f52806bd8c7"). InnerVolumeSpecName "config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 12:51:00.496367 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:51:00.496358    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/84b4b343-d43c-4639-a768-3f52806bd8c7-secret-prometheus-k8s-thanos-sidecar-tls" (OuterVolumeSpecName: "secret-prometheus-k8s-thanos-sidecar-tls") pod "84b4b343-d43c-4639-a768-3f52806bd8c7" (UID: "84b4b343-d43c-4639-a768-3f52806bd8c7"). InnerVolumeSpecName "secret-prometheus-k8s-thanos-sidecar-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 12:51:00.496638 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:51:00.496403    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/84b4b343-d43c-4639-a768-3f52806bd8c7-secret-prometheus-k8s-tls" (OuterVolumeSpecName: "secret-prometheus-k8s-tls") pod "84b4b343-d43c-4639-a768-3f52806bd8c7" (UID: "84b4b343-d43c-4639-a768-3f52806bd8c7"). InnerVolumeSpecName "secret-prometheus-k8s-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 12:51:00.496737 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:51:00.496687    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/84b4b343-d43c-4639-a768-3f52806bd8c7-kube-api-access-hxkpn" (OuterVolumeSpecName: "kube-api-access-hxkpn") pod "84b4b343-d43c-4639-a768-3f52806bd8c7" (UID: "84b4b343-d43c-4639-a768-3f52806bd8c7"). InnerVolumeSpecName "kube-api-access-hxkpn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 12:51:00.496962 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:51:00.496939    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/84b4b343-d43c-4639-a768-3f52806bd8c7-secret-kube-rbac-proxy" (OuterVolumeSpecName: "secret-kube-rbac-proxy") pod "84b4b343-d43c-4639-a768-3f52806bd8c7" (UID: "84b4b343-d43c-4639-a768-3f52806bd8c7"). InnerVolumeSpecName "secret-kube-rbac-proxy". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 12:51:00.497037 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:51:00.497009    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/84b4b343-d43c-4639-a768-3f52806bd8c7-secret-prometheus-k8s-kube-rbac-proxy-web" (OuterVolumeSpecName: "secret-prometheus-k8s-kube-rbac-proxy-web") pod "84b4b343-d43c-4639-a768-3f52806bd8c7" (UID: "84b4b343-d43c-4639-a768-3f52806bd8c7"). InnerVolumeSpecName "secret-prometheus-k8s-kube-rbac-proxy-web". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 12:51:00.497646 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:51:00.497622    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/84b4b343-d43c-4639-a768-3f52806bd8c7-secret-metrics-client-certs" (OuterVolumeSpecName: "secret-metrics-client-certs") pod "84b4b343-d43c-4639-a768-3f52806bd8c7" (UID: "84b4b343-d43c-4639-a768-3f52806bd8c7"). InnerVolumeSpecName "secret-metrics-client-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 12:51:00.497779 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:51:00.497742    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/84b4b343-d43c-4639-a768-3f52806bd8c7-thanos-prometheus-http-client-file" (OuterVolumeSpecName: "thanos-prometheus-http-client-file") pod "84b4b343-d43c-4639-a768-3f52806bd8c7" (UID: "84b4b343-d43c-4639-a768-3f52806bd8c7"). InnerVolumeSpecName "thanos-prometheus-http-client-file". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 12:51:00.498147 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:51:00.498122    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/84b4b343-d43c-4639-a768-3f52806bd8c7-config-out" (OuterVolumeSpecName: "config-out") pod "84b4b343-d43c-4639-a768-3f52806bd8c7" (UID: "84b4b343-d43c-4639-a768-3f52806bd8c7"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 12:51:00.502166 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:51:00.502149    2580 scope.go:117] "RemoveContainer" containerID="a90db64ac3c00006d0d70fb1a4756654bf3c583bc4468edeff701d86e7c05c31"
Sep 16 12:51:00.505537 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:51:00.505516    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/84b4b343-d43c-4639-a768-3f52806bd8c7-web-config" (OuterVolumeSpecName: "web-config") pod "84b4b343-d43c-4639-a768-3f52806bd8c7" (UID: "84b4b343-d43c-4639-a768-3f52806bd8c7"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 12:51:00.510513 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:51:00.510474    2580 scope.go:117] "RemoveContainer" containerID="e4c8b98a3e29dc5927fd6e25c39c6f6e8e5a52bc45b8d99817ec406280a1c6a0"
Sep 16 12:51:00.516841 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:51:00.516821    2580 scope.go:117] "RemoveContainer" containerID="84be625f3f386f8baeda5297f5efad680c1f6b97a7f91fa03f3523b58c9415ef"
Sep 16 12:51:00.517097 ip-10-0-133-218 kubenswrapper[2580]: E0916 12:51:00.517075    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"84be625f3f386f8baeda5297f5efad680c1f6b97a7f91fa03f3523b58c9415ef\": container with ID starting with 84be625f3f386f8baeda5297f5efad680c1f6b97a7f91fa03f3523b58c9415ef not found: ID does not exist" containerID="84be625f3f386f8baeda5297f5efad680c1f6b97a7f91fa03f3523b58c9415ef"
Sep 16 12:51:00.517159 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:51:00.517104    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"84be625f3f386f8baeda5297f5efad680c1f6b97a7f91fa03f3523b58c9415ef"} err="failed to get container status \"84be625f3f386f8baeda5297f5efad680c1f6b97a7f91fa03f3523b58c9415ef\": rpc error: code = NotFound desc = could not find container \"84be625f3f386f8baeda5297f5efad680c1f6b97a7f91fa03f3523b58c9415ef\": container with ID starting with 84be625f3f386f8baeda5297f5efad680c1f6b97a7f91fa03f3523b58c9415ef not found: ID does not exist"
Sep 16 12:51:00.517159 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:51:00.517120    2580 scope.go:117] "RemoveContainer" containerID="d25934a737be5b16f1c782d611e02f6bbbd06974fd86a136b102a4d77e134e92"
Sep 16 12:51:00.517371 ip-10-0-133-218 kubenswrapper[2580]: E0916 12:51:00.517354    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d25934a737be5b16f1c782d611e02f6bbbd06974fd86a136b102a4d77e134e92\": container with ID starting with d25934a737be5b16f1c782d611e02f6bbbd06974fd86a136b102a4d77e134e92 not found: ID does not exist" containerID="d25934a737be5b16f1c782d611e02f6bbbd06974fd86a136b102a4d77e134e92"
Sep 16 12:51:00.517417 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:51:00.517378    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d25934a737be5b16f1c782d611e02f6bbbd06974fd86a136b102a4d77e134e92"} err="failed to get container status \"d25934a737be5b16f1c782d611e02f6bbbd06974fd86a136b102a4d77e134e92\": rpc error: code = NotFound desc = could not find container \"d25934a737be5b16f1c782d611e02f6bbbd06974fd86a136b102a4d77e134e92\": container with ID starting with d25934a737be5b16f1c782d611e02f6bbbd06974fd86a136b102a4d77e134e92 not found: ID does not exist"
Sep 16 12:51:00.517417 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:51:00.517394    2580 scope.go:117] "RemoveContainer" containerID="d1f754c7c06351c83ed6e00e5bc4f31c69ca8c2e4fb704760304995862185e2a"
Sep 16 12:51:00.517698 ip-10-0-133-218 kubenswrapper[2580]: E0916 12:51:00.517667    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d1f754c7c06351c83ed6e00e5bc4f31c69ca8c2e4fb704760304995862185e2a\": container with ID starting with d1f754c7c06351c83ed6e00e5bc4f31c69ca8c2e4fb704760304995862185e2a not found: ID does not exist" containerID="d1f754c7c06351c83ed6e00e5bc4f31c69ca8c2e4fb704760304995862185e2a"
Sep 16 12:51:00.517768 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:51:00.517696    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d1f754c7c06351c83ed6e00e5bc4f31c69ca8c2e4fb704760304995862185e2a"} err="failed to get container status \"d1f754c7c06351c83ed6e00e5bc4f31c69ca8c2e4fb704760304995862185e2a\": rpc error: code = NotFound desc = could not find container \"d1f754c7c06351c83ed6e00e5bc4f31c69ca8c2e4fb704760304995862185e2a\": container with ID starting with d1f754c7c06351c83ed6e00e5bc4f31c69ca8c2e4fb704760304995862185e2a not found: ID does not exist"
Sep 16 12:51:00.517768 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:51:00.517715    2580 scope.go:117] "RemoveContainer" containerID="1418326e938a5682f35998260a697824f95c6fee3512f8fd106f74caadec39c8"
Sep 16 12:51:00.517962 ip-10-0-133-218 kubenswrapper[2580]: E0916 12:51:00.517949    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1418326e938a5682f35998260a697824f95c6fee3512f8fd106f74caadec39c8\": container with ID starting with 1418326e938a5682f35998260a697824f95c6fee3512f8fd106f74caadec39c8 not found: ID does not exist" containerID="1418326e938a5682f35998260a697824f95c6fee3512f8fd106f74caadec39c8"
Sep 16 12:51:00.518004 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:51:00.517965    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1418326e938a5682f35998260a697824f95c6fee3512f8fd106f74caadec39c8"} err="failed to get container status \"1418326e938a5682f35998260a697824f95c6fee3512f8fd106f74caadec39c8\": rpc error: code = NotFound desc = could not find container \"1418326e938a5682f35998260a697824f95c6fee3512f8fd106f74caadec39c8\": container with ID starting with 1418326e938a5682f35998260a697824f95c6fee3512f8fd106f74caadec39c8 not found: ID does not exist"
Sep 16 12:51:00.518004 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:51:00.517976    2580 scope.go:117] "RemoveContainer" containerID="935230d89b4c3cbbbdea303993cac3eb841c6b2fedfaf8ce16f5f9142794b6a8"
Sep 16 12:51:00.518254 ip-10-0-133-218 kubenswrapper[2580]: E0916 12:51:00.518236    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"935230d89b4c3cbbbdea303993cac3eb841c6b2fedfaf8ce16f5f9142794b6a8\": container with ID starting with 935230d89b4c3cbbbdea303993cac3eb841c6b2fedfaf8ce16f5f9142794b6a8 not found: ID does not exist" containerID="935230d89b4c3cbbbdea303993cac3eb841c6b2fedfaf8ce16f5f9142794b6a8"
Sep 16 12:51:00.518310 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:51:00.518259    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"935230d89b4c3cbbbdea303993cac3eb841c6b2fedfaf8ce16f5f9142794b6a8"} err="failed to get container status \"935230d89b4c3cbbbdea303993cac3eb841c6b2fedfaf8ce16f5f9142794b6a8\": rpc error: code = NotFound desc = could not find container \"935230d89b4c3cbbbdea303993cac3eb841c6b2fedfaf8ce16f5f9142794b6a8\": container with ID starting with 935230d89b4c3cbbbdea303993cac3eb841c6b2fedfaf8ce16f5f9142794b6a8 not found: ID does not exist"
Sep 16 12:51:00.518310 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:51:00.518273    2580 scope.go:117] "RemoveContainer" containerID="a90db64ac3c00006d0d70fb1a4756654bf3c583bc4468edeff701d86e7c05c31"
Sep 16 12:51:00.518530 ip-10-0-133-218 kubenswrapper[2580]: E0916 12:51:00.518515    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a90db64ac3c00006d0d70fb1a4756654bf3c583bc4468edeff701d86e7c05c31\": container with ID starting with a90db64ac3c00006d0d70fb1a4756654bf3c583bc4468edeff701d86e7c05c31 not found: ID does not exist" containerID="a90db64ac3c00006d0d70fb1a4756654bf3c583bc4468edeff701d86e7c05c31"
Sep 16 12:51:00.518583 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:51:00.518532    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a90db64ac3c00006d0d70fb1a4756654bf3c583bc4468edeff701d86e7c05c31"} err="failed to get container status \"a90db64ac3c00006d0d70fb1a4756654bf3c583bc4468edeff701d86e7c05c31\": rpc error: code = NotFound desc = could not find container \"a90db64ac3c00006d0d70fb1a4756654bf3c583bc4468edeff701d86e7c05c31\": container with ID starting with a90db64ac3c00006d0d70fb1a4756654bf3c583bc4468edeff701d86e7c05c31 not found: ID does not exist"
Sep 16 12:51:00.518583 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:51:00.518543    2580 scope.go:117] "RemoveContainer" containerID="e4c8b98a3e29dc5927fd6e25c39c6f6e8e5a52bc45b8d99817ec406280a1c6a0"
Sep 16 12:51:00.518761 ip-10-0-133-218 kubenswrapper[2580]: E0916 12:51:00.518740    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e4c8b98a3e29dc5927fd6e25c39c6f6e8e5a52bc45b8d99817ec406280a1c6a0\": container with ID starting with e4c8b98a3e29dc5927fd6e25c39c6f6e8e5a52bc45b8d99817ec406280a1c6a0 not found: ID does not exist" containerID="e4c8b98a3e29dc5927fd6e25c39c6f6e8e5a52bc45b8d99817ec406280a1c6a0"
Sep 16 12:51:00.518803 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:51:00.518762    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e4c8b98a3e29dc5927fd6e25c39c6f6e8e5a52bc45b8d99817ec406280a1c6a0"} err="failed to get container status \"e4c8b98a3e29dc5927fd6e25c39c6f6e8e5a52bc45b8d99817ec406280a1c6a0\": rpc error: code = NotFound desc = could not find container \"e4c8b98a3e29dc5927fd6e25c39c6f6e8e5a52bc45b8d99817ec406280a1c6a0\": container with ID starting with e4c8b98a3e29dc5927fd6e25c39c6f6e8e5a52bc45b8d99817ec406280a1c6a0 not found: ID does not exist"
Sep 16 12:51:00.518803 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:51:00.518772    2580 scope.go:117] "RemoveContainer" containerID="84be625f3f386f8baeda5297f5efad680c1f6b97a7f91fa03f3523b58c9415ef"
Sep 16 12:51:00.518999 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:51:00.518986    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"84be625f3f386f8baeda5297f5efad680c1f6b97a7f91fa03f3523b58c9415ef"} err="failed to get container status \"84be625f3f386f8baeda5297f5efad680c1f6b97a7f91fa03f3523b58c9415ef\": rpc error: code = NotFound desc = could not find container \"84be625f3f386f8baeda5297f5efad680c1f6b97a7f91fa03f3523b58c9415ef\": container with ID starting with 84be625f3f386f8baeda5297f5efad680c1f6b97a7f91fa03f3523b58c9415ef not found: ID does not exist"
Sep 16 12:51:00.519046 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:51:00.519000    2580 scope.go:117] "RemoveContainer" containerID="d25934a737be5b16f1c782d611e02f6bbbd06974fd86a136b102a4d77e134e92"
Sep 16 12:51:00.519187 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:51:00.519174    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d25934a737be5b16f1c782d611e02f6bbbd06974fd86a136b102a4d77e134e92"} err="failed to get container status \"d25934a737be5b16f1c782d611e02f6bbbd06974fd86a136b102a4d77e134e92\": rpc error: code = NotFound desc = could not find container \"d25934a737be5b16f1c782d611e02f6bbbd06974fd86a136b102a4d77e134e92\": container with ID starting with d25934a737be5b16f1c782d611e02f6bbbd06974fd86a136b102a4d77e134e92 not found: ID does not exist"
Sep 16 12:51:00.519229 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:51:00.519186    2580 scope.go:117] "RemoveContainer" containerID="d1f754c7c06351c83ed6e00e5bc4f31c69ca8c2e4fb704760304995862185e2a"
Sep 16 12:51:00.519406 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:51:00.519388    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d1f754c7c06351c83ed6e00e5bc4f31c69ca8c2e4fb704760304995862185e2a"} err="failed to get container status \"d1f754c7c06351c83ed6e00e5bc4f31c69ca8c2e4fb704760304995862185e2a\": rpc error: code = NotFound desc = could not find container \"d1f754c7c06351c83ed6e00e5bc4f31c69ca8c2e4fb704760304995862185e2a\": container with ID starting with d1f754c7c06351c83ed6e00e5bc4f31c69ca8c2e4fb704760304995862185e2a not found: ID does not exist"
Sep 16 12:51:00.519445 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:51:00.519407    2580 scope.go:117] "RemoveContainer" containerID="1418326e938a5682f35998260a697824f95c6fee3512f8fd106f74caadec39c8"
Sep 16 12:51:00.519634 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:51:00.519619    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1418326e938a5682f35998260a697824f95c6fee3512f8fd106f74caadec39c8"} err="failed to get container status \"1418326e938a5682f35998260a697824f95c6fee3512f8fd106f74caadec39c8\": rpc error: code = NotFound desc = could not find container \"1418326e938a5682f35998260a697824f95c6fee3512f8fd106f74caadec39c8\": container with ID starting with 1418326e938a5682f35998260a697824f95c6fee3512f8fd106f74caadec39c8 not found: ID does not exist"
Sep 16 12:51:00.519685 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:51:00.519634    2580 scope.go:117] "RemoveContainer" containerID="935230d89b4c3cbbbdea303993cac3eb841c6b2fedfaf8ce16f5f9142794b6a8"
Sep 16 12:51:00.519828 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:51:00.519813    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"935230d89b4c3cbbbdea303993cac3eb841c6b2fedfaf8ce16f5f9142794b6a8"} err="failed to get container status \"935230d89b4c3cbbbdea303993cac3eb841c6b2fedfaf8ce16f5f9142794b6a8\": rpc error: code = NotFound desc = could not find container \"935230d89b4c3cbbbdea303993cac3eb841c6b2fedfaf8ce16f5f9142794b6a8\": container with ID starting with 935230d89b4c3cbbbdea303993cac3eb841c6b2fedfaf8ce16f5f9142794b6a8 not found: ID does not exist"
Sep 16 12:51:00.519828 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:51:00.519827    2580 scope.go:117] "RemoveContainer" containerID="a90db64ac3c00006d0d70fb1a4756654bf3c583bc4468edeff701d86e7c05c31"
Sep 16 12:51:00.520010 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:51:00.519995    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a90db64ac3c00006d0d70fb1a4756654bf3c583bc4468edeff701d86e7c05c31"} err="failed to get container status \"a90db64ac3c00006d0d70fb1a4756654bf3c583bc4468edeff701d86e7c05c31\": rpc error: code = NotFound desc = could not find container \"a90db64ac3c00006d0d70fb1a4756654bf3c583bc4468edeff701d86e7c05c31\": container with ID starting with a90db64ac3c00006d0d70fb1a4756654bf3c583bc4468edeff701d86e7c05c31 not found: ID does not exist"
Sep 16 12:51:00.520046 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:51:00.520010    2580 scope.go:117] "RemoveContainer" containerID="e4c8b98a3e29dc5927fd6e25c39c6f6e8e5a52bc45b8d99817ec406280a1c6a0"
Sep 16 12:51:00.520228 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:51:00.520214    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e4c8b98a3e29dc5927fd6e25c39c6f6e8e5a52bc45b8d99817ec406280a1c6a0"} err="failed to get container status \"e4c8b98a3e29dc5927fd6e25c39c6f6e8e5a52bc45b8d99817ec406280a1c6a0\": rpc error: code = NotFound desc = could not find container \"e4c8b98a3e29dc5927fd6e25c39c6f6e8e5a52bc45b8d99817ec406280a1c6a0\": container with ID starting with e4c8b98a3e29dc5927fd6e25c39c6f6e8e5a52bc45b8d99817ec406280a1c6a0 not found: ID does not exist"
Sep 16 12:51:00.520272 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:51:00.520228    2580 scope.go:117] "RemoveContainer" containerID="84be625f3f386f8baeda5297f5efad680c1f6b97a7f91fa03f3523b58c9415ef"
Sep 16 12:51:00.520416 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:51:00.520401    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"84be625f3f386f8baeda5297f5efad680c1f6b97a7f91fa03f3523b58c9415ef"} err="failed to get container status \"84be625f3f386f8baeda5297f5efad680c1f6b97a7f91fa03f3523b58c9415ef\": rpc error: code = NotFound desc = could not find container \"84be625f3f386f8baeda5297f5efad680c1f6b97a7f91fa03f3523b58c9415ef\": container with ID starting with 84be625f3f386f8baeda5297f5efad680c1f6b97a7f91fa03f3523b58c9415ef not found: ID does not exist"
Sep 16 12:51:00.520460 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:51:00.520416    2580 scope.go:117] "RemoveContainer" containerID="d25934a737be5b16f1c782d611e02f6bbbd06974fd86a136b102a4d77e134e92"
Sep 16 12:51:00.520612 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:51:00.520597    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d25934a737be5b16f1c782d611e02f6bbbd06974fd86a136b102a4d77e134e92"} err="failed to get container status \"d25934a737be5b16f1c782d611e02f6bbbd06974fd86a136b102a4d77e134e92\": rpc error: code = NotFound desc = could not find container \"d25934a737be5b16f1c782d611e02f6bbbd06974fd86a136b102a4d77e134e92\": container with ID starting with d25934a737be5b16f1c782d611e02f6bbbd06974fd86a136b102a4d77e134e92 not found: ID does not exist"
Sep 16 12:51:00.520662 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:51:00.520612    2580 scope.go:117] "RemoveContainer" containerID="d1f754c7c06351c83ed6e00e5bc4f31c69ca8c2e4fb704760304995862185e2a"
Sep 16 12:51:00.520811 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:51:00.520796    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d1f754c7c06351c83ed6e00e5bc4f31c69ca8c2e4fb704760304995862185e2a"} err="failed to get container status \"d1f754c7c06351c83ed6e00e5bc4f31c69ca8c2e4fb704760304995862185e2a\": rpc error: code = NotFound desc = could not find container \"d1f754c7c06351c83ed6e00e5bc4f31c69ca8c2e4fb704760304995862185e2a\": container with ID starting with d1f754c7c06351c83ed6e00e5bc4f31c69ca8c2e4fb704760304995862185e2a not found: ID does not exist"
Sep 16 12:51:00.520853 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:51:00.520811    2580 scope.go:117] "RemoveContainer" containerID="1418326e938a5682f35998260a697824f95c6fee3512f8fd106f74caadec39c8"
Sep 16 12:51:00.521008 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:51:00.520993    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1418326e938a5682f35998260a697824f95c6fee3512f8fd106f74caadec39c8"} err="failed to get container status \"1418326e938a5682f35998260a697824f95c6fee3512f8fd106f74caadec39c8\": rpc error: code = NotFound desc = could not find container \"1418326e938a5682f35998260a697824f95c6fee3512f8fd106f74caadec39c8\": container with ID starting with 1418326e938a5682f35998260a697824f95c6fee3512f8fd106f74caadec39c8 not found: ID does not exist"
Sep 16 12:51:00.521049 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:51:00.521007    2580 scope.go:117] "RemoveContainer" containerID="935230d89b4c3cbbbdea303993cac3eb841c6b2fedfaf8ce16f5f9142794b6a8"
Sep 16 12:51:00.521196 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:51:00.521180    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"935230d89b4c3cbbbdea303993cac3eb841c6b2fedfaf8ce16f5f9142794b6a8"} err="failed to get container status \"935230d89b4c3cbbbdea303993cac3eb841c6b2fedfaf8ce16f5f9142794b6a8\": rpc error: code = NotFound desc = could not find container \"935230d89b4c3cbbbdea303993cac3eb841c6b2fedfaf8ce16f5f9142794b6a8\": container with ID starting with 935230d89b4c3cbbbdea303993cac3eb841c6b2fedfaf8ce16f5f9142794b6a8 not found: ID does not exist"
Sep 16 12:51:00.521243 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:51:00.521197    2580 scope.go:117] "RemoveContainer" containerID="a90db64ac3c00006d0d70fb1a4756654bf3c583bc4468edeff701d86e7c05c31"
Sep 16 12:51:00.521392 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:51:00.521370    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a90db64ac3c00006d0d70fb1a4756654bf3c583bc4468edeff701d86e7c05c31"} err="failed to get container status \"a90db64ac3c00006d0d70fb1a4756654bf3c583bc4468edeff701d86e7c05c31\": rpc error: code = NotFound desc = could not find container \"a90db64ac3c00006d0d70fb1a4756654bf3c583bc4468edeff701d86e7c05c31\": container with ID starting with a90db64ac3c00006d0d70fb1a4756654bf3c583bc4468edeff701d86e7c05c31 not found: ID does not exist"
Sep 16 12:51:00.521392 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:51:00.521388    2580 scope.go:117] "RemoveContainer" containerID="e4c8b98a3e29dc5927fd6e25c39c6f6e8e5a52bc45b8d99817ec406280a1c6a0"
Sep 16 12:51:00.521615 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:51:00.521598    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e4c8b98a3e29dc5927fd6e25c39c6f6e8e5a52bc45b8d99817ec406280a1c6a0"} err="failed to get container status \"e4c8b98a3e29dc5927fd6e25c39c6f6e8e5a52bc45b8d99817ec406280a1c6a0\": rpc error: code = NotFound desc = could not find container \"e4c8b98a3e29dc5927fd6e25c39c6f6e8e5a52bc45b8d99817ec406280a1c6a0\": container with ID starting with e4c8b98a3e29dc5927fd6e25c39c6f6e8e5a52bc45b8d99817ec406280a1c6a0 not found: ID does not exist"
Sep 16 12:51:00.521615 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:51:00.521614    2580 scope.go:117] "RemoveContainer" containerID="84be625f3f386f8baeda5297f5efad680c1f6b97a7f91fa03f3523b58c9415ef"
Sep 16 12:51:00.521818 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:51:00.521803    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"84be625f3f386f8baeda5297f5efad680c1f6b97a7f91fa03f3523b58c9415ef"} err="failed to get container status \"84be625f3f386f8baeda5297f5efad680c1f6b97a7f91fa03f3523b58c9415ef\": rpc error: code = NotFound desc = could not find container \"84be625f3f386f8baeda5297f5efad680c1f6b97a7f91fa03f3523b58c9415ef\": container with ID starting with 84be625f3f386f8baeda5297f5efad680c1f6b97a7f91fa03f3523b58c9415ef not found: ID does not exist"
Sep 16 12:51:00.521818 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:51:00.521817    2580 scope.go:117] "RemoveContainer" containerID="d25934a737be5b16f1c782d611e02f6bbbd06974fd86a136b102a4d77e134e92"
Sep 16 12:51:00.522017 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:51:00.522001    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d25934a737be5b16f1c782d611e02f6bbbd06974fd86a136b102a4d77e134e92"} err="failed to get container status \"d25934a737be5b16f1c782d611e02f6bbbd06974fd86a136b102a4d77e134e92\": rpc error: code = NotFound desc = could not find container \"d25934a737be5b16f1c782d611e02f6bbbd06974fd86a136b102a4d77e134e92\": container with ID starting with d25934a737be5b16f1c782d611e02f6bbbd06974fd86a136b102a4d77e134e92 not found: ID does not exist"
Sep 16 12:51:00.522017 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:51:00.522016    2580 scope.go:117] "RemoveContainer" containerID="d1f754c7c06351c83ed6e00e5bc4f31c69ca8c2e4fb704760304995862185e2a"
Sep 16 12:51:00.522207 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:51:00.522192    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d1f754c7c06351c83ed6e00e5bc4f31c69ca8c2e4fb704760304995862185e2a"} err="failed to get container status \"d1f754c7c06351c83ed6e00e5bc4f31c69ca8c2e4fb704760304995862185e2a\": rpc error: code = NotFound desc = could not find container \"d1f754c7c06351c83ed6e00e5bc4f31c69ca8c2e4fb704760304995862185e2a\": container with ID starting with d1f754c7c06351c83ed6e00e5bc4f31c69ca8c2e4fb704760304995862185e2a not found: ID does not exist"
Sep 16 12:51:00.522249 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:51:00.522206    2580 scope.go:117] "RemoveContainer" containerID="1418326e938a5682f35998260a697824f95c6fee3512f8fd106f74caadec39c8"
Sep 16 12:51:00.522407 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:51:00.522391    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1418326e938a5682f35998260a697824f95c6fee3512f8fd106f74caadec39c8"} err="failed to get container status \"1418326e938a5682f35998260a697824f95c6fee3512f8fd106f74caadec39c8\": rpc error: code = NotFound desc = could not find container \"1418326e938a5682f35998260a697824f95c6fee3512f8fd106f74caadec39c8\": container with ID starting with 1418326e938a5682f35998260a697824f95c6fee3512f8fd106f74caadec39c8 not found: ID does not exist"
Sep 16 12:51:00.522407 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:51:00.522406    2580 scope.go:117] "RemoveContainer" containerID="935230d89b4c3cbbbdea303993cac3eb841c6b2fedfaf8ce16f5f9142794b6a8"
Sep 16 12:51:00.522621 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:51:00.522600    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"935230d89b4c3cbbbdea303993cac3eb841c6b2fedfaf8ce16f5f9142794b6a8"} err="failed to get container status \"935230d89b4c3cbbbdea303993cac3eb841c6b2fedfaf8ce16f5f9142794b6a8\": rpc error: code = NotFound desc = could not find container \"935230d89b4c3cbbbdea303993cac3eb841c6b2fedfaf8ce16f5f9142794b6a8\": container with ID starting with 935230d89b4c3cbbbdea303993cac3eb841c6b2fedfaf8ce16f5f9142794b6a8 not found: ID does not exist"
Sep 16 12:51:00.522677 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:51:00.522623    2580 scope.go:117] "RemoveContainer" containerID="a90db64ac3c00006d0d70fb1a4756654bf3c583bc4468edeff701d86e7c05c31"
Sep 16 12:51:00.522821 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:51:00.522806    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a90db64ac3c00006d0d70fb1a4756654bf3c583bc4468edeff701d86e7c05c31"} err="failed to get container status \"a90db64ac3c00006d0d70fb1a4756654bf3c583bc4468edeff701d86e7c05c31\": rpc error: code = NotFound desc = could not find container \"a90db64ac3c00006d0d70fb1a4756654bf3c583bc4468edeff701d86e7c05c31\": container with ID starting with a90db64ac3c00006d0d70fb1a4756654bf3c583bc4468edeff701d86e7c05c31 not found: ID does not exist"
Sep 16 12:51:00.522860 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:51:00.522821    2580 scope.go:117] "RemoveContainer" containerID="e4c8b98a3e29dc5927fd6e25c39c6f6e8e5a52bc45b8d99817ec406280a1c6a0"
Sep 16 12:51:00.522993 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:51:00.522978    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e4c8b98a3e29dc5927fd6e25c39c6f6e8e5a52bc45b8d99817ec406280a1c6a0"} err="failed to get container status \"e4c8b98a3e29dc5927fd6e25c39c6f6e8e5a52bc45b8d99817ec406280a1c6a0\": rpc error: code = NotFound desc = could not find container \"e4c8b98a3e29dc5927fd6e25c39c6f6e8e5a52bc45b8d99817ec406280a1c6a0\": container with ID starting with e4c8b98a3e29dc5927fd6e25c39c6f6e8e5a52bc45b8d99817ec406280a1c6a0 not found: ID does not exist"
Sep 16 12:51:00.523031 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:51:00.522992    2580 scope.go:117] "RemoveContainer" containerID="84be625f3f386f8baeda5297f5efad680c1f6b97a7f91fa03f3523b58c9415ef"
Sep 16 12:51:00.523171 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:51:00.523157    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"84be625f3f386f8baeda5297f5efad680c1f6b97a7f91fa03f3523b58c9415ef"} err="failed to get container status \"84be625f3f386f8baeda5297f5efad680c1f6b97a7f91fa03f3523b58c9415ef\": rpc error: code = NotFound desc = could not find container \"84be625f3f386f8baeda5297f5efad680c1f6b97a7f91fa03f3523b58c9415ef\": container with ID starting with 84be625f3f386f8baeda5297f5efad680c1f6b97a7f91fa03f3523b58c9415ef not found: ID does not exist"
Sep 16 12:51:00.523255 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:51:00.523171    2580 scope.go:117] "RemoveContainer" containerID="d25934a737be5b16f1c782d611e02f6bbbd06974fd86a136b102a4d77e134e92"
Sep 16 12:51:00.523349 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:51:00.523335    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d25934a737be5b16f1c782d611e02f6bbbd06974fd86a136b102a4d77e134e92"} err="failed to get container status \"d25934a737be5b16f1c782d611e02f6bbbd06974fd86a136b102a4d77e134e92\": rpc error: code = NotFound desc = could not find container \"d25934a737be5b16f1c782d611e02f6bbbd06974fd86a136b102a4d77e134e92\": container with ID starting with d25934a737be5b16f1c782d611e02f6bbbd06974fd86a136b102a4d77e134e92 not found: ID does not exist"
Sep 16 12:51:00.523388 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:51:00.523349    2580 scope.go:117] "RemoveContainer" containerID="d1f754c7c06351c83ed6e00e5bc4f31c69ca8c2e4fb704760304995862185e2a"
Sep 16 12:51:00.523603 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:51:00.523587    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d1f754c7c06351c83ed6e00e5bc4f31c69ca8c2e4fb704760304995862185e2a"} err="failed to get container status \"d1f754c7c06351c83ed6e00e5bc4f31c69ca8c2e4fb704760304995862185e2a\": rpc error: code = NotFound desc = could not find container \"d1f754c7c06351c83ed6e00e5bc4f31c69ca8c2e4fb704760304995862185e2a\": container with ID starting with d1f754c7c06351c83ed6e00e5bc4f31c69ca8c2e4fb704760304995862185e2a not found: ID does not exist"
Sep 16 12:51:00.523603 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:51:00.523602    2580 scope.go:117] "RemoveContainer" containerID="1418326e938a5682f35998260a697824f95c6fee3512f8fd106f74caadec39c8"
Sep 16 12:51:00.523796 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:51:00.523777    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1418326e938a5682f35998260a697824f95c6fee3512f8fd106f74caadec39c8"} err="failed to get container status \"1418326e938a5682f35998260a697824f95c6fee3512f8fd106f74caadec39c8\": rpc error: code = NotFound desc = could not find container \"1418326e938a5682f35998260a697824f95c6fee3512f8fd106f74caadec39c8\": container with ID starting with 1418326e938a5682f35998260a697824f95c6fee3512f8fd106f74caadec39c8 not found: ID does not exist"
Sep 16 12:51:00.523839 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:51:00.523796    2580 scope.go:117] "RemoveContainer" containerID="935230d89b4c3cbbbdea303993cac3eb841c6b2fedfaf8ce16f5f9142794b6a8"
Sep 16 12:51:00.523980 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:51:00.523965    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"935230d89b4c3cbbbdea303993cac3eb841c6b2fedfaf8ce16f5f9142794b6a8"} err="failed to get container status \"935230d89b4c3cbbbdea303993cac3eb841c6b2fedfaf8ce16f5f9142794b6a8\": rpc error: code = NotFound desc = could not find container \"935230d89b4c3cbbbdea303993cac3eb841c6b2fedfaf8ce16f5f9142794b6a8\": container with ID starting with 935230d89b4c3cbbbdea303993cac3eb841c6b2fedfaf8ce16f5f9142794b6a8 not found: ID does not exist"
Sep 16 12:51:00.524017 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:51:00.523979    2580 scope.go:117] "RemoveContainer" containerID="a90db64ac3c00006d0d70fb1a4756654bf3c583bc4468edeff701d86e7c05c31"
Sep 16 12:51:00.524165 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:51:00.524150    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a90db64ac3c00006d0d70fb1a4756654bf3c583bc4468edeff701d86e7c05c31"} err="failed to get container status \"a90db64ac3c00006d0d70fb1a4756654bf3c583bc4468edeff701d86e7c05c31\": rpc error: code = NotFound desc = could not find container \"a90db64ac3c00006d0d70fb1a4756654bf3c583bc4468edeff701d86e7c05c31\": container with ID starting with a90db64ac3c00006d0d70fb1a4756654bf3c583bc4468edeff701d86e7c05c31 not found: ID does not exist"
Sep 16 12:51:00.524165 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:51:00.524164    2580 scope.go:117] "RemoveContainer" containerID="e4c8b98a3e29dc5927fd6e25c39c6f6e8e5a52bc45b8d99817ec406280a1c6a0"
Sep 16 12:51:00.524344 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:51:00.524328    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e4c8b98a3e29dc5927fd6e25c39c6f6e8e5a52bc45b8d99817ec406280a1c6a0"} err="failed to get container status \"e4c8b98a3e29dc5927fd6e25c39c6f6e8e5a52bc45b8d99817ec406280a1c6a0\": rpc error: code = NotFound desc = could not find container \"e4c8b98a3e29dc5927fd6e25c39c6f6e8e5a52bc45b8d99817ec406280a1c6a0\": container with ID starting with e4c8b98a3e29dc5927fd6e25c39c6f6e8e5a52bc45b8d99817ec406280a1c6a0 not found: ID does not exist"
Sep 16 12:51:00.524344 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:51:00.524343    2580 scope.go:117] "RemoveContainer" containerID="84be625f3f386f8baeda5297f5efad680c1f6b97a7f91fa03f3523b58c9415ef"
Sep 16 12:51:00.524540 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:51:00.524524    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"84be625f3f386f8baeda5297f5efad680c1f6b97a7f91fa03f3523b58c9415ef"} err="failed to get container status \"84be625f3f386f8baeda5297f5efad680c1f6b97a7f91fa03f3523b58c9415ef\": rpc error: code = NotFound desc = could not find container \"84be625f3f386f8baeda5297f5efad680c1f6b97a7f91fa03f3523b58c9415ef\": container with ID starting with 84be625f3f386f8baeda5297f5efad680c1f6b97a7f91fa03f3523b58c9415ef not found: ID does not exist"
Sep 16 12:51:00.524585 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:51:00.524539    2580 scope.go:117] "RemoveContainer" containerID="d25934a737be5b16f1c782d611e02f6bbbd06974fd86a136b102a4d77e134e92"
Sep 16 12:51:00.524715 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:51:00.524699    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d25934a737be5b16f1c782d611e02f6bbbd06974fd86a136b102a4d77e134e92"} err="failed to get container status \"d25934a737be5b16f1c782d611e02f6bbbd06974fd86a136b102a4d77e134e92\": rpc error: code = NotFound desc = could not find container \"d25934a737be5b16f1c782d611e02f6bbbd06974fd86a136b102a4d77e134e92\": container with ID starting with d25934a737be5b16f1c782d611e02f6bbbd06974fd86a136b102a4d77e134e92 not found: ID does not exist"
Sep 16 12:51:00.524755 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:51:00.524714    2580 scope.go:117] "RemoveContainer" containerID="d1f754c7c06351c83ed6e00e5bc4f31c69ca8c2e4fb704760304995862185e2a"
Sep 16 12:51:00.524895 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:51:00.524880    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d1f754c7c06351c83ed6e00e5bc4f31c69ca8c2e4fb704760304995862185e2a"} err="failed to get container status \"d1f754c7c06351c83ed6e00e5bc4f31c69ca8c2e4fb704760304995862185e2a\": rpc error: code = NotFound desc = could not find container \"d1f754c7c06351c83ed6e00e5bc4f31c69ca8c2e4fb704760304995862185e2a\": container with ID starting with d1f754c7c06351c83ed6e00e5bc4f31c69ca8c2e4fb704760304995862185e2a not found: ID does not exist"
Sep 16 12:51:00.524895 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:51:00.524894    2580 scope.go:117] "RemoveContainer" containerID="1418326e938a5682f35998260a697824f95c6fee3512f8fd106f74caadec39c8"
Sep 16 12:51:00.525075 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:51:00.525060    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1418326e938a5682f35998260a697824f95c6fee3512f8fd106f74caadec39c8"} err="failed to get container status \"1418326e938a5682f35998260a697824f95c6fee3512f8fd106f74caadec39c8\": rpc error: code = NotFound desc = could not find container \"1418326e938a5682f35998260a697824f95c6fee3512f8fd106f74caadec39c8\": container with ID starting with 1418326e938a5682f35998260a697824f95c6fee3512f8fd106f74caadec39c8 not found: ID does not exist"
Sep 16 12:51:00.525075 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:51:00.525074    2580 scope.go:117] "RemoveContainer" containerID="935230d89b4c3cbbbdea303993cac3eb841c6b2fedfaf8ce16f5f9142794b6a8"
Sep 16 12:51:00.525250 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:51:00.525235    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"935230d89b4c3cbbbdea303993cac3eb841c6b2fedfaf8ce16f5f9142794b6a8"} err="failed to get container status \"935230d89b4c3cbbbdea303993cac3eb841c6b2fedfaf8ce16f5f9142794b6a8\": rpc error: code = NotFound desc = could not find container \"935230d89b4c3cbbbdea303993cac3eb841c6b2fedfaf8ce16f5f9142794b6a8\": container with ID starting with 935230d89b4c3cbbbdea303993cac3eb841c6b2fedfaf8ce16f5f9142794b6a8 not found: ID does not exist"
Sep 16 12:51:00.525290 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:51:00.525250    2580 scope.go:117] "RemoveContainer" containerID="a90db64ac3c00006d0d70fb1a4756654bf3c583bc4468edeff701d86e7c05c31"
Sep 16 12:51:00.525435 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:51:00.525420    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a90db64ac3c00006d0d70fb1a4756654bf3c583bc4468edeff701d86e7c05c31"} err="failed to get container status \"a90db64ac3c00006d0d70fb1a4756654bf3c583bc4468edeff701d86e7c05c31\": rpc error: code = NotFound desc = could not find container \"a90db64ac3c00006d0d70fb1a4756654bf3c583bc4468edeff701d86e7c05c31\": container with ID starting with a90db64ac3c00006d0d70fb1a4756654bf3c583bc4468edeff701d86e7c05c31 not found: ID does not exist"
Sep 16 12:51:00.525475 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:51:00.525434    2580 scope.go:117] "RemoveContainer" containerID="e4c8b98a3e29dc5927fd6e25c39c6f6e8e5a52bc45b8d99817ec406280a1c6a0"
Sep 16 12:51:00.525626 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:51:00.525612    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e4c8b98a3e29dc5927fd6e25c39c6f6e8e5a52bc45b8d99817ec406280a1c6a0"} err="failed to get container status \"e4c8b98a3e29dc5927fd6e25c39c6f6e8e5a52bc45b8d99817ec406280a1c6a0\": rpc error: code = NotFound desc = could not find container \"e4c8b98a3e29dc5927fd6e25c39c6f6e8e5a52bc45b8d99817ec406280a1c6a0\": container with ID starting with e4c8b98a3e29dc5927fd6e25c39c6f6e8e5a52bc45b8d99817ec406280a1c6a0 not found: ID does not exist"
Sep 16 12:51:00.593758 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:51:00.593735    2580 reconciler_common.go:299] "Volume detached for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/84b4b343-d43c-4639-a768-3f52806bd8c7-configmap-metrics-client-ca\") on node \"ip-10-0-133-218.ec2.internal\" DevicePath \"\""
Sep 16 12:51:00.593758 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:51:00.593756    2580 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/84b4b343-d43c-4639-a768-3f52806bd8c7-secret-prometheus-k8s-tls\") on node \"ip-10-0-133-218.ec2.internal\" DevicePath \"\""
Sep 16 12:51:00.593873 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:51:00.593765    2580 reconciler_common.go:299] "Volume detached for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/84b4b343-d43c-4639-a768-3f52806bd8c7-secret-metrics-client-certs\") on node \"ip-10-0-133-218.ec2.internal\" DevicePath \"\""
Sep 16 12:51:00.593873 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:51:00.593773    2580 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/84b4b343-d43c-4639-a768-3f52806bd8c7-prometheus-k8s-rulefiles-0\") on node \"ip-10-0-133-218.ec2.internal\" DevicePath \"\""
Sep 16 12:51:00.593873 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:51:00.593782    2580 reconciler_common.go:299] "Volume detached for volume \"config\" (UniqueName: \"kubernetes.io/secret/84b4b343-d43c-4639-a768-3f52806bd8c7-config\") on node \"ip-10-0-133-218.ec2.internal\" DevicePath \"\""
Sep 16 12:51:00.593873 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:51:00.593790    2580 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/84b4b343-d43c-4639-a768-3f52806bd8c7-tls-assets\") on node \"ip-10-0-133-218.ec2.internal\" DevicePath \"\""
Sep 16 12:51:00.593873 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:51:00.593799    2580 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/84b4b343-d43c-4639-a768-3f52806bd8c7-secret-prometheus-k8s-thanos-sidecar-tls\") on node \"ip-10-0-133-218.ec2.internal\" DevicePath \"\""
Sep 16 12:51:00.593873 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:51:00.593807    2580 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/84b4b343-d43c-4639-a768-3f52806bd8c7-web-config\") on node \"ip-10-0-133-218.ec2.internal\" DevicePath \"\""
Sep 16 12:51:00.593873 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:51:00.593815    2580 reconciler_common.go:299] "Volume detached for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/84b4b343-d43c-4639-a768-3f52806bd8c7-configmap-kubelet-serving-ca-bundle\") on node \"ip-10-0-133-218.ec2.internal\" DevicePath \"\""
Sep 16 12:51:00.593873 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:51:00.593825    2580 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/84b4b343-d43c-4639-a768-3f52806bd8c7-prometheus-k8s-db\") on node \"ip-10-0-133-218.ec2.internal\" DevicePath \"\""
Sep 16 12:51:00.593873 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:51:00.593832    2580 reconciler_common.go:299] "Volume detached for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/84b4b343-d43c-4639-a768-3f52806bd8c7-thanos-prometheus-http-client-file\") on node \"ip-10-0-133-218.ec2.internal\" DevicePath \"\""
Sep 16 12:51:00.593873 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:51:00.593841    2580 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/84b4b343-d43c-4639-a768-3f52806bd8c7-secret-prometheus-k8s-kube-rbac-proxy-web\") on node \"ip-10-0-133-218.ec2.internal\" DevicePath \"\""
Sep 16 12:51:00.593873 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:51:00.593849    2580 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/84b4b343-d43c-4639-a768-3f52806bd8c7-prometheus-k8s-rulefiles-1\") on node \"ip-10-0-133-218.ec2.internal\" DevicePath \"\""
Sep 16 12:51:00.593873 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:51:00.593857    2580 reconciler_common.go:299] "Volume detached for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/84b4b343-d43c-4639-a768-3f52806bd8c7-prometheus-trusted-ca-bundle\") on node \"ip-10-0-133-218.ec2.internal\" DevicePath \"\""
Sep 16 12:51:00.593873 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:51:00.593866    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hxkpn\" (UniqueName: \"kubernetes.io/projected/84b4b343-d43c-4639-a768-3f52806bd8c7-kube-api-access-hxkpn\") on node \"ip-10-0-133-218.ec2.internal\" DevicePath \"\""
Sep 16 12:51:00.593873 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:51:00.593874    2580 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/84b4b343-d43c-4639-a768-3f52806bd8c7-prometheus-k8s-rulefiles-2\") on node \"ip-10-0-133-218.ec2.internal\" DevicePath \"\""
Sep 16 12:51:00.594263 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:51:00.593881    2580 reconciler_common.go:299] "Volume detached for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/84b4b343-d43c-4639-a768-3f52806bd8c7-secret-grpc-tls\") on node \"ip-10-0-133-218.ec2.internal\" DevicePath \"\""
Sep 16 12:51:00.594263 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:51:00.593889    2580 reconciler_common.go:299] "Volume detached for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/84b4b343-d43c-4639-a768-3f52806bd8c7-configmap-serving-certs-ca-bundle\") on node \"ip-10-0-133-218.ec2.internal\" DevicePath \"\""
Sep 16 12:51:00.594263 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:51:00.593898    2580 reconciler_common.go:299] "Volume detached for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/84b4b343-d43c-4639-a768-3f52806bd8c7-secret-kube-rbac-proxy\") on node \"ip-10-0-133-218.ec2.internal\" DevicePath \"\""
Sep 16 12:51:00.594263 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:51:00.593905    2580 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/84b4b343-d43c-4639-a768-3f52806bd8c7-config-out\") on node \"ip-10-0-133-218.ec2.internal\" DevicePath \"\""
Sep 16 12:51:00.788108 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:51:00.788075    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 16 12:51:00.792460 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:51:00.792437    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 16 12:51:00.822535 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:51:00.822505    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 16 12:51:00.822789 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:51:00.822776    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1273107e-4c61-4b7c-983d-1dbf0dfc99ad" containerName="init-config-reloader"
Sep 16 12:51:00.822832 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:51:00.822791    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="1273107e-4c61-4b7c-983d-1dbf0dfc99ad" containerName="init-config-reloader"
Sep 16 12:51:00.822832 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:51:00.822800    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="84b4b343-d43c-4639-a768-3f52806bd8c7" containerName="init-config-reloader"
Sep 16 12:51:00.822832 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:51:00.822806    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="84b4b343-d43c-4639-a768-3f52806bd8c7" containerName="init-config-reloader"
Sep 16 12:51:00.822832 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:51:00.822813    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="84b4b343-d43c-4639-a768-3f52806bd8c7" containerName="kube-rbac-proxy-web"
Sep 16 12:51:00.822832 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:51:00.822819    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="84b4b343-d43c-4639-a768-3f52806bd8c7" containerName="kube-rbac-proxy-web"
Sep 16 12:51:00.822832 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:51:00.822826    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="84b4b343-d43c-4639-a768-3f52806bd8c7" containerName="kube-rbac-proxy"
Sep 16 12:51:00.822832 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:51:00.822831    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="84b4b343-d43c-4639-a768-3f52806bd8c7" containerName="kube-rbac-proxy"
Sep 16 12:51:00.823034 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:51:00.822839    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1273107e-4c61-4b7c-983d-1dbf0dfc99ad" containerName="prom-label-proxy"
Sep 16 12:51:00.823034 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:51:00.822845    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="1273107e-4c61-4b7c-983d-1dbf0dfc99ad" containerName="prom-label-proxy"
Sep 16 12:51:00.823034 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:51:00.822850    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="84b4b343-d43c-4639-a768-3f52806bd8c7" containerName="kube-rbac-proxy-thanos"
Sep 16 12:51:00.823034 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:51:00.822856    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="84b4b343-d43c-4639-a768-3f52806bd8c7" containerName="kube-rbac-proxy-thanos"
Sep 16 12:51:00.823034 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:51:00.822863    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="84b4b343-d43c-4639-a768-3f52806bd8c7" containerName="thanos-sidecar"
Sep 16 12:51:00.823034 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:51:00.822868    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="84b4b343-d43c-4639-a768-3f52806bd8c7" containerName="thanos-sidecar"
Sep 16 12:51:00.823034 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:51:00.822877    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1273107e-4c61-4b7c-983d-1dbf0dfc99ad" containerName="kube-rbac-proxy-web"
Sep 16 12:51:00.823034 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:51:00.822882    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="1273107e-4c61-4b7c-983d-1dbf0dfc99ad" containerName="kube-rbac-proxy-web"
Sep 16 12:51:00.823034 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:51:00.822889    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3720e939-652a-4ad2-938d-72adb38487ad" containerName="console"
Sep 16 12:51:00.823034 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:51:00.822894    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="3720e939-652a-4ad2-938d-72adb38487ad" containerName="console"
Sep 16 12:51:00.823034 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:51:00.822900    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1273107e-4c61-4b7c-983d-1dbf0dfc99ad" containerName="kube-rbac-proxy"
Sep 16 12:51:00.823034 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:51:00.822905    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="1273107e-4c61-4b7c-983d-1dbf0dfc99ad" containerName="kube-rbac-proxy"
Sep 16 12:51:00.823034 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:51:00.822912    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1273107e-4c61-4b7c-983d-1dbf0dfc99ad" containerName="alertmanager"
Sep 16 12:51:00.823034 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:51:00.822916    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="1273107e-4c61-4b7c-983d-1dbf0dfc99ad" containerName="alertmanager"
Sep 16 12:51:00.823034 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:51:00.822925    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c5a49713-de16-4f73-b654-d4ed06ce1946" containerName="console"
Sep 16 12:51:00.823034 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:51:00.822930    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="c5a49713-de16-4f73-b654-d4ed06ce1946" containerName="console"
Sep 16 12:51:00.823034 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:51:00.822935    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1273107e-4c61-4b7c-983d-1dbf0dfc99ad" containerName="kube-rbac-proxy-metric"
Sep 16 12:51:00.823034 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:51:00.822940    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="1273107e-4c61-4b7c-983d-1dbf0dfc99ad" containerName="kube-rbac-proxy-metric"
Sep 16 12:51:00.823034 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:51:00.822946    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1273107e-4c61-4b7c-983d-1dbf0dfc99ad" containerName="config-reloader"
Sep 16 12:51:00.823034 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:51:00.822951    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="1273107e-4c61-4b7c-983d-1dbf0dfc99ad" containerName="config-reloader"
Sep 16 12:51:00.823034 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:51:00.822957    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="84b4b343-d43c-4639-a768-3f52806bd8c7" containerName="prometheus"
Sep 16 12:51:00.823034 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:51:00.822961    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="84b4b343-d43c-4639-a768-3f52806bd8c7" containerName="prometheus"
Sep 16 12:51:00.823034 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:51:00.822969    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="84b4b343-d43c-4639-a768-3f52806bd8c7" containerName="config-reloader"
Sep 16 12:51:00.823034 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:51:00.822974    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="84b4b343-d43c-4639-a768-3f52806bd8c7" containerName="config-reloader"
Sep 16 12:51:00.823034 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:51:00.823013    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="1273107e-4c61-4b7c-983d-1dbf0dfc99ad" containerName="kube-rbac-proxy"
Sep 16 12:51:00.823034 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:51:00.823020    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="1273107e-4c61-4b7c-983d-1dbf0dfc99ad" containerName="kube-rbac-proxy-metric"
Sep 16 12:51:00.823034 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:51:00.823026    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="84b4b343-d43c-4639-a768-3f52806bd8c7" containerName="kube-rbac-proxy"
Sep 16 12:51:00.823034 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:51:00.823032    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="84b4b343-d43c-4639-a768-3f52806bd8c7" containerName="thanos-sidecar"
Sep 16 12:51:00.823034 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:51:00.823039    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="1273107e-4c61-4b7c-983d-1dbf0dfc99ad" containerName="kube-rbac-proxy-web"
Sep 16 12:51:00.823034 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:51:00.823045    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="84b4b343-d43c-4639-a768-3f52806bd8c7" containerName="prometheus"
Sep 16 12:51:00.823954 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:51:00.823049    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="3720e939-652a-4ad2-938d-72adb38487ad" containerName="console"
Sep 16 12:51:00.823954 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:51:00.823055    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="84b4b343-d43c-4639-a768-3f52806bd8c7" containerName="config-reloader"
Sep 16 12:51:00.823954 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:51:00.823060    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="84b4b343-d43c-4639-a768-3f52806bd8c7" containerName="kube-rbac-proxy-thanos"
Sep 16 12:51:00.823954 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:51:00.823065    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="1273107e-4c61-4b7c-983d-1dbf0dfc99ad" containerName="prom-label-proxy"
Sep 16 12:51:00.823954 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:51:00.823071    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="1273107e-4c61-4b7c-983d-1dbf0dfc99ad" containerName="alertmanager"
Sep 16 12:51:00.823954 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:51:00.823076    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="1273107e-4c61-4b7c-983d-1dbf0dfc99ad" containerName="config-reloader"
Sep 16 12:51:00.823954 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:51:00.823081    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="c5a49713-de16-4f73-b654-d4ed06ce1946" containerName="console"
Sep 16 12:51:00.823954 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:51:00.823087    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="84b4b343-d43c-4639-a768-3f52806bd8c7" containerName="kube-rbac-proxy-web"
Sep 16 12:51:00.828302 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:51:00.828283    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 12:51:00.831212 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:51:00.831193    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\""
Sep 16 12:51:00.831212 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:51:00.831204    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\""
Sep 16 12:51:00.831369 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:51:00.831195    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\""
Sep 16 12:51:00.831480 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:51:00.831463    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\""
Sep 16 12:51:00.831572 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:51:00.831523    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\""
Sep 16 12:51:00.831628 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:51:00.831568    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\""
Sep 16 12:51:00.833516 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:51:00.832216    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\""
Sep 16 12:51:00.833516 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:51:00.833129    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\""
Sep 16 12:51:00.833516 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:51:00.833263    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\""
Sep 16 12:51:00.838199 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:51:00.838054    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\""
Sep 16 12:51:00.838199 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:51:00.838069    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\""
Sep 16 12:51:00.838199 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:51:00.838089    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\""
Sep 16 12:51:00.838199 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:51:00.838119    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-cf5t6v2n01b5c\""
Sep 16 12:51:00.838465 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:51:00.838226    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-9bkct\""
Sep 16 12:51:00.838465 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:51:00.838063    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\""
Sep 16 12:51:00.840220 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:51:00.840199    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\""
Sep 16 12:51:00.842730 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:51:00.842712    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\""
Sep 16 12:51:00.843260 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:51:00.843243    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 16 12:51:00.896394 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:51:00.896368    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/81b1531b-258a-4233-b637-b38a3de6d99d-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"81b1531b-258a-4233-b637-b38a3de6d99d\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 12:51:00.896559 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:51:00.896406    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/81b1531b-258a-4233-b637-b38a3de6d99d-web-config\") pod \"prometheus-k8s-0\" (UID: \"81b1531b-258a-4233-b637-b38a3de6d99d\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 12:51:00.896559 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:51:00.896434    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/81b1531b-258a-4233-b637-b38a3de6d99d-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"81b1531b-258a-4233-b637-b38a3de6d99d\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 12:51:00.896559 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:51:00.896520    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/81b1531b-258a-4233-b637-b38a3de6d99d-config-out\") pod \"prometheus-k8s-0\" (UID: \"81b1531b-258a-4233-b637-b38a3de6d99d\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 12:51:00.896559 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:51:00.896546    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/81b1531b-258a-4233-b637-b38a3de6d99d-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"81b1531b-258a-4233-b637-b38a3de6d99d\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 12:51:00.896706 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:51:00.896590    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/81b1531b-258a-4233-b637-b38a3de6d99d-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"81b1531b-258a-4233-b637-b38a3de6d99d\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 12:51:00.896706 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:51:00.896626    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/81b1531b-258a-4233-b637-b38a3de6d99d-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"81b1531b-258a-4233-b637-b38a3de6d99d\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 12:51:00.896706 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:51:00.896668    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/81b1531b-258a-4233-b637-b38a3de6d99d-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"81b1531b-258a-4233-b637-b38a3de6d99d\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 12:51:00.896706 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:51:00.896684    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/81b1531b-258a-4233-b637-b38a3de6d99d-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"81b1531b-258a-4233-b637-b38a3de6d99d\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 12:51:00.896706 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:51:00.896698    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/81b1531b-258a-4233-b637-b38a3de6d99d-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"81b1531b-258a-4233-b637-b38a3de6d99d\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 12:51:00.896896 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:51:00.896718    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/81b1531b-258a-4233-b637-b38a3de6d99d-config\") pod \"prometheus-k8s-0\" (UID: \"81b1531b-258a-4233-b637-b38a3de6d99d\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 12:51:00.896896 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:51:00.896739    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/81b1531b-258a-4233-b637-b38a3de6d99d-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"81b1531b-258a-4233-b637-b38a3de6d99d\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 12:51:00.896896 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:51:00.896824    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/81b1531b-258a-4233-b637-b38a3de6d99d-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"81b1531b-258a-4233-b637-b38a3de6d99d\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 12:51:00.896896 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:51:00.896875    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/81b1531b-258a-4233-b637-b38a3de6d99d-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"81b1531b-258a-4233-b637-b38a3de6d99d\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 12:51:00.896896 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:51:00.896893    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/81b1531b-258a-4233-b637-b38a3de6d99d-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"81b1531b-258a-4233-b637-b38a3de6d99d\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 12:51:00.897105 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:51:00.896932    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/81b1531b-258a-4233-b637-b38a3de6d99d-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"81b1531b-258a-4233-b637-b38a3de6d99d\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 12:51:00.897105 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:51:00.896958    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/81b1531b-258a-4233-b637-b38a3de6d99d-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"81b1531b-258a-4233-b637-b38a3de6d99d\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 12:51:00.897105 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:51:00.897009    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2vjxr\" (UniqueName: \"kubernetes.io/projected/81b1531b-258a-4233-b637-b38a3de6d99d-kube-api-access-2vjxr\") pod \"prometheus-k8s-0\" (UID: \"81b1531b-258a-4233-b637-b38a3de6d99d\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 12:51:00.897105 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:51:00.897058    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/81b1531b-258a-4233-b637-b38a3de6d99d-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"81b1531b-258a-4233-b637-b38a3de6d99d\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 12:51:00.897105 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:51:00.897093    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/81b1531b-258a-4233-b637-b38a3de6d99d-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"81b1531b-258a-4233-b637-b38a3de6d99d\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 12:51:00.998025 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:51:00.997985    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/81b1531b-258a-4233-b637-b38a3de6d99d-config-out\") pod \"prometheus-k8s-0\" (UID: \"81b1531b-258a-4233-b637-b38a3de6d99d\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 12:51:00.998025 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:51:00.998023    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/81b1531b-258a-4233-b637-b38a3de6d99d-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"81b1531b-258a-4233-b637-b38a3de6d99d\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 12:51:00.998266 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:51:00.998044    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/81b1531b-258a-4233-b637-b38a3de6d99d-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"81b1531b-258a-4233-b637-b38a3de6d99d\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 12:51:00.998266 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:51:00.998069    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/81b1531b-258a-4233-b637-b38a3de6d99d-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"81b1531b-258a-4233-b637-b38a3de6d99d\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 12:51:00.998266 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:51:00.998095    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/81b1531b-258a-4233-b637-b38a3de6d99d-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"81b1531b-258a-4233-b637-b38a3de6d99d\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 12:51:00.998266 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:51:00.998115    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/81b1531b-258a-4233-b637-b38a3de6d99d-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"81b1531b-258a-4233-b637-b38a3de6d99d\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 12:51:00.998266 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:51:00.998136    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/81b1531b-258a-4233-b637-b38a3de6d99d-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"81b1531b-258a-4233-b637-b38a3de6d99d\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 12:51:00.998266 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:51:00.998165    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/81b1531b-258a-4233-b637-b38a3de6d99d-config\") pod \"prometheus-k8s-0\" (UID: \"81b1531b-258a-4233-b637-b38a3de6d99d\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 12:51:00.998266 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:51:00.998186    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/81b1531b-258a-4233-b637-b38a3de6d99d-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"81b1531b-258a-4233-b637-b38a3de6d99d\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 12:51:00.998266 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:51:00.998215    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/81b1531b-258a-4233-b637-b38a3de6d99d-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"81b1531b-258a-4233-b637-b38a3de6d99d\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 12:51:00.998266 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:51:00.998251    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/81b1531b-258a-4233-b637-b38a3de6d99d-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"81b1531b-258a-4233-b637-b38a3de6d99d\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 12:51:00.998722 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:51:00.998275    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/81b1531b-258a-4233-b637-b38a3de6d99d-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"81b1531b-258a-4233-b637-b38a3de6d99d\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 12:51:00.998722 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:51:00.998309    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/81b1531b-258a-4233-b637-b38a3de6d99d-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"81b1531b-258a-4233-b637-b38a3de6d99d\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 12:51:00.998722 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:51:00.998346    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/81b1531b-258a-4233-b637-b38a3de6d99d-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"81b1531b-258a-4233-b637-b38a3de6d99d\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 12:51:00.998722 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:51:00.998385    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2vjxr\" (UniqueName: \"kubernetes.io/projected/81b1531b-258a-4233-b637-b38a3de6d99d-kube-api-access-2vjxr\") pod \"prometheus-k8s-0\" (UID: \"81b1531b-258a-4233-b637-b38a3de6d99d\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 12:51:00.998722 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:51:00.998413    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/81b1531b-258a-4233-b637-b38a3de6d99d-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"81b1531b-258a-4233-b637-b38a3de6d99d\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 12:51:00.998722 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:51:00.998445    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/81b1531b-258a-4233-b637-b38a3de6d99d-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"81b1531b-258a-4233-b637-b38a3de6d99d\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 12:51:00.998722 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:51:00.998479    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/81b1531b-258a-4233-b637-b38a3de6d99d-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"81b1531b-258a-4233-b637-b38a3de6d99d\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 12:51:00.998722 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:51:00.998534    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/81b1531b-258a-4233-b637-b38a3de6d99d-web-config\") pod \"prometheus-k8s-0\" (UID: \"81b1531b-258a-4233-b637-b38a3de6d99d\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 12:51:00.998722 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:51:00.998559    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/81b1531b-258a-4233-b637-b38a3de6d99d-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"81b1531b-258a-4233-b637-b38a3de6d99d\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 12:51:00.999166 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:51:00.998874    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/81b1531b-258a-4233-b637-b38a3de6d99d-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"81b1531b-258a-4233-b637-b38a3de6d99d\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 12:51:00.999398 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:51:00.999376    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/81b1531b-258a-4233-b637-b38a3de6d99d-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"81b1531b-258a-4233-b637-b38a3de6d99d\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 12:51:01.000299 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:51:00.999892    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/81b1531b-258a-4233-b637-b38a3de6d99d-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"81b1531b-258a-4233-b637-b38a3de6d99d\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 12:51:01.000299 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:51:00.999996    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/81b1531b-258a-4233-b637-b38a3de6d99d-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"81b1531b-258a-4233-b637-b38a3de6d99d\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 12:51:01.000299 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:51:01.000022    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/81b1531b-258a-4233-b637-b38a3de6d99d-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"81b1531b-258a-4233-b637-b38a3de6d99d\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 12:51:01.001253 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:51:01.000868    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/81b1531b-258a-4233-b637-b38a3de6d99d-config-out\") pod \"prometheus-k8s-0\" (UID: \"81b1531b-258a-4233-b637-b38a3de6d99d\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 12:51:01.001253 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:51:01.001126    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/81b1531b-258a-4233-b637-b38a3de6d99d-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"81b1531b-258a-4233-b637-b38a3de6d99d\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 12:51:01.001253 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:51:01.001242    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/81b1531b-258a-4233-b637-b38a3de6d99d-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"81b1531b-258a-4233-b637-b38a3de6d99d\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 12:51:01.001459 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:51:01.001279    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/81b1531b-258a-4233-b637-b38a3de6d99d-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"81b1531b-258a-4233-b637-b38a3de6d99d\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 12:51:01.002585 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:51:01.001976    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/81b1531b-258a-4233-b637-b38a3de6d99d-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"81b1531b-258a-4233-b637-b38a3de6d99d\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 12:51:01.002585 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:51:01.002283    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/81b1531b-258a-4233-b637-b38a3de6d99d-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"81b1531b-258a-4233-b637-b38a3de6d99d\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 12:51:01.002585 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:51:01.002342    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/81b1531b-258a-4233-b637-b38a3de6d99d-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"81b1531b-258a-4233-b637-b38a3de6d99d\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 12:51:01.002585 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:51:01.002423    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/81b1531b-258a-4233-b637-b38a3de6d99d-config\") pod \"prometheus-k8s-0\" (UID: \"81b1531b-258a-4233-b637-b38a3de6d99d\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 12:51:01.002807 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:51:01.002627    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/81b1531b-258a-4233-b637-b38a3de6d99d-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"81b1531b-258a-4233-b637-b38a3de6d99d\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 12:51:01.002843 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:51:01.002803    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/81b1531b-258a-4233-b637-b38a3de6d99d-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"81b1531b-258a-4233-b637-b38a3de6d99d\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 12:51:01.003298 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:51:01.003279    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/81b1531b-258a-4233-b637-b38a3de6d99d-web-config\") pod \"prometheus-k8s-0\" (UID: \"81b1531b-258a-4233-b637-b38a3de6d99d\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 12:51:01.003841 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:51:01.003817    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/81b1531b-258a-4233-b637-b38a3de6d99d-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"81b1531b-258a-4233-b637-b38a3de6d99d\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 12:51:01.003907 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:51:01.003884    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/81b1531b-258a-4233-b637-b38a3de6d99d-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"81b1531b-258a-4233-b637-b38a3de6d99d\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 12:51:01.004243 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:51:01.004227    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/81b1531b-258a-4233-b637-b38a3de6d99d-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"81b1531b-258a-4233-b637-b38a3de6d99d\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 12:51:01.014170 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:51:01.014152    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2vjxr\" (UniqueName: \"kubernetes.io/projected/81b1531b-258a-4233-b637-b38a3de6d99d-kube-api-access-2vjxr\") pod \"prometheus-k8s-0\" (UID: \"81b1531b-258a-4233-b637-b38a3de6d99d\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 12:51:01.140553 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:51:01.140524    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 12:51:01.420751 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:51:01.420724    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 16 12:51:01.423091 ip-10-0-133-218 kubenswrapper[2580]: W0916 12:51:01.423065    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod81b1531b_258a_4233_b637_b38a3de6d99d.slice/crio-75f1dc3c117290941adc02f138208b736bcce1610c62f5b29330d3ae399fcf73 WatchSource:0}: Error finding container 75f1dc3c117290941adc02f138208b736bcce1610c62f5b29330d3ae399fcf73: Status 404 returned error can't find the container with id 75f1dc3c117290941adc02f138208b736bcce1610c62f5b29330d3ae399fcf73
Sep 16 12:51:01.471257 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:51:01.471233    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"81b1531b-258a-4233-b637-b38a3de6d99d","Type":"ContainerStarted","Data":"75f1dc3c117290941adc02f138208b736bcce1610c62f5b29330d3ae399fcf73"}
Sep 16 12:51:01.598863 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:51:01.598833    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="84b4b343-d43c-4639-a768-3f52806bd8c7" path="/var/lib/kubelet/pods/84b4b343-d43c-4639-a768-3f52806bd8c7/volumes"
Sep 16 12:51:02.478169 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:51:02.478132    2580 generic.go:356] "Generic (PLEG): container finished" podID="81b1531b-258a-4233-b637-b38a3de6d99d" containerID="af350874ae55aabac23c228ce6c6d854f8a6bbf5588597a6f5f854c8c8af3746" exitCode=0
Sep 16 12:51:02.478555 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:51:02.478210    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"81b1531b-258a-4233-b637-b38a3de6d99d","Type":"ContainerDied","Data":"af350874ae55aabac23c228ce6c6d854f8a6bbf5588597a6f5f854c8c8af3746"}
Sep 16 12:51:03.485740 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:51:03.485705    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"81b1531b-258a-4233-b637-b38a3de6d99d","Type":"ContainerStarted","Data":"6830daaaed74f6dc57ac30af82cdcae62ef40fa64a6aa70a3f792399d6a24ea9"}
Sep 16 12:51:03.486086 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:51:03.485805    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"81b1531b-258a-4233-b637-b38a3de6d99d","Type":"ContainerStarted","Data":"3e845ed37eef1808331e78e3c1933b516edfe3118d3cd040d566fb6beea0756e"}
Sep 16 12:51:03.486086 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:51:03.485838    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"81b1531b-258a-4233-b637-b38a3de6d99d","Type":"ContainerStarted","Data":"0ef4d57e95963202431a32c8c73221dafe03e4671952fc554e9a1ff1bc00deb5"}
Sep 16 12:51:03.486086 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:51:03.485861    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"81b1531b-258a-4233-b637-b38a3de6d99d","Type":"ContainerStarted","Data":"1abb58b27a2d98921703d49a6e910d28c12f5e8e87b20e76eafdd9afe40ccfc5"}
Sep 16 12:51:03.486086 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:51:03.485892    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"81b1531b-258a-4233-b637-b38a3de6d99d","Type":"ContainerStarted","Data":"172258d59c75eb5377c2f75f7f287c1b11e2ccf1934a0a941ad21caacd691b96"}
Sep 16 12:51:03.486086 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:51:03.485925    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"81b1531b-258a-4233-b637-b38a3de6d99d","Type":"ContainerStarted","Data":"1a5a65988f93308fb4495fb909a78289292e1d7e40c9e8eff149ecf40e20ebd0"}
Sep 16 12:51:03.522875 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:51:03.522817    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-k8s-0" podStartSLOduration=3.503018091 podStartE2EDuration="3.52279794s" totalImagesPullingTime="0s" totalInitContainerRuntime="19.779849ms" isStatefulPod=false podCreationTimestamp="2026-09-16 12:51:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 12:51:03.519587069 +0000 UTC m=+202.442459537" watchObservedRunningTime="2026-09-16 12:51:03.52279794 +0000 UTC m=+202.445670406"
Sep 16 12:51:06.141641 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:51:06.141598    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 12:51:12.677531 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:51:12.677500    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-c9f8d47dd-z85ph"]
Sep 16 12:51:37.696462 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:51:37.696418    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-c9f8d47dd-z85ph" podUID="34893d6e-7a42-47d3-9ad8-cd1845c6ccd8" containerName="console" containerID="cri-o://d2f72e49ac444d32872f2cc454acc1a994309f2af2a91e85d0249fb78ab4a984" gracePeriod=15
Sep 16 12:51:37.932415 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:51:37.932391    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-c9f8d47dd-z85ph"
Sep 16 12:51:37.967835 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:51:37.967771    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/34893d6e-7a42-47d3-9ad8-cd1845c6ccd8-oauth-serving-cert\") pod \"34893d6e-7a42-47d3-9ad8-cd1845c6ccd8\" (UID: \"34893d6e-7a42-47d3-9ad8-cd1845c6ccd8\") "
Sep 16 12:51:37.967835 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:51:37.967803    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hccqg\" (UniqueName: \"kubernetes.io/projected/34893d6e-7a42-47d3-9ad8-cd1845c6ccd8-kube-api-access-hccqg\") pod \"34893d6e-7a42-47d3-9ad8-cd1845c6ccd8\" (UID: \"34893d6e-7a42-47d3-9ad8-cd1845c6ccd8\") "
Sep 16 12:51:37.967835 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:51:37.967827    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/34893d6e-7a42-47d3-9ad8-cd1845c6ccd8-console-serving-cert\") pod \"34893d6e-7a42-47d3-9ad8-cd1845c6ccd8\" (UID: \"34893d6e-7a42-47d3-9ad8-cd1845c6ccd8\") "
Sep 16 12:51:37.968106 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:51:37.967857    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/34893d6e-7a42-47d3-9ad8-cd1845c6ccd8-console-config\") pod \"34893d6e-7a42-47d3-9ad8-cd1845c6ccd8\" (UID: \"34893d6e-7a42-47d3-9ad8-cd1845c6ccd8\") "
Sep 16 12:51:37.968106 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:51:37.967876    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/34893d6e-7a42-47d3-9ad8-cd1845c6ccd8-trusted-ca-bundle\") pod \"34893d6e-7a42-47d3-9ad8-cd1845c6ccd8\" (UID: \"34893d6e-7a42-47d3-9ad8-cd1845c6ccd8\") "
Sep 16 12:51:37.968106 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:51:37.967894    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/34893d6e-7a42-47d3-9ad8-cd1845c6ccd8-console-oauth-config\") pod \"34893d6e-7a42-47d3-9ad8-cd1845c6ccd8\" (UID: \"34893d6e-7a42-47d3-9ad8-cd1845c6ccd8\") "
Sep 16 12:51:37.968106 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:51:37.967918    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/34893d6e-7a42-47d3-9ad8-cd1845c6ccd8-service-ca\") pod \"34893d6e-7a42-47d3-9ad8-cd1845c6ccd8\" (UID: \"34893d6e-7a42-47d3-9ad8-cd1845c6ccd8\") "
Sep 16 12:51:37.968307 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:51:37.968134    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/34893d6e-7a42-47d3-9ad8-cd1845c6ccd8-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "34893d6e-7a42-47d3-9ad8-cd1845c6ccd8" (UID: "34893d6e-7a42-47d3-9ad8-cd1845c6ccd8"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 12:51:37.968307 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:51:37.968252    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/34893d6e-7a42-47d3-9ad8-cd1845c6ccd8-console-config" (OuterVolumeSpecName: "console-config") pod "34893d6e-7a42-47d3-9ad8-cd1845c6ccd8" (UID: "34893d6e-7a42-47d3-9ad8-cd1845c6ccd8"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 12:51:37.968412 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:51:37.968315    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/34893d6e-7a42-47d3-9ad8-cd1845c6ccd8-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "34893d6e-7a42-47d3-9ad8-cd1845c6ccd8" (UID: "34893d6e-7a42-47d3-9ad8-cd1845c6ccd8"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 12:51:37.968412 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:51:37.968344    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/34893d6e-7a42-47d3-9ad8-cd1845c6ccd8-service-ca" (OuterVolumeSpecName: "service-ca") pod "34893d6e-7a42-47d3-9ad8-cd1845c6ccd8" (UID: "34893d6e-7a42-47d3-9ad8-cd1845c6ccd8"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 12:51:37.969946 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:51:37.969921    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/34893d6e-7a42-47d3-9ad8-cd1845c6ccd8-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "34893d6e-7a42-47d3-9ad8-cd1845c6ccd8" (UID: "34893d6e-7a42-47d3-9ad8-cd1845c6ccd8"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 12:51:37.970177 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:51:37.970153    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/34893d6e-7a42-47d3-9ad8-cd1845c6ccd8-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "34893d6e-7a42-47d3-9ad8-cd1845c6ccd8" (UID: "34893d6e-7a42-47d3-9ad8-cd1845c6ccd8"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 12:51:37.970297 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:51:37.970283    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/34893d6e-7a42-47d3-9ad8-cd1845c6ccd8-kube-api-access-hccqg" (OuterVolumeSpecName: "kube-api-access-hccqg") pod "34893d6e-7a42-47d3-9ad8-cd1845c6ccd8" (UID: "34893d6e-7a42-47d3-9ad8-cd1845c6ccd8"). InnerVolumeSpecName "kube-api-access-hccqg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 12:51:38.068758 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:51:38.068724    2580 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/34893d6e-7a42-47d3-9ad8-cd1845c6ccd8-console-serving-cert\") on node \"ip-10-0-133-218.ec2.internal\" DevicePath \"\""
Sep 16 12:51:38.068758 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:51:38.068755    2580 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/34893d6e-7a42-47d3-9ad8-cd1845c6ccd8-console-config\") on node \"ip-10-0-133-218.ec2.internal\" DevicePath \"\""
Sep 16 12:51:38.068758 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:51:38.068765    2580 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/34893d6e-7a42-47d3-9ad8-cd1845c6ccd8-trusted-ca-bundle\") on node \"ip-10-0-133-218.ec2.internal\" DevicePath \"\""
Sep 16 12:51:38.068984 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:51:38.068773    2580 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/34893d6e-7a42-47d3-9ad8-cd1845c6ccd8-console-oauth-config\") on node \"ip-10-0-133-218.ec2.internal\" DevicePath \"\""
Sep 16 12:51:38.068984 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:51:38.068782    2580 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/34893d6e-7a42-47d3-9ad8-cd1845c6ccd8-service-ca\") on node \"ip-10-0-133-218.ec2.internal\" DevicePath \"\""
Sep 16 12:51:38.068984 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:51:38.068790    2580 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/34893d6e-7a42-47d3-9ad8-cd1845c6ccd8-oauth-serving-cert\") on node \"ip-10-0-133-218.ec2.internal\" DevicePath \"\""
Sep 16 12:51:38.068984 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:51:38.068798    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hccqg\" (UniqueName: \"kubernetes.io/projected/34893d6e-7a42-47d3-9ad8-cd1845c6ccd8-kube-api-access-hccqg\") on node \"ip-10-0-133-218.ec2.internal\" DevicePath \"\""
Sep 16 12:51:38.630940 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:51:38.630907    2580 generic.go:356] "Generic (PLEG): container finished" podID="34893d6e-7a42-47d3-9ad8-cd1845c6ccd8" containerID="d2f72e49ac444d32872f2cc454acc1a994309f2af2a91e85d0249fb78ab4a984" exitCode=0
Sep 16 12:51:38.631114 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:51:38.630953    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-c9f8d47dd-z85ph"
Sep 16 12:51:38.631114 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:51:38.630990    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-c9f8d47dd-z85ph" event={"ID":"34893d6e-7a42-47d3-9ad8-cd1845c6ccd8","Type":"ContainerDied","Data":"d2f72e49ac444d32872f2cc454acc1a994309f2af2a91e85d0249fb78ab4a984"}
Sep 16 12:51:38.631114 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:51:38.631036    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-c9f8d47dd-z85ph" event={"ID":"34893d6e-7a42-47d3-9ad8-cd1845c6ccd8","Type":"ContainerDied","Data":"09af190660de48385938b802a9ef300f5cdb3ca8e20031fd0fab37ec32c3efba"}
Sep 16 12:51:38.631114 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:51:38.631056    2580 scope.go:117] "RemoveContainer" containerID="d2f72e49ac444d32872f2cc454acc1a994309f2af2a91e85d0249fb78ab4a984"
Sep 16 12:51:38.639642 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:51:38.639625    2580 scope.go:117] "RemoveContainer" containerID="d2f72e49ac444d32872f2cc454acc1a994309f2af2a91e85d0249fb78ab4a984"
Sep 16 12:51:38.639901 ip-10-0-133-218 kubenswrapper[2580]: E0916 12:51:38.639884    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d2f72e49ac444d32872f2cc454acc1a994309f2af2a91e85d0249fb78ab4a984\": container with ID starting with d2f72e49ac444d32872f2cc454acc1a994309f2af2a91e85d0249fb78ab4a984 not found: ID does not exist" containerID="d2f72e49ac444d32872f2cc454acc1a994309f2af2a91e85d0249fb78ab4a984"
Sep 16 12:51:38.639947 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:51:38.639910    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d2f72e49ac444d32872f2cc454acc1a994309f2af2a91e85d0249fb78ab4a984"} err="failed to get container status \"d2f72e49ac444d32872f2cc454acc1a994309f2af2a91e85d0249fb78ab4a984\": rpc error: code = NotFound desc = could not find container \"d2f72e49ac444d32872f2cc454acc1a994309f2af2a91e85d0249fb78ab4a984\": container with ID starting with d2f72e49ac444d32872f2cc454acc1a994309f2af2a91e85d0249fb78ab4a984 not found: ID does not exist"
Sep 16 12:51:38.651330 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:51:38.651308    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-c9f8d47dd-z85ph"]
Sep 16 12:51:38.655160 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:51:38.655139    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-c9f8d47dd-z85ph"]
Sep 16 12:51:39.599415 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:51:39.599381    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="34893d6e-7a42-47d3-9ad8-cd1845c6ccd8" path="/var/lib/kubelet/pods/34893d6e-7a42-47d3-9ad8-cd1845c6ccd8/volumes"
Sep 16 12:52:01.140768 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:52:01.140725    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 12:52:01.155837 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:52:01.155813    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 12:52:01.723701 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:52:01.723674    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 12:52:41.504150 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:52:41.504117    2580 kubelet.go:1648] "Image garbage collection succeeded"
Sep 16 12:56:14.246879 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:56:14.246845    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/odh-model-controller-7b966d7984-kkx92"]
Sep 16 12:56:14.247406 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:56:14.247170    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="34893d6e-7a42-47d3-9ad8-cd1845c6ccd8" containerName="console"
Sep 16 12:56:14.247406 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:56:14.247180    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="34893d6e-7a42-47d3-9ad8-cd1845c6ccd8" containerName="console"
Sep 16 12:56:14.247406 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:56:14.247235    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="34893d6e-7a42-47d3-9ad8-cd1845c6ccd8" containerName="console"
Sep 16 12:56:14.250119 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:56:14.250100    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/odh-model-controller-7b966d7984-kkx92"
Sep 16 12:56:14.253168 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:56:14.253140    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"odh-model-controller-metrics-tls\""
Sep 16 12:56:14.253416 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:56:14.253399    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"odh-model-controller-webhook-cert\""
Sep 16 12:56:14.253475 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:56:14.253401    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"odh-model-controller-dockercfg-8g7sq\""
Sep 16 12:56:14.254308 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:56:14.254295    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve\"/\"kube-root-ca.crt\""
Sep 16 12:56:14.254381 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:56:14.254335    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve\"/\"openshift-service-ca.crt\""
Sep 16 12:56:14.269434 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:56:14.269413    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/odh-model-controller-7b966d7984-kkx92"]
Sep 16 12:56:14.352836 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:56:14.352800    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/988887ee-5cc2-461e-a0e5-5338d5c59ab8-cert\") pod \"odh-model-controller-7b966d7984-kkx92\" (UID: \"988887ee-5cc2-461e-a0e5-5338d5c59ab8\") " pod="kserve/odh-model-controller-7b966d7984-kkx92"
Sep 16 12:56:14.353017 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:56:14.352856    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/988887ee-5cc2-461e-a0e5-5338d5c59ab8-metrics-certs\") pod \"odh-model-controller-7b966d7984-kkx92\" (UID: \"988887ee-5cc2-461e-a0e5-5338d5c59ab8\") " pod="kserve/odh-model-controller-7b966d7984-kkx92"
Sep 16 12:56:14.353017 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:56:14.352900    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6bmx6\" (UniqueName: \"kubernetes.io/projected/988887ee-5cc2-461e-a0e5-5338d5c59ab8-kube-api-access-6bmx6\") pod \"odh-model-controller-7b966d7984-kkx92\" (UID: \"988887ee-5cc2-461e-a0e5-5338d5c59ab8\") " pod="kserve/odh-model-controller-7b966d7984-kkx92"
Sep 16 12:56:14.454033 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:56:14.453997    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6bmx6\" (UniqueName: \"kubernetes.io/projected/988887ee-5cc2-461e-a0e5-5338d5c59ab8-kube-api-access-6bmx6\") pod \"odh-model-controller-7b966d7984-kkx92\" (UID: \"988887ee-5cc2-461e-a0e5-5338d5c59ab8\") " pod="kserve/odh-model-controller-7b966d7984-kkx92"
Sep 16 12:56:14.454192 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:56:14.454053    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/988887ee-5cc2-461e-a0e5-5338d5c59ab8-cert\") pod \"odh-model-controller-7b966d7984-kkx92\" (UID: \"988887ee-5cc2-461e-a0e5-5338d5c59ab8\") " pod="kserve/odh-model-controller-7b966d7984-kkx92"
Sep 16 12:56:14.454192 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:56:14.454085    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/988887ee-5cc2-461e-a0e5-5338d5c59ab8-metrics-certs\") pod \"odh-model-controller-7b966d7984-kkx92\" (UID: \"988887ee-5cc2-461e-a0e5-5338d5c59ab8\") " pod="kserve/odh-model-controller-7b966d7984-kkx92"
Sep 16 12:56:14.454192 ip-10-0-133-218 kubenswrapper[2580]: E0916 12:56:14.454163    2580 secret.go:189] Couldn't get secret kserve/odh-model-controller-metrics-tls: secret "odh-model-controller-metrics-tls" not found
Sep 16 12:56:14.454313 ip-10-0-133-218 kubenswrapper[2580]: E0916 12:56:14.454214    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/988887ee-5cc2-461e-a0e5-5338d5c59ab8-metrics-certs podName:988887ee-5cc2-461e-a0e5-5338d5c59ab8 nodeName:}" failed. No retries permitted until 2026-09-16 12:56:14.954200183 +0000 UTC m=+513.877072629 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/988887ee-5cc2-461e-a0e5-5338d5c59ab8-metrics-certs") pod "odh-model-controller-7b966d7984-kkx92" (UID: "988887ee-5cc2-461e-a0e5-5338d5c59ab8") : secret "odh-model-controller-metrics-tls" not found
Sep 16 12:56:14.456421 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:56:14.456401    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/988887ee-5cc2-461e-a0e5-5338d5c59ab8-cert\") pod \"odh-model-controller-7b966d7984-kkx92\" (UID: \"988887ee-5cc2-461e-a0e5-5338d5c59ab8\") " pod="kserve/odh-model-controller-7b966d7984-kkx92"
Sep 16 12:56:14.479546 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:56:14.479519    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6bmx6\" (UniqueName: \"kubernetes.io/projected/988887ee-5cc2-461e-a0e5-5338d5c59ab8-kube-api-access-6bmx6\") pod \"odh-model-controller-7b966d7984-kkx92\" (UID: \"988887ee-5cc2-461e-a0e5-5338d5c59ab8\") " pod="kserve/odh-model-controller-7b966d7984-kkx92"
Sep 16 12:56:14.959163 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:56:14.959129    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/988887ee-5cc2-461e-a0e5-5338d5c59ab8-metrics-certs\") pod \"odh-model-controller-7b966d7984-kkx92\" (UID: \"988887ee-5cc2-461e-a0e5-5338d5c59ab8\") " pod="kserve/odh-model-controller-7b966d7984-kkx92"
Sep 16 12:56:14.961381 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:56:14.961354    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/988887ee-5cc2-461e-a0e5-5338d5c59ab8-metrics-certs\") pod \"odh-model-controller-7b966d7984-kkx92\" (UID: \"988887ee-5cc2-461e-a0e5-5338d5c59ab8\") " pod="kserve/odh-model-controller-7b966d7984-kkx92"
Sep 16 12:56:15.160115 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:56:15.160073    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/odh-model-controller-7b966d7984-kkx92"
Sep 16 12:56:15.430642 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:56:15.430609    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/odh-model-controller-7b966d7984-kkx92"]
Sep 16 12:56:15.433277 ip-10-0-133-218 kubenswrapper[2580]: W0916 12:56:15.433248    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod988887ee_5cc2_461e_a0e5_5338d5c59ab8.slice/crio-9d5a867d9ab3ecb14f91323b2612761c59e57570e0de02160748a31cbf52d697 WatchSource:0}: Error finding container 9d5a867d9ab3ecb14f91323b2612761c59e57570e0de02160748a31cbf52d697: Status 404 returned error can't find the container with id 9d5a867d9ab3ecb14f91323b2612761c59e57570e0de02160748a31cbf52d697
Sep 16 12:56:15.434481 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:56:15.434464    2580 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 16 12:56:15.622330 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:56:15.622298    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/odh-model-controller-7b966d7984-kkx92" event={"ID":"988887ee-5cc2-461e-a0e5-5338d5c59ab8","Type":"ContainerStarted","Data":"9d5a867d9ab3ecb14f91323b2612761c59e57570e0de02160748a31cbf52d697"}
Sep 16 12:56:18.634709 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:56:18.634672    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/odh-model-controller-7b966d7984-kkx92" event={"ID":"988887ee-5cc2-461e-a0e5-5338d5c59ab8","Type":"ContainerStarted","Data":"375d025e0323cc8a6c09b1bb9fdf5a76060377bae35c3abf8baeb3a1942d3809"}
Sep 16 12:56:18.635141 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:56:18.634794    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve/odh-model-controller-7b966d7984-kkx92"
Sep 16 12:56:18.657813 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:56:18.657769    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/odh-model-controller-7b966d7984-kkx92" podStartSLOduration=2.0892036 podStartE2EDuration="4.65775535s" totalImagesPullingTime="2.56855175s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 12:56:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 12:56:18.655539179 +0000 UTC m=+517.578411648" watchObservedRunningTime="2026-09-16 12:56:18.65775535 +0000 UTC m=+517.580627867"
Sep 16 12:56:29.640294 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:56:29.640262    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve/odh-model-controller-7b966d7984-kkx92"
Sep 16 12:56:30.515049 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:56:30.515009    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/s3-init-zwqmj"]
Sep 16 12:56:30.524136 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:56:30.524108    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/s3-init-zwqmj"
Sep 16 12:56:30.528167 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:56:30.528145    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"hf-token\""
Sep 16 12:56:30.528288 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:56:30.528229    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"mlpipeline-s3-artifact\""
Sep 16 12:56:30.528433 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:56:30.528418    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"default-dockercfg-h4cfl\""
Sep 16 12:56:30.546142 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:56:30.546119    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/s3-init-zwqmj"]
Sep 16 12:56:30.683132 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:56:30.683102    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a1bd02a1-01d1-46d7-a28a-5b54fe56c086-model-cache\") pod \"s3-init-zwqmj\" (UID: \"a1bd02a1-01d1-46d7-a28a-5b54fe56c086\") " pod="kserve/s3-init-zwqmj"
Sep 16 12:56:30.683466 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:56:30.683172    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wf4cw\" (UniqueName: \"kubernetes.io/projected/a1bd02a1-01d1-46d7-a28a-5b54fe56c086-kube-api-access-wf4cw\") pod \"s3-init-zwqmj\" (UID: \"a1bd02a1-01d1-46d7-a28a-5b54fe56c086\") " pod="kserve/s3-init-zwqmj"
Sep 16 12:56:30.784450 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:56:30.784422    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wf4cw\" (UniqueName: \"kubernetes.io/projected/a1bd02a1-01d1-46d7-a28a-5b54fe56c086-kube-api-access-wf4cw\") pod \"s3-init-zwqmj\" (UID: \"a1bd02a1-01d1-46d7-a28a-5b54fe56c086\") " pod="kserve/s3-init-zwqmj"
Sep 16 12:56:30.784622 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:56:30.784470    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a1bd02a1-01d1-46d7-a28a-5b54fe56c086-model-cache\") pod \"s3-init-zwqmj\" (UID: \"a1bd02a1-01d1-46d7-a28a-5b54fe56c086\") " pod="kserve/s3-init-zwqmj"
Sep 16 12:56:30.784879 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:56:30.784853    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a1bd02a1-01d1-46d7-a28a-5b54fe56c086-model-cache\") pod \"s3-init-zwqmj\" (UID: \"a1bd02a1-01d1-46d7-a28a-5b54fe56c086\") " pod="kserve/s3-init-zwqmj"
Sep 16 12:56:30.792879 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:56:30.792859    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wf4cw\" (UniqueName: \"kubernetes.io/projected/a1bd02a1-01d1-46d7-a28a-5b54fe56c086-kube-api-access-wf4cw\") pod \"s3-init-zwqmj\" (UID: \"a1bd02a1-01d1-46d7-a28a-5b54fe56c086\") " pod="kserve/s3-init-zwqmj"
Sep 16 12:56:30.842433 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:56:30.842402    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/s3-init-zwqmj"
Sep 16 12:56:31.116119 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:56:31.116040    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/s3-init-zwqmj"]
Sep 16 12:56:31.119976 ip-10-0-133-218 kubenswrapper[2580]: W0916 12:56:31.119942    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poda1bd02a1_01d1_46d7_a28a_5b54fe56c086.slice/crio-24b1c5176c5c7d338ce1840944fdcd14fdc1dbc6edcae2d19906f0c038039ea3 WatchSource:0}: Error finding container 24b1c5176c5c7d338ce1840944fdcd14fdc1dbc6edcae2d19906f0c038039ea3: Status 404 returned error can't find the container with id 24b1c5176c5c7d338ce1840944fdcd14fdc1dbc6edcae2d19906f0c038039ea3
Sep 16 12:56:31.685366 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:56:31.685328    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-zwqmj" event={"ID":"a1bd02a1-01d1-46d7-a28a-5b54fe56c086","Type":"ContainerStarted","Data":"24b1c5176c5c7d338ce1840944fdcd14fdc1dbc6edcae2d19906f0c038039ea3"}
Sep 16 12:56:34.700216 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:56:34.700137    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-zwqmj" event={"ID":"a1bd02a1-01d1-46d7-a28a-5b54fe56c086","Type":"ContainerStarted","Data":"195a69713c5bad1905d5cf45d54c4d6ecae88aabfe1f63c8101f71f9fe24c780"}
Sep 16 12:56:36.708626 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:56:36.708533    2580 generic.go:356] "Generic (PLEG): container finished" podID="a1bd02a1-01d1-46d7-a28a-5b54fe56c086" containerID="195a69713c5bad1905d5cf45d54c4d6ecae88aabfe1f63c8101f71f9fe24c780" exitCode=0
Sep 16 12:56:36.708626 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:56:36.708612    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-zwqmj" event={"ID":"a1bd02a1-01d1-46d7-a28a-5b54fe56c086","Type":"ContainerDied","Data":"195a69713c5bad1905d5cf45d54c4d6ecae88aabfe1f63c8101f71f9fe24c780"}
Sep 16 12:56:41.732434 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:56:41.732397    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-zwqmj" event={"ID":"a1bd02a1-01d1-46d7-a28a-5b54fe56c086","Type":"ContainerStarted","Data":"68dd9e07f83d871fd9cf1c9472c0b269db280eb5746cad76a8fe4a4dc2bdceb6"}
Sep 16 12:57:46.978614 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:57:46.978579    2580 generic.go:356] "Generic (PLEG): container finished" podID="a1bd02a1-01d1-46d7-a28a-5b54fe56c086" containerID="68dd9e07f83d871fd9cf1c9472c0b269db280eb5746cad76a8fe4a4dc2bdceb6" exitCode=0
Sep 16 12:57:46.979045 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:57:46.978656    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-zwqmj" event={"ID":"a1bd02a1-01d1-46d7-a28a-5b54fe56c086","Type":"ContainerDied","Data":"68dd9e07f83d871fd9cf1c9472c0b269db280eb5746cad76a8fe4a4dc2bdceb6"}
Sep 16 12:57:48.100827 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:57:48.100803    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/s3-init-zwqmj"
Sep 16 12:57:48.186224 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:57:48.186189    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a1bd02a1-01d1-46d7-a28a-5b54fe56c086-model-cache\") pod \"a1bd02a1-01d1-46d7-a28a-5b54fe56c086\" (UID: \"a1bd02a1-01d1-46d7-a28a-5b54fe56c086\") "
Sep 16 12:57:48.186410 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:57:48.186294    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wf4cw\" (UniqueName: \"kubernetes.io/projected/a1bd02a1-01d1-46d7-a28a-5b54fe56c086-kube-api-access-wf4cw\") pod \"a1bd02a1-01d1-46d7-a28a-5b54fe56c086\" (UID: \"a1bd02a1-01d1-46d7-a28a-5b54fe56c086\") "
Sep 16 12:57:48.188410 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:57:48.188383    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a1bd02a1-01d1-46d7-a28a-5b54fe56c086-kube-api-access-wf4cw" (OuterVolumeSpecName: "kube-api-access-wf4cw") pod "a1bd02a1-01d1-46d7-a28a-5b54fe56c086" (UID: "a1bd02a1-01d1-46d7-a28a-5b54fe56c086"). InnerVolumeSpecName "kube-api-access-wf4cw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 12:57:48.204555 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:57:48.204527    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a1bd02a1-01d1-46d7-a28a-5b54fe56c086-model-cache" (OuterVolumeSpecName: "model-cache") pod "a1bd02a1-01d1-46d7-a28a-5b54fe56c086" (UID: "a1bd02a1-01d1-46d7-a28a-5b54fe56c086"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 12:57:48.286968 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:57:48.286937    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wf4cw\" (UniqueName: \"kubernetes.io/projected/a1bd02a1-01d1-46d7-a28a-5b54fe56c086-kube-api-access-wf4cw\") on node \"ip-10-0-133-218.ec2.internal\" DevicePath \"\""
Sep 16 12:57:48.286968 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:57:48.286968    2580 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a1bd02a1-01d1-46d7-a28a-5b54fe56c086-model-cache\") on node \"ip-10-0-133-218.ec2.internal\" DevicePath \"\""
Sep 16 12:57:48.987500 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:57:48.987474    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/s3-init-zwqmj"
Sep 16 12:57:48.987672 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:57:48.987471    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-zwqmj" event={"ID":"a1bd02a1-01d1-46d7-a28a-5b54fe56c086","Type":"ContainerDied","Data":"24b1c5176c5c7d338ce1840944fdcd14fdc1dbc6edcae2d19906f0c038039ea3"}
Sep 16 12:57:48.987672 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:57:48.987591    2580 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="24b1c5176c5c7d338ce1840944fdcd14fdc1dbc6edcae2d19906f0c038039ea3"
Sep 16 12:58:00.928336 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:58:00.928302    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-7ffd87447b-rmdk5"]
Sep 16 12:58:00.928834 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:58:00.928611    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a1bd02a1-01d1-46d7-a28a-5b54fe56c086" containerName="download-hf-model"
Sep 16 12:58:00.928834 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:58:00.928623    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="a1bd02a1-01d1-46d7-a28a-5b54fe56c086" containerName="download-hf-model"
Sep 16 12:58:00.928834 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:58:00.928639    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a1bd02a1-01d1-46d7-a28a-5b54fe56c086" containerName="s3-init"
Sep 16 12:58:00.928834 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:58:00.928644    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="a1bd02a1-01d1-46d7-a28a-5b54fe56c086" containerName="s3-init"
Sep 16 12:58:00.928834 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:58:00.928694    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="a1bd02a1-01d1-46d7-a28a-5b54fe56c086" containerName="s3-init"
Sep 16 12:58:00.930870 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:58:00.930847    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-7ffd87447b-rmdk5"
Sep 16 12:58:00.933462 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:58:00.933437    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Sep 16 12:58:00.933986 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:58:00.933969    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-rollback-kube-rbac-proxy-sar-config\""
Sep 16 12:58:00.934053 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:58:00.933991    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-9sk5b\""
Sep 16 12:58:00.934053 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:58:00.934003    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Sep 16 12:58:00.934053 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:58:00.934019    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-rollback-v2-predictor-serving-cert\""
Sep 16 12:58:00.934375 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:58:00.934359    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Sep 16 12:58:00.934440 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:58:00.934421    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Sep 16 12:58:00.945215 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:58:00.945195    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-7ffd87447b-rmdk5"]
Sep 16 12:58:00.988854 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:58:00.988827    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a65e5a0e-beb3-4b27-a076-130b38b51d49-proxy-tls\") pod \"isvc-canary-rollback-v2-predictor-7ffd87447b-rmdk5\" (UID: \"a65e5a0e-beb3-4b27-a076-130b38b51d49\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-7ffd87447b-rmdk5"
Sep 16 12:58:00.988984 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:58:00.988866    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/a65e5a0e-beb3-4b27-a076-130b38b51d49-cabundle-cert\") pod \"isvc-canary-rollback-v2-predictor-7ffd87447b-rmdk5\" (UID: \"a65e5a0e-beb3-4b27-a076-130b38b51d49\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-7ffd87447b-rmdk5"
Sep 16 12:58:00.988984 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:58:00.988886    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t2dp2\" (UniqueName: \"kubernetes.io/projected/a65e5a0e-beb3-4b27-a076-130b38b51d49-kube-api-access-t2dp2\") pod \"isvc-canary-rollback-v2-predictor-7ffd87447b-rmdk5\" (UID: \"a65e5a0e-beb3-4b27-a076-130b38b51d49\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-7ffd87447b-rmdk5"
Sep 16 12:58:00.988984 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:58:00.988948    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a65e5a0e-beb3-4b27-a076-130b38b51d49-kube-rbac-proxy-sar-config\") pod \"isvc-canary-rollback-v2-predictor-7ffd87447b-rmdk5\" (UID: \"a65e5a0e-beb3-4b27-a076-130b38b51d49\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-7ffd87447b-rmdk5"
Sep 16 12:58:00.989082 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:58:00.989000    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a65e5a0e-beb3-4b27-a076-130b38b51d49-kserve-provision-location\") pod \"isvc-canary-rollback-v2-predictor-7ffd87447b-rmdk5\" (UID: \"a65e5a0e-beb3-4b27-a076-130b38b51d49\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-7ffd87447b-rmdk5"
Sep 16 12:58:01.045981 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:58:01.045953    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-predictor-696c5cc6cc-9dh72"]
Sep 16 12:58:01.048719 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:58:01.048700    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-696c5cc6cc-9dh72"
Sep 16 12:58:01.050751 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:58:01.050729    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-rollback-predictor-serving-cert\""
Sep 16 12:58:01.064132 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:58:01.064110    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-predictor-696c5cc6cc-9dh72"]
Sep 16 12:58:01.089716 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:58:01.089690    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2dklf\" (UniqueName: \"kubernetes.io/projected/2abe9c6e-a97c-4f3b-b8eb-4c2697a6485d-kube-api-access-2dklf\") pod \"isvc-canary-rollback-predictor-696c5cc6cc-9dh72\" (UID: \"2abe9c6e-a97c-4f3b-b8eb-4c2697a6485d\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-696c5cc6cc-9dh72"
Sep 16 12:58:01.089860 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:58:01.089726    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/2abe9c6e-a97c-4f3b-b8eb-4c2697a6485d-cabundle-cert\") pod \"isvc-canary-rollback-predictor-696c5cc6cc-9dh72\" (UID: \"2abe9c6e-a97c-4f3b-b8eb-4c2697a6485d\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-696c5cc6cc-9dh72"
Sep 16 12:58:01.089860 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:58:01.089754    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a65e5a0e-beb3-4b27-a076-130b38b51d49-kserve-provision-location\") pod \"isvc-canary-rollback-v2-predictor-7ffd87447b-rmdk5\" (UID: \"a65e5a0e-beb3-4b27-a076-130b38b51d49\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-7ffd87447b-rmdk5"
Sep 16 12:58:01.089860 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:58:01.089802    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/2abe9c6e-a97c-4f3b-b8eb-4c2697a6485d-kube-rbac-proxy-sar-config\") pod \"isvc-canary-rollback-predictor-696c5cc6cc-9dh72\" (UID: \"2abe9c6e-a97c-4f3b-b8eb-4c2697a6485d\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-696c5cc6cc-9dh72"
Sep 16 12:58:01.089860 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:58:01.089848    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a65e5a0e-beb3-4b27-a076-130b38b51d49-proxy-tls\") pod \"isvc-canary-rollback-v2-predictor-7ffd87447b-rmdk5\" (UID: \"a65e5a0e-beb3-4b27-a076-130b38b51d49\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-7ffd87447b-rmdk5"
Sep 16 12:58:01.090076 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:58:01.089881    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/a65e5a0e-beb3-4b27-a076-130b38b51d49-cabundle-cert\") pod \"isvc-canary-rollback-v2-predictor-7ffd87447b-rmdk5\" (UID: \"a65e5a0e-beb3-4b27-a076-130b38b51d49\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-7ffd87447b-rmdk5"
Sep 16 12:58:01.090076 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:58:01.089899    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/2abe9c6e-a97c-4f3b-b8eb-4c2697a6485d-kserve-provision-location\") pod \"isvc-canary-rollback-predictor-696c5cc6cc-9dh72\" (UID: \"2abe9c6e-a97c-4f3b-b8eb-4c2697a6485d\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-696c5cc6cc-9dh72"
Sep 16 12:58:01.090076 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:58:01.089930    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t2dp2\" (UniqueName: \"kubernetes.io/projected/a65e5a0e-beb3-4b27-a076-130b38b51d49-kube-api-access-t2dp2\") pod \"isvc-canary-rollback-v2-predictor-7ffd87447b-rmdk5\" (UID: \"a65e5a0e-beb3-4b27-a076-130b38b51d49\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-7ffd87447b-rmdk5"
Sep 16 12:58:01.090076 ip-10-0-133-218 kubenswrapper[2580]: E0916 12:58:01.090028    2580 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-serving-cert: secret "isvc-canary-rollback-v2-predictor-serving-cert" not found
Sep 16 12:58:01.090076 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:58:01.090055    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a65e5a0e-beb3-4b27-a076-130b38b51d49-kube-rbac-proxy-sar-config\") pod \"isvc-canary-rollback-v2-predictor-7ffd87447b-rmdk5\" (UID: \"a65e5a0e-beb3-4b27-a076-130b38b51d49\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-7ffd87447b-rmdk5"
Sep 16 12:58:01.090257 ip-10-0-133-218 kubenswrapper[2580]: E0916 12:58:01.090083    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a65e5a0e-beb3-4b27-a076-130b38b51d49-proxy-tls podName:a65e5a0e-beb3-4b27-a076-130b38b51d49 nodeName:}" failed. No retries permitted until 2026-09-16 12:58:01.590065363 +0000 UTC m=+620.512937830 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/a65e5a0e-beb3-4b27-a076-130b38b51d49-proxy-tls") pod "isvc-canary-rollback-v2-predictor-7ffd87447b-rmdk5" (UID: "a65e5a0e-beb3-4b27-a076-130b38b51d49") : secret "isvc-canary-rollback-v2-predictor-serving-cert" not found
Sep 16 12:58:01.090257 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:58:01.090125    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/2abe9c6e-a97c-4f3b-b8eb-4c2697a6485d-proxy-tls\") pod \"isvc-canary-rollback-predictor-696c5cc6cc-9dh72\" (UID: \"2abe9c6e-a97c-4f3b-b8eb-4c2697a6485d\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-696c5cc6cc-9dh72"
Sep 16 12:58:01.090257 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:58:01.090152    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a65e5a0e-beb3-4b27-a076-130b38b51d49-kserve-provision-location\") pod \"isvc-canary-rollback-v2-predictor-7ffd87447b-rmdk5\" (UID: \"a65e5a0e-beb3-4b27-a076-130b38b51d49\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-7ffd87447b-rmdk5"
Sep 16 12:58:01.090767 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:58:01.090738    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a65e5a0e-beb3-4b27-a076-130b38b51d49-kube-rbac-proxy-sar-config\") pod \"isvc-canary-rollback-v2-predictor-7ffd87447b-rmdk5\" (UID: \"a65e5a0e-beb3-4b27-a076-130b38b51d49\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-7ffd87447b-rmdk5"
Sep 16 12:58:01.090944 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:58:01.090928    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/a65e5a0e-beb3-4b27-a076-130b38b51d49-cabundle-cert\") pod \"isvc-canary-rollback-v2-predictor-7ffd87447b-rmdk5\" (UID: \"a65e5a0e-beb3-4b27-a076-130b38b51d49\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-7ffd87447b-rmdk5"
Sep 16 12:58:01.098873 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:58:01.098848    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-t2dp2\" (UniqueName: \"kubernetes.io/projected/a65e5a0e-beb3-4b27-a076-130b38b51d49-kube-api-access-t2dp2\") pod \"isvc-canary-rollback-v2-predictor-7ffd87447b-rmdk5\" (UID: \"a65e5a0e-beb3-4b27-a076-130b38b51d49\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-7ffd87447b-rmdk5"
Sep 16 12:58:01.190606 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:58:01.190524    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2dklf\" (UniqueName: \"kubernetes.io/projected/2abe9c6e-a97c-4f3b-b8eb-4c2697a6485d-kube-api-access-2dklf\") pod \"isvc-canary-rollback-predictor-696c5cc6cc-9dh72\" (UID: \"2abe9c6e-a97c-4f3b-b8eb-4c2697a6485d\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-696c5cc6cc-9dh72"
Sep 16 12:58:01.190606 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:58:01.190564    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/2abe9c6e-a97c-4f3b-b8eb-4c2697a6485d-cabundle-cert\") pod \"isvc-canary-rollback-predictor-696c5cc6cc-9dh72\" (UID: \"2abe9c6e-a97c-4f3b-b8eb-4c2697a6485d\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-696c5cc6cc-9dh72"
Sep 16 12:58:01.190606 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:58:01.190587    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/2abe9c6e-a97c-4f3b-b8eb-4c2697a6485d-kube-rbac-proxy-sar-config\") pod \"isvc-canary-rollback-predictor-696c5cc6cc-9dh72\" (UID: \"2abe9c6e-a97c-4f3b-b8eb-4c2697a6485d\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-696c5cc6cc-9dh72"
Sep 16 12:58:01.190889 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:58:01.190646    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/2abe9c6e-a97c-4f3b-b8eb-4c2697a6485d-kserve-provision-location\") pod \"isvc-canary-rollback-predictor-696c5cc6cc-9dh72\" (UID: \"2abe9c6e-a97c-4f3b-b8eb-4c2697a6485d\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-696c5cc6cc-9dh72"
Sep 16 12:58:01.190889 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:58:01.190683    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/2abe9c6e-a97c-4f3b-b8eb-4c2697a6485d-proxy-tls\") pod \"isvc-canary-rollback-predictor-696c5cc6cc-9dh72\" (UID: \"2abe9c6e-a97c-4f3b-b8eb-4c2697a6485d\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-696c5cc6cc-9dh72"
Sep 16 12:58:01.191133 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:58:01.191107    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/2abe9c6e-a97c-4f3b-b8eb-4c2697a6485d-kserve-provision-location\") pod \"isvc-canary-rollback-predictor-696c5cc6cc-9dh72\" (UID: \"2abe9c6e-a97c-4f3b-b8eb-4c2697a6485d\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-696c5cc6cc-9dh72"
Sep 16 12:58:01.191349 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:58:01.191327    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/2abe9c6e-a97c-4f3b-b8eb-4c2697a6485d-kube-rbac-proxy-sar-config\") pod \"isvc-canary-rollback-predictor-696c5cc6cc-9dh72\" (UID: \"2abe9c6e-a97c-4f3b-b8eb-4c2697a6485d\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-696c5cc6cc-9dh72"
Sep 16 12:58:01.191411 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:58:01.191328    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/2abe9c6e-a97c-4f3b-b8eb-4c2697a6485d-cabundle-cert\") pod \"isvc-canary-rollback-predictor-696c5cc6cc-9dh72\" (UID: \"2abe9c6e-a97c-4f3b-b8eb-4c2697a6485d\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-696c5cc6cc-9dh72"
Sep 16 12:58:01.193146 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:58:01.193119    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/2abe9c6e-a97c-4f3b-b8eb-4c2697a6485d-proxy-tls\") pod \"isvc-canary-rollback-predictor-696c5cc6cc-9dh72\" (UID: \"2abe9c6e-a97c-4f3b-b8eb-4c2697a6485d\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-696c5cc6cc-9dh72"
Sep 16 12:58:01.209311 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:58:01.209288    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2dklf\" (UniqueName: \"kubernetes.io/projected/2abe9c6e-a97c-4f3b-b8eb-4c2697a6485d-kube-api-access-2dklf\") pod \"isvc-canary-rollback-predictor-696c5cc6cc-9dh72\" (UID: \"2abe9c6e-a97c-4f3b-b8eb-4c2697a6485d\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-696c5cc6cc-9dh72"
Sep 16 12:58:01.362010 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:58:01.361974    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-696c5cc6cc-9dh72"
Sep 16 12:58:01.594149 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:58:01.594112    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a65e5a0e-beb3-4b27-a076-130b38b51d49-proxy-tls\") pod \"isvc-canary-rollback-v2-predictor-7ffd87447b-rmdk5\" (UID: \"a65e5a0e-beb3-4b27-a076-130b38b51d49\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-7ffd87447b-rmdk5"
Sep 16 12:58:01.602981 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:58:01.602945    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a65e5a0e-beb3-4b27-a076-130b38b51d49-proxy-tls\") pod \"isvc-canary-rollback-v2-predictor-7ffd87447b-rmdk5\" (UID: \"a65e5a0e-beb3-4b27-a076-130b38b51d49\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-7ffd87447b-rmdk5"
Sep 16 12:58:01.634667 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:58:01.634642    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-predictor-696c5cc6cc-9dh72"]
Sep 16 12:58:01.637264 ip-10-0-133-218 kubenswrapper[2580]: W0916 12:58:01.637242    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2abe9c6e_a97c_4f3b_b8eb_4c2697a6485d.slice/crio-b51f7b9733bea584417f48154a9c9a1e2e1d8ad98d9fed7c0323f4088b6a6ad6 WatchSource:0}: Error finding container b51f7b9733bea584417f48154a9c9a1e2e1d8ad98d9fed7c0323f4088b6a6ad6: Status 404 returned error can't find the container with id b51f7b9733bea584417f48154a9c9a1e2e1d8ad98d9fed7c0323f4088b6a6ad6
Sep 16 12:58:01.841296 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:58:01.841256    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-7ffd87447b-rmdk5"
Sep 16 12:58:02.039526 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:58:02.039472    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-696c5cc6cc-9dh72" event={"ID":"2abe9c6e-a97c-4f3b-b8eb-4c2697a6485d","Type":"ContainerStarted","Data":"12ca280bad1ac1c8667647895c41239333364e5153baaf984805929aa4e567f6"}
Sep 16 12:58:02.039883 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:58:02.039548    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-696c5cc6cc-9dh72" event={"ID":"2abe9c6e-a97c-4f3b-b8eb-4c2697a6485d","Type":"ContainerStarted","Data":"b51f7b9733bea584417f48154a9c9a1e2e1d8ad98d9fed7c0323f4088b6a6ad6"}
Sep 16 12:58:02.139577 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:58:02.139472    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-7ffd87447b-rmdk5"]
Sep 16 12:58:02.142685 ip-10-0-133-218 kubenswrapper[2580]: W0916 12:58:02.142649    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda65e5a0e_beb3_4b27_a076_130b38b51d49.slice/crio-821a2007c904ef0f8007c65339e901e053aa8faf645ada653a88505706a13a5c WatchSource:0}: Error finding container 821a2007c904ef0f8007c65339e901e053aa8faf645ada653a88505706a13a5c: Status 404 returned error can't find the container with id 821a2007c904ef0f8007c65339e901e053aa8faf645ada653a88505706a13a5c
Sep 16 12:58:03.045747 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:58:03.045709    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-7ffd87447b-rmdk5" event={"ID":"a65e5a0e-beb3-4b27-a076-130b38b51d49","Type":"ContainerStarted","Data":"588dcbe5133407fcc21f1894f861b9df75546f4a518bf2b126a95d7d30afbccd"}
Sep 16 12:58:03.046137 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:58:03.045774    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-7ffd87447b-rmdk5" event={"ID":"a65e5a0e-beb3-4b27-a076-130b38b51d49","Type":"ContainerStarted","Data":"821a2007c904ef0f8007c65339e901e053aa8faf645ada653a88505706a13a5c"}
Sep 16 12:58:06.059596 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:58:06.059563    2580 generic.go:356] "Generic (PLEG): container finished" podID="a65e5a0e-beb3-4b27-a076-130b38b51d49" containerID="588dcbe5133407fcc21f1894f861b9df75546f4a518bf2b126a95d7d30afbccd" exitCode=0
Sep 16 12:58:06.059969 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:58:06.059640    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-7ffd87447b-rmdk5" event={"ID":"a65e5a0e-beb3-4b27-a076-130b38b51d49","Type":"ContainerDied","Data":"588dcbe5133407fcc21f1894f861b9df75546f4a518bf2b126a95d7d30afbccd"}
Sep 16 12:58:06.061161 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:58:06.061137    2580 generic.go:356] "Generic (PLEG): container finished" podID="2abe9c6e-a97c-4f3b-b8eb-4c2697a6485d" containerID="12ca280bad1ac1c8667647895c41239333364e5153baaf984805929aa4e567f6" exitCode=0
Sep 16 12:58:06.061236 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:58:06.061173    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-696c5cc6cc-9dh72" event={"ID":"2abe9c6e-a97c-4f3b-b8eb-4c2697a6485d","Type":"ContainerDied","Data":"12ca280bad1ac1c8667647895c41239333364e5153baaf984805929aa4e567f6"}
Sep 16 12:58:24.190171 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:58:24.190063    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-7ffd87447b-rmdk5" event={"ID":"a65e5a0e-beb3-4b27-a076-130b38b51d49","Type":"ContainerStarted","Data":"6f44dba2b7ae6da894ed1decc6ef145a411c318fcc172f07a6342203c585d876"}
Sep 16 12:58:24.194115 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:58:24.194047    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-696c5cc6cc-9dh72" event={"ID":"2abe9c6e-a97c-4f3b-b8eb-4c2697a6485d","Type":"ContainerStarted","Data":"8bd070f106d7fb83004929a7ecf5b2dfe29769749adbcc94d42c07444c9d1977"}
Sep 16 12:58:27.209889 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:58:27.209845    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-7ffd87447b-rmdk5" event={"ID":"a65e5a0e-beb3-4b27-a076-130b38b51d49","Type":"ContainerStarted","Data":"4e64dcdc464ed2b32a9d404bb945bf10ad11abe0d3240010e34d793c4d6b395d"}
Sep 16 12:58:27.210324 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:58:27.209943    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-7ffd87447b-rmdk5"
Sep 16 12:58:27.210324 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:58:27.210099    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-7ffd87447b-rmdk5"
Sep 16 12:58:27.211724 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:58:27.211687    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-7ffd87447b-rmdk5" podUID="a65e5a0e-beb3-4b27-a076-130b38b51d49" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.26:8080: connect: connection refused"
Sep 16 12:58:27.212142 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:58:27.212123    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-696c5cc6cc-9dh72" event={"ID":"2abe9c6e-a97c-4f3b-b8eb-4c2697a6485d","Type":"ContainerStarted","Data":"c13aaa95dea7777ff24b55a5f2c031f63868520058d55c5167a72e72b93bf5f8"}
Sep 16 12:58:27.212266 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:58:27.212248    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-696c5cc6cc-9dh72"
Sep 16 12:58:27.212333 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:58:27.212288    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-696c5cc6cc-9dh72"
Sep 16 12:58:27.213333 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:58:27.213311    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-696c5cc6cc-9dh72" podUID="2abe9c6e-a97c-4f3b-b8eb-4c2697a6485d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.27:8080: connect: connection refused"
Sep 16 12:58:27.278453 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:58:27.278389    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-7ffd87447b-rmdk5" podStartSLOduration=3.579979938 podStartE2EDuration="27.278372899s" totalImagesPullingTime="20.036133934s" totalInitContainerRuntime="3.662259027s" isStatefulPod=false podCreationTimestamp="2026-09-16 12:58:00 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 12:58:27.239695907 +0000 UTC m=+646.162568375" watchObservedRunningTime="2026-09-16 12:58:27.278372899 +0000 UTC m=+646.201245368"
Sep 16 12:58:27.279338 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:58:27.279305    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-696c5cc6cc-9dh72" podStartSLOduration=2.162860018 podStartE2EDuration="26.2792934s" totalImagesPullingTime="20.078952684s" totalInitContainerRuntime="4.037480698s" isStatefulPod=false podCreationTimestamp="2026-09-16 12:58:01 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 12:58:27.278989079 +0000 UTC m=+646.201861558" watchObservedRunningTime="2026-09-16 12:58:27.2792934 +0000 UTC m=+646.202165872"
Sep 16 12:58:28.216838 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:58:28.216798    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-696c5cc6cc-9dh72" podUID="2abe9c6e-a97c-4f3b-b8eb-4c2697a6485d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.27:8080: connect: connection refused"
Sep 16 12:58:28.217237 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:58:28.216937    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-7ffd87447b-rmdk5" podUID="a65e5a0e-beb3-4b27-a076-130b38b51d49" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.26:8080: connect: connection refused"
Sep 16 12:58:33.221041 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:58:33.221014    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-696c5cc6cc-9dh72"
Sep 16 12:58:33.221549 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:58:33.221400    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-7ffd87447b-rmdk5"
Sep 16 12:58:33.221549 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:58:33.221440    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-696c5cc6cc-9dh72" podUID="2abe9c6e-a97c-4f3b-b8eb-4c2697a6485d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.27:8080: connect: connection refused"
Sep 16 12:58:33.221732 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:58:33.221707    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-7ffd87447b-rmdk5" podUID="a65e5a0e-beb3-4b27-a076-130b38b51d49" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.26:8080: connect: connection refused"
Sep 16 12:58:43.221956 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:58:43.221912    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-696c5cc6cc-9dh72" podUID="2abe9c6e-a97c-4f3b-b8eb-4c2697a6485d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.27:8080: connect: connection refused"
Sep 16 12:58:43.222408 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:58:43.221919    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-7ffd87447b-rmdk5" podUID="a65e5a0e-beb3-4b27-a076-130b38b51d49" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.26:8080: connect: connection refused"
Sep 16 12:58:53.221687 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:58:53.221650    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-696c5cc6cc-9dh72" podUID="2abe9c6e-a97c-4f3b-b8eb-4c2697a6485d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.27:8080: connect: connection refused"
Sep 16 12:58:53.222115 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:58:53.221784    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-7ffd87447b-rmdk5" podUID="a65e5a0e-beb3-4b27-a076-130b38b51d49" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.26:8080: connect: connection refused"
Sep 16 12:59:03.222104 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:59:03.222057    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-7ffd87447b-rmdk5" podUID="a65e5a0e-beb3-4b27-a076-130b38b51d49" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.26:8080: connect: connection refused"
Sep 16 12:59:03.222647 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:59:03.222057    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-696c5cc6cc-9dh72" podUID="2abe9c6e-a97c-4f3b-b8eb-4c2697a6485d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.27:8080: connect: connection refused"
Sep 16 12:59:13.222026 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:59:13.221987    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-7ffd87447b-rmdk5" podUID="a65e5a0e-beb3-4b27-a076-130b38b51d49" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.26:8080: connect: connection refused"
Sep 16 12:59:13.222455 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:59:13.221986    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-696c5cc6cc-9dh72" podUID="2abe9c6e-a97c-4f3b-b8eb-4c2697a6485d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.27:8080: connect: connection refused"
Sep 16 12:59:23.221532 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:59:23.221483    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-696c5cc6cc-9dh72" podUID="2abe9c6e-a97c-4f3b-b8eb-4c2697a6485d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.27:8080: connect: connection refused"
Sep 16 12:59:23.221954 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:59:23.221683    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-7ffd87447b-rmdk5" podUID="a65e5a0e-beb3-4b27-a076-130b38b51d49" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.26:8080: connect: connection refused"
Sep 16 12:59:33.221531 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:59:33.221474    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-696c5cc6cc-9dh72" podUID="2abe9c6e-a97c-4f3b-b8eb-4c2697a6485d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.27:8080: connect: connection refused"
Sep 16 12:59:33.221998 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:59:33.221698    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-7ffd87447b-rmdk5" podUID="a65e5a0e-beb3-4b27-a076-130b38b51d49" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.26:8080: connect: connection refused"
Sep 16 12:59:43.222504 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:59:43.222411    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-7ffd87447b-rmdk5"
Sep 16 12:59:43.222950 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:59:43.222516    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-696c5cc6cc-9dh72"
Sep 16 12:59:43.403528 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:59:43.403478    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-predictor-696c5cc6cc-9dh72"]
Sep 16 12:59:43.517556 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:59:43.517416    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-696c5cc6cc-9dh72" podUID="2abe9c6e-a97c-4f3b-b8eb-4c2697a6485d" containerName="kserve-container" containerID="cri-o://8bd070f106d7fb83004929a7ecf5b2dfe29769749adbcc94d42c07444c9d1977" gracePeriod=30
Sep 16 12:59:43.517556 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:59:43.517440    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-696c5cc6cc-9dh72" podUID="2abe9c6e-a97c-4f3b-b8eb-4c2697a6485d" containerName="kube-rbac-proxy" containerID="cri-o://c13aaa95dea7777ff24b55a5f2c031f63868520058d55c5167a72e72b93bf5f8" gracePeriod=30
Sep 16 12:59:44.523722 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:59:44.523681    2580 generic.go:356] "Generic (PLEG): container finished" podID="2abe9c6e-a97c-4f3b-b8eb-4c2697a6485d" containerID="c13aaa95dea7777ff24b55a5f2c031f63868520058d55c5167a72e72b93bf5f8" exitCode=2
Sep 16 12:59:44.524101 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:59:44.523748    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-696c5cc6cc-9dh72" event={"ID":"2abe9c6e-a97c-4f3b-b8eb-4c2697a6485d","Type":"ContainerDied","Data":"c13aaa95dea7777ff24b55a5f2c031f63868520058d55c5167a72e72b93bf5f8"}
Sep 16 12:59:46.389170 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:59:46.389123    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-graph-raw-ec80e-predictor-77f5fb7585-jbvsj"]
Sep 16 12:59:46.391875 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:59:46.391850    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-ec80e-predictor-77f5fb7585-jbvsj"
Sep 16 12:59:46.394351 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:59:46.394330    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-xgboost-graph-raw-ec80e-predictor-serving-cert\""
Sep 16 12:59:46.394457 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:59:46.394412    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-xgboost-graph-raw-ec80e-kube-rbac-proxy-sar-config\""
Sep 16 12:59:46.404588 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:59:46.404566    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-graph-raw-ec80e-predictor-77f5fb7585-jbvsj"]
Sep 16 12:59:46.476469 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:59:46.476436    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/b6f97e15-4f04-4c9a-ae13-68aeaa8e85de-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-graph-raw-ec80e-predictor-77f5fb7585-jbvsj\" (UID: \"b6f97e15-4f04-4c9a-ae13-68aeaa8e85de\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-ec80e-predictor-77f5fb7585-jbvsj"
Sep 16 12:59:46.476469 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:59:46.476469    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b6f97e15-4f04-4c9a-ae13-68aeaa8e85de-kserve-provision-location\") pod \"isvc-xgboost-graph-raw-ec80e-predictor-77f5fb7585-jbvsj\" (UID: \"b6f97e15-4f04-4c9a-ae13-68aeaa8e85de\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-ec80e-predictor-77f5fb7585-jbvsj"
Sep 16 12:59:46.476733 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:59:46.476615    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k6j6v\" (UniqueName: \"kubernetes.io/projected/b6f97e15-4f04-4c9a-ae13-68aeaa8e85de-kube-api-access-k6j6v\") pod \"isvc-xgboost-graph-raw-ec80e-predictor-77f5fb7585-jbvsj\" (UID: \"b6f97e15-4f04-4c9a-ae13-68aeaa8e85de\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-ec80e-predictor-77f5fb7585-jbvsj"
Sep 16 12:59:46.476733 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:59:46.476650    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b6f97e15-4f04-4c9a-ae13-68aeaa8e85de-proxy-tls\") pod \"isvc-xgboost-graph-raw-ec80e-predictor-77f5fb7585-jbvsj\" (UID: \"b6f97e15-4f04-4c9a-ae13-68aeaa8e85de\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-ec80e-predictor-77f5fb7585-jbvsj"
Sep 16 12:59:46.476733 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:59:46.476676    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/b6f97e15-4f04-4c9a-ae13-68aeaa8e85de-cabundle-cert\") pod \"isvc-xgboost-graph-raw-ec80e-predictor-77f5fb7585-jbvsj\" (UID: \"b6f97e15-4f04-4c9a-ae13-68aeaa8e85de\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-ec80e-predictor-77f5fb7585-jbvsj"
Sep 16 12:59:46.577611 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:59:46.577579    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/b6f97e15-4f04-4c9a-ae13-68aeaa8e85de-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-graph-raw-ec80e-predictor-77f5fb7585-jbvsj\" (UID: \"b6f97e15-4f04-4c9a-ae13-68aeaa8e85de\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-ec80e-predictor-77f5fb7585-jbvsj"
Sep 16 12:59:46.577611 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:59:46.577615    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b6f97e15-4f04-4c9a-ae13-68aeaa8e85de-kserve-provision-location\") pod \"isvc-xgboost-graph-raw-ec80e-predictor-77f5fb7585-jbvsj\" (UID: \"b6f97e15-4f04-4c9a-ae13-68aeaa8e85de\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-ec80e-predictor-77f5fb7585-jbvsj"
Sep 16 12:59:46.577846 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:59:46.577673    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k6j6v\" (UniqueName: \"kubernetes.io/projected/b6f97e15-4f04-4c9a-ae13-68aeaa8e85de-kube-api-access-k6j6v\") pod \"isvc-xgboost-graph-raw-ec80e-predictor-77f5fb7585-jbvsj\" (UID: \"b6f97e15-4f04-4c9a-ae13-68aeaa8e85de\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-ec80e-predictor-77f5fb7585-jbvsj"
Sep 16 12:59:46.577846 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:59:46.577693    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b6f97e15-4f04-4c9a-ae13-68aeaa8e85de-proxy-tls\") pod \"isvc-xgboost-graph-raw-ec80e-predictor-77f5fb7585-jbvsj\" (UID: \"b6f97e15-4f04-4c9a-ae13-68aeaa8e85de\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-ec80e-predictor-77f5fb7585-jbvsj"
Sep 16 12:59:46.577846 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:59:46.577708    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/b6f97e15-4f04-4c9a-ae13-68aeaa8e85de-cabundle-cert\") pod \"isvc-xgboost-graph-raw-ec80e-predictor-77f5fb7585-jbvsj\" (UID: \"b6f97e15-4f04-4c9a-ae13-68aeaa8e85de\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-ec80e-predictor-77f5fb7585-jbvsj"
Sep 16 12:59:46.578149 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:59:46.578125    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b6f97e15-4f04-4c9a-ae13-68aeaa8e85de-kserve-provision-location\") pod \"isvc-xgboost-graph-raw-ec80e-predictor-77f5fb7585-jbvsj\" (UID: \"b6f97e15-4f04-4c9a-ae13-68aeaa8e85de\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-ec80e-predictor-77f5fb7585-jbvsj"
Sep 16 12:59:46.578323 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:59:46.578306    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/b6f97e15-4f04-4c9a-ae13-68aeaa8e85de-cabundle-cert\") pod \"isvc-xgboost-graph-raw-ec80e-predictor-77f5fb7585-jbvsj\" (UID: \"b6f97e15-4f04-4c9a-ae13-68aeaa8e85de\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-ec80e-predictor-77f5fb7585-jbvsj"
Sep 16 12:59:46.578409 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:59:46.578390    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/b6f97e15-4f04-4c9a-ae13-68aeaa8e85de-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-graph-raw-ec80e-predictor-77f5fb7585-jbvsj\" (UID: \"b6f97e15-4f04-4c9a-ae13-68aeaa8e85de\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-ec80e-predictor-77f5fb7585-jbvsj"
Sep 16 12:59:46.580105 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:59:46.580085    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b6f97e15-4f04-4c9a-ae13-68aeaa8e85de-proxy-tls\") pod \"isvc-xgboost-graph-raw-ec80e-predictor-77f5fb7585-jbvsj\" (UID: \"b6f97e15-4f04-4c9a-ae13-68aeaa8e85de\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-ec80e-predictor-77f5fb7585-jbvsj"
Sep 16 12:59:46.588968 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:59:46.588946    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-k6j6v\" (UniqueName: \"kubernetes.io/projected/b6f97e15-4f04-4c9a-ae13-68aeaa8e85de-kube-api-access-k6j6v\") pod \"isvc-xgboost-graph-raw-ec80e-predictor-77f5fb7585-jbvsj\" (UID: \"b6f97e15-4f04-4c9a-ae13-68aeaa8e85de\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-ec80e-predictor-77f5fb7585-jbvsj"
Sep 16 12:59:46.702199 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:59:46.702115    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-ec80e-predictor-77f5fb7585-jbvsj"
Sep 16 12:59:46.985223 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:59:46.985147    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-graph-raw-ec80e-predictor-77f5fb7585-jbvsj"]
Sep 16 12:59:46.988218 ip-10-0-133-218 kubenswrapper[2580]: W0916 12:59:46.988192    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb6f97e15_4f04_4c9a_ae13_68aeaa8e85de.slice/crio-9957d9e91a669c431fe07fe6aec78650786c91c6d9fc904a09b0c748b358cf3a WatchSource:0}: Error finding container 9957d9e91a669c431fe07fe6aec78650786c91c6d9fc904a09b0c748b358cf3a: Status 404 returned error can't find the container with id 9957d9e91a669c431fe07fe6aec78650786c91c6d9fc904a09b0c748b358cf3a
Sep 16 12:59:47.538010 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:59:47.537969    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-ec80e-predictor-77f5fb7585-jbvsj" event={"ID":"b6f97e15-4f04-4c9a-ae13-68aeaa8e85de","Type":"ContainerStarted","Data":"41892d1c5d85ca608f952cf88f41b7ae48fb7ff7f27f30de28cfa78006131481"}
Sep 16 12:59:47.538464 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:59:47.538053    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-ec80e-predictor-77f5fb7585-jbvsj" event={"ID":"b6f97e15-4f04-4c9a-ae13-68aeaa8e85de","Type":"ContainerStarted","Data":"9957d9e91a669c431fe07fe6aec78650786c91c6d9fc904a09b0c748b358cf3a"}
Sep 16 12:59:48.264673 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:59:48.264649    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-696c5cc6cc-9dh72"
Sep 16 12:59:48.393703 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:59:48.393620    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/2abe9c6e-a97c-4f3b-b8eb-4c2697a6485d-proxy-tls\") pod \"2abe9c6e-a97c-4f3b-b8eb-4c2697a6485d\" (UID: \"2abe9c6e-a97c-4f3b-b8eb-4c2697a6485d\") "
Sep 16 12:59:48.393703 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:59:48.393660    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2dklf\" (UniqueName: \"kubernetes.io/projected/2abe9c6e-a97c-4f3b-b8eb-4c2697a6485d-kube-api-access-2dklf\") pod \"2abe9c6e-a97c-4f3b-b8eb-4c2697a6485d\" (UID: \"2abe9c6e-a97c-4f3b-b8eb-4c2697a6485d\") "
Sep 16 12:59:48.393703 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:59:48.393684    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/2abe9c6e-a97c-4f3b-b8eb-4c2697a6485d-kube-rbac-proxy-sar-config\") pod \"2abe9c6e-a97c-4f3b-b8eb-4c2697a6485d\" (UID: \"2abe9c6e-a97c-4f3b-b8eb-4c2697a6485d\") "
Sep 16 12:59:48.393950 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:59:48.393740    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/2abe9c6e-a97c-4f3b-b8eb-4c2697a6485d-kserve-provision-location\") pod \"2abe9c6e-a97c-4f3b-b8eb-4c2697a6485d\" (UID: \"2abe9c6e-a97c-4f3b-b8eb-4c2697a6485d\") "
Sep 16 12:59:48.393950 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:59:48.393762    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/2abe9c6e-a97c-4f3b-b8eb-4c2697a6485d-cabundle-cert\") pod \"2abe9c6e-a97c-4f3b-b8eb-4c2697a6485d\" (UID: \"2abe9c6e-a97c-4f3b-b8eb-4c2697a6485d\") "
Sep 16 12:59:48.394103 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:59:48.394070    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2abe9c6e-a97c-4f3b-b8eb-4c2697a6485d-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "2abe9c6e-a97c-4f3b-b8eb-4c2697a6485d" (UID: "2abe9c6e-a97c-4f3b-b8eb-4c2697a6485d"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 12:59:48.394198 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:59:48.394104    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2abe9c6e-a97c-4f3b-b8eb-4c2697a6485d-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "2abe9c6e-a97c-4f3b-b8eb-4c2697a6485d" (UID: "2abe9c6e-a97c-4f3b-b8eb-4c2697a6485d"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 12:59:48.394260 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:59:48.394221    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2abe9c6e-a97c-4f3b-b8eb-4c2697a6485d-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "2abe9c6e-a97c-4f3b-b8eb-4c2697a6485d" (UID: "2abe9c6e-a97c-4f3b-b8eb-4c2697a6485d"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 12:59:48.395880 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:59:48.395859    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2abe9c6e-a97c-4f3b-b8eb-4c2697a6485d-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "2abe9c6e-a97c-4f3b-b8eb-4c2697a6485d" (UID: "2abe9c6e-a97c-4f3b-b8eb-4c2697a6485d"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 12:59:48.396031 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:59:48.396016    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2abe9c6e-a97c-4f3b-b8eb-4c2697a6485d-kube-api-access-2dklf" (OuterVolumeSpecName: "kube-api-access-2dklf") pod "2abe9c6e-a97c-4f3b-b8eb-4c2697a6485d" (UID: "2abe9c6e-a97c-4f3b-b8eb-4c2697a6485d"). InnerVolumeSpecName "kube-api-access-2dklf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 12:59:48.494660 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:59:48.494629    2580 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/2abe9c6e-a97c-4f3b-b8eb-4c2697a6485d-cabundle-cert\") on node \"ip-10-0-133-218.ec2.internal\" DevicePath \"\""
Sep 16 12:59:48.494660 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:59:48.494654    2580 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/2abe9c6e-a97c-4f3b-b8eb-4c2697a6485d-proxy-tls\") on node \"ip-10-0-133-218.ec2.internal\" DevicePath \"\""
Sep 16 12:59:48.494660 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:59:48.494663    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2dklf\" (UniqueName: \"kubernetes.io/projected/2abe9c6e-a97c-4f3b-b8eb-4c2697a6485d-kube-api-access-2dklf\") on node \"ip-10-0-133-218.ec2.internal\" DevicePath \"\""
Sep 16 12:59:48.494940 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:59:48.494675    2580 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/2abe9c6e-a97c-4f3b-b8eb-4c2697a6485d-kube-rbac-proxy-sar-config\") on node \"ip-10-0-133-218.ec2.internal\" DevicePath \"\""
Sep 16 12:59:48.494940 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:59:48.494687    2580 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/2abe9c6e-a97c-4f3b-b8eb-4c2697a6485d-kserve-provision-location\") on node \"ip-10-0-133-218.ec2.internal\" DevicePath \"\""
Sep 16 12:59:48.543787 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:59:48.543752    2580 generic.go:356] "Generic (PLEG): container finished" podID="2abe9c6e-a97c-4f3b-b8eb-4c2697a6485d" containerID="8bd070f106d7fb83004929a7ecf5b2dfe29769749adbcc94d42c07444c9d1977" exitCode=0
Sep 16 12:59:48.544179 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:59:48.543811    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-696c5cc6cc-9dh72"
Sep 16 12:59:48.544179 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:59:48.543835    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-696c5cc6cc-9dh72" event={"ID":"2abe9c6e-a97c-4f3b-b8eb-4c2697a6485d","Type":"ContainerDied","Data":"8bd070f106d7fb83004929a7ecf5b2dfe29769749adbcc94d42c07444c9d1977"}
Sep 16 12:59:48.544179 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:59:48.544091    2580 scope.go:117] "RemoveContainer" containerID="c13aaa95dea7777ff24b55a5f2c031f63868520058d55c5167a72e72b93bf5f8"
Sep 16 12:59:48.544179 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:59:48.543888    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-696c5cc6cc-9dh72" event={"ID":"2abe9c6e-a97c-4f3b-b8eb-4c2697a6485d","Type":"ContainerDied","Data":"b51f7b9733bea584417f48154a9c9a1e2e1d8ad98d9fed7c0323f4088b6a6ad6"}
Sep 16 12:59:48.558359 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:59:48.558342    2580 scope.go:117] "RemoveContainer" containerID="8bd070f106d7fb83004929a7ecf5b2dfe29769749adbcc94d42c07444c9d1977"
Sep 16 12:59:48.565536 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:59:48.565512    2580 scope.go:117] "RemoveContainer" containerID="12ca280bad1ac1c8667647895c41239333364e5153baaf984805929aa4e567f6"
Sep 16 12:59:48.572313 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:59:48.572286    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-predictor-696c5cc6cc-9dh72"]
Sep 16 12:59:48.573220 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:59:48.573199    2580 scope.go:117] "RemoveContainer" containerID="c13aaa95dea7777ff24b55a5f2c031f63868520058d55c5167a72e72b93bf5f8"
Sep 16 12:59:48.573515 ip-10-0-133-218 kubenswrapper[2580]: E0916 12:59:48.573496    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c13aaa95dea7777ff24b55a5f2c031f63868520058d55c5167a72e72b93bf5f8\": container with ID starting with c13aaa95dea7777ff24b55a5f2c031f63868520058d55c5167a72e72b93bf5f8 not found: ID does not exist" containerID="c13aaa95dea7777ff24b55a5f2c031f63868520058d55c5167a72e72b93bf5f8"
Sep 16 12:59:48.573561 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:59:48.573525    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c13aaa95dea7777ff24b55a5f2c031f63868520058d55c5167a72e72b93bf5f8"} err="failed to get container status \"c13aaa95dea7777ff24b55a5f2c031f63868520058d55c5167a72e72b93bf5f8\": rpc error: code = NotFound desc = could not find container \"c13aaa95dea7777ff24b55a5f2c031f63868520058d55c5167a72e72b93bf5f8\": container with ID starting with c13aaa95dea7777ff24b55a5f2c031f63868520058d55c5167a72e72b93bf5f8 not found: ID does not exist"
Sep 16 12:59:48.573561 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:59:48.573543    2580 scope.go:117] "RemoveContainer" containerID="8bd070f106d7fb83004929a7ecf5b2dfe29769749adbcc94d42c07444c9d1977"
Sep 16 12:59:48.573782 ip-10-0-133-218 kubenswrapper[2580]: E0916 12:59:48.573765    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8bd070f106d7fb83004929a7ecf5b2dfe29769749adbcc94d42c07444c9d1977\": container with ID starting with 8bd070f106d7fb83004929a7ecf5b2dfe29769749adbcc94d42c07444c9d1977 not found: ID does not exist" containerID="8bd070f106d7fb83004929a7ecf5b2dfe29769749adbcc94d42c07444c9d1977"
Sep 16 12:59:48.573828 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:59:48.573786    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8bd070f106d7fb83004929a7ecf5b2dfe29769749adbcc94d42c07444c9d1977"} err="failed to get container status \"8bd070f106d7fb83004929a7ecf5b2dfe29769749adbcc94d42c07444c9d1977\": rpc error: code = NotFound desc = could not find container \"8bd070f106d7fb83004929a7ecf5b2dfe29769749adbcc94d42c07444c9d1977\": container with ID starting with 8bd070f106d7fb83004929a7ecf5b2dfe29769749adbcc94d42c07444c9d1977 not found: ID does not exist"
Sep 16 12:59:48.573828 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:59:48.573800    2580 scope.go:117] "RemoveContainer" containerID="12ca280bad1ac1c8667647895c41239333364e5153baaf984805929aa4e567f6"
Sep 16 12:59:48.574042 ip-10-0-133-218 kubenswrapper[2580]: E0916 12:59:48.574023    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"12ca280bad1ac1c8667647895c41239333364e5153baaf984805929aa4e567f6\": container with ID starting with 12ca280bad1ac1c8667647895c41239333364e5153baaf984805929aa4e567f6 not found: ID does not exist" containerID="12ca280bad1ac1c8667647895c41239333364e5153baaf984805929aa4e567f6"
Sep 16 12:59:48.574090 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:59:48.574051    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"12ca280bad1ac1c8667647895c41239333364e5153baaf984805929aa4e567f6"} err="failed to get container status \"12ca280bad1ac1c8667647895c41239333364e5153baaf984805929aa4e567f6\": rpc error: code = NotFound desc = could not find container \"12ca280bad1ac1c8667647895c41239333364e5153baaf984805929aa4e567f6\": container with ID starting with 12ca280bad1ac1c8667647895c41239333364e5153baaf984805929aa4e567f6 not found: ID does not exist"
Sep 16 12:59:48.587869 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:59:48.587840    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-predictor-696c5cc6cc-9dh72"]
Sep 16 12:59:49.217822 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:59:49.217778    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-696c5cc6cc-9dh72" podUID="2abe9c6e-a97c-4f3b-b8eb-4c2697a6485d" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.27:8643/healthz\": context deadline exceeded"
Sep 16 12:59:49.599124 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:59:49.599088    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2abe9c6e-a97c-4f3b-b8eb-4c2697a6485d" path="/var/lib/kubelet/pods/2abe9c6e-a97c-4f3b-b8eb-4c2697a6485d/volumes"
Sep 16 12:59:50.787542 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:59:50.787507    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-7ffd87447b-rmdk5"]
Sep 16 12:59:50.787901 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:59:50.787796    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-7ffd87447b-rmdk5" podUID="a65e5a0e-beb3-4b27-a076-130b38b51d49" containerName="kserve-container" containerID="cri-o://6f44dba2b7ae6da894ed1decc6ef145a411c318fcc172f07a6342203c585d876" gracePeriod=30
Sep 16 12:59:50.787901 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:59:50.787827    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-7ffd87447b-rmdk5" podUID="a65e5a0e-beb3-4b27-a076-130b38b51d49" containerName="kube-rbac-proxy" containerID="cri-o://4e64dcdc464ed2b32a9d404bb945bf10ad11abe0d3240010e34d793c4d6b395d" gracePeriod=30
Sep 16 12:59:50.950229 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:59:50.950192    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-predictor-696c5cc6cc-5dsjm"]
Sep 16 12:59:50.950544 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:59:50.950531    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2abe9c6e-a97c-4f3b-b8eb-4c2697a6485d" containerName="kserve-container"
Sep 16 12:59:50.950609 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:59:50.950545    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="2abe9c6e-a97c-4f3b-b8eb-4c2697a6485d" containerName="kserve-container"
Sep 16 12:59:50.950609 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:59:50.950553    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2abe9c6e-a97c-4f3b-b8eb-4c2697a6485d" containerName="kube-rbac-proxy"
Sep 16 12:59:50.950609 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:59:50.950558    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="2abe9c6e-a97c-4f3b-b8eb-4c2697a6485d" containerName="kube-rbac-proxy"
Sep 16 12:59:50.950609 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:59:50.950566    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2abe9c6e-a97c-4f3b-b8eb-4c2697a6485d" containerName="storage-initializer"
Sep 16 12:59:50.950609 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:59:50.950572    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="2abe9c6e-a97c-4f3b-b8eb-4c2697a6485d" containerName="storage-initializer"
Sep 16 12:59:50.950777 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:59:50.950629    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="2abe9c6e-a97c-4f3b-b8eb-4c2697a6485d" containerName="kube-rbac-proxy"
Sep 16 12:59:50.950777 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:59:50.950636    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="2abe9c6e-a97c-4f3b-b8eb-4c2697a6485d" containerName="kserve-container"
Sep 16 12:59:50.953807 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:59:50.953791    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-696c5cc6cc-5dsjm"
Sep 16 12:59:50.957894 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:59:50.957872    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-rollback-predictor-serving-cert\""
Sep 16 12:59:50.969363 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:59:50.969342    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-predictor-696c5cc6cc-5dsjm"]
Sep 16 12:59:51.116180 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:59:51.116095    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/3cf459e0-cf33-4718-98aa-9bd7e302c9f4-kserve-provision-location\") pod \"isvc-canary-rollback-predictor-696c5cc6cc-5dsjm\" (UID: \"3cf459e0-cf33-4718-98aa-9bd7e302c9f4\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-696c5cc6cc-5dsjm"
Sep 16 12:59:51.116180 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:59:51.116147    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/3cf459e0-cf33-4718-98aa-9bd7e302c9f4-kube-rbac-proxy-sar-config\") pod \"isvc-canary-rollback-predictor-696c5cc6cc-5dsjm\" (UID: \"3cf459e0-cf33-4718-98aa-9bd7e302c9f4\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-696c5cc6cc-5dsjm"
Sep 16 12:59:51.116180 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:59:51.116174    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/3cf459e0-cf33-4718-98aa-9bd7e302c9f4-proxy-tls\") pod \"isvc-canary-rollback-predictor-696c5cc6cc-5dsjm\" (UID: \"3cf459e0-cf33-4718-98aa-9bd7e302c9f4\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-696c5cc6cc-5dsjm"
Sep 16 12:59:51.116426 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:59:51.116260    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m28bx\" (UniqueName: \"kubernetes.io/projected/3cf459e0-cf33-4718-98aa-9bd7e302c9f4-kube-api-access-m28bx\") pod \"isvc-canary-rollback-predictor-696c5cc6cc-5dsjm\" (UID: \"3cf459e0-cf33-4718-98aa-9bd7e302c9f4\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-696c5cc6cc-5dsjm"
Sep 16 12:59:51.116426 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:59:51.116306    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/3cf459e0-cf33-4718-98aa-9bd7e302c9f4-cabundle-cert\") pod \"isvc-canary-rollback-predictor-696c5cc6cc-5dsjm\" (UID: \"3cf459e0-cf33-4718-98aa-9bd7e302c9f4\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-696c5cc6cc-5dsjm"
Sep 16 12:59:51.217732 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:59:51.217695    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/3cf459e0-cf33-4718-98aa-9bd7e302c9f4-kube-rbac-proxy-sar-config\") pod \"isvc-canary-rollback-predictor-696c5cc6cc-5dsjm\" (UID: \"3cf459e0-cf33-4718-98aa-9bd7e302c9f4\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-696c5cc6cc-5dsjm"
Sep 16 12:59:51.217908 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:59:51.217740    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/3cf459e0-cf33-4718-98aa-9bd7e302c9f4-proxy-tls\") pod \"isvc-canary-rollback-predictor-696c5cc6cc-5dsjm\" (UID: \"3cf459e0-cf33-4718-98aa-9bd7e302c9f4\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-696c5cc6cc-5dsjm"
Sep 16 12:59:51.217908 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:59:51.217790    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m28bx\" (UniqueName: \"kubernetes.io/projected/3cf459e0-cf33-4718-98aa-9bd7e302c9f4-kube-api-access-m28bx\") pod \"isvc-canary-rollback-predictor-696c5cc6cc-5dsjm\" (UID: \"3cf459e0-cf33-4718-98aa-9bd7e302c9f4\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-696c5cc6cc-5dsjm"
Sep 16 12:59:51.217908 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:59:51.217822    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/3cf459e0-cf33-4718-98aa-9bd7e302c9f4-cabundle-cert\") pod \"isvc-canary-rollback-predictor-696c5cc6cc-5dsjm\" (UID: \"3cf459e0-cf33-4718-98aa-9bd7e302c9f4\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-696c5cc6cc-5dsjm"
Sep 16 12:59:51.217908 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:59:51.217872    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/3cf459e0-cf33-4718-98aa-9bd7e302c9f4-kserve-provision-location\") pod \"isvc-canary-rollback-predictor-696c5cc6cc-5dsjm\" (UID: \"3cf459e0-cf33-4718-98aa-9bd7e302c9f4\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-696c5cc6cc-5dsjm"
Sep 16 12:59:51.218240 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:59:51.218216    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/3cf459e0-cf33-4718-98aa-9bd7e302c9f4-kserve-provision-location\") pod \"isvc-canary-rollback-predictor-696c5cc6cc-5dsjm\" (UID: \"3cf459e0-cf33-4718-98aa-9bd7e302c9f4\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-696c5cc6cc-5dsjm"
Sep 16 12:59:51.218423 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:59:51.218405    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/3cf459e0-cf33-4718-98aa-9bd7e302c9f4-cabundle-cert\") pod \"isvc-canary-rollback-predictor-696c5cc6cc-5dsjm\" (UID: \"3cf459e0-cf33-4718-98aa-9bd7e302c9f4\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-696c5cc6cc-5dsjm"
Sep 16 12:59:51.218466 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:59:51.218433    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/3cf459e0-cf33-4718-98aa-9bd7e302c9f4-kube-rbac-proxy-sar-config\") pod \"isvc-canary-rollback-predictor-696c5cc6cc-5dsjm\" (UID: \"3cf459e0-cf33-4718-98aa-9bd7e302c9f4\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-696c5cc6cc-5dsjm"
Sep 16 12:59:51.220121 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:59:51.220099    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/3cf459e0-cf33-4718-98aa-9bd7e302c9f4-proxy-tls\") pod \"isvc-canary-rollback-predictor-696c5cc6cc-5dsjm\" (UID: \"3cf459e0-cf33-4718-98aa-9bd7e302c9f4\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-696c5cc6cc-5dsjm"
Sep 16 12:59:51.227637 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:59:51.227617    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-m28bx\" (UniqueName: \"kubernetes.io/projected/3cf459e0-cf33-4718-98aa-9bd7e302c9f4-kube-api-access-m28bx\") pod \"isvc-canary-rollback-predictor-696c5cc6cc-5dsjm\" (UID: \"3cf459e0-cf33-4718-98aa-9bd7e302c9f4\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-696c5cc6cc-5dsjm"
Sep 16 12:59:51.263619 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:59:51.263596    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-696c5cc6cc-5dsjm"
Sep 16 12:59:51.558477 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:59:51.558449    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-predictor-696c5cc6cc-5dsjm"]
Sep 16 12:59:51.559782 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:59:51.559756    2580 generic.go:356] "Generic (PLEG): container finished" podID="a65e5a0e-beb3-4b27-a076-130b38b51d49" containerID="4e64dcdc464ed2b32a9d404bb945bf10ad11abe0d3240010e34d793c4d6b395d" exitCode=2
Sep 16 12:59:51.559884 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:59:51.559822    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-7ffd87447b-rmdk5" event={"ID":"a65e5a0e-beb3-4b27-a076-130b38b51d49","Type":"ContainerDied","Data":"4e64dcdc464ed2b32a9d404bb945bf10ad11abe0d3240010e34d793c4d6b395d"}
Sep 16 12:59:51.561323 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:59:51.561299    2580 generic.go:356] "Generic (PLEG): container finished" podID="b6f97e15-4f04-4c9a-ae13-68aeaa8e85de" containerID="41892d1c5d85ca608f952cf88f41b7ae48fb7ff7f27f30de28cfa78006131481" exitCode=0
Sep 16 12:59:51.561412 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:59:51.561344    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-ec80e-predictor-77f5fb7585-jbvsj" event={"ID":"b6f97e15-4f04-4c9a-ae13-68aeaa8e85de","Type":"ContainerDied","Data":"41892d1c5d85ca608f952cf88f41b7ae48fb7ff7f27f30de28cfa78006131481"}
Sep 16 12:59:51.561763 ip-10-0-133-218 kubenswrapper[2580]: W0916 12:59:51.561744    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3cf459e0_cf33_4718_98aa_9bd7e302c9f4.slice/crio-95da654d3182953e061c17733f17299df9b0a8cbf27ad2b14cfc9b4de4654d93 WatchSource:0}: Error finding container 95da654d3182953e061c17733f17299df9b0a8cbf27ad2b14cfc9b4de4654d93: Status 404 returned error can't find the container with id 95da654d3182953e061c17733f17299df9b0a8cbf27ad2b14cfc9b4de4654d93
Sep 16 12:59:52.568925 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:59:52.568857    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-696c5cc6cc-5dsjm" event={"ID":"3cf459e0-cf33-4718-98aa-9bd7e302c9f4","Type":"ContainerStarted","Data":"967010abd177dfe621fc37f4ac96e8c7f0b8cb70d9ac9fff3da73125e694b78d"}
Sep 16 12:59:52.569403 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:59:52.568971    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-696c5cc6cc-5dsjm" event={"ID":"3cf459e0-cf33-4718-98aa-9bd7e302c9f4","Type":"ContainerStarted","Data":"95da654d3182953e061c17733f17299df9b0a8cbf27ad2b14cfc9b4de4654d93"}
Sep 16 12:59:53.217945 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:59:53.217905    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-7ffd87447b-rmdk5" podUID="a65e5a0e-beb3-4b27-a076-130b38b51d49" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.26:8643/healthz\": dial tcp 10.133.0.26:8643: connect: connection refused"
Sep 16 12:59:53.221824 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:59:53.221789    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-7ffd87447b-rmdk5" podUID="a65e5a0e-beb3-4b27-a076-130b38b51d49" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.26:8080: connect: connection refused"
Sep 16 12:59:56.044555 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:59:56.044528    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-7ffd87447b-rmdk5"
Sep 16 12:59:56.160298 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:59:56.160217    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/a65e5a0e-beb3-4b27-a076-130b38b51d49-cabundle-cert\") pod \"a65e5a0e-beb3-4b27-a076-130b38b51d49\" (UID: \"a65e5a0e-beb3-4b27-a076-130b38b51d49\") "
Sep 16 12:59:56.160298 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:59:56.160281    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-t2dp2\" (UniqueName: \"kubernetes.io/projected/a65e5a0e-beb3-4b27-a076-130b38b51d49-kube-api-access-t2dp2\") pod \"a65e5a0e-beb3-4b27-a076-130b38b51d49\" (UID: \"a65e5a0e-beb3-4b27-a076-130b38b51d49\") "
Sep 16 12:59:56.160528 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:59:56.160323    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a65e5a0e-beb3-4b27-a076-130b38b51d49-kserve-provision-location\") pod \"a65e5a0e-beb3-4b27-a076-130b38b51d49\" (UID: \"a65e5a0e-beb3-4b27-a076-130b38b51d49\") "
Sep 16 12:59:56.160528 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:59:56.160372    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a65e5a0e-beb3-4b27-a076-130b38b51d49-kube-rbac-proxy-sar-config\") pod \"a65e5a0e-beb3-4b27-a076-130b38b51d49\" (UID: \"a65e5a0e-beb3-4b27-a076-130b38b51d49\") "
Sep 16 12:59:56.160528 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:59:56.160409    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a65e5a0e-beb3-4b27-a076-130b38b51d49-proxy-tls\") pod \"a65e5a0e-beb3-4b27-a076-130b38b51d49\" (UID: \"a65e5a0e-beb3-4b27-a076-130b38b51d49\") "
Sep 16 12:59:56.160760 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:59:56.160648    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a65e5a0e-beb3-4b27-a076-130b38b51d49-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "a65e5a0e-beb3-4b27-a076-130b38b51d49" (UID: "a65e5a0e-beb3-4b27-a076-130b38b51d49"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 12:59:56.160760 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:59:56.160731    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a65e5a0e-beb3-4b27-a076-130b38b51d49-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "a65e5a0e-beb3-4b27-a076-130b38b51d49" (UID: "a65e5a0e-beb3-4b27-a076-130b38b51d49"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 12:59:56.160980 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:59:56.160822    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a65e5a0e-beb3-4b27-a076-130b38b51d49-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "a65e5a0e-beb3-4b27-a076-130b38b51d49" (UID: "a65e5a0e-beb3-4b27-a076-130b38b51d49"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 12:59:56.162542 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:59:56.162515    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a65e5a0e-beb3-4b27-a076-130b38b51d49-kube-api-access-t2dp2" (OuterVolumeSpecName: "kube-api-access-t2dp2") pod "a65e5a0e-beb3-4b27-a076-130b38b51d49" (UID: "a65e5a0e-beb3-4b27-a076-130b38b51d49"). InnerVolumeSpecName "kube-api-access-t2dp2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 12:59:56.162790 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:59:56.162769    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a65e5a0e-beb3-4b27-a076-130b38b51d49-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "a65e5a0e-beb3-4b27-a076-130b38b51d49" (UID: "a65e5a0e-beb3-4b27-a076-130b38b51d49"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 12:59:56.261431 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:59:56.261389    2580 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a65e5a0e-beb3-4b27-a076-130b38b51d49-kube-rbac-proxy-sar-config\") on node \"ip-10-0-133-218.ec2.internal\" DevicePath \"\""
Sep 16 12:59:56.261431 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:59:56.261426    2580 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a65e5a0e-beb3-4b27-a076-130b38b51d49-proxy-tls\") on node \"ip-10-0-133-218.ec2.internal\" DevicePath \"\""
Sep 16 12:59:56.261650 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:59:56.261442    2580 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/a65e5a0e-beb3-4b27-a076-130b38b51d49-cabundle-cert\") on node \"ip-10-0-133-218.ec2.internal\" DevicePath \"\""
Sep 16 12:59:56.261650 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:59:56.261456    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t2dp2\" (UniqueName: \"kubernetes.io/projected/a65e5a0e-beb3-4b27-a076-130b38b51d49-kube-api-access-t2dp2\") on node \"ip-10-0-133-218.ec2.internal\" DevicePath \"\""
Sep 16 12:59:56.261650 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:59:56.261470    2580 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a65e5a0e-beb3-4b27-a076-130b38b51d49-kserve-provision-location\") on node \"ip-10-0-133-218.ec2.internal\" DevicePath \"\""
Sep 16 12:59:56.590467 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:59:56.590432    2580 generic.go:356] "Generic (PLEG): container finished" podID="a65e5a0e-beb3-4b27-a076-130b38b51d49" containerID="6f44dba2b7ae6da894ed1decc6ef145a411c318fcc172f07a6342203c585d876" exitCode=0
Sep 16 12:59:56.590660 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:59:56.590523    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-7ffd87447b-rmdk5" event={"ID":"a65e5a0e-beb3-4b27-a076-130b38b51d49","Type":"ContainerDied","Data":"6f44dba2b7ae6da894ed1decc6ef145a411c318fcc172f07a6342203c585d876"}
Sep 16 12:59:56.590660 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:59:56.590528    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-7ffd87447b-rmdk5"
Sep 16 12:59:56.590660 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:59:56.590585    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-7ffd87447b-rmdk5" event={"ID":"a65e5a0e-beb3-4b27-a076-130b38b51d49","Type":"ContainerDied","Data":"821a2007c904ef0f8007c65339e901e053aa8faf645ada653a88505706a13a5c"}
Sep 16 12:59:56.590660 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:59:56.590623    2580 scope.go:117] "RemoveContainer" containerID="4e64dcdc464ed2b32a9d404bb945bf10ad11abe0d3240010e34d793c4d6b395d"
Sep 16 12:59:56.592622 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:59:56.592593    2580 generic.go:356] "Generic (PLEG): container finished" podID="3cf459e0-cf33-4718-98aa-9bd7e302c9f4" containerID="967010abd177dfe621fc37f4ac96e8c7f0b8cb70d9ac9fff3da73125e694b78d" exitCode=0
Sep 16 12:59:56.592737 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:59:56.592668    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-696c5cc6cc-5dsjm" event={"ID":"3cf459e0-cf33-4718-98aa-9bd7e302c9f4","Type":"ContainerDied","Data":"967010abd177dfe621fc37f4ac96e8c7f0b8cb70d9ac9fff3da73125e694b78d"}
Sep 16 12:59:56.600844 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:59:56.600769    2580 scope.go:117] "RemoveContainer" containerID="6f44dba2b7ae6da894ed1decc6ef145a411c318fcc172f07a6342203c585d876"
Sep 16 12:59:56.609992 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:59:56.609972    2580 scope.go:117] "RemoveContainer" containerID="588dcbe5133407fcc21f1894f861b9df75546f4a518bf2b126a95d7d30afbccd"
Sep 16 12:59:56.620738 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:59:56.620717    2580 scope.go:117] "RemoveContainer" containerID="4e64dcdc464ed2b32a9d404bb945bf10ad11abe0d3240010e34d793c4d6b395d"
Sep 16 12:59:56.621057 ip-10-0-133-218 kubenswrapper[2580]: E0916 12:59:56.621033    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4e64dcdc464ed2b32a9d404bb945bf10ad11abe0d3240010e34d793c4d6b395d\": container with ID starting with 4e64dcdc464ed2b32a9d404bb945bf10ad11abe0d3240010e34d793c4d6b395d not found: ID does not exist" containerID="4e64dcdc464ed2b32a9d404bb945bf10ad11abe0d3240010e34d793c4d6b395d"
Sep 16 12:59:56.621145 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:59:56.621069    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4e64dcdc464ed2b32a9d404bb945bf10ad11abe0d3240010e34d793c4d6b395d"} err="failed to get container status \"4e64dcdc464ed2b32a9d404bb945bf10ad11abe0d3240010e34d793c4d6b395d\": rpc error: code = NotFound desc = could not find container \"4e64dcdc464ed2b32a9d404bb945bf10ad11abe0d3240010e34d793c4d6b395d\": container with ID starting with 4e64dcdc464ed2b32a9d404bb945bf10ad11abe0d3240010e34d793c4d6b395d not found: ID does not exist"
Sep 16 12:59:56.621145 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:59:56.621093    2580 scope.go:117] "RemoveContainer" containerID="6f44dba2b7ae6da894ed1decc6ef145a411c318fcc172f07a6342203c585d876"
Sep 16 12:59:56.621386 ip-10-0-133-218 kubenswrapper[2580]: E0916 12:59:56.621361    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6f44dba2b7ae6da894ed1decc6ef145a411c318fcc172f07a6342203c585d876\": container with ID starting with 6f44dba2b7ae6da894ed1decc6ef145a411c318fcc172f07a6342203c585d876 not found: ID does not exist" containerID="6f44dba2b7ae6da894ed1decc6ef145a411c318fcc172f07a6342203c585d876"
Sep 16 12:59:56.621514 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:59:56.621392    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6f44dba2b7ae6da894ed1decc6ef145a411c318fcc172f07a6342203c585d876"} err="failed to get container status \"6f44dba2b7ae6da894ed1decc6ef145a411c318fcc172f07a6342203c585d876\": rpc error: code = NotFound desc = could not find container \"6f44dba2b7ae6da894ed1decc6ef145a411c318fcc172f07a6342203c585d876\": container with ID starting with 6f44dba2b7ae6da894ed1decc6ef145a411c318fcc172f07a6342203c585d876 not found: ID does not exist"
Sep 16 12:59:56.621514 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:59:56.621413    2580 scope.go:117] "RemoveContainer" containerID="588dcbe5133407fcc21f1894f861b9df75546f4a518bf2b126a95d7d30afbccd"
Sep 16 12:59:56.621685 ip-10-0-133-218 kubenswrapper[2580]: E0916 12:59:56.621665    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"588dcbe5133407fcc21f1894f861b9df75546f4a518bf2b126a95d7d30afbccd\": container with ID starting with 588dcbe5133407fcc21f1894f861b9df75546f4a518bf2b126a95d7d30afbccd not found: ID does not exist" containerID="588dcbe5133407fcc21f1894f861b9df75546f4a518bf2b126a95d7d30afbccd"
Sep 16 12:59:56.621742 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:59:56.621691    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"588dcbe5133407fcc21f1894f861b9df75546f4a518bf2b126a95d7d30afbccd"} err="failed to get container status \"588dcbe5133407fcc21f1894f861b9df75546f4a518bf2b126a95d7d30afbccd\": rpc error: code = NotFound desc = could not find container \"588dcbe5133407fcc21f1894f861b9df75546f4a518bf2b126a95d7d30afbccd\": container with ID starting with 588dcbe5133407fcc21f1894f861b9df75546f4a518bf2b126a95d7d30afbccd not found: ID does not exist"
Sep 16 12:59:56.648161 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:59:56.647869    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-7ffd87447b-rmdk5"]
Sep 16 12:59:56.657131 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:59:56.657106    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-7ffd87447b-rmdk5"]
Sep 16 12:59:57.602868 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:59:57.602823    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a65e5a0e-beb3-4b27-a076-130b38b51d49" path="/var/lib/kubelet/pods/a65e5a0e-beb3-4b27-a076-130b38b51d49/volumes"
Sep 16 12:59:57.603529 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:59:57.603301    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-696c5cc6cc-5dsjm" event={"ID":"3cf459e0-cf33-4718-98aa-9bd7e302c9f4","Type":"ContainerStarted","Data":"c9f090b190d9b014402681488bf7fd8f3c452ca763be42c53d42cba767d42825"}
Sep 16 12:59:57.603529 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:59:57.603377    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-696c5cc6cc-5dsjm" event={"ID":"3cf459e0-cf33-4718-98aa-9bd7e302c9f4","Type":"ContainerStarted","Data":"c31d4e06b9797d68bf8f7c4116085454603489d6f16dc824fca43e00d32656ef"}
Sep 16 12:59:57.604196 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:59:57.604174    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-696c5cc6cc-5dsjm"
Sep 16 12:59:57.604288 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:59:57.604246    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-696c5cc6cc-5dsjm"
Sep 16 12:59:57.605162 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:59:57.605126    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-696c5cc6cc-5dsjm" podUID="3cf459e0-cf33-4718-98aa-9bd7e302c9f4" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.29:8080: connect: connection refused"
Sep 16 12:59:57.627788 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:59:57.627743    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-696c5cc6cc-5dsjm" podStartSLOduration=3.620345257 podStartE2EDuration="7.627725422s" totalImagesPullingTime="0s" totalInitContainerRuntime="4.007380165s" isStatefulPod=false podCreationTimestamp="2026-09-16 12:59:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 12:59:57.626564377 +0000 UTC m=+736.549436845" watchObservedRunningTime="2026-09-16 12:59:57.627725422 +0000 UTC m=+736.550597892"
Sep 16 12:59:58.609298 ip-10-0-133-218 kubenswrapper[2580]: I0916 12:59:58.609255    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-696c5cc6cc-5dsjm" podUID="3cf459e0-cf33-4718-98aa-9bd7e302c9f4" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.29:8080: connect: connection refused"
Sep 16 13:00:03.613939 ip-10-0-133-218 kubenswrapper[2580]: I0916 13:00:03.613912    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-696c5cc6cc-5dsjm"
Sep 16 13:00:03.614504 ip-10-0-133-218 kubenswrapper[2580]: I0916 13:00:03.614456    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-696c5cc6cc-5dsjm" podUID="3cf459e0-cf33-4718-98aa-9bd7e302c9f4" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.29:8080: connect: connection refused"
Sep 16 13:00:09.659426 ip-10-0-133-218 kubenswrapper[2580]: I0916 13:00:09.659389    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-ec80e-predictor-77f5fb7585-jbvsj" event={"ID":"b6f97e15-4f04-4c9a-ae13-68aeaa8e85de","Type":"ContainerStarted","Data":"4dc8be51c30bf3de2279e1e58b5e61b95be379e8bfa94209ca8c17fffda42866"}
Sep 16 13:00:09.659822 ip-10-0-133-218 kubenswrapper[2580]: I0916 13:00:09.659451    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-ec80e-predictor-77f5fb7585-jbvsj" event={"ID":"b6f97e15-4f04-4c9a-ae13-68aeaa8e85de","Type":"ContainerStarted","Data":"8b0cd92b7874d06dc1552b779c41f8f1ae09ddee5a0ee25de24a6d3d7370a94e"}
Sep 16 13:00:09.659822 ip-10-0-133-218 kubenswrapper[2580]: I0916 13:00:09.659652    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-ec80e-predictor-77f5fb7585-jbvsj"
Sep 16 13:00:09.713345 ip-10-0-133-218 kubenswrapper[2580]: I0916 13:00:09.713290    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-ec80e-predictor-77f5fb7585-jbvsj" podStartSLOduration=2.443081279 podStartE2EDuration="23.713274208s" totalImagesPullingTime="17.489345948s" totalInitContainerRuntime="3.780846981s" isStatefulPod=false podCreationTimestamp="2026-09-16 12:59:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 13:00:09.712148628 +0000 UTC m=+748.635021102" watchObservedRunningTime="2026-09-16 13:00:09.713274208 +0000 UTC m=+748.636146679"
Sep 16 13:00:10.664263 ip-10-0-133-218 kubenswrapper[2580]: I0916 13:00:10.664229    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-ec80e-predictor-77f5fb7585-jbvsj"
Sep 16 13:00:10.665548 ip-10-0-133-218 kubenswrapper[2580]: I0916 13:00:10.665515    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-ec80e-predictor-77f5fb7585-jbvsj" podUID="b6f97e15-4f04-4c9a-ae13-68aeaa8e85de" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.28:8080: connect: connection refused"
Sep 16 13:00:11.668277 ip-10-0-133-218 kubenswrapper[2580]: I0916 13:00:11.668230    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-ec80e-predictor-77f5fb7585-jbvsj" podUID="b6f97e15-4f04-4c9a-ae13-68aeaa8e85de" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.28:8080: connect: connection refused"
Sep 16 13:00:13.614412 ip-10-0-133-218 kubenswrapper[2580]: I0916 13:00:13.614378    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-696c5cc6cc-5dsjm" podUID="3cf459e0-cf33-4718-98aa-9bd7e302c9f4" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.29:8080: connect: connection refused"
Sep 16 13:00:16.673340 ip-10-0-133-218 kubenswrapper[2580]: I0916 13:00:16.673310    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-ec80e-predictor-77f5fb7585-jbvsj"
Sep 16 13:00:16.673799 ip-10-0-133-218 kubenswrapper[2580]: I0916 13:00:16.673685    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-ec80e-predictor-77f5fb7585-jbvsj" podUID="b6f97e15-4f04-4c9a-ae13-68aeaa8e85de" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.28:8080: connect: connection refused"
Sep 16 13:00:23.615094 ip-10-0-133-218 kubenswrapper[2580]: I0916 13:00:23.615051    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-696c5cc6cc-5dsjm" podUID="3cf459e0-cf33-4718-98aa-9bd7e302c9f4" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.29:8080: connect: connection refused"
Sep 16 13:00:26.674692 ip-10-0-133-218 kubenswrapper[2580]: I0916 13:00:26.674647    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-ec80e-predictor-77f5fb7585-jbvsj" podUID="b6f97e15-4f04-4c9a-ae13-68aeaa8e85de" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.28:8080: connect: connection refused"
Sep 16 13:00:33.614447 ip-10-0-133-218 kubenswrapper[2580]: I0916 13:00:33.614405    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-696c5cc6cc-5dsjm" podUID="3cf459e0-cf33-4718-98aa-9bd7e302c9f4" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.29:8080: connect: connection refused"
Sep 16 13:00:36.674526 ip-10-0-133-218 kubenswrapper[2580]: I0916 13:00:36.674469    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-ec80e-predictor-77f5fb7585-jbvsj" podUID="b6f97e15-4f04-4c9a-ae13-68aeaa8e85de" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.28:8080: connect: connection refused"
Sep 16 13:00:43.615181 ip-10-0-133-218 kubenswrapper[2580]: I0916 13:00:43.615139    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-696c5cc6cc-5dsjm" podUID="3cf459e0-cf33-4718-98aa-9bd7e302c9f4" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.29:8080: connect: connection refused"
Sep 16 13:00:46.674034 ip-10-0-133-218 kubenswrapper[2580]: I0916 13:00:46.673990    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-ec80e-predictor-77f5fb7585-jbvsj" podUID="b6f97e15-4f04-4c9a-ae13-68aeaa8e85de" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.28:8080: connect: connection refused"
Sep 16 13:00:53.615068 ip-10-0-133-218 kubenswrapper[2580]: I0916 13:00:53.615026    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-696c5cc6cc-5dsjm" podUID="3cf459e0-cf33-4718-98aa-9bd7e302c9f4" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.29:8080: connect: connection refused"
Sep 16 13:00:56.674295 ip-10-0-133-218 kubenswrapper[2580]: I0916 13:00:56.674253    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-ec80e-predictor-77f5fb7585-jbvsj" podUID="b6f97e15-4f04-4c9a-ae13-68aeaa8e85de" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.28:8080: connect: connection refused"
Sep 16 13:01:03.615014 ip-10-0-133-218 kubenswrapper[2580]: I0916 13:01:03.614973    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-696c5cc6cc-5dsjm" podUID="3cf459e0-cf33-4718-98aa-9bd7e302c9f4" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.29:8080: connect: connection refused"
Sep 16 13:01:06.674747 ip-10-0-133-218 kubenswrapper[2580]: I0916 13:01:06.674707    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-ec80e-predictor-77f5fb7585-jbvsj" podUID="b6f97e15-4f04-4c9a-ae13-68aeaa8e85de" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.28:8080: connect: connection refused"
Sep 16 13:01:13.615281 ip-10-0-133-218 kubenswrapper[2580]: I0916 13:01:13.615250    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-696c5cc6cc-5dsjm"
Sep 16 13:01:16.674438 ip-10-0-133-218 kubenswrapper[2580]: I0916 13:01:16.674411    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-ec80e-predictor-77f5fb7585-jbvsj"
Sep 16 13:01:21.017904 ip-10-0-133-218 kubenswrapper[2580]: I0916 13:01:21.017873    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-predictor-696c5cc6cc-5dsjm"]
Sep 16 13:01:21.018405 ip-10-0-133-218 kubenswrapper[2580]: I0916 13:01:21.018120    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-696c5cc6cc-5dsjm" podUID="3cf459e0-cf33-4718-98aa-9bd7e302c9f4" containerName="kserve-container" containerID="cri-o://c31d4e06b9797d68bf8f7c4116085454603489d6f16dc824fca43e00d32656ef" gracePeriod=30
Sep 16 13:01:21.018405 ip-10-0-133-218 kubenswrapper[2580]: I0916 13:01:21.018142    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-696c5cc6cc-5dsjm" podUID="3cf459e0-cf33-4718-98aa-9bd7e302c9f4" containerName="kube-rbac-proxy" containerID="cri-o://c9f090b190d9b014402681488bf7fd8f3c452ca763be42c53d42cba767d42825" gracePeriod=30
Sep 16 13:01:21.950930 ip-10-0-133-218 kubenswrapper[2580]: I0916 13:01:21.950894    2580 generic.go:356] "Generic (PLEG): container finished" podID="3cf459e0-cf33-4718-98aa-9bd7e302c9f4" containerID="c9f090b190d9b014402681488bf7fd8f3c452ca763be42c53d42cba767d42825" exitCode=2
Sep 16 13:01:21.951098 ip-10-0-133-218 kubenswrapper[2580]: I0916 13:01:21.950960    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-696c5cc6cc-5dsjm" event={"ID":"3cf459e0-cf33-4718-98aa-9bd7e302c9f4","Type":"ContainerDied","Data":"c9f090b190d9b014402681488bf7fd8f3c452ca763be42c53d42cba767d42825"}
Sep 16 13:01:23.609848 ip-10-0-133-218 kubenswrapper[2580]: I0916 13:01:23.609811    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-696c5cc6cc-5dsjm" podUID="3cf459e0-cf33-4718-98aa-9bd7e302c9f4" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.29:8643/healthz\": dial tcp 10.133.0.29:8643: connect: connection refused"
Sep 16 13:01:23.615131 ip-10-0-133-218 kubenswrapper[2580]: I0916 13:01:23.615110    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-696c5cc6cc-5dsjm" podUID="3cf459e0-cf33-4718-98aa-9bd7e302c9f4" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.29:8080: connect: connection refused"
Sep 16 13:01:25.550561 ip-10-0-133-218 kubenswrapper[2580]: I0916 13:01:25.550537    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-696c5cc6cc-5dsjm"
Sep 16 13:01:25.609352 ip-10-0-133-218 kubenswrapper[2580]: I0916 13:01:25.609327    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/3cf459e0-cf33-4718-98aa-9bd7e302c9f4-proxy-tls\") pod \"3cf459e0-cf33-4718-98aa-9bd7e302c9f4\" (UID: \"3cf459e0-cf33-4718-98aa-9bd7e302c9f4\") "
Sep 16 13:01:25.609501 ip-10-0-133-218 kubenswrapper[2580]: I0916 13:01:25.609370    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/3cf459e0-cf33-4718-98aa-9bd7e302c9f4-kube-rbac-proxy-sar-config\") pod \"3cf459e0-cf33-4718-98aa-9bd7e302c9f4\" (UID: \"3cf459e0-cf33-4718-98aa-9bd7e302c9f4\") "
Sep 16 13:01:25.609501 ip-10-0-133-218 kubenswrapper[2580]: I0916 13:01:25.609410    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/3cf459e0-cf33-4718-98aa-9bd7e302c9f4-kserve-provision-location\") pod \"3cf459e0-cf33-4718-98aa-9bd7e302c9f4\" (UID: \"3cf459e0-cf33-4718-98aa-9bd7e302c9f4\") "
Sep 16 13:01:25.609501 ip-10-0-133-218 kubenswrapper[2580]: I0916 13:01:25.609427    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-m28bx\" (UniqueName: \"kubernetes.io/projected/3cf459e0-cf33-4718-98aa-9bd7e302c9f4-kube-api-access-m28bx\") pod \"3cf459e0-cf33-4718-98aa-9bd7e302c9f4\" (UID: \"3cf459e0-cf33-4718-98aa-9bd7e302c9f4\") "
Sep 16 13:01:25.609501 ip-10-0-133-218 kubenswrapper[2580]: I0916 13:01:25.609445    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/3cf459e0-cf33-4718-98aa-9bd7e302c9f4-cabundle-cert\") pod \"3cf459e0-cf33-4718-98aa-9bd7e302c9f4\" (UID: \"3cf459e0-cf33-4718-98aa-9bd7e302c9f4\") "
Sep 16 13:01:25.609788 ip-10-0-133-218 kubenswrapper[2580]: I0916 13:01:25.609756    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3cf459e0-cf33-4718-98aa-9bd7e302c9f4-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "3cf459e0-cf33-4718-98aa-9bd7e302c9f4" (UID: "3cf459e0-cf33-4718-98aa-9bd7e302c9f4"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 13:01:25.609907 ip-10-0-133-218 kubenswrapper[2580]: I0916 13:01:25.609847    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3cf459e0-cf33-4718-98aa-9bd7e302c9f4-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "3cf459e0-cf33-4718-98aa-9bd7e302c9f4" (UID: "3cf459e0-cf33-4718-98aa-9bd7e302c9f4"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 13:01:25.609907 ip-10-0-133-218 kubenswrapper[2580]: I0916 13:01:25.609862    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3cf459e0-cf33-4718-98aa-9bd7e302c9f4-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "3cf459e0-cf33-4718-98aa-9bd7e302c9f4" (UID: "3cf459e0-cf33-4718-98aa-9bd7e302c9f4"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 13:01:25.611443 ip-10-0-133-218 kubenswrapper[2580]: I0916 13:01:25.611427    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3cf459e0-cf33-4718-98aa-9bd7e302c9f4-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "3cf459e0-cf33-4718-98aa-9bd7e302c9f4" (UID: "3cf459e0-cf33-4718-98aa-9bd7e302c9f4"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 13:01:25.611586 ip-10-0-133-218 kubenswrapper[2580]: I0916 13:01:25.611568    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3cf459e0-cf33-4718-98aa-9bd7e302c9f4-kube-api-access-m28bx" (OuterVolumeSpecName: "kube-api-access-m28bx") pod "3cf459e0-cf33-4718-98aa-9bd7e302c9f4" (UID: "3cf459e0-cf33-4718-98aa-9bd7e302c9f4"). InnerVolumeSpecName "kube-api-access-m28bx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 13:01:25.710215 ip-10-0-133-218 kubenswrapper[2580]: I0916 13:01:25.710119    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m28bx\" (UniqueName: \"kubernetes.io/projected/3cf459e0-cf33-4718-98aa-9bd7e302c9f4-kube-api-access-m28bx\") on node \"ip-10-0-133-218.ec2.internal\" DevicePath \"\""
Sep 16 13:01:25.710215 ip-10-0-133-218 kubenswrapper[2580]: I0916 13:01:25.710152    2580 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/3cf459e0-cf33-4718-98aa-9bd7e302c9f4-cabundle-cert\") on node \"ip-10-0-133-218.ec2.internal\" DevicePath \"\""
Sep 16 13:01:25.710215 ip-10-0-133-218 kubenswrapper[2580]: I0916 13:01:25.710165    2580 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/3cf459e0-cf33-4718-98aa-9bd7e302c9f4-proxy-tls\") on node \"ip-10-0-133-218.ec2.internal\" DevicePath \"\""
Sep 16 13:01:25.710215 ip-10-0-133-218 kubenswrapper[2580]: I0916 13:01:25.710176    2580 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/3cf459e0-cf33-4718-98aa-9bd7e302c9f4-kube-rbac-proxy-sar-config\") on node \"ip-10-0-133-218.ec2.internal\" DevicePath \"\""
Sep 16 13:01:25.710215 ip-10-0-133-218 kubenswrapper[2580]: I0916 13:01:25.710188    2580 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/3cf459e0-cf33-4718-98aa-9bd7e302c9f4-kserve-provision-location\") on node \"ip-10-0-133-218.ec2.internal\" DevicePath \"\""
Sep 16 13:01:25.969108 ip-10-0-133-218 kubenswrapper[2580]: I0916 13:01:25.969023    2580 generic.go:356] "Generic (PLEG): container finished" podID="3cf459e0-cf33-4718-98aa-9bd7e302c9f4" containerID="c31d4e06b9797d68bf8f7c4116085454603489d6f16dc824fca43e00d32656ef" exitCode=0
Sep 16 13:01:25.969108 ip-10-0-133-218 kubenswrapper[2580]: I0916 13:01:25.969068    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-696c5cc6cc-5dsjm" event={"ID":"3cf459e0-cf33-4718-98aa-9bd7e302c9f4","Type":"ContainerDied","Data":"c31d4e06b9797d68bf8f7c4116085454603489d6f16dc824fca43e00d32656ef"}
Sep 16 13:01:25.969108 ip-10-0-133-218 kubenswrapper[2580]: I0916 13:01:25.969086    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-696c5cc6cc-5dsjm"
Sep 16 13:01:25.969340 ip-10-0-133-218 kubenswrapper[2580]: I0916 13:01:25.969129    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-696c5cc6cc-5dsjm" event={"ID":"3cf459e0-cf33-4718-98aa-9bd7e302c9f4","Type":"ContainerDied","Data":"95da654d3182953e061c17733f17299df9b0a8cbf27ad2b14cfc9b4de4654d93"}
Sep 16 13:01:25.969340 ip-10-0-133-218 kubenswrapper[2580]: I0916 13:01:25.969167    2580 scope.go:117] "RemoveContainer" containerID="c9f090b190d9b014402681488bf7fd8f3c452ca763be42c53d42cba767d42825"
Sep 16 13:01:25.977900 ip-10-0-133-218 kubenswrapper[2580]: I0916 13:01:25.977883    2580 scope.go:117] "RemoveContainer" containerID="c31d4e06b9797d68bf8f7c4116085454603489d6f16dc824fca43e00d32656ef"
Sep 16 13:01:25.984816 ip-10-0-133-218 kubenswrapper[2580]: I0916 13:01:25.984800    2580 scope.go:117] "RemoveContainer" containerID="967010abd177dfe621fc37f4ac96e8c7f0b8cb70d9ac9fff3da73125e694b78d"
Sep 16 13:01:25.991682 ip-10-0-133-218 kubenswrapper[2580]: I0916 13:01:25.991662    2580 scope.go:117] "RemoveContainer" containerID="c9f090b190d9b014402681488bf7fd8f3c452ca763be42c53d42cba767d42825"
Sep 16 13:01:25.991926 ip-10-0-133-218 kubenswrapper[2580]: E0916 13:01:25.991908    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c9f090b190d9b014402681488bf7fd8f3c452ca763be42c53d42cba767d42825\": container with ID starting with c9f090b190d9b014402681488bf7fd8f3c452ca763be42c53d42cba767d42825 not found: ID does not exist" containerID="c9f090b190d9b014402681488bf7fd8f3c452ca763be42c53d42cba767d42825"
Sep 16 13:01:25.991974 ip-10-0-133-218 kubenswrapper[2580]: I0916 13:01:25.991934    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c9f090b190d9b014402681488bf7fd8f3c452ca763be42c53d42cba767d42825"} err="failed to get container status \"c9f090b190d9b014402681488bf7fd8f3c452ca763be42c53d42cba767d42825\": rpc error: code = NotFound desc = could not find container \"c9f090b190d9b014402681488bf7fd8f3c452ca763be42c53d42cba767d42825\": container with ID starting with c9f090b190d9b014402681488bf7fd8f3c452ca763be42c53d42cba767d42825 not found: ID does not exist"
Sep 16 13:01:25.991974 ip-10-0-133-218 kubenswrapper[2580]: I0916 13:01:25.991951    2580 scope.go:117] "RemoveContainer" containerID="c31d4e06b9797d68bf8f7c4116085454603489d6f16dc824fca43e00d32656ef"
Sep 16 13:01:25.992335 ip-10-0-133-218 kubenswrapper[2580]: E0916 13:01:25.992178    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c31d4e06b9797d68bf8f7c4116085454603489d6f16dc824fca43e00d32656ef\": container with ID starting with c31d4e06b9797d68bf8f7c4116085454603489d6f16dc824fca43e00d32656ef not found: ID does not exist" containerID="c31d4e06b9797d68bf8f7c4116085454603489d6f16dc824fca43e00d32656ef"
Sep 16 13:01:25.992335 ip-10-0-133-218 kubenswrapper[2580]: I0916 13:01:25.992199    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c31d4e06b9797d68bf8f7c4116085454603489d6f16dc824fca43e00d32656ef"} err="failed to get container status \"c31d4e06b9797d68bf8f7c4116085454603489d6f16dc824fca43e00d32656ef\": rpc error: code = NotFound desc = could not find container \"c31d4e06b9797d68bf8f7c4116085454603489d6f16dc824fca43e00d32656ef\": container with ID starting with c31d4e06b9797d68bf8f7c4116085454603489d6f16dc824fca43e00d32656ef not found: ID does not exist"
Sep 16 13:01:25.992335 ip-10-0-133-218 kubenswrapper[2580]: I0916 13:01:25.992211    2580 scope.go:117] "RemoveContainer" containerID="967010abd177dfe621fc37f4ac96e8c7f0b8cb70d9ac9fff3da73125e694b78d"
Sep 16 13:01:25.992449 ip-10-0-133-218 kubenswrapper[2580]: E0916 13:01:25.992426    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"967010abd177dfe621fc37f4ac96e8c7f0b8cb70d9ac9fff3da73125e694b78d\": container with ID starting with 967010abd177dfe621fc37f4ac96e8c7f0b8cb70d9ac9fff3da73125e694b78d not found: ID does not exist" containerID="967010abd177dfe621fc37f4ac96e8c7f0b8cb70d9ac9fff3da73125e694b78d"
Sep 16 13:01:25.992449 ip-10-0-133-218 kubenswrapper[2580]: I0916 13:01:25.992440    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"967010abd177dfe621fc37f4ac96e8c7f0b8cb70d9ac9fff3da73125e694b78d"} err="failed to get container status \"967010abd177dfe621fc37f4ac96e8c7f0b8cb70d9ac9fff3da73125e694b78d\": rpc error: code = NotFound desc = could not find container \"967010abd177dfe621fc37f4ac96e8c7f0b8cb70d9ac9fff3da73125e694b78d\": container with ID starting with 967010abd177dfe621fc37f4ac96e8c7f0b8cb70d9ac9fff3da73125e694b78d not found: ID does not exist"
Sep 16 13:01:25.997704 ip-10-0-133-218 kubenswrapper[2580]: I0916 13:01:25.997683    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-predictor-696c5cc6cc-5dsjm"]
Sep 16 13:01:26.005248 ip-10-0-133-218 kubenswrapper[2580]: I0916 13:01:26.005229    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-predictor-696c5cc6cc-5dsjm"]
Sep 16 13:01:26.260965 ip-10-0-133-218 kubenswrapper[2580]: I0916 13:01:26.260888    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-775495c6b4-hl5vj"]
Sep 16 13:01:26.261224 ip-10-0-133-218 kubenswrapper[2580]: I0916 13:01:26.261209    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a65e5a0e-beb3-4b27-a076-130b38b51d49" containerName="storage-initializer"
Sep 16 13:01:26.261270 ip-10-0-133-218 kubenswrapper[2580]: I0916 13:01:26.261225    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="a65e5a0e-beb3-4b27-a076-130b38b51d49" containerName="storage-initializer"
Sep 16 13:01:26.261270 ip-10-0-133-218 kubenswrapper[2580]: I0916 13:01:26.261235    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3cf459e0-cf33-4718-98aa-9bd7e302c9f4" containerName="kube-rbac-proxy"
Sep 16 13:01:26.261270 ip-10-0-133-218 kubenswrapper[2580]: I0916 13:01:26.261241    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="3cf459e0-cf33-4718-98aa-9bd7e302c9f4" containerName="kube-rbac-proxy"
Sep 16 13:01:26.261270 ip-10-0-133-218 kubenswrapper[2580]: I0916 13:01:26.261250    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a65e5a0e-beb3-4b27-a076-130b38b51d49" containerName="kube-rbac-proxy"
Sep 16 13:01:26.261270 ip-10-0-133-218 kubenswrapper[2580]: I0916 13:01:26.261255    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="a65e5a0e-beb3-4b27-a076-130b38b51d49" containerName="kube-rbac-proxy"
Sep 16 13:01:26.261270 ip-10-0-133-218 kubenswrapper[2580]: I0916 13:01:26.261267    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3cf459e0-cf33-4718-98aa-9bd7e302c9f4" containerName="storage-initializer"
Sep 16 13:01:26.261270 ip-10-0-133-218 kubenswrapper[2580]: I0916 13:01:26.261273    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="3cf459e0-cf33-4718-98aa-9bd7e302c9f4" containerName="storage-initializer"
Sep 16 13:01:26.261562 ip-10-0-133-218 kubenswrapper[2580]: I0916 13:01:26.261282    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3cf459e0-cf33-4718-98aa-9bd7e302c9f4" containerName="kserve-container"
Sep 16 13:01:26.261562 ip-10-0-133-218 kubenswrapper[2580]: I0916 13:01:26.261287    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="3cf459e0-cf33-4718-98aa-9bd7e302c9f4" containerName="kserve-container"
Sep 16 13:01:26.261562 ip-10-0-133-218 kubenswrapper[2580]: I0916 13:01:26.261295    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a65e5a0e-beb3-4b27-a076-130b38b51d49" containerName="kserve-container"
Sep 16 13:01:26.261562 ip-10-0-133-218 kubenswrapper[2580]: I0916 13:01:26.261300    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="a65e5a0e-beb3-4b27-a076-130b38b51d49" containerName="kserve-container"
Sep 16 13:01:26.261562 ip-10-0-133-218 kubenswrapper[2580]: I0916 13:01:26.261346    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="a65e5a0e-beb3-4b27-a076-130b38b51d49" containerName="kserve-container"
Sep 16 13:01:26.261562 ip-10-0-133-218 kubenswrapper[2580]: I0916 13:01:26.261353    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="3cf459e0-cf33-4718-98aa-9bd7e302c9f4" containerName="kserve-container"
Sep 16 13:01:26.261562 ip-10-0-133-218 kubenswrapper[2580]: I0916 13:01:26.261361    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="a65e5a0e-beb3-4b27-a076-130b38b51d49" containerName="kube-rbac-proxy"
Sep 16 13:01:26.261562 ip-10-0-133-218 kubenswrapper[2580]: I0916 13:01:26.261367    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="3cf459e0-cf33-4718-98aa-9bd7e302c9f4" containerName="kube-rbac-proxy"
Sep 16 13:01:26.264480 ip-10-0-133-218 kubenswrapper[2580]: I0916 13:01:26.264464    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-775495c6b4-hl5vj"
Sep 16 13:01:26.267026 ip-10-0-133-218 kubenswrapper[2580]: I0916 13:01:26.267008    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-stop-kube-rbac-proxy-sar-config\""
Sep 16 13:01:26.267235 ip-10-0-133-218 kubenswrapper[2580]: I0916 13:01:26.267221    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-stop-v2-predictor-serving-cert\""
Sep 16 13:01:26.280970 ip-10-0-133-218 kubenswrapper[2580]: I0916 13:01:26.280946    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-775495c6b4-hl5vj"]
Sep 16 13:01:26.415823 ip-10-0-133-218 kubenswrapper[2580]: I0916 13:01:26.415786    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hwpb8\" (UniqueName: \"kubernetes.io/projected/a099972a-5c02-432b-a4bf-40850f65fe4f-kube-api-access-hwpb8\") pod \"isvc-canary-stop-v2-predictor-775495c6b4-hl5vj\" (UID: \"a099972a-5c02-432b-a4bf-40850f65fe4f\") " pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-775495c6b4-hl5vj"
Sep 16 13:01:26.415823 ip-10-0-133-218 kubenswrapper[2580]: I0916 13:01:26.415828    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a099972a-5c02-432b-a4bf-40850f65fe4f-proxy-tls\") pod \"isvc-canary-stop-v2-predictor-775495c6b4-hl5vj\" (UID: \"a099972a-5c02-432b-a4bf-40850f65fe4f\") " pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-775495c6b4-hl5vj"
Sep 16 13:01:26.416047 ip-10-0-133-218 kubenswrapper[2580]: I0916 13:01:26.415851    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/a099972a-5c02-432b-a4bf-40850f65fe4f-cabundle-cert\") pod \"isvc-canary-stop-v2-predictor-775495c6b4-hl5vj\" (UID: \"a099972a-5c02-432b-a4bf-40850f65fe4f\") " pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-775495c6b4-hl5vj"
Sep 16 13:01:26.416047 ip-10-0-133-218 kubenswrapper[2580]: I0916 13:01:26.415937    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a099972a-5c02-432b-a4bf-40850f65fe4f-kube-rbac-proxy-sar-config\") pod \"isvc-canary-stop-v2-predictor-775495c6b4-hl5vj\" (UID: \"a099972a-5c02-432b-a4bf-40850f65fe4f\") " pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-775495c6b4-hl5vj"
Sep 16 13:01:26.416047 ip-10-0-133-218 kubenswrapper[2580]: I0916 13:01:26.415988    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a099972a-5c02-432b-a4bf-40850f65fe4f-kserve-provision-location\") pod \"isvc-canary-stop-v2-predictor-775495c6b4-hl5vj\" (UID: \"a099972a-5c02-432b-a4bf-40850f65fe4f\") " pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-775495c6b4-hl5vj"
Sep 16 13:01:26.516820 ip-10-0-133-218 kubenswrapper[2580]: I0916 13:01:26.516742    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a099972a-5c02-432b-a4bf-40850f65fe4f-kserve-provision-location\") pod \"isvc-canary-stop-v2-predictor-775495c6b4-hl5vj\" (UID: \"a099972a-5c02-432b-a4bf-40850f65fe4f\") " pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-775495c6b4-hl5vj"
Sep 16 13:01:26.516820 ip-10-0-133-218 kubenswrapper[2580]: I0916 13:01:26.516797    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hwpb8\" (UniqueName: \"kubernetes.io/projected/a099972a-5c02-432b-a4bf-40850f65fe4f-kube-api-access-hwpb8\") pod \"isvc-canary-stop-v2-predictor-775495c6b4-hl5vj\" (UID: \"a099972a-5c02-432b-a4bf-40850f65fe4f\") " pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-775495c6b4-hl5vj"
Sep 16 13:01:26.516996 ip-10-0-133-218 kubenswrapper[2580]: I0916 13:01:26.516910    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a099972a-5c02-432b-a4bf-40850f65fe4f-proxy-tls\") pod \"isvc-canary-stop-v2-predictor-775495c6b4-hl5vj\" (UID: \"a099972a-5c02-432b-a4bf-40850f65fe4f\") " pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-775495c6b4-hl5vj"
Sep 16 13:01:26.516996 ip-10-0-133-218 kubenswrapper[2580]: I0916 13:01:26.516952    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/a099972a-5c02-432b-a4bf-40850f65fe4f-cabundle-cert\") pod \"isvc-canary-stop-v2-predictor-775495c6b4-hl5vj\" (UID: \"a099972a-5c02-432b-a4bf-40850f65fe4f\") " pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-775495c6b4-hl5vj"
Sep 16 13:01:26.517125 ip-10-0-133-218 kubenswrapper[2580]: I0916 13:01:26.517100    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a099972a-5c02-432b-a4bf-40850f65fe4f-kube-rbac-proxy-sar-config\") pod \"isvc-canary-stop-v2-predictor-775495c6b4-hl5vj\" (UID: \"a099972a-5c02-432b-a4bf-40850f65fe4f\") " pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-775495c6b4-hl5vj"
Sep 16 13:01:26.517187 ip-10-0-133-218 kubenswrapper[2580]: I0916 13:01:26.517161    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a099972a-5c02-432b-a4bf-40850f65fe4f-kserve-provision-location\") pod \"isvc-canary-stop-v2-predictor-775495c6b4-hl5vj\" (UID: \"a099972a-5c02-432b-a4bf-40850f65fe4f\") " pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-775495c6b4-hl5vj"
Sep 16 13:01:26.517717 ip-10-0-133-218 kubenswrapper[2580]: I0916 13:01:26.517695    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/a099972a-5c02-432b-a4bf-40850f65fe4f-cabundle-cert\") pod \"isvc-canary-stop-v2-predictor-775495c6b4-hl5vj\" (UID: \"a099972a-5c02-432b-a4bf-40850f65fe4f\") " pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-775495c6b4-hl5vj"
Sep 16 13:01:26.517788 ip-10-0-133-218 kubenswrapper[2580]: I0916 13:01:26.517775    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a099972a-5c02-432b-a4bf-40850f65fe4f-kube-rbac-proxy-sar-config\") pod \"isvc-canary-stop-v2-predictor-775495c6b4-hl5vj\" (UID: \"a099972a-5c02-432b-a4bf-40850f65fe4f\") " pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-775495c6b4-hl5vj"
Sep 16 13:01:26.519275 ip-10-0-133-218 kubenswrapper[2580]: I0916 13:01:26.519259    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a099972a-5c02-432b-a4bf-40850f65fe4f-proxy-tls\") pod \"isvc-canary-stop-v2-predictor-775495c6b4-hl5vj\" (UID: \"a099972a-5c02-432b-a4bf-40850f65fe4f\") " pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-775495c6b4-hl5vj"
Sep 16 13:01:26.525183 ip-10-0-133-218 kubenswrapper[2580]: I0916 13:01:26.525157    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hwpb8\" (UniqueName: \"kubernetes.io/projected/a099972a-5c02-432b-a4bf-40850f65fe4f-kube-api-access-hwpb8\") pod \"isvc-canary-stop-v2-predictor-775495c6b4-hl5vj\" (UID: \"a099972a-5c02-432b-a4bf-40850f65fe4f\") " pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-775495c6b4-hl5vj"
Sep 16 13:01:26.574986 ip-10-0-133-218 kubenswrapper[2580]: I0916 13:01:26.574953    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-775495c6b4-hl5vj"
Sep 16 13:01:26.854268 ip-10-0-133-218 kubenswrapper[2580]: I0916 13:01:26.854239    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-775495c6b4-hl5vj"]
Sep 16 13:01:26.856321 ip-10-0-133-218 kubenswrapper[2580]: W0916 13:01:26.856298    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda099972a_5c02_432b_a4bf_40850f65fe4f.slice/crio-302a13c5f6e4702c02bc9a743668cde37cf2c07fd84cd2a08566e7ee2d4975b6 WatchSource:0}: Error finding container 302a13c5f6e4702c02bc9a743668cde37cf2c07fd84cd2a08566e7ee2d4975b6: Status 404 returned error can't find the container with id 302a13c5f6e4702c02bc9a743668cde37cf2c07fd84cd2a08566e7ee2d4975b6
Sep 16 13:01:26.974999 ip-10-0-133-218 kubenswrapper[2580]: I0916 13:01:26.974961    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-775495c6b4-hl5vj" event={"ID":"a099972a-5c02-432b-a4bf-40850f65fe4f","Type":"ContainerStarted","Data":"d129c93cb5588a8d90dd6c1ba10ca9f13c871af2c0fa345808d9948d8a4fba98"}
Sep 16 13:01:26.975178 ip-10-0-133-218 kubenswrapper[2580]: I0916 13:01:26.975027    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-775495c6b4-hl5vj" event={"ID":"a099972a-5c02-432b-a4bf-40850f65fe4f","Type":"ContainerStarted","Data":"302a13c5f6e4702c02bc9a743668cde37cf2c07fd84cd2a08566e7ee2d4975b6"}
Sep 16 13:01:27.599195 ip-10-0-133-218 kubenswrapper[2580]: I0916 13:01:27.599162    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3cf459e0-cf33-4718-98aa-9bd7e302c9f4" path="/var/lib/kubelet/pods/3cf459e0-cf33-4718-98aa-9bd7e302c9f4/volumes"
Sep 16 13:01:32.002583 ip-10-0-133-218 kubenswrapper[2580]: I0916 13:01:32.002548    2580 generic.go:356] "Generic (PLEG): container finished" podID="a099972a-5c02-432b-a4bf-40850f65fe4f" containerID="d129c93cb5588a8d90dd6c1ba10ca9f13c871af2c0fa345808d9948d8a4fba98" exitCode=0
Sep 16 13:01:32.002958 ip-10-0-133-218 kubenswrapper[2580]: I0916 13:01:32.002624    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-775495c6b4-hl5vj" event={"ID":"a099972a-5c02-432b-a4bf-40850f65fe4f","Type":"ContainerDied","Data":"d129c93cb5588a8d90dd6c1ba10ca9f13c871af2c0fa345808d9948d8a4fba98"}
Sep 16 13:01:33.008928 ip-10-0-133-218 kubenswrapper[2580]: I0916 13:01:33.008895    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-775495c6b4-hl5vj" event={"ID":"a099972a-5c02-432b-a4bf-40850f65fe4f","Type":"ContainerStarted","Data":"116e86bb1fdc70046712810272d808a9328fd3ea0e34b93ac370bbea77b653d3"}
Sep 16 13:01:33.009327 ip-10-0-133-218 kubenswrapper[2580]: I0916 13:01:33.008954    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-775495c6b4-hl5vj" event={"ID":"a099972a-5c02-432b-a4bf-40850f65fe4f","Type":"ContainerStarted","Data":"033b6f7e03e03dd672c2932453f57f4c7d82dbd3de3350fac67e2a13879d89e6"}
Sep 16 13:01:33.009327 ip-10-0-133-218 kubenswrapper[2580]: I0916 13:01:33.009141    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-775495c6b4-hl5vj"
Sep 16 13:01:33.032578 ip-10-0-133-218 kubenswrapper[2580]: I0916 13:01:33.032528    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-775495c6b4-hl5vj" podStartSLOduration=2.558089768 podStartE2EDuration="7.032512055s" totalImagesPullingTime="0s" totalInitContainerRuntime="4.474422287s" isStatefulPod=false podCreationTimestamp="2026-09-16 13:01:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 13:01:33.031282949 +0000 UTC m=+831.954155428" watchObservedRunningTime="2026-09-16 13:01:33.032512055 +0000 UTC m=+831.955384910"
Sep 16 13:01:34.016106 ip-10-0-133-218 kubenswrapper[2580]: I0916 13:01:34.016074    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-775495c6b4-hl5vj"
Sep 16 13:01:34.016708 ip-10-0-133-218 kubenswrapper[2580]: I0916 13:01:34.016105    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-775495c6b4-hl5vj" podUID="a099972a-5c02-432b-a4bf-40850f65fe4f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.30:8080: connect: connection refused"
Sep 16 13:01:35.017051 ip-10-0-133-218 kubenswrapper[2580]: I0916 13:01:35.017012    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-775495c6b4-hl5vj" podUID="a099972a-5c02-432b-a4bf-40850f65fe4f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.30:8080: connect: connection refused"
Sep 16 13:01:36.702396 ip-10-0-133-218 kubenswrapper[2580]: I0916 13:01:36.702231    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-5b43d-predictor-5c567f8fb-9gwrq"]
Sep 16 13:01:36.705275 ip-10-0-133-218 kubenswrapper[2580]: I0916 13:01:36.705246    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-5b43d-predictor-5c567f8fb-9gwrq"
Sep 16 13:01:36.709845 ip-10-0-133-218 kubenswrapper[2580]: I0916 13:01:36.709814    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-graph-raw-hpa-5b43d-kube-rbac-proxy-sar-config\""
Sep 16 13:01:36.709988 ip-10-0-133-218 kubenswrapper[2580]: I0916 13:01:36.709851    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-graph-raw-hpa-5b43d-predictor-serving-cert\""
Sep 16 13:01:36.729347 ip-10-0-133-218 kubenswrapper[2580]: I0916 13:01:36.729316    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-5b43d-predictor-5c567f8fb-9gwrq"]
Sep 16 13:01:36.771954 ip-10-0-133-218 kubenswrapper[2580]: I0916 13:01:36.771924    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-graph-raw-ec80e-predictor-77f5fb7585-jbvsj"]
Sep 16 13:01:36.772160 ip-10-0-133-218 kubenswrapper[2580]: I0916 13:01:36.772134    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-ec80e-predictor-77f5fb7585-jbvsj" podUID="b6f97e15-4f04-4c9a-ae13-68aeaa8e85de" containerName="kserve-container" containerID="cri-o://8b0cd92b7874d06dc1552b779c41f8f1ae09ddee5a0ee25de24a6d3d7370a94e" gracePeriod=30
Sep 16 13:01:36.772305 ip-10-0-133-218 kubenswrapper[2580]: I0916 13:01:36.772172    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-ec80e-predictor-77f5fb7585-jbvsj" podUID="b6f97e15-4f04-4c9a-ae13-68aeaa8e85de" containerName="kube-rbac-proxy" containerID="cri-o://4dc8be51c30bf3de2279e1e58b5e61b95be379e8bfa94209ca8c17fffda42866" gracePeriod=30
Sep 16 13:01:36.802325 ip-10-0-133-218 kubenswrapper[2580]: I0916 13:01:36.802297    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/ccf0e9b1-5f46-45e7-8192-65d957d2ffd6-cabundle-cert\") pod \"isvc-sklearn-graph-raw-hpa-5b43d-predictor-5c567f8fb-9gwrq\" (UID: \"ccf0e9b1-5f46-45e7-8192-65d957d2ffd6\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-5b43d-predictor-5c567f8fb-9gwrq"
Sep 16 13:01:36.802436 ip-10-0-133-218 kubenswrapper[2580]: I0916 13:01:36.802339    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ccf0e9b1-5f46-45e7-8192-65d957d2ffd6-kserve-provision-location\") pod \"isvc-sklearn-graph-raw-hpa-5b43d-predictor-5c567f8fb-9gwrq\" (UID: \"ccf0e9b1-5f46-45e7-8192-65d957d2ffd6\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-5b43d-predictor-5c567f8fb-9gwrq"
Sep 16 13:01:36.802481 ip-10-0-133-218 kubenswrapper[2580]: I0916 13:01:36.802441    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ccf0e9b1-5f46-45e7-8192-65d957d2ffd6-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-graph-raw-hpa-5b43d-predictor-5c567f8fb-9gwrq\" (UID: \"ccf0e9b1-5f46-45e7-8192-65d957d2ffd6\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-5b43d-predictor-5c567f8fb-9gwrq"
Sep 16 13:01:36.802481 ip-10-0-133-218 kubenswrapper[2580]: I0916 13:01:36.802467    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cnjpn\" (UniqueName: \"kubernetes.io/projected/ccf0e9b1-5f46-45e7-8192-65d957d2ffd6-kube-api-access-cnjpn\") pod \"isvc-sklearn-graph-raw-hpa-5b43d-predictor-5c567f8fb-9gwrq\" (UID: \"ccf0e9b1-5f46-45e7-8192-65d957d2ffd6\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-5b43d-predictor-5c567f8fb-9gwrq"
Sep 16 13:01:36.802576 ip-10-0-133-218 kubenswrapper[2580]: I0916 13:01:36.802510    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ccf0e9b1-5f46-45e7-8192-65d957d2ffd6-proxy-tls\") pod \"isvc-sklearn-graph-raw-hpa-5b43d-predictor-5c567f8fb-9gwrq\" (UID: \"ccf0e9b1-5f46-45e7-8192-65d957d2ffd6\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-5b43d-predictor-5c567f8fb-9gwrq"
Sep 16 13:01:36.903378 ip-10-0-133-218 kubenswrapper[2580]: I0916 13:01:36.903346    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ccf0e9b1-5f46-45e7-8192-65d957d2ffd6-kserve-provision-location\") pod \"isvc-sklearn-graph-raw-hpa-5b43d-predictor-5c567f8fb-9gwrq\" (UID: \"ccf0e9b1-5f46-45e7-8192-65d957d2ffd6\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-5b43d-predictor-5c567f8fb-9gwrq"
Sep 16 13:01:36.903553 ip-10-0-133-218 kubenswrapper[2580]: I0916 13:01:36.903476    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ccf0e9b1-5f46-45e7-8192-65d957d2ffd6-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-graph-raw-hpa-5b43d-predictor-5c567f8fb-9gwrq\" (UID: \"ccf0e9b1-5f46-45e7-8192-65d957d2ffd6\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-5b43d-predictor-5c567f8fb-9gwrq"
Sep 16 13:01:36.903553 ip-10-0-133-218 kubenswrapper[2580]: I0916 13:01:36.903522    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cnjpn\" (UniqueName: \"kubernetes.io/projected/ccf0e9b1-5f46-45e7-8192-65d957d2ffd6-kube-api-access-cnjpn\") pod \"isvc-sklearn-graph-raw-hpa-5b43d-predictor-5c567f8fb-9gwrq\" (UID: \"ccf0e9b1-5f46-45e7-8192-65d957d2ffd6\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-5b43d-predictor-5c567f8fb-9gwrq"
Sep 16 13:01:36.903553 ip-10-0-133-218 kubenswrapper[2580]: I0916 13:01:36.903548    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ccf0e9b1-5f46-45e7-8192-65d957d2ffd6-proxy-tls\") pod \"isvc-sklearn-graph-raw-hpa-5b43d-predictor-5c567f8fb-9gwrq\" (UID: \"ccf0e9b1-5f46-45e7-8192-65d957d2ffd6\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-5b43d-predictor-5c567f8fb-9gwrq"
Sep 16 13:01:36.903731 ip-10-0-133-218 kubenswrapper[2580]: I0916 13:01:36.903595    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/ccf0e9b1-5f46-45e7-8192-65d957d2ffd6-cabundle-cert\") pod \"isvc-sklearn-graph-raw-hpa-5b43d-predictor-5c567f8fb-9gwrq\" (UID: \"ccf0e9b1-5f46-45e7-8192-65d957d2ffd6\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-5b43d-predictor-5c567f8fb-9gwrq"
Sep 16 13:01:36.903860 ip-10-0-133-218 kubenswrapper[2580]: I0916 13:01:36.903831    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ccf0e9b1-5f46-45e7-8192-65d957d2ffd6-kserve-provision-location\") pod \"isvc-sklearn-graph-raw-hpa-5b43d-predictor-5c567f8fb-9gwrq\" (UID: \"ccf0e9b1-5f46-45e7-8192-65d957d2ffd6\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-5b43d-predictor-5c567f8fb-9gwrq"
Sep 16 13:01:36.904121 ip-10-0-133-218 kubenswrapper[2580]: I0916 13:01:36.904098    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ccf0e9b1-5f46-45e7-8192-65d957d2ffd6-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-graph-raw-hpa-5b43d-predictor-5c567f8fb-9gwrq\" (UID: \"ccf0e9b1-5f46-45e7-8192-65d957d2ffd6\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-5b43d-predictor-5c567f8fb-9gwrq"
Sep 16 13:01:36.904196 ip-10-0-133-218 kubenswrapper[2580]: I0916 13:01:36.904178    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/ccf0e9b1-5f46-45e7-8192-65d957d2ffd6-cabundle-cert\") pod \"isvc-sklearn-graph-raw-hpa-5b43d-predictor-5c567f8fb-9gwrq\" (UID: \"ccf0e9b1-5f46-45e7-8192-65d957d2ffd6\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-5b43d-predictor-5c567f8fb-9gwrq"
Sep 16 13:01:36.905816 ip-10-0-133-218 kubenswrapper[2580]: I0916 13:01:36.905795    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ccf0e9b1-5f46-45e7-8192-65d957d2ffd6-proxy-tls\") pod \"isvc-sklearn-graph-raw-hpa-5b43d-predictor-5c567f8fb-9gwrq\" (UID: \"ccf0e9b1-5f46-45e7-8192-65d957d2ffd6\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-5b43d-predictor-5c567f8fb-9gwrq"
Sep 16 13:01:36.914272 ip-10-0-133-218 kubenswrapper[2580]: I0916 13:01:36.914251    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cnjpn\" (UniqueName: \"kubernetes.io/projected/ccf0e9b1-5f46-45e7-8192-65d957d2ffd6-kube-api-access-cnjpn\") pod \"isvc-sklearn-graph-raw-hpa-5b43d-predictor-5c567f8fb-9gwrq\" (UID: \"ccf0e9b1-5f46-45e7-8192-65d957d2ffd6\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-5b43d-predictor-5c567f8fb-9gwrq"
Sep 16 13:01:37.018719 ip-10-0-133-218 kubenswrapper[2580]: I0916 13:01:37.018634    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-5b43d-predictor-5c567f8fb-9gwrq"
Sep 16 13:01:37.027369 ip-10-0-133-218 kubenswrapper[2580]: I0916 13:01:37.027343    2580 generic.go:356] "Generic (PLEG): container finished" podID="b6f97e15-4f04-4c9a-ae13-68aeaa8e85de" containerID="4dc8be51c30bf3de2279e1e58b5e61b95be379e8bfa94209ca8c17fffda42866" exitCode=2
Sep 16 13:01:37.027470 ip-10-0-133-218 kubenswrapper[2580]: I0916 13:01:37.027408    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-ec80e-predictor-77f5fb7585-jbvsj" event={"ID":"b6f97e15-4f04-4c9a-ae13-68aeaa8e85de","Type":"ContainerDied","Data":"4dc8be51c30bf3de2279e1e58b5e61b95be379e8bfa94209ca8c17fffda42866"}
Sep 16 13:01:37.306611 ip-10-0-133-218 kubenswrapper[2580]: I0916 13:01:37.306583    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-5b43d-predictor-5c567f8fb-9gwrq"]
Sep 16 13:01:37.308577 ip-10-0-133-218 kubenswrapper[2580]: W0916 13:01:37.308550    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podccf0e9b1_5f46_45e7_8192_65d957d2ffd6.slice/crio-d6b49bdbe5ae5150f54e5816c431df0f9a82ee6570342a81f7d90248557de04b WatchSource:0}: Error finding container d6b49bdbe5ae5150f54e5816c431df0f9a82ee6570342a81f7d90248557de04b: Status 404 returned error can't find the container with id d6b49bdbe5ae5150f54e5816c431df0f9a82ee6570342a81f7d90248557de04b
Sep 16 13:01:38.032845 ip-10-0-133-218 kubenswrapper[2580]: I0916 13:01:38.032805    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-5b43d-predictor-5c567f8fb-9gwrq" event={"ID":"ccf0e9b1-5f46-45e7-8192-65d957d2ffd6","Type":"ContainerStarted","Data":"bdd7bc6c714d04afb0cd976fe336d11c78ec4d1e58905361b30db5263fe3d496"}
Sep 16 13:01:38.033280 ip-10-0-133-218 kubenswrapper[2580]: I0916 13:01:38.032869    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-5b43d-predictor-5c567f8fb-9gwrq" event={"ID":"ccf0e9b1-5f46-45e7-8192-65d957d2ffd6","Type":"ContainerStarted","Data":"d6b49bdbe5ae5150f54e5816c431df0f9a82ee6570342a81f7d90248557de04b"}
Sep 16 13:01:40.021727 ip-10-0-133-218 kubenswrapper[2580]: I0916 13:01:40.021697    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-775495c6b4-hl5vj"
Sep 16 13:01:40.022376 ip-10-0-133-218 kubenswrapper[2580]: I0916 13:01:40.022346    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-775495c6b4-hl5vj" podUID="a099972a-5c02-432b-a4bf-40850f65fe4f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.30:8080: connect: connection refused"
Sep 16 13:01:40.315883 ip-10-0-133-218 kubenswrapper[2580]: I0916 13:01:40.315853    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-ec80e-predictor-77f5fb7585-jbvsj"
Sep 16 13:01:40.437417 ip-10-0-133-218 kubenswrapper[2580]: I0916 13:01:40.437365    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/b6f97e15-4f04-4c9a-ae13-68aeaa8e85de-cabundle-cert\") pod \"b6f97e15-4f04-4c9a-ae13-68aeaa8e85de\" (UID: \"b6f97e15-4f04-4c9a-ae13-68aeaa8e85de\") "
Sep 16 13:01:40.437417 ip-10-0-133-218 kubenswrapper[2580]: I0916 13:01:40.437428    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/b6f97e15-4f04-4c9a-ae13-68aeaa8e85de-kube-rbac-proxy-sar-config\") pod \"b6f97e15-4f04-4c9a-ae13-68aeaa8e85de\" (UID: \"b6f97e15-4f04-4c9a-ae13-68aeaa8e85de\") "
Sep 16 13:01:40.437677 ip-10-0-133-218 kubenswrapper[2580]: I0916 13:01:40.437470    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-k6j6v\" (UniqueName: \"kubernetes.io/projected/b6f97e15-4f04-4c9a-ae13-68aeaa8e85de-kube-api-access-k6j6v\") pod \"b6f97e15-4f04-4c9a-ae13-68aeaa8e85de\" (UID: \"b6f97e15-4f04-4c9a-ae13-68aeaa8e85de\") "
Sep 16 13:01:40.437677 ip-10-0-133-218 kubenswrapper[2580]: I0916 13:01:40.437538    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b6f97e15-4f04-4c9a-ae13-68aeaa8e85de-proxy-tls\") pod \"b6f97e15-4f04-4c9a-ae13-68aeaa8e85de\" (UID: \"b6f97e15-4f04-4c9a-ae13-68aeaa8e85de\") "
Sep 16 13:01:40.437771 ip-10-0-133-218 kubenswrapper[2580]: I0916 13:01:40.437706    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b6f97e15-4f04-4c9a-ae13-68aeaa8e85de-kserve-provision-location\") pod \"b6f97e15-4f04-4c9a-ae13-68aeaa8e85de\" (UID: \"b6f97e15-4f04-4c9a-ae13-68aeaa8e85de\") "
Sep 16 13:01:40.437871 ip-10-0-133-218 kubenswrapper[2580]: I0916 13:01:40.437846    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b6f97e15-4f04-4c9a-ae13-68aeaa8e85de-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "b6f97e15-4f04-4c9a-ae13-68aeaa8e85de" (UID: "b6f97e15-4f04-4c9a-ae13-68aeaa8e85de"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 13:01:40.437940 ip-10-0-133-218 kubenswrapper[2580]: I0916 13:01:40.437860    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b6f97e15-4f04-4c9a-ae13-68aeaa8e85de-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "b6f97e15-4f04-4c9a-ae13-68aeaa8e85de" (UID: "b6f97e15-4f04-4c9a-ae13-68aeaa8e85de"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 13:01:40.437940 ip-10-0-133-218 kubenswrapper[2580]: I0916 13:01:40.437922    2580 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/b6f97e15-4f04-4c9a-ae13-68aeaa8e85de-cabundle-cert\") on node \"ip-10-0-133-218.ec2.internal\" DevicePath \"\""
Sep 16 13:01:40.437940 ip-10-0-133-218 kubenswrapper[2580]: I0916 13:01:40.437936    2580 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/b6f97e15-4f04-4c9a-ae13-68aeaa8e85de-kube-rbac-proxy-sar-config\") on node \"ip-10-0-133-218.ec2.internal\" DevicePath \"\""
Sep 16 13:01:40.438095 ip-10-0-133-218 kubenswrapper[2580]: I0916 13:01:40.438063    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b6f97e15-4f04-4c9a-ae13-68aeaa8e85de-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "b6f97e15-4f04-4c9a-ae13-68aeaa8e85de" (UID: "b6f97e15-4f04-4c9a-ae13-68aeaa8e85de"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 13:01:40.439678 ip-10-0-133-218 kubenswrapper[2580]: I0916 13:01:40.439661    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b6f97e15-4f04-4c9a-ae13-68aeaa8e85de-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "b6f97e15-4f04-4c9a-ae13-68aeaa8e85de" (UID: "b6f97e15-4f04-4c9a-ae13-68aeaa8e85de"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 13:01:40.439753 ip-10-0-133-218 kubenswrapper[2580]: I0916 13:01:40.439738    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b6f97e15-4f04-4c9a-ae13-68aeaa8e85de-kube-api-access-k6j6v" (OuterVolumeSpecName: "kube-api-access-k6j6v") pod "b6f97e15-4f04-4c9a-ae13-68aeaa8e85de" (UID: "b6f97e15-4f04-4c9a-ae13-68aeaa8e85de"). InnerVolumeSpecName "kube-api-access-k6j6v". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 13:01:40.539162 ip-10-0-133-218 kubenswrapper[2580]: I0916 13:01:40.539129    2580 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b6f97e15-4f04-4c9a-ae13-68aeaa8e85de-kserve-provision-location\") on node \"ip-10-0-133-218.ec2.internal\" DevicePath \"\""
Sep 16 13:01:40.539162 ip-10-0-133-218 kubenswrapper[2580]: I0916 13:01:40.539154    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k6j6v\" (UniqueName: \"kubernetes.io/projected/b6f97e15-4f04-4c9a-ae13-68aeaa8e85de-kube-api-access-k6j6v\") on node \"ip-10-0-133-218.ec2.internal\" DevicePath \"\""
Sep 16 13:01:40.539162 ip-10-0-133-218 kubenswrapper[2580]: I0916 13:01:40.539165    2580 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b6f97e15-4f04-4c9a-ae13-68aeaa8e85de-proxy-tls\") on node \"ip-10-0-133-218.ec2.internal\" DevicePath \"\""
Sep 16 13:01:41.047154 ip-10-0-133-218 kubenswrapper[2580]: I0916 13:01:41.047120    2580 generic.go:356] "Generic (PLEG): container finished" podID="b6f97e15-4f04-4c9a-ae13-68aeaa8e85de" containerID="8b0cd92b7874d06dc1552b779c41f8f1ae09ddee5a0ee25de24a6d3d7370a94e" exitCode=0
Sep 16 13:01:41.047533 ip-10-0-133-218 kubenswrapper[2580]: I0916 13:01:41.047170    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-ec80e-predictor-77f5fb7585-jbvsj"
Sep 16 13:01:41.047533 ip-10-0-133-218 kubenswrapper[2580]: I0916 13:01:41.047200    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-ec80e-predictor-77f5fb7585-jbvsj" event={"ID":"b6f97e15-4f04-4c9a-ae13-68aeaa8e85de","Type":"ContainerDied","Data":"8b0cd92b7874d06dc1552b779c41f8f1ae09ddee5a0ee25de24a6d3d7370a94e"}
Sep 16 13:01:41.047533 ip-10-0-133-218 kubenswrapper[2580]: I0916 13:01:41.047269    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-ec80e-predictor-77f5fb7585-jbvsj" event={"ID":"b6f97e15-4f04-4c9a-ae13-68aeaa8e85de","Type":"ContainerDied","Data":"9957d9e91a669c431fe07fe6aec78650786c91c6d9fc904a09b0c748b358cf3a"}
Sep 16 13:01:41.047533 ip-10-0-133-218 kubenswrapper[2580]: I0916 13:01:41.047307    2580 scope.go:117] "RemoveContainer" containerID="4dc8be51c30bf3de2279e1e58b5e61b95be379e8bfa94209ca8c17fffda42866"
Sep 16 13:01:41.055701 ip-10-0-133-218 kubenswrapper[2580]: I0916 13:01:41.055669    2580 scope.go:117] "RemoveContainer" containerID="8b0cd92b7874d06dc1552b779c41f8f1ae09ddee5a0ee25de24a6d3d7370a94e"
Sep 16 13:01:41.062951 ip-10-0-133-218 kubenswrapper[2580]: I0916 13:01:41.062928    2580 scope.go:117] "RemoveContainer" containerID="41892d1c5d85ca608f952cf88f41b7ae48fb7ff7f27f30de28cfa78006131481"
Sep 16 13:01:41.068790 ip-10-0-133-218 kubenswrapper[2580]: I0916 13:01:41.068769    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-graph-raw-ec80e-predictor-77f5fb7585-jbvsj"]
Sep 16 13:01:41.070724 ip-10-0-133-218 kubenswrapper[2580]: I0916 13:01:41.070704    2580 scope.go:117] "RemoveContainer" containerID="4dc8be51c30bf3de2279e1e58b5e61b95be379e8bfa94209ca8c17fffda42866"
Sep 16 13:01:41.071011 ip-10-0-133-218 kubenswrapper[2580]: E0916 13:01:41.070989    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4dc8be51c30bf3de2279e1e58b5e61b95be379e8bfa94209ca8c17fffda42866\": container with ID starting with 4dc8be51c30bf3de2279e1e58b5e61b95be379e8bfa94209ca8c17fffda42866 not found: ID does not exist" containerID="4dc8be51c30bf3de2279e1e58b5e61b95be379e8bfa94209ca8c17fffda42866"
Sep 16 13:01:41.071068 ip-10-0-133-218 kubenswrapper[2580]: I0916 13:01:41.071019    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4dc8be51c30bf3de2279e1e58b5e61b95be379e8bfa94209ca8c17fffda42866"} err="failed to get container status \"4dc8be51c30bf3de2279e1e58b5e61b95be379e8bfa94209ca8c17fffda42866\": rpc error: code = NotFound desc = could not find container \"4dc8be51c30bf3de2279e1e58b5e61b95be379e8bfa94209ca8c17fffda42866\": container with ID starting with 4dc8be51c30bf3de2279e1e58b5e61b95be379e8bfa94209ca8c17fffda42866 not found: ID does not exist"
Sep 16 13:01:41.071068 ip-10-0-133-218 kubenswrapper[2580]: I0916 13:01:41.071034    2580 scope.go:117] "RemoveContainer" containerID="8b0cd92b7874d06dc1552b779c41f8f1ae09ddee5a0ee25de24a6d3d7370a94e"
Sep 16 13:01:41.071301 ip-10-0-133-218 kubenswrapper[2580]: E0916 13:01:41.071280    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8b0cd92b7874d06dc1552b779c41f8f1ae09ddee5a0ee25de24a6d3d7370a94e\": container with ID starting with 8b0cd92b7874d06dc1552b779c41f8f1ae09ddee5a0ee25de24a6d3d7370a94e not found: ID does not exist" containerID="8b0cd92b7874d06dc1552b779c41f8f1ae09ddee5a0ee25de24a6d3d7370a94e"
Sep 16 13:01:41.071340 ip-10-0-133-218 kubenswrapper[2580]: I0916 13:01:41.071308    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8b0cd92b7874d06dc1552b779c41f8f1ae09ddee5a0ee25de24a6d3d7370a94e"} err="failed to get container status \"8b0cd92b7874d06dc1552b779c41f8f1ae09ddee5a0ee25de24a6d3d7370a94e\": rpc error: code = NotFound desc = could not find container \"8b0cd92b7874d06dc1552b779c41f8f1ae09ddee5a0ee25de24a6d3d7370a94e\": container with ID starting with 8b0cd92b7874d06dc1552b779c41f8f1ae09ddee5a0ee25de24a6d3d7370a94e not found: ID does not exist"
Sep 16 13:01:41.071340 ip-10-0-133-218 kubenswrapper[2580]: I0916 13:01:41.071324    2580 scope.go:117] "RemoveContainer" containerID="41892d1c5d85ca608f952cf88f41b7ae48fb7ff7f27f30de28cfa78006131481"
Sep 16 13:01:41.071624 ip-10-0-133-218 kubenswrapper[2580]: E0916 13:01:41.071604    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"41892d1c5d85ca608f952cf88f41b7ae48fb7ff7f27f30de28cfa78006131481\": container with ID starting with 41892d1c5d85ca608f952cf88f41b7ae48fb7ff7f27f30de28cfa78006131481 not found: ID does not exist" containerID="41892d1c5d85ca608f952cf88f41b7ae48fb7ff7f27f30de28cfa78006131481"
Sep 16 13:01:41.071703 ip-10-0-133-218 kubenswrapper[2580]: I0916 13:01:41.071627    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"41892d1c5d85ca608f952cf88f41b7ae48fb7ff7f27f30de28cfa78006131481"} err="failed to get container status \"41892d1c5d85ca608f952cf88f41b7ae48fb7ff7f27f30de28cfa78006131481\": rpc error: code = NotFound desc = could not find container \"41892d1c5d85ca608f952cf88f41b7ae48fb7ff7f27f30de28cfa78006131481\": container with ID starting with 41892d1c5d85ca608f952cf88f41b7ae48fb7ff7f27f30de28cfa78006131481 not found: ID does not exist"
Sep 16 13:01:41.077882 ip-10-0-133-218 kubenswrapper[2580]: I0916 13:01:41.077862    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-graph-raw-ec80e-predictor-77f5fb7585-jbvsj"]
Sep 16 13:01:41.600864 ip-10-0-133-218 kubenswrapper[2580]: I0916 13:01:41.600752    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b6f97e15-4f04-4c9a-ae13-68aeaa8e85de" path="/var/lib/kubelet/pods/b6f97e15-4f04-4c9a-ae13-68aeaa8e85de/volumes"
Sep 16 13:01:42.053561 ip-10-0-133-218 kubenswrapper[2580]: I0916 13:01:42.053531    2580 generic.go:356] "Generic (PLEG): container finished" podID="ccf0e9b1-5f46-45e7-8192-65d957d2ffd6" containerID="bdd7bc6c714d04afb0cd976fe336d11c78ec4d1e58905361b30db5263fe3d496" exitCode=0
Sep 16 13:01:42.053942 ip-10-0-133-218 kubenswrapper[2580]: I0916 13:01:42.053615    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-5b43d-predictor-5c567f8fb-9gwrq" event={"ID":"ccf0e9b1-5f46-45e7-8192-65d957d2ffd6","Type":"ContainerDied","Data":"bdd7bc6c714d04afb0cd976fe336d11c78ec4d1e58905361b30db5263fe3d496"}
Sep 16 13:01:43.059556 ip-10-0-133-218 kubenswrapper[2580]: I0916 13:01:43.059525    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-5b43d-predictor-5c567f8fb-9gwrq" event={"ID":"ccf0e9b1-5f46-45e7-8192-65d957d2ffd6","Type":"ContainerStarted","Data":"8a829daa187e448fec74c99efa9e49c5b13d2909199926702640535182249df4"}
Sep 16 13:01:43.059935 ip-10-0-133-218 kubenswrapper[2580]: I0916 13:01:43.059587    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-5b43d-predictor-5c567f8fb-9gwrq" event={"ID":"ccf0e9b1-5f46-45e7-8192-65d957d2ffd6","Type":"ContainerStarted","Data":"acd22281ec06859151ab18a7f795d88b5df52345531fde35150a40645e3a6412"}
Sep 16 13:01:43.059935 ip-10-0-133-218 kubenswrapper[2580]: I0916 13:01:43.059859    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-5b43d-predictor-5c567f8fb-9gwrq"
Sep 16 13:01:43.060063 ip-10-0-133-218 kubenswrapper[2580]: I0916 13:01:43.060015    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-5b43d-predictor-5c567f8fb-9gwrq"
Sep 16 13:01:43.061452 ip-10-0-133-218 kubenswrapper[2580]: I0916 13:01:43.061424    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-5b43d-predictor-5c567f8fb-9gwrq" podUID="ccf0e9b1-5f46-45e7-8192-65d957d2ffd6" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.31:8080: connect: connection refused"
Sep 16 13:01:43.080977 ip-10-0-133-218 kubenswrapper[2580]: I0916 13:01:43.080931    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-5b43d-predictor-5c567f8fb-9gwrq" podStartSLOduration=3.283591884 podStartE2EDuration="7.080916254s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.79732437s" isStatefulPod=false podCreationTimestamp="2026-09-16 13:01:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 13:01:43.079409902 +0000 UTC m=+842.002282370" watchObservedRunningTime="2026-09-16 13:01:43.080916254 +0000 UTC m=+842.003788722"
Sep 16 13:01:44.064720 ip-10-0-133-218 kubenswrapper[2580]: I0916 13:01:44.064681    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-5b43d-predictor-5c567f8fb-9gwrq" podUID="ccf0e9b1-5f46-45e7-8192-65d957d2ffd6" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.31:8080: connect: connection refused"
Sep 16 13:01:49.069649 ip-10-0-133-218 kubenswrapper[2580]: I0916 13:01:49.069618    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-5b43d-predictor-5c567f8fb-9gwrq"
Sep 16 13:01:49.070123 ip-10-0-133-218 kubenswrapper[2580]: I0916 13:01:49.070084    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-5b43d-predictor-5c567f8fb-9gwrq" podUID="ccf0e9b1-5f46-45e7-8192-65d957d2ffd6" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.31:8080: connect: connection refused"
Sep 16 13:01:50.022305 ip-10-0-133-218 kubenswrapper[2580]: I0916 13:01:50.022265    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-775495c6b4-hl5vj" podUID="a099972a-5c02-432b-a4bf-40850f65fe4f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.30:8080: connect: connection refused"
Sep 16 13:01:59.070113 ip-10-0-133-218 kubenswrapper[2580]: I0916 13:01:59.070069    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-5b43d-predictor-5c567f8fb-9gwrq" podUID="ccf0e9b1-5f46-45e7-8192-65d957d2ffd6" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.31:8080: connect: connection refused"
Sep 16 13:02:00.022255 ip-10-0-133-218 kubenswrapper[2580]: I0916 13:02:00.022208    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-775495c6b4-hl5vj" podUID="a099972a-5c02-432b-a4bf-40850f65fe4f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.30:8080: connect: connection refused"
Sep 16 13:02:09.070520 ip-10-0-133-218 kubenswrapper[2580]: I0916 13:02:09.070461    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-5b43d-predictor-5c567f8fb-9gwrq" podUID="ccf0e9b1-5f46-45e7-8192-65d957d2ffd6" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.31:8080: connect: connection refused"
Sep 16 13:02:10.023068 ip-10-0-133-218 kubenswrapper[2580]: I0916 13:02:10.023026    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-775495c6b4-hl5vj" podUID="a099972a-5c02-432b-a4bf-40850f65fe4f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.30:8080: connect: connection refused"
Sep 16 13:02:19.070986 ip-10-0-133-218 kubenswrapper[2580]: I0916 13:02:19.070941    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-5b43d-predictor-5c567f8fb-9gwrq" podUID="ccf0e9b1-5f46-45e7-8192-65d957d2ffd6" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.31:8080: connect: connection refused"
Sep 16 13:02:20.022445 ip-10-0-133-218 kubenswrapper[2580]: I0916 13:02:20.022408    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-775495c6b4-hl5vj" podUID="a099972a-5c02-432b-a4bf-40850f65fe4f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.30:8080: connect: connection refused"
Sep 16 13:02:29.070175 ip-10-0-133-218 kubenswrapper[2580]: I0916 13:02:29.070130    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-5b43d-predictor-5c567f8fb-9gwrq" podUID="ccf0e9b1-5f46-45e7-8192-65d957d2ffd6" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.31:8080: connect: connection refused"
Sep 16 13:02:30.022390 ip-10-0-133-218 kubenswrapper[2580]: I0916 13:02:30.022348    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-775495c6b4-hl5vj" podUID="a099972a-5c02-432b-a4bf-40850f65fe4f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.30:8080: connect: connection refused"
Sep 16 13:02:39.071106 ip-10-0-133-218 kubenswrapper[2580]: I0916 13:02:39.071008    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-5b43d-predictor-5c567f8fb-9gwrq" podUID="ccf0e9b1-5f46-45e7-8192-65d957d2ffd6" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.31:8080: connect: connection refused"
Sep 16 13:02:40.023029 ip-10-0-133-218 kubenswrapper[2580]: I0916 13:02:40.022988    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-775495c6b4-hl5vj" podUID="a099972a-5c02-432b-a4bf-40850f65fe4f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.30:8080: connect: connection refused"
Sep 16 13:02:49.070672 ip-10-0-133-218 kubenswrapper[2580]: I0916 13:02:49.070632    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-5b43d-predictor-5c567f8fb-9gwrq" podUID="ccf0e9b1-5f46-45e7-8192-65d957d2ffd6" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.31:8080: connect: connection refused"
Sep 16 13:02:50.023648 ip-10-0-133-218 kubenswrapper[2580]: I0916 13:02:50.023620    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-775495c6b4-hl5vj"
Sep 16 13:02:56.180126 ip-10-0-133-218 kubenswrapper[2580]: I0916 13:02:56.180096    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-775495c6b4-hl5vj"]
Sep 16 13:02:56.180604 ip-10-0-133-218 kubenswrapper[2580]: I0916 13:02:56.180415    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-775495c6b4-hl5vj" podUID="a099972a-5c02-432b-a4bf-40850f65fe4f" containerName="kube-rbac-proxy" containerID="cri-o://116e86bb1fdc70046712810272d808a9328fd3ea0e34b93ac370bbea77b653d3" gracePeriod=30
Sep 16 13:02:56.180604 ip-10-0-133-218 kubenswrapper[2580]: I0916 13:02:56.180402    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-775495c6b4-hl5vj" podUID="a099972a-5c02-432b-a4bf-40850f65fe4f" containerName="kserve-container" containerID="cri-o://033b6f7e03e03dd672c2932453f57f4c7d82dbd3de3350fac67e2a13879d89e6" gracePeriod=30
Sep 16 13:02:56.351157 ip-10-0-133-218 kubenswrapper[2580]: I0916 13:02:56.351124    2580 generic.go:356] "Generic (PLEG): container finished" podID="a099972a-5c02-432b-a4bf-40850f65fe4f" containerID="116e86bb1fdc70046712810272d808a9328fd3ea0e34b93ac370bbea77b653d3" exitCode=2
Sep 16 13:02:56.351319 ip-10-0-133-218 kubenswrapper[2580]: I0916 13:02:56.351194    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-775495c6b4-hl5vj" event={"ID":"a099972a-5c02-432b-a4bf-40850f65fe4f","Type":"ContainerDied","Data":"116e86bb1fdc70046712810272d808a9328fd3ea0e34b93ac370bbea77b653d3"}
Sep 16 13:02:59.071160 ip-10-0-133-218 kubenswrapper[2580]: I0916 13:02:59.071130    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-5b43d-predictor-5c567f8fb-9gwrq"
Sep 16 13:03:00.018171 ip-10-0-133-218 kubenswrapper[2580]: I0916 13:03:00.018121    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-775495c6b4-hl5vj" podUID="a099972a-5c02-432b-a4bf-40850f65fe4f" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.30:8643/healthz\": dial tcp 10.133.0.30:8643: connect: connection refused"
Sep 16 13:03:00.022368 ip-10-0-133-218 kubenswrapper[2580]: I0916 13:03:00.022338    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-775495c6b4-hl5vj" podUID="a099972a-5c02-432b-a4bf-40850f65fe4f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.30:8080: connect: connection refused"
Sep 16 13:03:00.721899 ip-10-0-133-218 kubenswrapper[2580]: I0916 13:03:00.721869    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-775495c6b4-hl5vj"
Sep 16 13:03:00.873541 ip-10-0-133-218 kubenswrapper[2580]: I0916 13:03:00.873431    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/a099972a-5c02-432b-a4bf-40850f65fe4f-cabundle-cert\") pod \"a099972a-5c02-432b-a4bf-40850f65fe4f\" (UID: \"a099972a-5c02-432b-a4bf-40850f65fe4f\") "
Sep 16 13:03:00.873541 ip-10-0-133-218 kubenswrapper[2580]: I0916 13:03:00.873501    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hwpb8\" (UniqueName: \"kubernetes.io/projected/a099972a-5c02-432b-a4bf-40850f65fe4f-kube-api-access-hwpb8\") pod \"a099972a-5c02-432b-a4bf-40850f65fe4f\" (UID: \"a099972a-5c02-432b-a4bf-40850f65fe4f\") "
Sep 16 13:03:00.873541 ip-10-0-133-218 kubenswrapper[2580]: I0916 13:03:00.873542    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a099972a-5c02-432b-a4bf-40850f65fe4f-proxy-tls\") pod \"a099972a-5c02-432b-a4bf-40850f65fe4f\" (UID: \"a099972a-5c02-432b-a4bf-40850f65fe4f\") "
Sep 16 13:03:00.873816 ip-10-0-133-218 kubenswrapper[2580]: I0916 13:03:00.873581    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a099972a-5c02-432b-a4bf-40850f65fe4f-kube-rbac-proxy-sar-config\") pod \"a099972a-5c02-432b-a4bf-40850f65fe4f\" (UID: \"a099972a-5c02-432b-a4bf-40850f65fe4f\") "
Sep 16 13:03:00.873816 ip-10-0-133-218 kubenswrapper[2580]: I0916 13:03:00.873616    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a099972a-5c02-432b-a4bf-40850f65fe4f-kserve-provision-location\") pod \"a099972a-5c02-432b-a4bf-40850f65fe4f\" (UID: \"a099972a-5c02-432b-a4bf-40850f65fe4f\") "
Sep 16 13:03:00.873904 ip-10-0-133-218 kubenswrapper[2580]: I0916 13:03:00.873848    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a099972a-5c02-432b-a4bf-40850f65fe4f-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "a099972a-5c02-432b-a4bf-40850f65fe4f" (UID: "a099972a-5c02-432b-a4bf-40850f65fe4f"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 13:03:00.873980 ip-10-0-133-218 kubenswrapper[2580]: I0916 13:03:00.873957    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a099972a-5c02-432b-a4bf-40850f65fe4f-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "a099972a-5c02-432b-a4bf-40850f65fe4f" (UID: "a099972a-5c02-432b-a4bf-40850f65fe4f"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 13:03:00.874044 ip-10-0-133-218 kubenswrapper[2580]: I0916 13:03:00.874022    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a099972a-5c02-432b-a4bf-40850f65fe4f-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "a099972a-5c02-432b-a4bf-40850f65fe4f" (UID: "a099972a-5c02-432b-a4bf-40850f65fe4f"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 13:03:00.875697 ip-10-0-133-218 kubenswrapper[2580]: I0916 13:03:00.875676    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a099972a-5c02-432b-a4bf-40850f65fe4f-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "a099972a-5c02-432b-a4bf-40850f65fe4f" (UID: "a099972a-5c02-432b-a4bf-40850f65fe4f"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 13:03:00.875799 ip-10-0-133-218 kubenswrapper[2580]: I0916 13:03:00.875726    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a099972a-5c02-432b-a4bf-40850f65fe4f-kube-api-access-hwpb8" (OuterVolumeSpecName: "kube-api-access-hwpb8") pod "a099972a-5c02-432b-a4bf-40850f65fe4f" (UID: "a099972a-5c02-432b-a4bf-40850f65fe4f"). InnerVolumeSpecName "kube-api-access-hwpb8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 13:03:00.974627 ip-10-0-133-218 kubenswrapper[2580]: I0916 13:03:00.974592    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hwpb8\" (UniqueName: \"kubernetes.io/projected/a099972a-5c02-432b-a4bf-40850f65fe4f-kube-api-access-hwpb8\") on node \"ip-10-0-133-218.ec2.internal\" DevicePath \"\""
Sep 16 13:03:00.974627 ip-10-0-133-218 kubenswrapper[2580]: I0916 13:03:00.974619    2580 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a099972a-5c02-432b-a4bf-40850f65fe4f-proxy-tls\") on node \"ip-10-0-133-218.ec2.internal\" DevicePath \"\""
Sep 16 13:03:00.974627 ip-10-0-133-218 kubenswrapper[2580]: I0916 13:03:00.974629    2580 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a099972a-5c02-432b-a4bf-40850f65fe4f-kube-rbac-proxy-sar-config\") on node \"ip-10-0-133-218.ec2.internal\" DevicePath \"\""
Sep 16 13:03:00.974826 ip-10-0-133-218 kubenswrapper[2580]: I0916 13:03:00.974638    2580 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a099972a-5c02-432b-a4bf-40850f65fe4f-kserve-provision-location\") on node \"ip-10-0-133-218.ec2.internal\" DevicePath \"\""
Sep 16 13:03:00.974826 ip-10-0-133-218 kubenswrapper[2580]: I0916 13:03:00.974647    2580 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/a099972a-5c02-432b-a4bf-40850f65fe4f-cabundle-cert\") on node \"ip-10-0-133-218.ec2.internal\" DevicePath \"\""
Sep 16 13:03:01.372611 ip-10-0-133-218 kubenswrapper[2580]: I0916 13:03:01.372569    2580 generic.go:356] "Generic (PLEG): container finished" podID="a099972a-5c02-432b-a4bf-40850f65fe4f" containerID="033b6f7e03e03dd672c2932453f57f4c7d82dbd3de3350fac67e2a13879d89e6" exitCode=0
Sep 16 13:03:01.372788 ip-10-0-133-218 kubenswrapper[2580]: I0916 13:03:01.372652    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-775495c6b4-hl5vj" event={"ID":"a099972a-5c02-432b-a4bf-40850f65fe4f","Type":"ContainerDied","Data":"033b6f7e03e03dd672c2932453f57f4c7d82dbd3de3350fac67e2a13879d89e6"}
Sep 16 13:03:01.372788 ip-10-0-133-218 kubenswrapper[2580]: I0916 13:03:01.372666    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-775495c6b4-hl5vj"
Sep 16 13:03:01.372788 ip-10-0-133-218 kubenswrapper[2580]: I0916 13:03:01.372740    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-775495c6b4-hl5vj" event={"ID":"a099972a-5c02-432b-a4bf-40850f65fe4f","Type":"ContainerDied","Data":"302a13c5f6e4702c02bc9a743668cde37cf2c07fd84cd2a08566e7ee2d4975b6"}
Sep 16 13:03:01.372788 ip-10-0-133-218 kubenswrapper[2580]: I0916 13:03:01.372768    2580 scope.go:117] "RemoveContainer" containerID="116e86bb1fdc70046712810272d808a9328fd3ea0e34b93ac370bbea77b653d3"
Sep 16 13:03:01.381238 ip-10-0-133-218 kubenswrapper[2580]: I0916 13:03:01.381217    2580 scope.go:117] "RemoveContainer" containerID="033b6f7e03e03dd672c2932453f57f4c7d82dbd3de3350fac67e2a13879d89e6"
Sep 16 13:03:01.388505 ip-10-0-133-218 kubenswrapper[2580]: I0916 13:03:01.388473    2580 scope.go:117] "RemoveContainer" containerID="d129c93cb5588a8d90dd6c1ba10ca9f13c871af2c0fa345808d9948d8a4fba98"
Sep 16 13:03:01.395422 ip-10-0-133-218 kubenswrapper[2580]: I0916 13:03:01.395398    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-775495c6b4-hl5vj"]
Sep 16 13:03:01.396607 ip-10-0-133-218 kubenswrapper[2580]: I0916 13:03:01.396551    2580 scope.go:117] "RemoveContainer" containerID="116e86bb1fdc70046712810272d808a9328fd3ea0e34b93ac370bbea77b653d3"
Sep 16 13:03:01.396985 ip-10-0-133-218 kubenswrapper[2580]: E0916 13:03:01.396963    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"116e86bb1fdc70046712810272d808a9328fd3ea0e34b93ac370bbea77b653d3\": container with ID starting with 116e86bb1fdc70046712810272d808a9328fd3ea0e34b93ac370bbea77b653d3 not found: ID does not exist" containerID="116e86bb1fdc70046712810272d808a9328fd3ea0e34b93ac370bbea77b653d3"
Sep 16 13:03:01.397073 ip-10-0-133-218 kubenswrapper[2580]: I0916 13:03:01.396994    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"116e86bb1fdc70046712810272d808a9328fd3ea0e34b93ac370bbea77b653d3"} err="failed to get container status \"116e86bb1fdc70046712810272d808a9328fd3ea0e34b93ac370bbea77b653d3\": rpc error: code = NotFound desc = could not find container \"116e86bb1fdc70046712810272d808a9328fd3ea0e34b93ac370bbea77b653d3\": container with ID starting with 116e86bb1fdc70046712810272d808a9328fd3ea0e34b93ac370bbea77b653d3 not found: ID does not exist"
Sep 16 13:03:01.397073 ip-10-0-133-218 kubenswrapper[2580]: I0916 13:03:01.397018    2580 scope.go:117] "RemoveContainer" containerID="033b6f7e03e03dd672c2932453f57f4c7d82dbd3de3350fac67e2a13879d89e6"
Sep 16 13:03:01.397296 ip-10-0-133-218 kubenswrapper[2580]: E0916 13:03:01.397274    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"033b6f7e03e03dd672c2932453f57f4c7d82dbd3de3350fac67e2a13879d89e6\": container with ID starting with 033b6f7e03e03dd672c2932453f57f4c7d82dbd3de3350fac67e2a13879d89e6 not found: ID does not exist" containerID="033b6f7e03e03dd672c2932453f57f4c7d82dbd3de3350fac67e2a13879d89e6"
Sep 16 13:03:01.397342 ip-10-0-133-218 kubenswrapper[2580]: I0916 13:03:01.397306    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"033b6f7e03e03dd672c2932453f57f4c7d82dbd3de3350fac67e2a13879d89e6"} err="failed to get container status \"033b6f7e03e03dd672c2932453f57f4c7d82dbd3de3350fac67e2a13879d89e6\": rpc error: code = NotFound desc = could not find container \"033b6f7e03e03dd672c2932453f57f4c7d82dbd3de3350fac67e2a13879d89e6\": container with ID starting with 033b6f7e03e03dd672c2932453f57f4c7d82dbd3de3350fac67e2a13879d89e6 not found: ID does not exist"
Sep 16 13:03:01.397342 ip-10-0-133-218 kubenswrapper[2580]: I0916 13:03:01.397329    2580 scope.go:117] "RemoveContainer" containerID="d129c93cb5588a8d90dd6c1ba10ca9f13c871af2c0fa345808d9948d8a4fba98"
Sep 16 13:03:01.397637 ip-10-0-133-218 kubenswrapper[2580]: E0916 13:03:01.397620    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d129c93cb5588a8d90dd6c1ba10ca9f13c871af2c0fa345808d9948d8a4fba98\": container with ID starting with d129c93cb5588a8d90dd6c1ba10ca9f13c871af2c0fa345808d9948d8a4fba98 not found: ID does not exist" containerID="d129c93cb5588a8d90dd6c1ba10ca9f13c871af2c0fa345808d9948d8a4fba98"
Sep 16 13:03:01.397693 ip-10-0-133-218 kubenswrapper[2580]: I0916 13:03:01.397644    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d129c93cb5588a8d90dd6c1ba10ca9f13c871af2c0fa345808d9948d8a4fba98"} err="failed to get container status \"d129c93cb5588a8d90dd6c1ba10ca9f13c871af2c0fa345808d9948d8a4fba98\": rpc error: code = NotFound desc = could not find container \"d129c93cb5588a8d90dd6c1ba10ca9f13c871af2c0fa345808d9948d8a4fba98\": container with ID starting with d129c93cb5588a8d90dd6c1ba10ca9f13c871af2c0fa345808d9948d8a4fba98 not found: ID does not exist"
Sep 16 13:03:01.398442 ip-10-0-133-218 kubenswrapper[2580]: I0916 13:03:01.398426    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-775495c6b4-hl5vj"]
Sep 16 13:03:01.599107 ip-10-0-133-218 kubenswrapper[2580]: I0916 13:03:01.599073    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a099972a-5c02-432b-a4bf-40850f65fe4f" path="/var/lib/kubelet/pods/a099972a-5c02-432b-a4bf-40850f65fe4f/volumes"
Sep 16 13:03:06.551568 ip-10-0-133-218 kubenswrapper[2580]: I0916 13:03:06.551533    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-primary-d93042-predictor-868fb57669-bcjqh"]
Sep 16 13:03:06.551937 ip-10-0-133-218 kubenswrapper[2580]: I0916 13:03:06.551869    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b6f97e15-4f04-4c9a-ae13-68aeaa8e85de" containerName="kserve-container"
Sep 16 13:03:06.551937 ip-10-0-133-218 kubenswrapper[2580]: I0916 13:03:06.551880    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="b6f97e15-4f04-4c9a-ae13-68aeaa8e85de" containerName="kserve-container"
Sep 16 13:03:06.551937 ip-10-0-133-218 kubenswrapper[2580]: I0916 13:03:06.551894    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a099972a-5c02-432b-a4bf-40850f65fe4f" containerName="kserve-container"
Sep 16 13:03:06.551937 ip-10-0-133-218 kubenswrapper[2580]: I0916 13:03:06.551899    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="a099972a-5c02-432b-a4bf-40850f65fe4f" containerName="kserve-container"
Sep 16 13:03:06.551937 ip-10-0-133-218 kubenswrapper[2580]: I0916 13:03:06.551912    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a099972a-5c02-432b-a4bf-40850f65fe4f" containerName="kube-rbac-proxy"
Sep 16 13:03:06.551937 ip-10-0-133-218 kubenswrapper[2580]: I0916 13:03:06.551917    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="a099972a-5c02-432b-a4bf-40850f65fe4f" containerName="kube-rbac-proxy"
Sep 16 13:03:06.551937 ip-10-0-133-218 kubenswrapper[2580]: I0916 13:03:06.551924    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b6f97e15-4f04-4c9a-ae13-68aeaa8e85de" containerName="storage-initializer"
Sep 16 13:03:06.551937 ip-10-0-133-218 kubenswrapper[2580]: I0916 13:03:06.551929    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="b6f97e15-4f04-4c9a-ae13-68aeaa8e85de" containerName="storage-initializer"
Sep 16 13:03:06.551937 ip-10-0-133-218 kubenswrapper[2580]: I0916 13:03:06.551934    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b6f97e15-4f04-4c9a-ae13-68aeaa8e85de" containerName="kube-rbac-proxy"
Sep 16 13:03:06.551937 ip-10-0-133-218 kubenswrapper[2580]: I0916 13:03:06.551940    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="b6f97e15-4f04-4c9a-ae13-68aeaa8e85de" containerName="kube-rbac-proxy"
Sep 16 13:03:06.552237 ip-10-0-133-218 kubenswrapper[2580]: I0916 13:03:06.551946    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a099972a-5c02-432b-a4bf-40850f65fe4f" containerName="storage-initializer"
Sep 16 13:03:06.552237 ip-10-0-133-218 kubenswrapper[2580]: I0916 13:03:06.551951    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="a099972a-5c02-432b-a4bf-40850f65fe4f" containerName="storage-initializer"
Sep 16 13:03:06.552237 ip-10-0-133-218 kubenswrapper[2580]: I0916 13:03:06.551997    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="a099972a-5c02-432b-a4bf-40850f65fe4f" containerName="kube-rbac-proxy"
Sep 16 13:03:06.552237 ip-10-0-133-218 kubenswrapper[2580]: I0916 13:03:06.552006    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="b6f97e15-4f04-4c9a-ae13-68aeaa8e85de" containerName="kserve-container"
Sep 16 13:03:06.552237 ip-10-0-133-218 kubenswrapper[2580]: I0916 13:03:06.552012    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="b6f97e15-4f04-4c9a-ae13-68aeaa8e85de" containerName="kube-rbac-proxy"
Sep 16 13:03:06.552237 ip-10-0-133-218 kubenswrapper[2580]: I0916 13:03:06.552020    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="a099972a-5c02-432b-a4bf-40850f65fe4f" containerName="kserve-container"
Sep 16 13:03:06.555537 ip-10-0-133-218 kubenswrapper[2580]: I0916 13:03:06.555518    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-primary-d93042-predictor-868fb57669-bcjqh"
Sep 16 13:03:06.558768 ip-10-0-133-218 kubenswrapper[2580]: I0916 13:03:06.558744    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-primary-d93042-predictor-serving-cert\""
Sep 16 13:03:06.558901 ip-10-0-133-218 kubenswrapper[2580]: I0916 13:03:06.558829    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-primary-d93042-kube-rbac-proxy-sar-config\""
Sep 16 13:03:06.570774 ip-10-0-133-218 kubenswrapper[2580]: I0916 13:03:06.570753    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-primary-d93042-predictor-868fb57669-bcjqh"]
Sep 16 13:03:06.612266 ip-10-0-133-218 kubenswrapper[2580]: I0916 13:03:06.612242    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/691ee328-ef29-46a2-a634-30213fa7acee-kube-rbac-proxy-sar-config\") pod \"isvc-primary-d93042-predictor-868fb57669-bcjqh\" (UID: \"691ee328-ef29-46a2-a634-30213fa7acee\") " pod="kserve-ci-e2e-test/isvc-primary-d93042-predictor-868fb57669-bcjqh"
Sep 16 13:03:06.612392 ip-10-0-133-218 kubenswrapper[2580]: I0916 13:03:06.612293    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/691ee328-ef29-46a2-a634-30213fa7acee-proxy-tls\") pod \"isvc-primary-d93042-predictor-868fb57669-bcjqh\" (UID: \"691ee328-ef29-46a2-a634-30213fa7acee\") " pod="kserve-ci-e2e-test/isvc-primary-d93042-predictor-868fb57669-bcjqh"
Sep 16 13:03:06.612392 ip-10-0-133-218 kubenswrapper[2580]: I0916 13:03:06.612319    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v6tv9\" (UniqueName: \"kubernetes.io/projected/691ee328-ef29-46a2-a634-30213fa7acee-kube-api-access-v6tv9\") pod \"isvc-primary-d93042-predictor-868fb57669-bcjqh\" (UID: \"691ee328-ef29-46a2-a634-30213fa7acee\") " pod="kserve-ci-e2e-test/isvc-primary-d93042-predictor-868fb57669-bcjqh"
Sep 16 13:03:06.612392 ip-10-0-133-218 kubenswrapper[2580]: I0916 13:03:06.612355    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/691ee328-ef29-46a2-a634-30213fa7acee-cabundle-cert\") pod \"isvc-primary-d93042-predictor-868fb57669-bcjqh\" (UID: \"691ee328-ef29-46a2-a634-30213fa7acee\") " pod="kserve-ci-e2e-test/isvc-primary-d93042-predictor-868fb57669-bcjqh"
Sep 16 13:03:06.612392 ip-10-0-133-218 kubenswrapper[2580]: I0916 13:03:06.612386    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/691ee328-ef29-46a2-a634-30213fa7acee-kserve-provision-location\") pod \"isvc-primary-d93042-predictor-868fb57669-bcjqh\" (UID: \"691ee328-ef29-46a2-a634-30213fa7acee\") " pod="kserve-ci-e2e-test/isvc-primary-d93042-predictor-868fb57669-bcjqh"
Sep 16 13:03:06.713447 ip-10-0-133-218 kubenswrapper[2580]: I0916 13:03:06.713407    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/691ee328-ef29-46a2-a634-30213fa7acee-kserve-provision-location\") pod \"isvc-primary-d93042-predictor-868fb57669-bcjqh\" (UID: \"691ee328-ef29-46a2-a634-30213fa7acee\") " pod="kserve-ci-e2e-test/isvc-primary-d93042-predictor-868fb57669-bcjqh"
Sep 16 13:03:06.713613 ip-10-0-133-218 kubenswrapper[2580]: I0916 13:03:06.713515    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/691ee328-ef29-46a2-a634-30213fa7acee-kube-rbac-proxy-sar-config\") pod \"isvc-primary-d93042-predictor-868fb57669-bcjqh\" (UID: \"691ee328-ef29-46a2-a634-30213fa7acee\") " pod="kserve-ci-e2e-test/isvc-primary-d93042-predictor-868fb57669-bcjqh"
Sep 16 13:03:06.713613 ip-10-0-133-218 kubenswrapper[2580]: I0916 13:03:06.713579    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/691ee328-ef29-46a2-a634-30213fa7acee-proxy-tls\") pod \"isvc-primary-d93042-predictor-868fb57669-bcjqh\" (UID: \"691ee328-ef29-46a2-a634-30213fa7acee\") " pod="kserve-ci-e2e-test/isvc-primary-d93042-predictor-868fb57669-bcjqh"
Sep 16 13:03:06.713789 ip-10-0-133-218 kubenswrapper[2580]: I0916 13:03:06.713611    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v6tv9\" (UniqueName: \"kubernetes.io/projected/691ee328-ef29-46a2-a634-30213fa7acee-kube-api-access-v6tv9\") pod \"isvc-primary-d93042-predictor-868fb57669-bcjqh\" (UID: \"691ee328-ef29-46a2-a634-30213fa7acee\") " pod="kserve-ci-e2e-test/isvc-primary-d93042-predictor-868fb57669-bcjqh"
Sep 16 13:03:06.713789 ip-10-0-133-218 kubenswrapper[2580]: I0916 13:03:06.713648    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/691ee328-ef29-46a2-a634-30213fa7acee-cabundle-cert\") pod \"isvc-primary-d93042-predictor-868fb57669-bcjqh\" (UID: \"691ee328-ef29-46a2-a634-30213fa7acee\") " pod="kserve-ci-e2e-test/isvc-primary-d93042-predictor-868fb57669-bcjqh"
Sep 16 13:03:06.713897 ip-10-0-133-218 kubenswrapper[2580]: I0916 13:03:06.713789    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/691ee328-ef29-46a2-a634-30213fa7acee-kserve-provision-location\") pod \"isvc-primary-d93042-predictor-868fb57669-bcjqh\" (UID: \"691ee328-ef29-46a2-a634-30213fa7acee\") " pod="kserve-ci-e2e-test/isvc-primary-d93042-predictor-868fb57669-bcjqh"
Sep 16 13:03:06.714248 ip-10-0-133-218 kubenswrapper[2580]: I0916 13:03:06.714227    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/691ee328-ef29-46a2-a634-30213fa7acee-kube-rbac-proxy-sar-config\") pod \"isvc-primary-d93042-predictor-868fb57669-bcjqh\" (UID: \"691ee328-ef29-46a2-a634-30213fa7acee\") " pod="kserve-ci-e2e-test/isvc-primary-d93042-predictor-868fb57669-bcjqh"
Sep 16 13:03:06.714365 ip-10-0-133-218 kubenswrapper[2580]: I0916 13:03:06.714344    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/691ee328-ef29-46a2-a634-30213fa7acee-cabundle-cert\") pod \"isvc-primary-d93042-predictor-868fb57669-bcjqh\" (UID: \"691ee328-ef29-46a2-a634-30213fa7acee\") " pod="kserve-ci-e2e-test/isvc-primary-d93042-predictor-868fb57669-bcjqh"
Sep 16 13:03:06.715842 ip-10-0-133-218 kubenswrapper[2580]: I0916 13:03:06.715821    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/691ee328-ef29-46a2-a634-30213fa7acee-proxy-tls\") pod \"isvc-primary-d93042-predictor-868fb57669-bcjqh\" (UID: \"691ee328-ef29-46a2-a634-30213fa7acee\") " pod="kserve-ci-e2e-test/isvc-primary-d93042-predictor-868fb57669-bcjqh"
Sep 16 13:03:06.727608 ip-10-0-133-218 kubenswrapper[2580]: I0916 13:03:06.727588    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-v6tv9\" (UniqueName: \"kubernetes.io/projected/691ee328-ef29-46a2-a634-30213fa7acee-kube-api-access-v6tv9\") pod \"isvc-primary-d93042-predictor-868fb57669-bcjqh\" (UID: \"691ee328-ef29-46a2-a634-30213fa7acee\") " pod="kserve-ci-e2e-test/isvc-primary-d93042-predictor-868fb57669-bcjqh"
Sep 16 13:03:06.864842 ip-10-0-133-218 kubenswrapper[2580]: I0916 13:03:06.864764    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-primary-d93042-predictor-868fb57669-bcjqh"
Sep 16 13:03:07.142696 ip-10-0-133-218 kubenswrapper[2580]: I0916 13:03:07.142618    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-primary-d93042-predictor-868fb57669-bcjqh"]
Sep 16 13:03:07.145880 ip-10-0-133-218 kubenswrapper[2580]: W0916 13:03:07.145851    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod691ee328_ef29_46a2_a634_30213fa7acee.slice/crio-39da15870fe3b8b72ec44f2ad5566f7f766879085655fb4976dfa01239e2970e WatchSource:0}: Error finding container 39da15870fe3b8b72ec44f2ad5566f7f766879085655fb4976dfa01239e2970e: Status 404 returned error can't find the container with id 39da15870fe3b8b72ec44f2ad5566f7f766879085655fb4976dfa01239e2970e
Sep 16 13:03:07.404829 ip-10-0-133-218 kubenswrapper[2580]: I0916 13:03:07.404741    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-primary-d93042-predictor-868fb57669-bcjqh" event={"ID":"691ee328-ef29-46a2-a634-30213fa7acee","Type":"ContainerStarted","Data":"9abe19f1a094279a87fa8bdc6d2b0c089f957bdb65148d0edfb45a5840990453"}
Sep 16 13:03:07.404829 ip-10-0-133-218 kubenswrapper[2580]: I0916 13:03:07.404811    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-primary-d93042-predictor-868fb57669-bcjqh" event={"ID":"691ee328-ef29-46a2-a634-30213fa7acee","Type":"ContainerStarted","Data":"39da15870fe3b8b72ec44f2ad5566f7f766879085655fb4976dfa01239e2970e"}
Sep 16 13:03:11.421963 ip-10-0-133-218 kubenswrapper[2580]: I0916 13:03:11.421873    2580 generic.go:356] "Generic (PLEG): container finished" podID="691ee328-ef29-46a2-a634-30213fa7acee" containerID="9abe19f1a094279a87fa8bdc6d2b0c089f957bdb65148d0edfb45a5840990453" exitCode=0
Sep 16 13:03:11.421963 ip-10-0-133-218 kubenswrapper[2580]: I0916 13:03:11.421946    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-primary-d93042-predictor-868fb57669-bcjqh" event={"ID":"691ee328-ef29-46a2-a634-30213fa7acee","Type":"ContainerDied","Data":"9abe19f1a094279a87fa8bdc6d2b0c089f957bdb65148d0edfb45a5840990453"}
Sep 16 13:03:12.428195 ip-10-0-133-218 kubenswrapper[2580]: I0916 13:03:12.428158    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-primary-d93042-predictor-868fb57669-bcjqh" event={"ID":"691ee328-ef29-46a2-a634-30213fa7acee","Type":"ContainerStarted","Data":"a8b50f7ba5799c5b9508f177944b88d183f382dab1c69656d84f417bf78dd5ec"}
Sep 16 13:03:12.428619 ip-10-0-133-218 kubenswrapper[2580]: I0916 13:03:12.428245    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-primary-d93042-predictor-868fb57669-bcjqh" event={"ID":"691ee328-ef29-46a2-a634-30213fa7acee","Type":"ContainerStarted","Data":"ac84066aa22b5f2f7e92a2bffe32dc5ef88c093104f1c91fe4a6b8c9672cfc86"}
Sep 16 13:03:12.428619 ip-10-0-133-218 kubenswrapper[2580]: I0916 13:03:12.428526    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-primary-d93042-predictor-868fb57669-bcjqh"
Sep 16 13:03:12.428730 ip-10-0-133-218 kubenswrapper[2580]: I0916 13:03:12.428665    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-primary-d93042-predictor-868fb57669-bcjqh"
Sep 16 13:03:12.430106 ip-10-0-133-218 kubenswrapper[2580]: I0916 13:03:12.430078    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-d93042-predictor-868fb57669-bcjqh" podUID="691ee328-ef29-46a2-a634-30213fa7acee" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.32:8080: connect: connection refused"
Sep 16 13:03:13.432272 ip-10-0-133-218 kubenswrapper[2580]: I0916 13:03:13.432230    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-d93042-predictor-868fb57669-bcjqh" podUID="691ee328-ef29-46a2-a634-30213fa7acee" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.32:8080: connect: connection refused"
Sep 16 13:03:18.436518 ip-10-0-133-218 kubenswrapper[2580]: I0916 13:03:18.436475    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-primary-d93042-predictor-868fb57669-bcjqh"
Sep 16 13:03:18.437052 ip-10-0-133-218 kubenswrapper[2580]: I0916 13:03:18.437025    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-d93042-predictor-868fb57669-bcjqh" podUID="691ee328-ef29-46a2-a634-30213fa7acee" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.32:8080: connect: connection refused"
Sep 16 13:03:18.473939 ip-10-0-133-218 kubenswrapper[2580]: I0916 13:03:18.473892    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-primary-d93042-predictor-868fb57669-bcjqh" podStartSLOduration=8.630327251 podStartE2EDuration="12.473878785s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.843551534s" isStatefulPod=false podCreationTimestamp="2026-09-16 13:03:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 13:03:12.467619867 +0000 UTC m=+931.390492348" watchObservedRunningTime="2026-09-16 13:03:18.473878785 +0000 UTC m=+937.396751253"
Sep 16 13:03:26.915364 ip-10-0-133-218 kubenswrapper[2580]: I0916 13:03:26.915332    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-5b43d-predictor-5c567f8fb-9gwrq"]
Sep 16 13:03:26.917989 ip-10-0-133-218 kubenswrapper[2580]: I0916 13:03:26.915595    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-5b43d-predictor-5c567f8fb-9gwrq" podUID="ccf0e9b1-5f46-45e7-8192-65d957d2ffd6" containerName="kserve-container" containerID="cri-o://acd22281ec06859151ab18a7f795d88b5df52345531fde35150a40645e3a6412" gracePeriod=30
Sep 16 13:03:26.917989 ip-10-0-133-218 kubenswrapper[2580]: I0916 13:03:26.915635    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-5b43d-predictor-5c567f8fb-9gwrq" podUID="ccf0e9b1-5f46-45e7-8192-65d957d2ffd6" containerName="kube-rbac-proxy" containerID="cri-o://8a829daa187e448fec74c99efa9e49c5b13d2909199926702640535182249df4" gracePeriod=30
Sep 16 13:03:27.490576 ip-10-0-133-218 kubenswrapper[2580]: I0916 13:03:27.490543    2580 generic.go:356] "Generic (PLEG): container finished" podID="ccf0e9b1-5f46-45e7-8192-65d957d2ffd6" containerID="8a829daa187e448fec74c99efa9e49c5b13d2909199926702640535182249df4" exitCode=2
Sep 16 13:03:27.490753 ip-10-0-133-218 kubenswrapper[2580]: I0916 13:03:27.490615    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-5b43d-predictor-5c567f8fb-9gwrq" event={"ID":"ccf0e9b1-5f46-45e7-8192-65d957d2ffd6","Type":"ContainerDied","Data":"8a829daa187e448fec74c99efa9e49c5b13d2909199926702640535182249df4"}
Sep 16 13:03:28.437745 ip-10-0-133-218 kubenswrapper[2580]: I0916 13:03:28.437705    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-d93042-predictor-868fb57669-bcjqh" podUID="691ee328-ef29-46a2-a634-30213fa7acee" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.32:8080: connect: connection refused"
Sep 16 13:03:29.064890 ip-10-0-133-218 kubenswrapper[2580]: I0916 13:03:29.064851    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-5b43d-predictor-5c567f8fb-9gwrq" podUID="ccf0e9b1-5f46-45e7-8192-65d957d2ffd6" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.31:8643/healthz\": dial tcp 10.133.0.31:8643: connect: connection refused"
Sep 16 13:03:29.070141 ip-10-0-133-218 kubenswrapper[2580]: I0916 13:03:29.070106    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-5b43d-predictor-5c567f8fb-9gwrq" podUID="ccf0e9b1-5f46-45e7-8192-65d957d2ffd6" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.31:8080: connect: connection refused"
Sep 16 13:03:31.354222 ip-10-0-133-218 kubenswrapper[2580]: I0916 13:03:31.354198    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-5b43d-predictor-5c567f8fb-9gwrq"
Sep 16 13:03:31.423409 ip-10-0-133-218 kubenswrapper[2580]: I0916 13:03:31.423326    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ccf0e9b1-5f46-45e7-8192-65d957d2ffd6-kube-rbac-proxy-sar-config\") pod \"ccf0e9b1-5f46-45e7-8192-65d957d2ffd6\" (UID: \"ccf0e9b1-5f46-45e7-8192-65d957d2ffd6\") "
Sep 16 13:03:31.423409 ip-10-0-133-218 kubenswrapper[2580]: I0916 13:03:31.423374    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ccf0e9b1-5f46-45e7-8192-65d957d2ffd6-proxy-tls\") pod \"ccf0e9b1-5f46-45e7-8192-65d957d2ffd6\" (UID: \"ccf0e9b1-5f46-45e7-8192-65d957d2ffd6\") "
Sep 16 13:03:31.423632 ip-10-0-133-218 kubenswrapper[2580]: I0916 13:03:31.423443    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/ccf0e9b1-5f46-45e7-8192-65d957d2ffd6-cabundle-cert\") pod \"ccf0e9b1-5f46-45e7-8192-65d957d2ffd6\" (UID: \"ccf0e9b1-5f46-45e7-8192-65d957d2ffd6\") "
Sep 16 13:03:31.423632 ip-10-0-133-218 kubenswrapper[2580]: I0916 13:03:31.423505    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ccf0e9b1-5f46-45e7-8192-65d957d2ffd6-kserve-provision-location\") pod \"ccf0e9b1-5f46-45e7-8192-65d957d2ffd6\" (UID: \"ccf0e9b1-5f46-45e7-8192-65d957d2ffd6\") "
Sep 16 13:03:31.423632 ip-10-0-133-218 kubenswrapper[2580]: I0916 13:03:31.423559    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-cnjpn\" (UniqueName: \"kubernetes.io/projected/ccf0e9b1-5f46-45e7-8192-65d957d2ffd6-kube-api-access-cnjpn\") pod \"ccf0e9b1-5f46-45e7-8192-65d957d2ffd6\" (UID: \"ccf0e9b1-5f46-45e7-8192-65d957d2ffd6\") "
Sep 16 13:03:31.423849 ip-10-0-133-218 kubenswrapper[2580]: I0916 13:03:31.423823    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ccf0e9b1-5f46-45e7-8192-65d957d2ffd6-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "ccf0e9b1-5f46-45e7-8192-65d957d2ffd6" (UID: "ccf0e9b1-5f46-45e7-8192-65d957d2ffd6"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 13:03:31.423899 ip-10-0-133-218 kubenswrapper[2580]: I0916 13:03:31.423844    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ccf0e9b1-5f46-45e7-8192-65d957d2ffd6-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "ccf0e9b1-5f46-45e7-8192-65d957d2ffd6" (UID: "ccf0e9b1-5f46-45e7-8192-65d957d2ffd6"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 13:03:31.423899 ip-10-0-133-218 kubenswrapper[2580]: I0916 13:03:31.423842    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ccf0e9b1-5f46-45e7-8192-65d957d2ffd6-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "ccf0e9b1-5f46-45e7-8192-65d957d2ffd6" (UID: "ccf0e9b1-5f46-45e7-8192-65d957d2ffd6"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 13:03:31.425605 ip-10-0-133-218 kubenswrapper[2580]: I0916 13:03:31.425583    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ccf0e9b1-5f46-45e7-8192-65d957d2ffd6-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "ccf0e9b1-5f46-45e7-8192-65d957d2ffd6" (UID: "ccf0e9b1-5f46-45e7-8192-65d957d2ffd6"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 13:03:31.425713 ip-10-0-133-218 kubenswrapper[2580]: I0916 13:03:31.425694    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ccf0e9b1-5f46-45e7-8192-65d957d2ffd6-kube-api-access-cnjpn" (OuterVolumeSpecName: "kube-api-access-cnjpn") pod "ccf0e9b1-5f46-45e7-8192-65d957d2ffd6" (UID: "ccf0e9b1-5f46-45e7-8192-65d957d2ffd6"). InnerVolumeSpecName "kube-api-access-cnjpn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 13:03:31.508555 ip-10-0-133-218 kubenswrapper[2580]: I0916 13:03:31.508521    2580 generic.go:356] "Generic (PLEG): container finished" podID="ccf0e9b1-5f46-45e7-8192-65d957d2ffd6" containerID="acd22281ec06859151ab18a7f795d88b5df52345531fde35150a40645e3a6412" exitCode=0
Sep 16 13:03:31.508705 ip-10-0-133-218 kubenswrapper[2580]: I0916 13:03:31.508577    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-5b43d-predictor-5c567f8fb-9gwrq"
Sep 16 13:03:31.508705 ip-10-0-133-218 kubenswrapper[2580]: I0916 13:03:31.508612    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-5b43d-predictor-5c567f8fb-9gwrq" event={"ID":"ccf0e9b1-5f46-45e7-8192-65d957d2ffd6","Type":"ContainerDied","Data":"acd22281ec06859151ab18a7f795d88b5df52345531fde35150a40645e3a6412"}
Sep 16 13:03:31.508705 ip-10-0-133-218 kubenswrapper[2580]: I0916 13:03:31.508672    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-5b43d-predictor-5c567f8fb-9gwrq" event={"ID":"ccf0e9b1-5f46-45e7-8192-65d957d2ffd6","Type":"ContainerDied","Data":"d6b49bdbe5ae5150f54e5816c431df0f9a82ee6570342a81f7d90248557de04b"}
Sep 16 13:03:31.508705 ip-10-0-133-218 kubenswrapper[2580]: I0916 13:03:31.508700    2580 scope.go:117] "RemoveContainer" containerID="8a829daa187e448fec74c99efa9e49c5b13d2909199926702640535182249df4"
Sep 16 13:03:31.522090 ip-10-0-133-218 kubenswrapper[2580]: I0916 13:03:31.522069    2580 scope.go:117] "RemoveContainer" containerID="acd22281ec06859151ab18a7f795d88b5df52345531fde35150a40645e3a6412"
Sep 16 13:03:31.524265 ip-10-0-133-218 kubenswrapper[2580]: I0916 13:03:31.524245    2580 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ccf0e9b1-5f46-45e7-8192-65d957d2ffd6-kserve-provision-location\") on node \"ip-10-0-133-218.ec2.internal\" DevicePath \"\""
Sep 16 13:03:31.524369 ip-10-0-133-218 kubenswrapper[2580]: I0916 13:03:31.524270    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cnjpn\" (UniqueName: \"kubernetes.io/projected/ccf0e9b1-5f46-45e7-8192-65d957d2ffd6-kube-api-access-cnjpn\") on node \"ip-10-0-133-218.ec2.internal\" DevicePath \"\""
Sep 16 13:03:31.524369 ip-10-0-133-218 kubenswrapper[2580]: I0916 13:03:31.524280    2580 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ccf0e9b1-5f46-45e7-8192-65d957d2ffd6-kube-rbac-proxy-sar-config\") on node \"ip-10-0-133-218.ec2.internal\" DevicePath \"\""
Sep 16 13:03:31.524369 ip-10-0-133-218 kubenswrapper[2580]: I0916 13:03:31.524289    2580 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ccf0e9b1-5f46-45e7-8192-65d957d2ffd6-proxy-tls\") on node \"ip-10-0-133-218.ec2.internal\" DevicePath \"\""
Sep 16 13:03:31.524369 ip-10-0-133-218 kubenswrapper[2580]: I0916 13:03:31.524298    2580 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/ccf0e9b1-5f46-45e7-8192-65d957d2ffd6-cabundle-cert\") on node \"ip-10-0-133-218.ec2.internal\" DevicePath \"\""
Sep 16 13:03:31.529355 ip-10-0-133-218 kubenswrapper[2580]: I0916 13:03:31.529337    2580 scope.go:117] "RemoveContainer" containerID="bdd7bc6c714d04afb0cd976fe336d11c78ec4d1e58905361b30db5263fe3d496"
Sep 16 13:03:31.534689 ip-10-0-133-218 kubenswrapper[2580]: I0916 13:03:31.534669    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-5b43d-predictor-5c567f8fb-9gwrq"]
Sep 16 13:03:31.537147 ip-10-0-133-218 kubenswrapper[2580]: I0916 13:03:31.537124    2580 scope.go:117] "RemoveContainer" containerID="8a829daa187e448fec74c99efa9e49c5b13d2909199926702640535182249df4"
Sep 16 13:03:31.537460 ip-10-0-133-218 kubenswrapper[2580]: E0916 13:03:31.537441    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8a829daa187e448fec74c99efa9e49c5b13d2909199926702640535182249df4\": container with ID starting with 8a829daa187e448fec74c99efa9e49c5b13d2909199926702640535182249df4 not found: ID does not exist" containerID="8a829daa187e448fec74c99efa9e49c5b13d2909199926702640535182249df4"
Sep 16 13:03:31.537579 ip-10-0-133-218 kubenswrapper[2580]: I0916 13:03:31.537467    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8a829daa187e448fec74c99efa9e49c5b13d2909199926702640535182249df4"} err="failed to get container status \"8a829daa187e448fec74c99efa9e49c5b13d2909199926702640535182249df4\": rpc error: code = NotFound desc = could not find container \"8a829daa187e448fec74c99efa9e49c5b13d2909199926702640535182249df4\": container with ID starting with 8a829daa187e448fec74c99efa9e49c5b13d2909199926702640535182249df4 not found: ID does not exist"
Sep 16 13:03:31.537579 ip-10-0-133-218 kubenswrapper[2580]: I0916 13:03:31.537557    2580 scope.go:117] "RemoveContainer" containerID="acd22281ec06859151ab18a7f795d88b5df52345531fde35150a40645e3a6412"
Sep 16 13:03:31.537855 ip-10-0-133-218 kubenswrapper[2580]: E0916 13:03:31.537831    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"acd22281ec06859151ab18a7f795d88b5df52345531fde35150a40645e3a6412\": container with ID starting with acd22281ec06859151ab18a7f795d88b5df52345531fde35150a40645e3a6412 not found: ID does not exist" containerID="acd22281ec06859151ab18a7f795d88b5df52345531fde35150a40645e3a6412"
Sep 16 13:03:31.537917 ip-10-0-133-218 kubenswrapper[2580]: I0916 13:03:31.537862    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"acd22281ec06859151ab18a7f795d88b5df52345531fde35150a40645e3a6412"} err="failed to get container status \"acd22281ec06859151ab18a7f795d88b5df52345531fde35150a40645e3a6412\": rpc error: code = NotFound desc = could not find container \"acd22281ec06859151ab18a7f795d88b5df52345531fde35150a40645e3a6412\": container with ID starting with acd22281ec06859151ab18a7f795d88b5df52345531fde35150a40645e3a6412 not found: ID does not exist"
Sep 16 13:03:31.537917 ip-10-0-133-218 kubenswrapper[2580]: I0916 13:03:31.537884    2580 scope.go:117] "RemoveContainer" containerID="bdd7bc6c714d04afb0cd976fe336d11c78ec4d1e58905361b30db5263fe3d496"
Sep 16 13:03:31.538137 ip-10-0-133-218 kubenswrapper[2580]: E0916 13:03:31.538117    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bdd7bc6c714d04afb0cd976fe336d11c78ec4d1e58905361b30db5263fe3d496\": container with ID starting with bdd7bc6c714d04afb0cd976fe336d11c78ec4d1e58905361b30db5263fe3d496 not found: ID does not exist" containerID="bdd7bc6c714d04afb0cd976fe336d11c78ec4d1e58905361b30db5263fe3d496"
Sep 16 13:03:31.538179 ip-10-0-133-218 kubenswrapper[2580]: I0916 13:03:31.538146    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bdd7bc6c714d04afb0cd976fe336d11c78ec4d1e58905361b30db5263fe3d496"} err="failed to get container status \"bdd7bc6c714d04afb0cd976fe336d11c78ec4d1e58905361b30db5263fe3d496\": rpc error: code = NotFound desc = could not find container \"bdd7bc6c714d04afb0cd976fe336d11c78ec4d1e58905361b30db5263fe3d496\": container with ID starting with bdd7bc6c714d04afb0cd976fe336d11c78ec4d1e58905361b30db5263fe3d496 not found: ID does not exist"
Sep 16 13:03:31.538510 ip-10-0-133-218 kubenswrapper[2580]: I0916 13:03:31.538478    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-5b43d-predictor-5c567f8fb-9gwrq"]
Sep 16 13:03:31.599350 ip-10-0-133-218 kubenswrapper[2580]: I0916 13:03:31.599321    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ccf0e9b1-5f46-45e7-8192-65d957d2ffd6" path="/var/lib/kubelet/pods/ccf0e9b1-5f46-45e7-8192-65d957d2ffd6/volumes"
Sep 16 13:03:38.437618 ip-10-0-133-218 kubenswrapper[2580]: I0916 13:03:38.437583    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-d93042-predictor-868fb57669-bcjqh" podUID="691ee328-ef29-46a2-a634-30213fa7acee" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.32:8080: connect: connection refused"
Sep 16 13:03:48.437554 ip-10-0-133-218 kubenswrapper[2580]: I0916 13:03:48.437511    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-d93042-predictor-868fb57669-bcjqh" podUID="691ee328-ef29-46a2-a634-30213fa7acee" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.32:8080: connect: connection refused"
Sep 16 13:03:58.437063 ip-10-0-133-218 kubenswrapper[2580]: I0916 13:03:58.437023    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-d93042-predictor-868fb57669-bcjqh" podUID="691ee328-ef29-46a2-a634-30213fa7acee" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.32:8080: connect: connection refused"
Sep 16 13:04:08.437619 ip-10-0-133-218 kubenswrapper[2580]: I0916 13:04:08.437526    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-d93042-predictor-868fb57669-bcjqh" podUID="691ee328-ef29-46a2-a634-30213fa7acee" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.32:8080: connect: connection refused"
Sep 16 13:04:18.437104 ip-10-0-133-218 kubenswrapper[2580]: I0916 13:04:18.437059    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-d93042-predictor-868fb57669-bcjqh" podUID="691ee328-ef29-46a2-a634-30213fa7acee" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.32:8080: connect: connection refused"
Sep 16 13:04:28.437946 ip-10-0-133-218 kubenswrapper[2580]: I0916 13:04:28.437915    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-primary-d93042-predictor-868fb57669-bcjqh"
Sep 16 13:04:51.669665 ip-10-0-133-218 kubenswrapper[2580]: I0916 13:04:51.669633    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-secondary-d93042-predictor-5d44554d5d-qqj6w"]
Sep 16 13:04:51.670098 ip-10-0-133-218 kubenswrapper[2580]: I0916 13:04:51.669973    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ccf0e9b1-5f46-45e7-8192-65d957d2ffd6" containerName="storage-initializer"
Sep 16 13:04:51.670098 ip-10-0-133-218 kubenswrapper[2580]: I0916 13:04:51.669985    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="ccf0e9b1-5f46-45e7-8192-65d957d2ffd6" containerName="storage-initializer"
Sep 16 13:04:51.670098 ip-10-0-133-218 kubenswrapper[2580]: I0916 13:04:51.669992    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ccf0e9b1-5f46-45e7-8192-65d957d2ffd6" containerName="kube-rbac-proxy"
Sep 16 13:04:51.670098 ip-10-0-133-218 kubenswrapper[2580]: I0916 13:04:51.669997    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="ccf0e9b1-5f46-45e7-8192-65d957d2ffd6" containerName="kube-rbac-proxy"
Sep 16 13:04:51.670098 ip-10-0-133-218 kubenswrapper[2580]: I0916 13:04:51.670014    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ccf0e9b1-5f46-45e7-8192-65d957d2ffd6" containerName="kserve-container"
Sep 16 13:04:51.670098 ip-10-0-133-218 kubenswrapper[2580]: I0916 13:04:51.670020    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="ccf0e9b1-5f46-45e7-8192-65d957d2ffd6" containerName="kserve-container"
Sep 16 13:04:51.670098 ip-10-0-133-218 kubenswrapper[2580]: I0916 13:04:51.670068    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="ccf0e9b1-5f46-45e7-8192-65d957d2ffd6" containerName="kserve-container"
Sep 16 13:04:51.670098 ip-10-0-133-218 kubenswrapper[2580]: I0916 13:04:51.670077    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="ccf0e9b1-5f46-45e7-8192-65d957d2ffd6" containerName="kube-rbac-proxy"
Sep 16 13:04:51.673416 ip-10-0-133-218 kubenswrapper[2580]: I0916 13:04:51.673389    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-secondary-d93042-predictor-5d44554d5d-qqj6w"
Sep 16 13:04:51.676107 ip-10-0-133-218 kubenswrapper[2580]: I0916 13:04:51.676079    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-secondary-d93042-predictor-serving-cert\""
Sep 16 13:04:51.676241 ip-10-0-133-218 kubenswrapper[2580]: I0916 13:04:51.676123    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"invalid-s3-sa-d93042-dockercfg-b8lcj\""
Sep 16 13:04:51.676241 ip-10-0-133-218 kubenswrapper[2580]: I0916 13:04:51.676135    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"invalid-s3-secret-d93042\""
Sep 16 13:04:51.676341 ip-10-0-133-218 kubenswrapper[2580]: I0916 13:04:51.676303    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-secondary-d93042-kube-rbac-proxy-sar-config\""
Sep 16 13:04:51.681810 ip-10-0-133-218 kubenswrapper[2580]: I0916 13:04:51.681782    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-secondary-d93042-predictor-5d44554d5d-qqj6w"]
Sep 16 13:04:51.756537 ip-10-0-133-218 kubenswrapper[2580]: I0916 13:04:51.756507    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/7bf1e9ad-3b6c-458b-a5d2-46e7cf659053-cabundle-cert\") pod \"isvc-secondary-d93042-predictor-5d44554d5d-qqj6w\" (UID: \"7bf1e9ad-3b6c-458b-a5d2-46e7cf659053\") " pod="kserve-ci-e2e-test/isvc-secondary-d93042-predictor-5d44554d5d-qqj6w"
Sep 16 13:04:51.756672 ip-10-0-133-218 kubenswrapper[2580]: I0916 13:04:51.756551    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/7bf1e9ad-3b6c-458b-a5d2-46e7cf659053-kserve-provision-location\") pod \"isvc-secondary-d93042-predictor-5d44554d5d-qqj6w\" (UID: \"7bf1e9ad-3b6c-458b-a5d2-46e7cf659053\") " pod="kserve-ci-e2e-test/isvc-secondary-d93042-predictor-5d44554d5d-qqj6w"
Sep 16 13:04:51.756672 ip-10-0-133-218 kubenswrapper[2580]: I0916 13:04:51.756578    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pltvz\" (UniqueName: \"kubernetes.io/projected/7bf1e9ad-3b6c-458b-a5d2-46e7cf659053-kube-api-access-pltvz\") pod \"isvc-secondary-d93042-predictor-5d44554d5d-qqj6w\" (UID: \"7bf1e9ad-3b6c-458b-a5d2-46e7cf659053\") " pod="kserve-ci-e2e-test/isvc-secondary-d93042-predictor-5d44554d5d-qqj6w"
Sep 16 13:04:51.756672 ip-10-0-133-218 kubenswrapper[2580]: I0916 13:04:51.756604    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/7bf1e9ad-3b6c-458b-a5d2-46e7cf659053-kube-rbac-proxy-sar-config\") pod \"isvc-secondary-d93042-predictor-5d44554d5d-qqj6w\" (UID: \"7bf1e9ad-3b6c-458b-a5d2-46e7cf659053\") " pod="kserve-ci-e2e-test/isvc-secondary-d93042-predictor-5d44554d5d-qqj6w"
Sep 16 13:04:51.756672 ip-10-0-133-218 kubenswrapper[2580]: I0916 13:04:51.756639    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/7bf1e9ad-3b6c-458b-a5d2-46e7cf659053-proxy-tls\") pod \"isvc-secondary-d93042-predictor-5d44554d5d-qqj6w\" (UID: \"7bf1e9ad-3b6c-458b-a5d2-46e7cf659053\") " pod="kserve-ci-e2e-test/isvc-secondary-d93042-predictor-5d44554d5d-qqj6w"
Sep 16 13:04:51.856963 ip-10-0-133-218 kubenswrapper[2580]: I0916 13:04:51.856934    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/7bf1e9ad-3b6c-458b-a5d2-46e7cf659053-kserve-provision-location\") pod \"isvc-secondary-d93042-predictor-5d44554d5d-qqj6w\" (UID: \"7bf1e9ad-3b6c-458b-a5d2-46e7cf659053\") " pod="kserve-ci-e2e-test/isvc-secondary-d93042-predictor-5d44554d5d-qqj6w"
Sep 16 13:04:51.857156 ip-10-0-133-218 kubenswrapper[2580]: I0916 13:04:51.856969    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pltvz\" (UniqueName: \"kubernetes.io/projected/7bf1e9ad-3b6c-458b-a5d2-46e7cf659053-kube-api-access-pltvz\") pod \"isvc-secondary-d93042-predictor-5d44554d5d-qqj6w\" (UID: \"7bf1e9ad-3b6c-458b-a5d2-46e7cf659053\") " pod="kserve-ci-e2e-test/isvc-secondary-d93042-predictor-5d44554d5d-qqj6w"
Sep 16 13:04:51.857156 ip-10-0-133-218 kubenswrapper[2580]: I0916 13:04:51.856986    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/7bf1e9ad-3b6c-458b-a5d2-46e7cf659053-kube-rbac-proxy-sar-config\") pod \"isvc-secondary-d93042-predictor-5d44554d5d-qqj6w\" (UID: \"7bf1e9ad-3b6c-458b-a5d2-46e7cf659053\") " pod="kserve-ci-e2e-test/isvc-secondary-d93042-predictor-5d44554d5d-qqj6w"
Sep 16 13:04:51.857156 ip-10-0-133-218 kubenswrapper[2580]: I0916 13:04:51.857020    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/7bf1e9ad-3b6c-458b-a5d2-46e7cf659053-proxy-tls\") pod \"isvc-secondary-d93042-predictor-5d44554d5d-qqj6w\" (UID: \"7bf1e9ad-3b6c-458b-a5d2-46e7cf659053\") " pod="kserve-ci-e2e-test/isvc-secondary-d93042-predictor-5d44554d5d-qqj6w"
Sep 16 13:04:51.857156 ip-10-0-133-218 kubenswrapper[2580]: I0916 13:04:51.857076    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/7bf1e9ad-3b6c-458b-a5d2-46e7cf659053-cabundle-cert\") pod \"isvc-secondary-d93042-predictor-5d44554d5d-qqj6w\" (UID: \"7bf1e9ad-3b6c-458b-a5d2-46e7cf659053\") " pod="kserve-ci-e2e-test/isvc-secondary-d93042-predictor-5d44554d5d-qqj6w"
Sep 16 13:04:51.857390 ip-10-0-133-218 kubenswrapper[2580]: E0916 13:04:51.857175    2580 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-secondary-d93042-predictor-serving-cert: secret "isvc-secondary-d93042-predictor-serving-cert" not found
Sep 16 13:04:51.857390 ip-10-0-133-218 kubenswrapper[2580]: E0916 13:04:51.857225    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7bf1e9ad-3b6c-458b-a5d2-46e7cf659053-proxy-tls podName:7bf1e9ad-3b6c-458b-a5d2-46e7cf659053 nodeName:}" failed. No retries permitted until 2026-09-16 13:04:52.357211383 +0000 UTC m=+1031.280083829 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/7bf1e9ad-3b6c-458b-a5d2-46e7cf659053-proxy-tls") pod "isvc-secondary-d93042-predictor-5d44554d5d-qqj6w" (UID: "7bf1e9ad-3b6c-458b-a5d2-46e7cf659053") : secret "isvc-secondary-d93042-predictor-serving-cert" not found
Sep 16 13:04:51.857390 ip-10-0-133-218 kubenswrapper[2580]: I0916 13:04:51.857370    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/7bf1e9ad-3b6c-458b-a5d2-46e7cf659053-kserve-provision-location\") pod \"isvc-secondary-d93042-predictor-5d44554d5d-qqj6w\" (UID: \"7bf1e9ad-3b6c-458b-a5d2-46e7cf659053\") " pod="kserve-ci-e2e-test/isvc-secondary-d93042-predictor-5d44554d5d-qqj6w"
Sep 16 13:04:51.857691 ip-10-0-133-218 kubenswrapper[2580]: I0916 13:04:51.857673    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/7bf1e9ad-3b6c-458b-a5d2-46e7cf659053-kube-rbac-proxy-sar-config\") pod \"isvc-secondary-d93042-predictor-5d44554d5d-qqj6w\" (UID: \"7bf1e9ad-3b6c-458b-a5d2-46e7cf659053\") " pod="kserve-ci-e2e-test/isvc-secondary-d93042-predictor-5d44554d5d-qqj6w"
Sep 16 13:04:51.857731 ip-10-0-133-218 kubenswrapper[2580]: I0916 13:04:51.857689    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/7bf1e9ad-3b6c-458b-a5d2-46e7cf659053-cabundle-cert\") pod \"isvc-secondary-d93042-predictor-5d44554d5d-qqj6w\" (UID: \"7bf1e9ad-3b6c-458b-a5d2-46e7cf659053\") " pod="kserve-ci-e2e-test/isvc-secondary-d93042-predictor-5d44554d5d-qqj6w"
Sep 16 13:04:51.865064 ip-10-0-133-218 kubenswrapper[2580]: I0916 13:04:51.865041    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pltvz\" (UniqueName: \"kubernetes.io/projected/7bf1e9ad-3b6c-458b-a5d2-46e7cf659053-kube-api-access-pltvz\") pod \"isvc-secondary-d93042-predictor-5d44554d5d-qqj6w\" (UID: \"7bf1e9ad-3b6c-458b-a5d2-46e7cf659053\") " pod="kserve-ci-e2e-test/isvc-secondary-d93042-predictor-5d44554d5d-qqj6w"
Sep 16 13:04:52.362105 ip-10-0-133-218 kubenswrapper[2580]: I0916 13:04:52.362068    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/7bf1e9ad-3b6c-458b-a5d2-46e7cf659053-proxy-tls\") pod \"isvc-secondary-d93042-predictor-5d44554d5d-qqj6w\" (UID: \"7bf1e9ad-3b6c-458b-a5d2-46e7cf659053\") " pod="kserve-ci-e2e-test/isvc-secondary-d93042-predictor-5d44554d5d-qqj6w"
Sep 16 13:04:52.362274 ip-10-0-133-218 kubenswrapper[2580]: E0916 13:04:52.362221    2580 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-secondary-d93042-predictor-serving-cert: secret "isvc-secondary-d93042-predictor-serving-cert" not found
Sep 16 13:04:52.362318 ip-10-0-133-218 kubenswrapper[2580]: E0916 13:04:52.362292    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7bf1e9ad-3b6c-458b-a5d2-46e7cf659053-proxy-tls podName:7bf1e9ad-3b6c-458b-a5d2-46e7cf659053 nodeName:}" failed. No retries permitted until 2026-09-16 13:04:53.362276819 +0000 UTC m=+1032.285149266 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/7bf1e9ad-3b6c-458b-a5d2-46e7cf659053-proxy-tls") pod "isvc-secondary-d93042-predictor-5d44554d5d-qqj6w" (UID: "7bf1e9ad-3b6c-458b-a5d2-46e7cf659053") : secret "isvc-secondary-d93042-predictor-serving-cert" not found
Sep 16 13:04:53.370272 ip-10-0-133-218 kubenswrapper[2580]: I0916 13:04:53.370234    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/7bf1e9ad-3b6c-458b-a5d2-46e7cf659053-proxy-tls\") pod \"isvc-secondary-d93042-predictor-5d44554d5d-qqj6w\" (UID: \"7bf1e9ad-3b6c-458b-a5d2-46e7cf659053\") " pod="kserve-ci-e2e-test/isvc-secondary-d93042-predictor-5d44554d5d-qqj6w"
Sep 16 13:04:53.372722 ip-10-0-133-218 kubenswrapper[2580]: I0916 13:04:53.372700    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/7bf1e9ad-3b6c-458b-a5d2-46e7cf659053-proxy-tls\") pod \"isvc-secondary-d93042-predictor-5d44554d5d-qqj6w\" (UID: \"7bf1e9ad-3b6c-458b-a5d2-46e7cf659053\") " pod="kserve-ci-e2e-test/isvc-secondary-d93042-predictor-5d44554d5d-qqj6w"
Sep 16 13:04:53.484579 ip-10-0-133-218 kubenswrapper[2580]: I0916 13:04:53.484530    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-secondary-d93042-predictor-5d44554d5d-qqj6w"
Sep 16 13:04:53.755012 ip-10-0-133-218 kubenswrapper[2580]: I0916 13:04:53.754987    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-secondary-d93042-predictor-5d44554d5d-qqj6w"]
Sep 16 13:04:53.763277 ip-10-0-133-218 kubenswrapper[2580]: W0916 13:04:53.763245    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7bf1e9ad_3b6c_458b_a5d2_46e7cf659053.slice/crio-4b772f61af7bcef6b5dc338db9190e136230173ad64eb3b060b664877f0da086 WatchSource:0}: Error finding container 4b772f61af7bcef6b5dc338db9190e136230173ad64eb3b060b664877f0da086: Status 404 returned error can't find the container with id 4b772f61af7bcef6b5dc338db9190e136230173ad64eb3b060b664877f0da086
Sep 16 13:04:53.824551 ip-10-0-133-218 kubenswrapper[2580]: I0916 13:04:53.824514    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-secondary-d93042-predictor-5d44554d5d-qqj6w" event={"ID":"7bf1e9ad-3b6c-458b-a5d2-46e7cf659053","Type":"ContainerStarted","Data":"0c17baeb1bcf161aaf68096fb561cbc27645294aa5ea6a11e9b2a0f3aa4e7db4"}
Sep 16 13:04:53.824666 ip-10-0-133-218 kubenswrapper[2580]: I0916 13:04:53.824614    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-secondary-d93042-predictor-5d44554d5d-qqj6w" event={"ID":"7bf1e9ad-3b6c-458b-a5d2-46e7cf659053","Type":"ContainerStarted","Data":"4b772f61af7bcef6b5dc338db9190e136230173ad64eb3b060b664877f0da086"}
Sep 16 13:05:00.856320 ip-10-0-133-218 kubenswrapper[2580]: I0916 13:05:00.856288    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-secondary-d93042-predictor-5d44554d5d-qqj6w_7bf1e9ad-3b6c-458b-a5d2-46e7cf659053/storage-initializer/0.log"
Sep 16 13:05:00.856701 ip-10-0-133-218 kubenswrapper[2580]: I0916 13:05:00.856337    2580 generic.go:356] "Generic (PLEG): container finished" podID="7bf1e9ad-3b6c-458b-a5d2-46e7cf659053" containerID="0c17baeb1bcf161aaf68096fb561cbc27645294aa5ea6a11e9b2a0f3aa4e7db4" exitCode=1
Sep 16 13:05:00.856701 ip-10-0-133-218 kubenswrapper[2580]: I0916 13:05:00.856381    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-secondary-d93042-predictor-5d44554d5d-qqj6w" event={"ID":"7bf1e9ad-3b6c-458b-a5d2-46e7cf659053","Type":"ContainerDied","Data":"0c17baeb1bcf161aaf68096fb561cbc27645294aa5ea6a11e9b2a0f3aa4e7db4"}
Sep 16 13:05:01.861553 ip-10-0-133-218 kubenswrapper[2580]: I0916 13:05:01.861525    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-secondary-d93042-predictor-5d44554d5d-qqj6w_7bf1e9ad-3b6c-458b-a5d2-46e7cf659053/storage-initializer/0.log"
Sep 16 13:05:01.861934 ip-10-0-133-218 kubenswrapper[2580]: I0916 13:05:01.861643    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-secondary-d93042-predictor-5d44554d5d-qqj6w" event={"ID":"7bf1e9ad-3b6c-458b-a5d2-46e7cf659053","Type":"ContainerStarted","Data":"b3bc06e33919a7c464b9be8eabe3e5ee472c1f118578d91b5e7f8f7cf66118ec"}
Sep 16 13:05:06.884426 ip-10-0-133-218 kubenswrapper[2580]: I0916 13:05:06.884395    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-secondary-d93042-predictor-5d44554d5d-qqj6w_7bf1e9ad-3b6c-458b-a5d2-46e7cf659053/storage-initializer/1.log"
Sep 16 13:05:06.884837 ip-10-0-133-218 kubenswrapper[2580]: I0916 13:05:06.884805    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-secondary-d93042-predictor-5d44554d5d-qqj6w_7bf1e9ad-3b6c-458b-a5d2-46e7cf659053/storage-initializer/0.log"
Sep 16 13:05:06.884880 ip-10-0-133-218 kubenswrapper[2580]: I0916 13:05:06.884837    2580 generic.go:356] "Generic (PLEG): container finished" podID="7bf1e9ad-3b6c-458b-a5d2-46e7cf659053" containerID="b3bc06e33919a7c464b9be8eabe3e5ee472c1f118578d91b5e7f8f7cf66118ec" exitCode=1
Sep 16 13:05:06.884938 ip-10-0-133-218 kubenswrapper[2580]: I0916 13:05:06.884917    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-secondary-d93042-predictor-5d44554d5d-qqj6w" event={"ID":"7bf1e9ad-3b6c-458b-a5d2-46e7cf659053","Type":"ContainerDied","Data":"b3bc06e33919a7c464b9be8eabe3e5ee472c1f118578d91b5e7f8f7cf66118ec"}
Sep 16 13:05:06.884997 ip-10-0-133-218 kubenswrapper[2580]: I0916 13:05:06.884986    2580 scope.go:117] "RemoveContainer" containerID="0c17baeb1bcf161aaf68096fb561cbc27645294aa5ea6a11e9b2a0f3aa4e7db4"
Sep 16 13:05:06.885336 ip-10-0-133-218 kubenswrapper[2580]: I0916 13:05:06.885309    2580 scope.go:117] "RemoveContainer" containerID="0c17baeb1bcf161aaf68096fb561cbc27645294aa5ea6a11e9b2a0f3aa4e7db4"
Sep 16 13:05:06.897749 ip-10-0-133-218 kubenswrapper[2580]: E0916 13:05:06.897708    2580 log.go:32] "RemoveContainer from runtime service failed" err="rpc error: code = Unknown desc = failed to delete container k8s_storage-initializer_isvc-secondary-d93042-predictor-5d44554d5d-qqj6w_kserve-ci-e2e-test_7bf1e9ad-3b6c-458b-a5d2-46e7cf659053_0 in pod sandbox 4b772f61af7bcef6b5dc338db9190e136230173ad64eb3b060b664877f0da086 from index: no such id: '0c17baeb1bcf161aaf68096fb561cbc27645294aa5ea6a11e9b2a0f3aa4e7db4'" containerID="0c17baeb1bcf161aaf68096fb561cbc27645294aa5ea6a11e9b2a0f3aa4e7db4"
Sep 16 13:05:06.897831 ip-10-0-133-218 kubenswrapper[2580]: E0916 13:05:06.897769    2580 kuberuntime_container.go:962] "Unhandled Error" err="failed to remove pod init container \"storage-initializer\": rpc error: code = Unknown desc = failed to delete container k8s_storage-initializer_isvc-secondary-d93042-predictor-5d44554d5d-qqj6w_kserve-ci-e2e-test_7bf1e9ad-3b6c-458b-a5d2-46e7cf659053_0 in pod sandbox 4b772f61af7bcef6b5dc338db9190e136230173ad64eb3b060b664877f0da086 from index: no such id: '0c17baeb1bcf161aaf68096fb561cbc27645294aa5ea6a11e9b2a0f3aa4e7db4'; Skipping pod \"isvc-secondary-d93042-predictor-5d44554d5d-qqj6w_kserve-ci-e2e-test(7bf1e9ad-3b6c-458b-a5d2-46e7cf659053)\"" logger="UnhandledError"
Sep 16 13:05:06.898012 ip-10-0-133-218 kubenswrapper[2580]: E0916 13:05:06.897994    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"storage-initializer\" with CrashLoopBackOff: \"back-off 10s restarting failed container=storage-initializer pod=isvc-secondary-d93042-predictor-5d44554d5d-qqj6w_kserve-ci-e2e-test(7bf1e9ad-3b6c-458b-a5d2-46e7cf659053)\"" pod="kserve-ci-e2e-test/isvc-secondary-d93042-predictor-5d44554d5d-qqj6w" podUID="7bf1e9ad-3b6c-458b-a5d2-46e7cf659053"
Sep 16 13:05:07.890649 ip-10-0-133-218 kubenswrapper[2580]: I0916 13:05:07.890620    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-secondary-d93042-predictor-5d44554d5d-qqj6w_7bf1e9ad-3b6c-458b-a5d2-46e7cf659053/storage-initializer/1.log"
Sep 16 13:05:11.699954 ip-10-0-133-218 kubenswrapper[2580]: I0916 13:05:11.699925    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-secondary-d93042-predictor-5d44554d5d-qqj6w"]
Sep 16 13:05:11.741514 ip-10-0-133-218 kubenswrapper[2580]: I0916 13:05:11.737905    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-primary-d93042-predictor-868fb57669-bcjqh"]
Sep 16 13:05:11.741514 ip-10-0-133-218 kubenswrapper[2580]: I0916 13:05:11.738338    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-primary-d93042-predictor-868fb57669-bcjqh" podUID="691ee328-ef29-46a2-a634-30213fa7acee" containerName="kserve-container" containerID="cri-o://ac84066aa22b5f2f7e92a2bffe32dc5ef88c093104f1c91fe4a6b8c9672cfc86" gracePeriod=30
Sep 16 13:05:11.741514 ip-10-0-133-218 kubenswrapper[2580]: I0916 13:05:11.738738    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-primary-d93042-predictor-868fb57669-bcjqh" podUID="691ee328-ef29-46a2-a634-30213fa7acee" containerName="kube-rbac-proxy" containerID="cri-o://a8b50f7ba5799c5b9508f177944b88d183f382dab1c69656d84f417bf78dd5ec" gracePeriod=30
Sep 16 13:05:11.860614 ip-10-0-133-218 kubenswrapper[2580]: I0916 13:05:11.860593    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-secondary-d93042-predictor-5d44554d5d-qqj6w_7bf1e9ad-3b6c-458b-a5d2-46e7cf659053/storage-initializer/1.log"
Sep 16 13:05:11.862567 ip-10-0-133-218 kubenswrapper[2580]: I0916 13:05:11.862553    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-secondary-d93042-predictor-5d44554d5d-qqj6w"
Sep 16 13:05:11.908630 ip-10-0-133-218 kubenswrapper[2580]: I0916 13:05:11.908599    2580 generic.go:356] "Generic (PLEG): container finished" podID="691ee328-ef29-46a2-a634-30213fa7acee" containerID="a8b50f7ba5799c5b9508f177944b88d183f382dab1c69656d84f417bf78dd5ec" exitCode=2
Sep 16 13:05:11.908785 ip-10-0-133-218 kubenswrapper[2580]: I0916 13:05:11.908675    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-primary-d93042-predictor-868fb57669-bcjqh" event={"ID":"691ee328-ef29-46a2-a634-30213fa7acee","Type":"ContainerDied","Data":"a8b50f7ba5799c5b9508f177944b88d183f382dab1c69656d84f417bf78dd5ec"}
Sep 16 13:05:11.909908 ip-10-0-133-218 kubenswrapper[2580]: I0916 13:05:11.909890    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-secondary-d93042-predictor-5d44554d5d-qqj6w_7bf1e9ad-3b6c-458b-a5d2-46e7cf659053/storage-initializer/1.log"
Sep 16 13:05:11.910001 ip-10-0-133-218 kubenswrapper[2580]: I0916 13:05:11.909953    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-secondary-d93042-predictor-5d44554d5d-qqj6w" event={"ID":"7bf1e9ad-3b6c-458b-a5d2-46e7cf659053","Type":"ContainerDied","Data":"4b772f61af7bcef6b5dc338db9190e136230173ad64eb3b060b664877f0da086"}
Sep 16 13:05:11.910001 ip-10-0-133-218 kubenswrapper[2580]: I0916 13:05:11.909984    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-secondary-d93042-predictor-5d44554d5d-qqj6w"
Sep 16 13:05:11.910076 ip-10-0-133-218 kubenswrapper[2580]: I0916 13:05:11.910008    2580 scope.go:117] "RemoveContainer" containerID="b3bc06e33919a7c464b9be8eabe3e5ee472c1f118578d91b5e7f8f7cf66118ec"
Sep 16 13:05:11.913519 ip-10-0-133-218 kubenswrapper[2580]: I0916 13:05:11.913479    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/7bf1e9ad-3b6c-458b-a5d2-46e7cf659053-cabundle-cert\") pod \"7bf1e9ad-3b6c-458b-a5d2-46e7cf659053\" (UID: \"7bf1e9ad-3b6c-458b-a5d2-46e7cf659053\") "
Sep 16 13:05:11.913595 ip-10-0-133-218 kubenswrapper[2580]: I0916 13:05:11.913536    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/7bf1e9ad-3b6c-458b-a5d2-46e7cf659053-proxy-tls\") pod \"7bf1e9ad-3b6c-458b-a5d2-46e7cf659053\" (UID: \"7bf1e9ad-3b6c-458b-a5d2-46e7cf659053\") "
Sep 16 13:05:11.913595 ip-10-0-133-218 kubenswrapper[2580]: I0916 13:05:11.913572    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-pltvz\" (UniqueName: \"kubernetes.io/projected/7bf1e9ad-3b6c-458b-a5d2-46e7cf659053-kube-api-access-pltvz\") pod \"7bf1e9ad-3b6c-458b-a5d2-46e7cf659053\" (UID: \"7bf1e9ad-3b6c-458b-a5d2-46e7cf659053\") "
Sep 16 13:05:11.913701 ip-10-0-133-218 kubenswrapper[2580]: I0916 13:05:11.913602    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/7bf1e9ad-3b6c-458b-a5d2-46e7cf659053-kube-rbac-proxy-sar-config\") pod \"7bf1e9ad-3b6c-458b-a5d2-46e7cf659053\" (UID: \"7bf1e9ad-3b6c-458b-a5d2-46e7cf659053\") "
Sep 16 13:05:11.913869 ip-10-0-133-218 kubenswrapper[2580]: I0916 13:05:11.913846    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7bf1e9ad-3b6c-458b-a5d2-46e7cf659053-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "7bf1e9ad-3b6c-458b-a5d2-46e7cf659053" (UID: "7bf1e9ad-3b6c-458b-a5d2-46e7cf659053"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 13:05:11.913869 ip-10-0-133-218 kubenswrapper[2580]: I0916 13:05:11.913859    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/7bf1e9ad-3b6c-458b-a5d2-46e7cf659053-kserve-provision-location\") pod \"7bf1e9ad-3b6c-458b-a5d2-46e7cf659053\" (UID: \"7bf1e9ad-3b6c-458b-a5d2-46e7cf659053\") "
Sep 16 13:05:11.914021 ip-10-0-133-218 kubenswrapper[2580]: I0916 13:05:11.913946    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7bf1e9ad-3b6c-458b-a5d2-46e7cf659053-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "7bf1e9ad-3b6c-458b-a5d2-46e7cf659053" (UID: "7bf1e9ad-3b6c-458b-a5d2-46e7cf659053"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 13:05:11.914076 ip-10-0-133-218 kubenswrapper[2580]: I0916 13:05:11.914051    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7bf1e9ad-3b6c-458b-a5d2-46e7cf659053-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "7bf1e9ad-3b6c-458b-a5d2-46e7cf659053" (UID: "7bf1e9ad-3b6c-458b-a5d2-46e7cf659053"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 13:05:11.914189 ip-10-0-133-218 kubenswrapper[2580]: I0916 13:05:11.914142    2580 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/7bf1e9ad-3b6c-458b-a5d2-46e7cf659053-kube-rbac-proxy-sar-config\") on node \"ip-10-0-133-218.ec2.internal\" DevicePath \"\""
Sep 16 13:05:11.914189 ip-10-0-133-218 kubenswrapper[2580]: I0916 13:05:11.914159    2580 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/7bf1e9ad-3b6c-458b-a5d2-46e7cf659053-kserve-provision-location\") on node \"ip-10-0-133-218.ec2.internal\" DevicePath \"\""
Sep 16 13:05:11.914296 ip-10-0-133-218 kubenswrapper[2580]: I0916 13:05:11.914197    2580 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/7bf1e9ad-3b6c-458b-a5d2-46e7cf659053-cabundle-cert\") on node \"ip-10-0-133-218.ec2.internal\" DevicePath \"\""
Sep 16 13:05:11.915732 ip-10-0-133-218 kubenswrapper[2580]: I0916 13:05:11.915709    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7bf1e9ad-3b6c-458b-a5d2-46e7cf659053-kube-api-access-pltvz" (OuterVolumeSpecName: "kube-api-access-pltvz") pod "7bf1e9ad-3b6c-458b-a5d2-46e7cf659053" (UID: "7bf1e9ad-3b6c-458b-a5d2-46e7cf659053"). InnerVolumeSpecName "kube-api-access-pltvz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 13:05:11.915732 ip-10-0-133-218 kubenswrapper[2580]: I0916 13:05:11.915723    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7bf1e9ad-3b6c-458b-a5d2-46e7cf659053-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "7bf1e9ad-3b6c-458b-a5d2-46e7cf659053" (UID: "7bf1e9ad-3b6c-458b-a5d2-46e7cf659053"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 13:05:12.014605 ip-10-0-133-218 kubenswrapper[2580]: I0916 13:05:12.014538    2580 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/7bf1e9ad-3b6c-458b-a5d2-46e7cf659053-proxy-tls\") on node \"ip-10-0-133-218.ec2.internal\" DevicePath \"\""
Sep 16 13:05:12.014605 ip-10-0-133-218 kubenswrapper[2580]: I0916 13:05:12.014562    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pltvz\" (UniqueName: \"kubernetes.io/projected/7bf1e9ad-3b6c-458b-a5d2-46e7cf659053-kube-api-access-pltvz\") on node \"ip-10-0-133-218.ec2.internal\" DevicePath \"\""
Sep 16 13:05:12.250590 ip-10-0-133-218 kubenswrapper[2580]: I0916 13:05:12.250561    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-secondary-d93042-predictor-5d44554d5d-qqj6w"]
Sep 16 13:05:12.256480 ip-10-0-133-218 kubenswrapper[2580]: I0916 13:05:12.256448    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-secondary-d93042-predictor-5d44554d5d-qqj6w"]
Sep 16 13:05:13.433208 ip-10-0-133-218 kubenswrapper[2580]: I0916 13:05:13.433167    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-d93042-predictor-868fb57669-bcjqh" podUID="691ee328-ef29-46a2-a634-30213fa7acee" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.32:8643/healthz\": dial tcp 10.133.0.32:8643: connect: connection refused"
Sep 16 13:05:13.598868 ip-10-0-133-218 kubenswrapper[2580]: I0916 13:05:13.598837    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7bf1e9ad-3b6c-458b-a5d2-46e7cf659053" path="/var/lib/kubelet/pods/7bf1e9ad-3b6c-458b-a5d2-46e7cf659053/volumes"
Sep 16 13:05:16.383834 ip-10-0-133-218 kubenswrapper[2580]: I0916 13:05:16.383807    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-primary-d93042-predictor-868fb57669-bcjqh"
Sep 16 13:05:16.448617 ip-10-0-133-218 kubenswrapper[2580]: I0916 13:05:16.448554    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/691ee328-ef29-46a2-a634-30213fa7acee-kserve-provision-location\") pod \"691ee328-ef29-46a2-a634-30213fa7acee\" (UID: \"691ee328-ef29-46a2-a634-30213fa7acee\") "
Sep 16 13:05:16.448617 ip-10-0-133-218 kubenswrapper[2580]: I0916 13:05:16.448590    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/691ee328-ef29-46a2-a634-30213fa7acee-cabundle-cert\") pod \"691ee328-ef29-46a2-a634-30213fa7acee\" (UID: \"691ee328-ef29-46a2-a634-30213fa7acee\") "
Sep 16 13:05:16.448773 ip-10-0-133-218 kubenswrapper[2580]: I0916 13:05:16.448632    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/691ee328-ef29-46a2-a634-30213fa7acee-proxy-tls\") pod \"691ee328-ef29-46a2-a634-30213fa7acee\" (UID: \"691ee328-ef29-46a2-a634-30213fa7acee\") "
Sep 16 13:05:16.448773 ip-10-0-133-218 kubenswrapper[2580]: I0916 13:05:16.448685    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/691ee328-ef29-46a2-a634-30213fa7acee-kube-rbac-proxy-sar-config\") pod \"691ee328-ef29-46a2-a634-30213fa7acee\" (UID: \"691ee328-ef29-46a2-a634-30213fa7acee\") "
Sep 16 13:05:16.448773 ip-10-0-133-218 kubenswrapper[2580]: I0916 13:05:16.448706    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-v6tv9\" (UniqueName: \"kubernetes.io/projected/691ee328-ef29-46a2-a634-30213fa7acee-kube-api-access-v6tv9\") pod \"691ee328-ef29-46a2-a634-30213fa7acee\" (UID: \"691ee328-ef29-46a2-a634-30213fa7acee\") "
Sep 16 13:05:16.448894 ip-10-0-133-218 kubenswrapper[2580]: I0916 13:05:16.448835    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/691ee328-ef29-46a2-a634-30213fa7acee-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "691ee328-ef29-46a2-a634-30213fa7acee" (UID: "691ee328-ef29-46a2-a634-30213fa7acee"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 13:05:16.449036 ip-10-0-133-218 kubenswrapper[2580]: I0916 13:05:16.449009    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/691ee328-ef29-46a2-a634-30213fa7acee-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "691ee328-ef29-46a2-a634-30213fa7acee" (UID: "691ee328-ef29-46a2-a634-30213fa7acee"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 13:05:16.449106 ip-10-0-133-218 kubenswrapper[2580]: I0916 13:05:16.449049    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/691ee328-ef29-46a2-a634-30213fa7acee-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "691ee328-ef29-46a2-a634-30213fa7acee" (UID: "691ee328-ef29-46a2-a634-30213fa7acee"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 13:05:16.450670 ip-10-0-133-218 kubenswrapper[2580]: I0916 13:05:16.450641    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/691ee328-ef29-46a2-a634-30213fa7acee-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "691ee328-ef29-46a2-a634-30213fa7acee" (UID: "691ee328-ef29-46a2-a634-30213fa7acee"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 13:05:16.450783 ip-10-0-133-218 kubenswrapper[2580]: I0916 13:05:16.450730    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/691ee328-ef29-46a2-a634-30213fa7acee-kube-api-access-v6tv9" (OuterVolumeSpecName: "kube-api-access-v6tv9") pod "691ee328-ef29-46a2-a634-30213fa7acee" (UID: "691ee328-ef29-46a2-a634-30213fa7acee"). InnerVolumeSpecName "kube-api-access-v6tv9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 13:05:16.550029 ip-10-0-133-218 kubenswrapper[2580]: I0916 13:05:16.549985    2580 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/691ee328-ef29-46a2-a634-30213fa7acee-cabundle-cert\") on node \"ip-10-0-133-218.ec2.internal\" DevicePath \"\""
Sep 16 13:05:16.550029 ip-10-0-133-218 kubenswrapper[2580]: I0916 13:05:16.550024    2580 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/691ee328-ef29-46a2-a634-30213fa7acee-proxy-tls\") on node \"ip-10-0-133-218.ec2.internal\" DevicePath \"\""
Sep 16 13:05:16.550029 ip-10-0-133-218 kubenswrapper[2580]: I0916 13:05:16.550033    2580 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/691ee328-ef29-46a2-a634-30213fa7acee-kube-rbac-proxy-sar-config\") on node \"ip-10-0-133-218.ec2.internal\" DevicePath \"\""
Sep 16 13:05:16.550242 ip-10-0-133-218 kubenswrapper[2580]: I0916 13:05:16.550043    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v6tv9\" (UniqueName: \"kubernetes.io/projected/691ee328-ef29-46a2-a634-30213fa7acee-kube-api-access-v6tv9\") on node \"ip-10-0-133-218.ec2.internal\" DevicePath \"\""
Sep 16 13:05:16.550242 ip-10-0-133-218 kubenswrapper[2580]: I0916 13:05:16.550052    2580 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/691ee328-ef29-46a2-a634-30213fa7acee-kserve-provision-location\") on node \"ip-10-0-133-218.ec2.internal\" DevicePath \"\""
Sep 16 13:05:16.932592 ip-10-0-133-218 kubenswrapper[2580]: I0916 13:05:16.932553    2580 generic.go:356] "Generic (PLEG): container finished" podID="691ee328-ef29-46a2-a634-30213fa7acee" containerID="ac84066aa22b5f2f7e92a2bffe32dc5ef88c093104f1c91fe4a6b8c9672cfc86" exitCode=0
Sep 16 13:05:16.932778 ip-10-0-133-218 kubenswrapper[2580]: I0916 13:05:16.932637    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-primary-d93042-predictor-868fb57669-bcjqh"
Sep 16 13:05:16.932778 ip-10-0-133-218 kubenswrapper[2580]: I0916 13:05:16.932654    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-primary-d93042-predictor-868fb57669-bcjqh" event={"ID":"691ee328-ef29-46a2-a634-30213fa7acee","Type":"ContainerDied","Data":"ac84066aa22b5f2f7e92a2bffe32dc5ef88c093104f1c91fe4a6b8c9672cfc86"}
Sep 16 13:05:16.932778 ip-10-0-133-218 kubenswrapper[2580]: I0916 13:05:16.932728    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-primary-d93042-predictor-868fb57669-bcjqh" event={"ID":"691ee328-ef29-46a2-a634-30213fa7acee","Type":"ContainerDied","Data":"39da15870fe3b8b72ec44f2ad5566f7f766879085655fb4976dfa01239e2970e"}
Sep 16 13:05:16.932778 ip-10-0-133-218 kubenswrapper[2580]: I0916 13:05:16.932753    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-init-fail-fb44b2-predictor-64566c8d6d-kkzjx"]
Sep 16 13:05:16.932778 ip-10-0-133-218 kubenswrapper[2580]: I0916 13:05:16.932773    2580 scope.go:117] "RemoveContainer" containerID="a8b50f7ba5799c5b9508f177944b88d183f382dab1c69656d84f417bf78dd5ec"
Sep 16 13:05:16.933362 ip-10-0-133-218 kubenswrapper[2580]: I0916 13:05:16.933342    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="691ee328-ef29-46a2-a634-30213fa7acee" containerName="kserve-container"
Sep 16 13:05:16.933362 ip-10-0-133-218 kubenswrapper[2580]: I0916 13:05:16.933362    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="691ee328-ef29-46a2-a634-30213fa7acee" containerName="kserve-container"
Sep 16 13:05:16.933521 ip-10-0-133-218 kubenswrapper[2580]: I0916 13:05:16.933386    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="691ee328-ef29-46a2-a634-30213fa7acee" containerName="storage-initializer"
Sep 16 13:05:16.933521 ip-10-0-133-218 kubenswrapper[2580]: I0916 13:05:16.933395    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="691ee328-ef29-46a2-a634-30213fa7acee" containerName="storage-initializer"
Sep 16 13:05:16.933521 ip-10-0-133-218 kubenswrapper[2580]: I0916 13:05:16.933406    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7bf1e9ad-3b6c-458b-a5d2-46e7cf659053" containerName="storage-initializer"
Sep 16 13:05:16.933521 ip-10-0-133-218 kubenswrapper[2580]: I0916 13:05:16.933414    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="7bf1e9ad-3b6c-458b-a5d2-46e7cf659053" containerName="storage-initializer"
Sep 16 13:05:16.933521 ip-10-0-133-218 kubenswrapper[2580]: I0916 13:05:16.933424    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7bf1e9ad-3b6c-458b-a5d2-46e7cf659053" containerName="storage-initializer"
Sep 16 13:05:16.933521 ip-10-0-133-218 kubenswrapper[2580]: I0916 13:05:16.933430    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="7bf1e9ad-3b6c-458b-a5d2-46e7cf659053" containerName="storage-initializer"
Sep 16 13:05:16.933521 ip-10-0-133-218 kubenswrapper[2580]: I0916 13:05:16.933444    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="691ee328-ef29-46a2-a634-30213fa7acee" containerName="kube-rbac-proxy"
Sep 16 13:05:16.933521 ip-10-0-133-218 kubenswrapper[2580]: I0916 13:05:16.933449    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="691ee328-ef29-46a2-a634-30213fa7acee" containerName="kube-rbac-proxy"
Sep 16 13:05:16.933521 ip-10-0-133-218 kubenswrapper[2580]: I0916 13:05:16.933525    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="691ee328-ef29-46a2-a634-30213fa7acee" containerName="kube-rbac-proxy"
Sep 16 13:05:16.933917 ip-10-0-133-218 kubenswrapper[2580]: I0916 13:05:16.933540    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="7bf1e9ad-3b6c-458b-a5d2-46e7cf659053" containerName="storage-initializer"
Sep 16 13:05:16.933917 ip-10-0-133-218 kubenswrapper[2580]: I0916 13:05:16.933550    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="7bf1e9ad-3b6c-458b-a5d2-46e7cf659053" containerName="storage-initializer"
Sep 16 13:05:16.933917 ip-10-0-133-218 kubenswrapper[2580]: I0916 13:05:16.933559    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="691ee328-ef29-46a2-a634-30213fa7acee" containerName="kserve-container"
Sep 16 13:05:16.936777 ip-10-0-133-218 kubenswrapper[2580]: I0916 13:05:16.936756    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-init-fail-fb44b2-predictor-64566c8d6d-kkzjx"
Sep 16 13:05:16.940536 ip-10-0-133-218 kubenswrapper[2580]: I0916 13:05:16.940517    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Sep 16 13:05:16.940767 ip-10-0-133-218 kubenswrapper[2580]: I0916 13:05:16.940752    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-init-fail-fb44b2-predictor-serving-cert\""
Sep 16 13:05:16.941060 ip-10-0-133-218 kubenswrapper[2580]: I0916 13:05:16.941041    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Sep 16 13:05:16.941472 ip-10-0-133-218 kubenswrapper[2580]: I0916 13:05:16.941451    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-init-fail-fb44b2-kube-rbac-proxy-sar-config\""
Sep 16 13:05:16.941756 ip-10-0-133-218 kubenswrapper[2580]: I0916 13:05:16.941573    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"fail-s3-secret-fb44b2\""
Sep 16 13:05:16.941899 ip-10-0-133-218 kubenswrapper[2580]: I0916 13:05:16.941576    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Sep 16 13:05:16.942015 ip-10-0-133-218 kubenswrapper[2580]: I0916 13:05:16.941699    2580 scope.go:117] "RemoveContainer" containerID="ac84066aa22b5f2f7e92a2bffe32dc5ef88c093104f1c91fe4a6b8c9672cfc86"
Sep 16 13:05:16.942165 ip-10-0-133-218 kubenswrapper[2580]: I0916 13:05:16.942142    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"fail-s3-sa-fb44b2-dockercfg-qr8vq\""
Sep 16 13:05:16.951616 ip-10-0-133-218 kubenswrapper[2580]: I0916 13:05:16.951603    2580 scope.go:117] "RemoveContainer" containerID="9abe19f1a094279a87fa8bdc6d2b0c089f957bdb65148d0edfb45a5840990453"
Sep 16 13:05:16.953478 ip-10-0-133-218 kubenswrapper[2580]: I0916 13:05:16.953460    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f07fc368-24b1-40bf-b097-3cdd9137ec2f-proxy-tls\") pod \"isvc-init-fail-fb44b2-predictor-64566c8d6d-kkzjx\" (UID: \"f07fc368-24b1-40bf-b097-3cdd9137ec2f\") " pod="kserve-ci-e2e-test/isvc-init-fail-fb44b2-predictor-64566c8d6d-kkzjx"
Sep 16 13:05:16.953985 ip-10-0-133-218 kubenswrapper[2580]: I0916 13:05:16.953553    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f07fc368-24b1-40bf-b097-3cdd9137ec2f-kserve-provision-location\") pod \"isvc-init-fail-fb44b2-predictor-64566c8d6d-kkzjx\" (UID: \"f07fc368-24b1-40bf-b097-3cdd9137ec2f\") " pod="kserve-ci-e2e-test/isvc-init-fail-fb44b2-predictor-64566c8d6d-kkzjx"
Sep 16 13:05:16.953985 ip-10-0-133-218 kubenswrapper[2580]: I0916 13:05:16.953649    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/f07fc368-24b1-40bf-b097-3cdd9137ec2f-cabundle-cert\") pod \"isvc-init-fail-fb44b2-predictor-64566c8d6d-kkzjx\" (UID: \"f07fc368-24b1-40bf-b097-3cdd9137ec2f\") " pod="kserve-ci-e2e-test/isvc-init-fail-fb44b2-predictor-64566c8d6d-kkzjx"
Sep 16 13:05:16.953985 ip-10-0-133-218 kubenswrapper[2580]: I0916 13:05:16.953721    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/f07fc368-24b1-40bf-b097-3cdd9137ec2f-kube-rbac-proxy-sar-config\") pod \"isvc-init-fail-fb44b2-predictor-64566c8d6d-kkzjx\" (UID: \"f07fc368-24b1-40bf-b097-3cdd9137ec2f\") " pod="kserve-ci-e2e-test/isvc-init-fail-fb44b2-predictor-64566c8d6d-kkzjx"
Sep 16 13:05:16.953985 ip-10-0-133-218 kubenswrapper[2580]: I0916 13:05:16.953776    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x5q5g\" (UniqueName: \"kubernetes.io/projected/f07fc368-24b1-40bf-b097-3cdd9137ec2f-kube-api-access-x5q5g\") pod \"isvc-init-fail-fb44b2-predictor-64566c8d6d-kkzjx\" (UID: \"f07fc368-24b1-40bf-b097-3cdd9137ec2f\") " pod="kserve-ci-e2e-test/isvc-init-fail-fb44b2-predictor-64566c8d6d-kkzjx"
Sep 16 13:05:16.956124 ip-10-0-133-218 kubenswrapper[2580]: I0916 13:05:16.956106    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-init-fail-fb44b2-predictor-64566c8d6d-kkzjx"]
Sep 16 13:05:16.959780 ip-10-0-133-218 kubenswrapper[2580]: I0916 13:05:16.959764    2580 scope.go:117] "RemoveContainer" containerID="a8b50f7ba5799c5b9508f177944b88d183f382dab1c69656d84f417bf78dd5ec"
Sep 16 13:05:16.960066 ip-10-0-133-218 kubenswrapper[2580]: E0916 13:05:16.960048    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a8b50f7ba5799c5b9508f177944b88d183f382dab1c69656d84f417bf78dd5ec\": container with ID starting with a8b50f7ba5799c5b9508f177944b88d183f382dab1c69656d84f417bf78dd5ec not found: ID does not exist" containerID="a8b50f7ba5799c5b9508f177944b88d183f382dab1c69656d84f417bf78dd5ec"
Sep 16 13:05:16.960121 ip-10-0-133-218 kubenswrapper[2580]: I0916 13:05:16.960074    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a8b50f7ba5799c5b9508f177944b88d183f382dab1c69656d84f417bf78dd5ec"} err="failed to get container status \"a8b50f7ba5799c5b9508f177944b88d183f382dab1c69656d84f417bf78dd5ec\": rpc error: code = NotFound desc = could not find container \"a8b50f7ba5799c5b9508f177944b88d183f382dab1c69656d84f417bf78dd5ec\": container with ID starting with a8b50f7ba5799c5b9508f177944b88d183f382dab1c69656d84f417bf78dd5ec not found: ID does not exist"
Sep 16 13:05:16.960121 ip-10-0-133-218 kubenswrapper[2580]: I0916 13:05:16.960090    2580 scope.go:117] "RemoveContainer" containerID="ac84066aa22b5f2f7e92a2bffe32dc5ef88c093104f1c91fe4a6b8c9672cfc86"
Sep 16 13:05:16.960316 ip-10-0-133-218 kubenswrapper[2580]: E0916 13:05:16.960300    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ac84066aa22b5f2f7e92a2bffe32dc5ef88c093104f1c91fe4a6b8c9672cfc86\": container with ID starting with ac84066aa22b5f2f7e92a2bffe32dc5ef88c093104f1c91fe4a6b8c9672cfc86 not found: ID does not exist" containerID="ac84066aa22b5f2f7e92a2bffe32dc5ef88c093104f1c91fe4a6b8c9672cfc86"
Sep 16 13:05:16.960359 ip-10-0-133-218 kubenswrapper[2580]: I0916 13:05:16.960319    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ac84066aa22b5f2f7e92a2bffe32dc5ef88c093104f1c91fe4a6b8c9672cfc86"} err="failed to get container status \"ac84066aa22b5f2f7e92a2bffe32dc5ef88c093104f1c91fe4a6b8c9672cfc86\": rpc error: code = NotFound desc = could not find container \"ac84066aa22b5f2f7e92a2bffe32dc5ef88c093104f1c91fe4a6b8c9672cfc86\": container with ID starting with ac84066aa22b5f2f7e92a2bffe32dc5ef88c093104f1c91fe4a6b8c9672cfc86 not found: ID does not exist"
Sep 16 13:05:16.960359 ip-10-0-133-218 kubenswrapper[2580]: I0916 13:05:16.960332    2580 scope.go:117] "RemoveContainer" containerID="9abe19f1a094279a87fa8bdc6d2b0c089f957bdb65148d0edfb45a5840990453"
Sep 16 13:05:16.960664 ip-10-0-133-218 kubenswrapper[2580]: E0916 13:05:16.960645    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9abe19f1a094279a87fa8bdc6d2b0c089f957bdb65148d0edfb45a5840990453\": container with ID starting with 9abe19f1a094279a87fa8bdc6d2b0c089f957bdb65148d0edfb45a5840990453 not found: ID does not exist" containerID="9abe19f1a094279a87fa8bdc6d2b0c089f957bdb65148d0edfb45a5840990453"
Sep 16 13:05:16.960724 ip-10-0-133-218 kubenswrapper[2580]: I0916 13:05:16.960672    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9abe19f1a094279a87fa8bdc6d2b0c089f957bdb65148d0edfb45a5840990453"} err="failed to get container status \"9abe19f1a094279a87fa8bdc6d2b0c089f957bdb65148d0edfb45a5840990453\": rpc error: code = NotFound desc = could not find container \"9abe19f1a094279a87fa8bdc6d2b0c089f957bdb65148d0edfb45a5840990453\": container with ID starting with 9abe19f1a094279a87fa8bdc6d2b0c089f957bdb65148d0edfb45a5840990453 not found: ID does not exist"
Sep 16 13:05:16.998184 ip-10-0-133-218 kubenswrapper[2580]: I0916 13:05:16.998155    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-primary-d93042-predictor-868fb57669-bcjqh"]
Sep 16 13:05:17.002110 ip-10-0-133-218 kubenswrapper[2580]: I0916 13:05:17.002089    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-primary-d93042-predictor-868fb57669-bcjqh"]
Sep 16 13:05:17.054391 ip-10-0-133-218 kubenswrapper[2580]: I0916 13:05:17.054367    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/f07fc368-24b1-40bf-b097-3cdd9137ec2f-cabundle-cert\") pod \"isvc-init-fail-fb44b2-predictor-64566c8d6d-kkzjx\" (UID: \"f07fc368-24b1-40bf-b097-3cdd9137ec2f\") " pod="kserve-ci-e2e-test/isvc-init-fail-fb44b2-predictor-64566c8d6d-kkzjx"
Sep 16 13:05:17.054526 ip-10-0-133-218 kubenswrapper[2580]: I0916 13:05:17.054401    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/f07fc368-24b1-40bf-b097-3cdd9137ec2f-kube-rbac-proxy-sar-config\") pod \"isvc-init-fail-fb44b2-predictor-64566c8d6d-kkzjx\" (UID: \"f07fc368-24b1-40bf-b097-3cdd9137ec2f\") " pod="kserve-ci-e2e-test/isvc-init-fail-fb44b2-predictor-64566c8d6d-kkzjx"
Sep 16 13:05:17.054526 ip-10-0-133-218 kubenswrapper[2580]: I0916 13:05:17.054419    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x5q5g\" (UniqueName: \"kubernetes.io/projected/f07fc368-24b1-40bf-b097-3cdd9137ec2f-kube-api-access-x5q5g\") pod \"isvc-init-fail-fb44b2-predictor-64566c8d6d-kkzjx\" (UID: \"f07fc368-24b1-40bf-b097-3cdd9137ec2f\") " pod="kserve-ci-e2e-test/isvc-init-fail-fb44b2-predictor-64566c8d6d-kkzjx"
Sep 16 13:05:17.054526 ip-10-0-133-218 kubenswrapper[2580]: I0916 13:05:17.054456    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f07fc368-24b1-40bf-b097-3cdd9137ec2f-proxy-tls\") pod \"isvc-init-fail-fb44b2-predictor-64566c8d6d-kkzjx\" (UID: \"f07fc368-24b1-40bf-b097-3cdd9137ec2f\") " pod="kserve-ci-e2e-test/isvc-init-fail-fb44b2-predictor-64566c8d6d-kkzjx"
Sep 16 13:05:17.054526 ip-10-0-133-218 kubenswrapper[2580]: I0916 13:05:17.054476    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f07fc368-24b1-40bf-b097-3cdd9137ec2f-kserve-provision-location\") pod \"isvc-init-fail-fb44b2-predictor-64566c8d6d-kkzjx\" (UID: \"f07fc368-24b1-40bf-b097-3cdd9137ec2f\") " pod="kserve-ci-e2e-test/isvc-init-fail-fb44b2-predictor-64566c8d6d-kkzjx"
Sep 16 13:05:17.054929 ip-10-0-133-218 kubenswrapper[2580]: I0916 13:05:17.054906    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f07fc368-24b1-40bf-b097-3cdd9137ec2f-kserve-provision-location\") pod \"isvc-init-fail-fb44b2-predictor-64566c8d6d-kkzjx\" (UID: \"f07fc368-24b1-40bf-b097-3cdd9137ec2f\") " pod="kserve-ci-e2e-test/isvc-init-fail-fb44b2-predictor-64566c8d6d-kkzjx"
Sep 16 13:05:17.055098 ip-10-0-133-218 kubenswrapper[2580]: I0916 13:05:17.055078    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/f07fc368-24b1-40bf-b097-3cdd9137ec2f-cabundle-cert\") pod \"isvc-init-fail-fb44b2-predictor-64566c8d6d-kkzjx\" (UID: \"f07fc368-24b1-40bf-b097-3cdd9137ec2f\") " pod="kserve-ci-e2e-test/isvc-init-fail-fb44b2-predictor-64566c8d6d-kkzjx"
Sep 16 13:05:17.055161 ip-10-0-133-218 kubenswrapper[2580]: I0916 13:05:17.055094    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/f07fc368-24b1-40bf-b097-3cdd9137ec2f-kube-rbac-proxy-sar-config\") pod \"isvc-init-fail-fb44b2-predictor-64566c8d6d-kkzjx\" (UID: \"f07fc368-24b1-40bf-b097-3cdd9137ec2f\") " pod="kserve-ci-e2e-test/isvc-init-fail-fb44b2-predictor-64566c8d6d-kkzjx"
Sep 16 13:05:17.056795 ip-10-0-133-218 kubenswrapper[2580]: I0916 13:05:17.056777    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f07fc368-24b1-40bf-b097-3cdd9137ec2f-proxy-tls\") pod \"isvc-init-fail-fb44b2-predictor-64566c8d6d-kkzjx\" (UID: \"f07fc368-24b1-40bf-b097-3cdd9137ec2f\") " pod="kserve-ci-e2e-test/isvc-init-fail-fb44b2-predictor-64566c8d6d-kkzjx"
Sep 16 13:05:17.062463 ip-10-0-133-218 kubenswrapper[2580]: I0916 13:05:17.062430    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-x5q5g\" (UniqueName: \"kubernetes.io/projected/f07fc368-24b1-40bf-b097-3cdd9137ec2f-kube-api-access-x5q5g\") pod \"isvc-init-fail-fb44b2-predictor-64566c8d6d-kkzjx\" (UID: \"f07fc368-24b1-40bf-b097-3cdd9137ec2f\") " pod="kserve-ci-e2e-test/isvc-init-fail-fb44b2-predictor-64566c8d6d-kkzjx"
Sep 16 13:05:17.249197 ip-10-0-133-218 kubenswrapper[2580]: I0916 13:05:17.249123    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-init-fail-fb44b2-predictor-64566c8d6d-kkzjx"
Sep 16 13:05:17.526062 ip-10-0-133-218 kubenswrapper[2580]: I0916 13:05:17.526037    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-init-fail-fb44b2-predictor-64566c8d6d-kkzjx"]
Sep 16 13:05:17.528193 ip-10-0-133-218 kubenswrapper[2580]: W0916 13:05:17.528168    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf07fc368_24b1_40bf_b097_3cdd9137ec2f.slice/crio-6bbf1fd61343e22882458572d42d262977cc347cb216fd240ecc1254baae950a WatchSource:0}: Error finding container 6bbf1fd61343e22882458572d42d262977cc347cb216fd240ecc1254baae950a: Status 404 returned error can't find the container with id 6bbf1fd61343e22882458572d42d262977cc347cb216fd240ecc1254baae950a
Sep 16 13:05:17.599891 ip-10-0-133-218 kubenswrapper[2580]: I0916 13:05:17.599866    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="691ee328-ef29-46a2-a634-30213fa7acee" path="/var/lib/kubelet/pods/691ee328-ef29-46a2-a634-30213fa7acee/volumes"
Sep 16 13:05:17.939752 ip-10-0-133-218 kubenswrapper[2580]: I0916 13:05:17.939716    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-init-fail-fb44b2-predictor-64566c8d6d-kkzjx" event={"ID":"f07fc368-24b1-40bf-b097-3cdd9137ec2f","Type":"ContainerStarted","Data":"e9584b74bb4b30d4deadee5d845ce8a4f3b16cdf8f3c6b1f336be0e20d73bb56"}
Sep 16 13:05:17.939920 ip-10-0-133-218 kubenswrapper[2580]: I0916 13:05:17.939772    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-init-fail-fb44b2-predictor-64566c8d6d-kkzjx" event={"ID":"f07fc368-24b1-40bf-b097-3cdd9137ec2f","Type":"ContainerStarted","Data":"6bbf1fd61343e22882458572d42d262977cc347cb216fd240ecc1254baae950a"}
Sep 16 13:05:22.959545 ip-10-0-133-218 kubenswrapper[2580]: I0916 13:05:22.959515    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-init-fail-fb44b2-predictor-64566c8d6d-kkzjx_f07fc368-24b1-40bf-b097-3cdd9137ec2f/storage-initializer/0.log"
Sep 16 13:05:22.959922 ip-10-0-133-218 kubenswrapper[2580]: I0916 13:05:22.959555    2580 generic.go:356] "Generic (PLEG): container finished" podID="f07fc368-24b1-40bf-b097-3cdd9137ec2f" containerID="e9584b74bb4b30d4deadee5d845ce8a4f3b16cdf8f3c6b1f336be0e20d73bb56" exitCode=1
Sep 16 13:05:22.959922 ip-10-0-133-218 kubenswrapper[2580]: I0916 13:05:22.959613    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-init-fail-fb44b2-predictor-64566c8d6d-kkzjx" event={"ID":"f07fc368-24b1-40bf-b097-3cdd9137ec2f","Type":"ContainerDied","Data":"e9584b74bb4b30d4deadee5d845ce8a4f3b16cdf8f3c6b1f336be0e20d73bb56"}
Sep 16 13:05:23.965210 ip-10-0-133-218 kubenswrapper[2580]: I0916 13:05:23.965176    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-init-fail-fb44b2-predictor-64566c8d6d-kkzjx_f07fc368-24b1-40bf-b097-3cdd9137ec2f/storage-initializer/0.log"
Sep 16 13:05:23.965679 ip-10-0-133-218 kubenswrapper[2580]: I0916 13:05:23.965295    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-init-fail-fb44b2-predictor-64566c8d6d-kkzjx" event={"ID":"f07fc368-24b1-40bf-b097-3cdd9137ec2f","Type":"ContainerStarted","Data":"ce2a2f62e40ffac2494db9938a9d23a6e8abd40d314153866cccd32143a16730"}
Sep 16 13:05:26.892422 ip-10-0-133-218 kubenswrapper[2580]: I0916 13:05:26.892390    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-init-fail-fb44b2-predictor-64566c8d6d-kkzjx"]
Sep 16 13:05:26.892907 ip-10-0-133-218 kubenswrapper[2580]: I0916 13:05:26.892656    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-init-fail-fb44b2-predictor-64566c8d6d-kkzjx" podUID="f07fc368-24b1-40bf-b097-3cdd9137ec2f" containerName="storage-initializer" containerID="cri-o://ce2a2f62e40ffac2494db9938a9d23a6e8abd40d314153866cccd32143a16730" gracePeriod=30
Sep 16 13:05:27.431779 ip-10-0-133-218 kubenswrapper[2580]: I0916 13:05:27.431756    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-init-fail-fb44b2-predictor-64566c8d6d-kkzjx_f07fc368-24b1-40bf-b097-3cdd9137ec2f/storage-initializer/1.log"
Sep 16 13:05:27.432156 ip-10-0-133-218 kubenswrapper[2580]: I0916 13:05:27.432141    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-init-fail-fb44b2-predictor-64566c8d6d-kkzjx_f07fc368-24b1-40bf-b097-3cdd9137ec2f/storage-initializer/0.log"
Sep 16 13:05:27.434455 ip-10-0-133-218 kubenswrapper[2580]: I0916 13:05:27.434440    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-init-fail-fb44b2-predictor-64566c8d6d-kkzjx"
Sep 16 13:05:27.543439 ip-10-0-133-218 kubenswrapper[2580]: I0916 13:05:27.543411    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/f07fc368-24b1-40bf-b097-3cdd9137ec2f-cabundle-cert\") pod \"f07fc368-24b1-40bf-b097-3cdd9137ec2f\" (UID: \"f07fc368-24b1-40bf-b097-3cdd9137ec2f\") "
Sep 16 13:05:27.543626 ip-10-0-133-218 kubenswrapper[2580]: I0916 13:05:27.543477    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f07fc368-24b1-40bf-b097-3cdd9137ec2f-proxy-tls\") pod \"f07fc368-24b1-40bf-b097-3cdd9137ec2f\" (UID: \"f07fc368-24b1-40bf-b097-3cdd9137ec2f\") "
Sep 16 13:05:27.543626 ip-10-0-133-218 kubenswrapper[2580]: I0916 13:05:27.543521    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f07fc368-24b1-40bf-b097-3cdd9137ec2f-kserve-provision-location\") pod \"f07fc368-24b1-40bf-b097-3cdd9137ec2f\" (UID: \"f07fc368-24b1-40bf-b097-3cdd9137ec2f\") "
Sep 16 13:05:27.543626 ip-10-0-133-218 kubenswrapper[2580]: I0916 13:05:27.543559    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/f07fc368-24b1-40bf-b097-3cdd9137ec2f-kube-rbac-proxy-sar-config\") pod \"f07fc368-24b1-40bf-b097-3cdd9137ec2f\" (UID: \"f07fc368-24b1-40bf-b097-3cdd9137ec2f\") "
Sep 16 13:05:27.543626 ip-10-0-133-218 kubenswrapper[2580]: I0916 13:05:27.543578    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-x5q5g\" (UniqueName: \"kubernetes.io/projected/f07fc368-24b1-40bf-b097-3cdd9137ec2f-kube-api-access-x5q5g\") pod \"f07fc368-24b1-40bf-b097-3cdd9137ec2f\" (UID: \"f07fc368-24b1-40bf-b097-3cdd9137ec2f\") "
Sep 16 13:05:27.543861 ip-10-0-133-218 kubenswrapper[2580]: I0916 13:05:27.543840    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f07fc368-24b1-40bf-b097-3cdd9137ec2f-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "f07fc368-24b1-40bf-b097-3cdd9137ec2f" (UID: "f07fc368-24b1-40bf-b097-3cdd9137ec2f"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 13:05:27.543924 ip-10-0-133-218 kubenswrapper[2580]: I0916 13:05:27.543900    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f07fc368-24b1-40bf-b097-3cdd9137ec2f-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "f07fc368-24b1-40bf-b097-3cdd9137ec2f" (UID: "f07fc368-24b1-40bf-b097-3cdd9137ec2f"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 13:05:27.543967 ip-10-0-133-218 kubenswrapper[2580]: I0916 13:05:27.543948    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f07fc368-24b1-40bf-b097-3cdd9137ec2f-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "f07fc368-24b1-40bf-b097-3cdd9137ec2f" (UID: "f07fc368-24b1-40bf-b097-3cdd9137ec2f"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 13:05:27.545616 ip-10-0-133-218 kubenswrapper[2580]: I0916 13:05:27.545588    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f07fc368-24b1-40bf-b097-3cdd9137ec2f-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "f07fc368-24b1-40bf-b097-3cdd9137ec2f" (UID: "f07fc368-24b1-40bf-b097-3cdd9137ec2f"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 13:05:27.545715 ip-10-0-133-218 kubenswrapper[2580]: I0916 13:05:27.545622    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f07fc368-24b1-40bf-b097-3cdd9137ec2f-kube-api-access-x5q5g" (OuterVolumeSpecName: "kube-api-access-x5q5g") pod "f07fc368-24b1-40bf-b097-3cdd9137ec2f" (UID: "f07fc368-24b1-40bf-b097-3cdd9137ec2f"). InnerVolumeSpecName "kube-api-access-x5q5g". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 13:05:27.644805 ip-10-0-133-218 kubenswrapper[2580]: I0916 13:05:27.644773    2580 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/f07fc368-24b1-40bf-b097-3cdd9137ec2f-cabundle-cert\") on node \"ip-10-0-133-218.ec2.internal\" DevicePath \"\""
Sep 16 13:05:27.644805 ip-10-0-133-218 kubenswrapper[2580]: I0916 13:05:27.644802    2580 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f07fc368-24b1-40bf-b097-3cdd9137ec2f-proxy-tls\") on node \"ip-10-0-133-218.ec2.internal\" DevicePath \"\""
Sep 16 13:05:27.644984 ip-10-0-133-218 kubenswrapper[2580]: I0916 13:05:27.644813    2580 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f07fc368-24b1-40bf-b097-3cdd9137ec2f-kserve-provision-location\") on node \"ip-10-0-133-218.ec2.internal\" DevicePath \"\""
Sep 16 13:05:27.644984 ip-10-0-133-218 kubenswrapper[2580]: I0916 13:05:27.644823    2580 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/f07fc368-24b1-40bf-b097-3cdd9137ec2f-kube-rbac-proxy-sar-config\") on node \"ip-10-0-133-218.ec2.internal\" DevicePath \"\""
Sep 16 13:05:27.644984 ip-10-0-133-218 kubenswrapper[2580]: I0916 13:05:27.644831    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x5q5g\" (UniqueName: \"kubernetes.io/projected/f07fc368-24b1-40bf-b097-3cdd9137ec2f-kube-api-access-x5q5g\") on node \"ip-10-0-133-218.ec2.internal\" DevicePath \"\""
Sep 16 13:05:27.982039 ip-10-0-133-218 kubenswrapper[2580]: I0916 13:05:27.981965    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-init-fail-fb44b2-predictor-64566c8d6d-kkzjx_f07fc368-24b1-40bf-b097-3cdd9137ec2f/storage-initializer/1.log"
Sep 16 13:05:27.982423 ip-10-0-133-218 kubenswrapper[2580]: I0916 13:05:27.982354    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-init-fail-fb44b2-predictor-64566c8d6d-kkzjx_f07fc368-24b1-40bf-b097-3cdd9137ec2f/storage-initializer/0.log"
Sep 16 13:05:27.982423 ip-10-0-133-218 kubenswrapper[2580]: I0916 13:05:27.982388    2580 generic.go:356] "Generic (PLEG): container finished" podID="f07fc368-24b1-40bf-b097-3cdd9137ec2f" containerID="ce2a2f62e40ffac2494db9938a9d23a6e8abd40d314153866cccd32143a16730" exitCode=1
Sep 16 13:05:27.982524 ip-10-0-133-218 kubenswrapper[2580]: I0916 13:05:27.982424    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-init-fail-fb44b2-predictor-64566c8d6d-kkzjx" event={"ID":"f07fc368-24b1-40bf-b097-3cdd9137ec2f","Type":"ContainerDied","Data":"ce2a2f62e40ffac2494db9938a9d23a6e8abd40d314153866cccd32143a16730"}
Sep 16 13:05:27.982524 ip-10-0-133-218 kubenswrapper[2580]: I0916 13:05:27.982445    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-init-fail-fb44b2-predictor-64566c8d6d-kkzjx"
Sep 16 13:05:27.982524 ip-10-0-133-218 kubenswrapper[2580]: I0916 13:05:27.982468    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-init-fail-fb44b2-predictor-64566c8d6d-kkzjx" event={"ID":"f07fc368-24b1-40bf-b097-3cdd9137ec2f","Type":"ContainerDied","Data":"6bbf1fd61343e22882458572d42d262977cc347cb216fd240ecc1254baae950a"}
Sep 16 13:05:27.982628 ip-10-0-133-218 kubenswrapper[2580]: I0916 13:05:27.982522    2580 scope.go:117] "RemoveContainer" containerID="ce2a2f62e40ffac2494db9938a9d23a6e8abd40d314153866cccd32143a16730"
Sep 16 13:05:27.991080 ip-10-0-133-218 kubenswrapper[2580]: I0916 13:05:27.991058    2580 scope.go:117] "RemoveContainer" containerID="e9584b74bb4b30d4deadee5d845ce8a4f3b16cdf8f3c6b1f336be0e20d73bb56"
Sep 16 13:05:27.997902 ip-10-0-133-218 kubenswrapper[2580]: I0916 13:05:27.997884    2580 scope.go:117] "RemoveContainer" containerID="ce2a2f62e40ffac2494db9938a9d23a6e8abd40d314153866cccd32143a16730"
Sep 16 13:05:27.998164 ip-10-0-133-218 kubenswrapper[2580]: E0916 13:05:27.998144    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ce2a2f62e40ffac2494db9938a9d23a6e8abd40d314153866cccd32143a16730\": container with ID starting with ce2a2f62e40ffac2494db9938a9d23a6e8abd40d314153866cccd32143a16730 not found: ID does not exist" containerID="ce2a2f62e40ffac2494db9938a9d23a6e8abd40d314153866cccd32143a16730"
Sep 16 13:05:27.998237 ip-10-0-133-218 kubenswrapper[2580]: I0916 13:05:27.998175    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ce2a2f62e40ffac2494db9938a9d23a6e8abd40d314153866cccd32143a16730"} err="failed to get container status \"ce2a2f62e40ffac2494db9938a9d23a6e8abd40d314153866cccd32143a16730\": rpc error: code = NotFound desc = could not find container \"ce2a2f62e40ffac2494db9938a9d23a6e8abd40d314153866cccd32143a16730\": container with ID starting with ce2a2f62e40ffac2494db9938a9d23a6e8abd40d314153866cccd32143a16730 not found: ID does not exist"
Sep 16 13:05:27.998237 ip-10-0-133-218 kubenswrapper[2580]: I0916 13:05:27.998198    2580 scope.go:117] "RemoveContainer" containerID="e9584b74bb4b30d4deadee5d845ce8a4f3b16cdf8f3c6b1f336be0e20d73bb56"
Sep 16 13:05:27.998435 ip-10-0-133-218 kubenswrapper[2580]: E0916 13:05:27.998417    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e9584b74bb4b30d4deadee5d845ce8a4f3b16cdf8f3c6b1f336be0e20d73bb56\": container with ID starting with e9584b74bb4b30d4deadee5d845ce8a4f3b16cdf8f3c6b1f336be0e20d73bb56 not found: ID does not exist" containerID="e9584b74bb4b30d4deadee5d845ce8a4f3b16cdf8f3c6b1f336be0e20d73bb56"
Sep 16 13:05:27.998509 ip-10-0-133-218 kubenswrapper[2580]: I0916 13:05:27.998442    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e9584b74bb4b30d4deadee5d845ce8a4f3b16cdf8f3c6b1f336be0e20d73bb56"} err="failed to get container status \"e9584b74bb4b30d4deadee5d845ce8a4f3b16cdf8f3c6b1f336be0e20d73bb56\": rpc error: code = NotFound desc = could not find container \"e9584b74bb4b30d4deadee5d845ce8a4f3b16cdf8f3c6b1f336be0e20d73bb56\": container with ID starting with e9584b74bb4b30d4deadee5d845ce8a4f3b16cdf8f3c6b1f336be0e20d73bb56 not found: ID does not exist"
Sep 16 13:05:28.012108 ip-10-0-133-218 kubenswrapper[2580]: I0916 13:05:28.012087    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-init-fail-fb44b2-predictor-64566c8d6d-kkzjx"]
Sep 16 13:05:28.014828 ip-10-0-133-218 kubenswrapper[2580]: I0916 13:05:28.014808    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-init-fail-fb44b2-predictor-64566c8d6d-kkzjx"]
Sep 16 13:05:29.599220 ip-10-0-133-218 kubenswrapper[2580]: I0916 13:05:29.599187    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f07fc368-24b1-40bf-b097-3cdd9137ec2f" path="/var/lib/kubelet/pods/f07fc368-24b1-40bf-b097-3cdd9137ec2f/volumes"
Sep 16 13:07:47.562580 ip-10-0-133-218 kubenswrapper[2580]: I0916 13:07:47.562543    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-v2-predictor-7768896644-sm48p"]
Sep 16 13:07:47.563166 ip-10-0-133-218 kubenswrapper[2580]: I0916 13:07:47.562985    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f07fc368-24b1-40bf-b097-3cdd9137ec2f" containerName="storage-initializer"
Sep 16 13:07:47.563166 ip-10-0-133-218 kubenswrapper[2580]: I0916 13:07:47.563003    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="f07fc368-24b1-40bf-b097-3cdd9137ec2f" containerName="storage-initializer"
Sep 16 13:07:47.563166 ip-10-0-133-218 kubenswrapper[2580]: I0916 13:07:47.563034    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f07fc368-24b1-40bf-b097-3cdd9137ec2f" containerName="storage-initializer"
Sep 16 13:07:47.563166 ip-10-0-133-218 kubenswrapper[2580]: I0916 13:07:47.563043    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="f07fc368-24b1-40bf-b097-3cdd9137ec2f" containerName="storage-initializer"
Sep 16 13:07:47.563166 ip-10-0-133-218 kubenswrapper[2580]: I0916 13:07:47.563121    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="f07fc368-24b1-40bf-b097-3cdd9137ec2f" containerName="storage-initializer"
Sep 16 13:07:47.563446 ip-10-0-133-218 kubenswrapper[2580]: I0916 13:07:47.563274    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="f07fc368-24b1-40bf-b097-3cdd9137ec2f" containerName="storage-initializer"
Sep 16 13:07:47.566303 ip-10-0-133-218 kubenswrapper[2580]: I0916 13:07:47.566282    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-7768896644-sm48p"
Sep 16 13:07:47.568647 ip-10-0-133-218 kubenswrapper[2580]: I0916 13:07:47.568626    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Sep 16 13:07:47.568913 ip-10-0-133-218 kubenswrapper[2580]: I0916 13:07:47.568868    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-create-kube-rbac-proxy-sar-config\""
Sep 16 13:07:47.569687 ip-10-0-133-218 kubenswrapper[2580]: I0916 13:07:47.569664    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Sep 16 13:07:47.569801 ip-10-0-133-218 kubenswrapper[2580]: I0916 13:07:47.569663    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-create-v2-predictor-serving-cert\""
Sep 16 13:07:47.569801 ip-10-0-133-218 kubenswrapper[2580]: I0916 13:07:47.569697    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-9sk5b\""
Sep 16 13:07:47.569801 ip-10-0-133-218 kubenswrapper[2580]: I0916 13:07:47.569666    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Sep 16 13:07:47.569801 ip-10-0-133-218 kubenswrapper[2580]: I0916 13:07:47.569730    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Sep 16 13:07:47.576317 ip-10-0-133-218 kubenswrapper[2580]: I0916 13:07:47.576294    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-v2-predictor-7768896644-sm48p"]
Sep 16 13:07:47.599876 ip-10-0-133-218 kubenswrapper[2580]: I0916 13:07:47.599856    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-predictor-8cb968449-w8qp8"]
Sep 16 13:07:47.603133 ip-10-0-133-218 kubenswrapper[2580]: I0916 13:07:47.603115    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-8cb968449-w8qp8"
Sep 16 13:07:47.605021 ip-10-0-133-218 kubenswrapper[2580]: I0916 13:07:47.605000    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-create-predictor-serving-cert\""
Sep 16 13:07:47.610389 ip-10-0-133-218 kubenswrapper[2580]: I0916 13:07:47.610369    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-predictor-8cb968449-w8qp8"]
Sep 16 13:07:47.638965 ip-10-0-133-218 kubenswrapper[2580]: I0916 13:07:47.638937    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/21450712-a128-4e3d-bf9b-997dcfd74263-proxy-tls\") pod \"isvc-canary-create-v2-predictor-7768896644-sm48p\" (UID: \"21450712-a128-4e3d-bf9b-997dcfd74263\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-7768896644-sm48p"
Sep 16 13:07:47.639099 ip-10-0-133-218 kubenswrapper[2580]: I0916 13:07:47.638985    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r5zjk\" (UniqueName: \"kubernetes.io/projected/21450712-a128-4e3d-bf9b-997dcfd74263-kube-api-access-r5zjk\") pod \"isvc-canary-create-v2-predictor-7768896644-sm48p\" (UID: \"21450712-a128-4e3d-bf9b-997dcfd74263\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-7768896644-sm48p"
Sep 16 13:07:47.639099 ip-10-0-133-218 kubenswrapper[2580]: I0916 13:07:47.639040    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/21450712-a128-4e3d-bf9b-997dcfd74263-kube-rbac-proxy-sar-config\") pod \"isvc-canary-create-v2-predictor-7768896644-sm48p\" (UID: \"21450712-a128-4e3d-bf9b-997dcfd74263\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-7768896644-sm48p"
Sep 16 13:07:47.639099 ip-10-0-133-218 kubenswrapper[2580]: I0916 13:07:47.639057    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/21450712-a128-4e3d-bf9b-997dcfd74263-kserve-provision-location\") pod \"isvc-canary-create-v2-predictor-7768896644-sm48p\" (UID: \"21450712-a128-4e3d-bf9b-997dcfd74263\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-7768896644-sm48p"
Sep 16 13:07:47.639218 ip-10-0-133-218 kubenswrapper[2580]: I0916 13:07:47.639113    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/21450712-a128-4e3d-bf9b-997dcfd74263-cabundle-cert\") pod \"isvc-canary-create-v2-predictor-7768896644-sm48p\" (UID: \"21450712-a128-4e3d-bf9b-997dcfd74263\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-7768896644-sm48p"
Sep 16 13:07:47.740373 ip-10-0-133-218 kubenswrapper[2580]: I0916 13:07:47.740342    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/62dc1c6d-fc62-4468-9a5f-3a6f0c86a652-proxy-tls\") pod \"isvc-canary-create-predictor-8cb968449-w8qp8\" (UID: \"62dc1c6d-fc62-4468-9a5f-3a6f0c86a652\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-8cb968449-w8qp8"
Sep 16 13:07:47.740568 ip-10-0-133-218 kubenswrapper[2580]: I0916 13:07:47.740382    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/21450712-a128-4e3d-bf9b-997dcfd74263-proxy-tls\") pod \"isvc-canary-create-v2-predictor-7768896644-sm48p\" (UID: \"21450712-a128-4e3d-bf9b-997dcfd74263\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-7768896644-sm48p"
Sep 16 13:07:47.740568 ip-10-0-133-218 kubenswrapper[2580]: I0916 13:07:47.740420    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/62dc1c6d-fc62-4468-9a5f-3a6f0c86a652-cabundle-cert\") pod \"isvc-canary-create-predictor-8cb968449-w8qp8\" (UID: \"62dc1c6d-fc62-4468-9a5f-3a6f0c86a652\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-8cb968449-w8qp8"
Sep 16 13:07:47.740568 ip-10-0-133-218 kubenswrapper[2580]: I0916 13:07:47.740470    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r5zjk\" (UniqueName: \"kubernetes.io/projected/21450712-a128-4e3d-bf9b-997dcfd74263-kube-api-access-r5zjk\") pod \"isvc-canary-create-v2-predictor-7768896644-sm48p\" (UID: \"21450712-a128-4e3d-bf9b-997dcfd74263\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-7768896644-sm48p"
Sep 16 13:07:47.740568 ip-10-0-133-218 kubenswrapper[2580]: I0916 13:07:47.740546    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j7qvn\" (UniqueName: \"kubernetes.io/projected/62dc1c6d-fc62-4468-9a5f-3a6f0c86a652-kube-api-access-j7qvn\") pod \"isvc-canary-create-predictor-8cb968449-w8qp8\" (UID: \"62dc1c6d-fc62-4468-9a5f-3a6f0c86a652\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-8cb968449-w8qp8"
Sep 16 13:07:47.740736 ip-10-0-133-218 kubenswrapper[2580]: I0916 13:07:47.740597    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/62dc1c6d-fc62-4468-9a5f-3a6f0c86a652-kube-rbac-proxy-sar-config\") pod \"isvc-canary-create-predictor-8cb968449-w8qp8\" (UID: \"62dc1c6d-fc62-4468-9a5f-3a6f0c86a652\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-8cb968449-w8qp8"
Sep 16 13:07:47.740736 ip-10-0-133-218 kubenswrapper[2580]: I0916 13:07:47.740640    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/62dc1c6d-fc62-4468-9a5f-3a6f0c86a652-kserve-provision-location\") pod \"isvc-canary-create-predictor-8cb968449-w8qp8\" (UID: \"62dc1c6d-fc62-4468-9a5f-3a6f0c86a652\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-8cb968449-w8qp8"
Sep 16 13:07:47.740736 ip-10-0-133-218 kubenswrapper[2580]: I0916 13:07:47.740663    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/21450712-a128-4e3d-bf9b-997dcfd74263-kube-rbac-proxy-sar-config\") pod \"isvc-canary-create-v2-predictor-7768896644-sm48p\" (UID: \"21450712-a128-4e3d-bf9b-997dcfd74263\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-7768896644-sm48p"
Sep 16 13:07:47.740736 ip-10-0-133-218 kubenswrapper[2580]: I0916 13:07:47.740681    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/21450712-a128-4e3d-bf9b-997dcfd74263-kserve-provision-location\") pod \"isvc-canary-create-v2-predictor-7768896644-sm48p\" (UID: \"21450712-a128-4e3d-bf9b-997dcfd74263\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-7768896644-sm48p"
Sep 16 13:07:47.740736 ip-10-0-133-218 kubenswrapper[2580]: I0916 13:07:47.740732    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/21450712-a128-4e3d-bf9b-997dcfd74263-cabundle-cert\") pod \"isvc-canary-create-v2-predictor-7768896644-sm48p\" (UID: \"21450712-a128-4e3d-bf9b-997dcfd74263\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-7768896644-sm48p"
Sep 16 13:07:47.740997 ip-10-0-133-218 kubenswrapper[2580]: I0916 13:07:47.740977    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/21450712-a128-4e3d-bf9b-997dcfd74263-kserve-provision-location\") pod \"isvc-canary-create-v2-predictor-7768896644-sm48p\" (UID: \"21450712-a128-4e3d-bf9b-997dcfd74263\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-7768896644-sm48p"
Sep 16 13:07:47.741382 ip-10-0-133-218 kubenswrapper[2580]: I0916 13:07:47.741362    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/21450712-a128-4e3d-bf9b-997dcfd74263-cabundle-cert\") pod \"isvc-canary-create-v2-predictor-7768896644-sm48p\" (UID: \"21450712-a128-4e3d-bf9b-997dcfd74263\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-7768896644-sm48p"
Sep 16 13:07:47.741421 ip-10-0-133-218 kubenswrapper[2580]: I0916 13:07:47.741362    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/21450712-a128-4e3d-bf9b-997dcfd74263-kube-rbac-proxy-sar-config\") pod \"isvc-canary-create-v2-predictor-7768896644-sm48p\" (UID: \"21450712-a128-4e3d-bf9b-997dcfd74263\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-7768896644-sm48p"
Sep 16 13:07:47.742708 ip-10-0-133-218 kubenswrapper[2580]: I0916 13:07:47.742692    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/21450712-a128-4e3d-bf9b-997dcfd74263-proxy-tls\") pod \"isvc-canary-create-v2-predictor-7768896644-sm48p\" (UID: \"21450712-a128-4e3d-bf9b-997dcfd74263\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-7768896644-sm48p"
Sep 16 13:07:47.747462 ip-10-0-133-218 kubenswrapper[2580]: I0916 13:07:47.747440    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-r5zjk\" (UniqueName: \"kubernetes.io/projected/21450712-a128-4e3d-bf9b-997dcfd74263-kube-api-access-r5zjk\") pod \"isvc-canary-create-v2-predictor-7768896644-sm48p\" (UID: \"21450712-a128-4e3d-bf9b-997dcfd74263\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-7768896644-sm48p"
Sep 16 13:07:47.841794 ip-10-0-133-218 kubenswrapper[2580]: I0916 13:07:47.841695    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/62dc1c6d-fc62-4468-9a5f-3a6f0c86a652-kube-rbac-proxy-sar-config\") pod \"isvc-canary-create-predictor-8cb968449-w8qp8\" (UID: \"62dc1c6d-fc62-4468-9a5f-3a6f0c86a652\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-8cb968449-w8qp8"
Sep 16 13:07:47.841794 ip-10-0-133-218 kubenswrapper[2580]: I0916 13:07:47.841739    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/62dc1c6d-fc62-4468-9a5f-3a6f0c86a652-kserve-provision-location\") pod \"isvc-canary-create-predictor-8cb968449-w8qp8\" (UID: \"62dc1c6d-fc62-4468-9a5f-3a6f0c86a652\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-8cb968449-w8qp8"
Sep 16 13:07:47.841794 ip-10-0-133-218 kubenswrapper[2580]: I0916 13:07:47.841779    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/62dc1c6d-fc62-4468-9a5f-3a6f0c86a652-proxy-tls\") pod \"isvc-canary-create-predictor-8cb968449-w8qp8\" (UID: \"62dc1c6d-fc62-4468-9a5f-3a6f0c86a652\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-8cb968449-w8qp8"
Sep 16 13:07:47.842093 ip-10-0-133-218 kubenswrapper[2580]: I0916 13:07:47.841821    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/62dc1c6d-fc62-4468-9a5f-3a6f0c86a652-cabundle-cert\") pod \"isvc-canary-create-predictor-8cb968449-w8qp8\" (UID: \"62dc1c6d-fc62-4468-9a5f-3a6f0c86a652\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-8cb968449-w8qp8"
Sep 16 13:07:47.842093 ip-10-0-133-218 kubenswrapper[2580]: I0916 13:07:47.841869    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j7qvn\" (UniqueName: \"kubernetes.io/projected/62dc1c6d-fc62-4468-9a5f-3a6f0c86a652-kube-api-access-j7qvn\") pod \"isvc-canary-create-predictor-8cb968449-w8qp8\" (UID: \"62dc1c6d-fc62-4468-9a5f-3a6f0c86a652\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-8cb968449-w8qp8"
Sep 16 13:07:47.842093 ip-10-0-133-218 kubenswrapper[2580]: E0916 13:07:47.841963    2580 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-canary-create-predictor-serving-cert: secret "isvc-canary-create-predictor-serving-cert" not found
Sep 16 13:07:47.842093 ip-10-0-133-218 kubenswrapper[2580]: E0916 13:07:47.842051    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/62dc1c6d-fc62-4468-9a5f-3a6f0c86a652-proxy-tls podName:62dc1c6d-fc62-4468-9a5f-3a6f0c86a652 nodeName:}" failed. No retries permitted until 2026-09-16 13:07:48.342028691 +0000 UTC m=+1207.264901139 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/62dc1c6d-fc62-4468-9a5f-3a6f0c86a652-proxy-tls") pod "isvc-canary-create-predictor-8cb968449-w8qp8" (UID: "62dc1c6d-fc62-4468-9a5f-3a6f0c86a652") : secret "isvc-canary-create-predictor-serving-cert" not found
Sep 16 13:07:47.842309 ip-10-0-133-218 kubenswrapper[2580]: I0916 13:07:47.842245    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/62dc1c6d-fc62-4468-9a5f-3a6f0c86a652-kserve-provision-location\") pod \"isvc-canary-create-predictor-8cb968449-w8qp8\" (UID: \"62dc1c6d-fc62-4468-9a5f-3a6f0c86a652\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-8cb968449-w8qp8"
Sep 16 13:07:47.842464 ip-10-0-133-218 kubenswrapper[2580]: I0916 13:07:47.842445    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/62dc1c6d-fc62-4468-9a5f-3a6f0c86a652-cabundle-cert\") pod \"isvc-canary-create-predictor-8cb968449-w8qp8\" (UID: \"62dc1c6d-fc62-4468-9a5f-3a6f0c86a652\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-8cb968449-w8qp8"
Sep 16 13:07:47.842569 ip-10-0-133-218 kubenswrapper[2580]: I0916 13:07:47.842473    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/62dc1c6d-fc62-4468-9a5f-3a6f0c86a652-kube-rbac-proxy-sar-config\") pod \"isvc-canary-create-predictor-8cb968449-w8qp8\" (UID: \"62dc1c6d-fc62-4468-9a5f-3a6f0c86a652\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-8cb968449-w8qp8"
Sep 16 13:07:47.849705 ip-10-0-133-218 kubenswrapper[2580]: I0916 13:07:47.849684    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-j7qvn\" (UniqueName: \"kubernetes.io/projected/62dc1c6d-fc62-4468-9a5f-3a6f0c86a652-kube-api-access-j7qvn\") pod \"isvc-canary-create-predictor-8cb968449-w8qp8\" (UID: \"62dc1c6d-fc62-4468-9a5f-3a6f0c86a652\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-8cb968449-w8qp8"
Sep 16 13:07:47.877265 ip-10-0-133-218 kubenswrapper[2580]: I0916 13:07:47.877227    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-7768896644-sm48p"
Sep 16 13:07:48.148319 ip-10-0-133-218 kubenswrapper[2580]: I0916 13:07:48.148284    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-v2-predictor-7768896644-sm48p"]
Sep 16 13:07:48.150644 ip-10-0-133-218 kubenswrapper[2580]: W0916 13:07:48.150620    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod21450712_a128_4e3d_bf9b_997dcfd74263.slice/crio-068717b3db85e6b69d545ba167aa12e54ae1b43d6ae7014b2dbbfa0394b51bf5 WatchSource:0}: Error finding container 068717b3db85e6b69d545ba167aa12e54ae1b43d6ae7014b2dbbfa0394b51bf5: Status 404 returned error can't find the container with id 068717b3db85e6b69d545ba167aa12e54ae1b43d6ae7014b2dbbfa0394b51bf5
Sep 16 13:07:48.346779 ip-10-0-133-218 kubenswrapper[2580]: I0916 13:07:48.346730    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/62dc1c6d-fc62-4468-9a5f-3a6f0c86a652-proxy-tls\") pod \"isvc-canary-create-predictor-8cb968449-w8qp8\" (UID: \"62dc1c6d-fc62-4468-9a5f-3a6f0c86a652\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-8cb968449-w8qp8"
Sep 16 13:07:48.349034 ip-10-0-133-218 kubenswrapper[2580]: I0916 13:07:48.349012    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/62dc1c6d-fc62-4468-9a5f-3a6f0c86a652-proxy-tls\") pod \"isvc-canary-create-predictor-8cb968449-w8qp8\" (UID: \"62dc1c6d-fc62-4468-9a5f-3a6f0c86a652\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-8cb968449-w8qp8"
Sep 16 13:07:48.512450 ip-10-0-133-218 kubenswrapper[2580]: I0916 13:07:48.512368    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-8cb968449-w8qp8"
Sep 16 13:07:48.520422 ip-10-0-133-218 kubenswrapper[2580]: I0916 13:07:48.520384    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-7768896644-sm48p" event={"ID":"21450712-a128-4e3d-bf9b-997dcfd74263","Type":"ContainerStarted","Data":"21b327d65b50291ac52b63f3664942b1f4b58ee8aa6f207568aeb0abd3da53a0"}
Sep 16 13:07:48.520568 ip-10-0-133-218 kubenswrapper[2580]: I0916 13:07:48.520507    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-7768896644-sm48p" event={"ID":"21450712-a128-4e3d-bf9b-997dcfd74263","Type":"ContainerStarted","Data":"068717b3db85e6b69d545ba167aa12e54ae1b43d6ae7014b2dbbfa0394b51bf5"}
Sep 16 13:07:48.785603 ip-10-0-133-218 kubenswrapper[2580]: I0916 13:07:48.785581    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-predictor-8cb968449-w8qp8"]
Sep 16 13:07:48.788762 ip-10-0-133-218 kubenswrapper[2580]: W0916 13:07:48.788737    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod62dc1c6d_fc62_4468_9a5f_3a6f0c86a652.slice/crio-7e371eb728da4877f264dc34926a10ab0c2be1f164be835a42b7373d1a6dae3d WatchSource:0}: Error finding container 7e371eb728da4877f264dc34926a10ab0c2be1f164be835a42b7373d1a6dae3d: Status 404 returned error can't find the container with id 7e371eb728da4877f264dc34926a10ab0c2be1f164be835a42b7373d1a6dae3d
Sep 16 13:07:49.525240 ip-10-0-133-218 kubenswrapper[2580]: I0916 13:07:49.525206    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-8cb968449-w8qp8" event={"ID":"62dc1c6d-fc62-4468-9a5f-3a6f0c86a652","Type":"ContainerStarted","Data":"d74df3d806605baa8a03075eb5b41064c346a77a2c4286c13aa5fcf504e02c6b"}
Sep 16 13:07:49.525410 ip-10-0-133-218 kubenswrapper[2580]: I0916 13:07:49.525258    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-8cb968449-w8qp8" event={"ID":"62dc1c6d-fc62-4468-9a5f-3a6f0c86a652","Type":"ContainerStarted","Data":"7e371eb728da4877f264dc34926a10ab0c2be1f164be835a42b7373d1a6dae3d"}
Sep 16 13:07:52.537322 ip-10-0-133-218 kubenswrapper[2580]: I0916 13:07:52.537284    2580 generic.go:356] "Generic (PLEG): container finished" podID="21450712-a128-4e3d-bf9b-997dcfd74263" containerID="21b327d65b50291ac52b63f3664942b1f4b58ee8aa6f207568aeb0abd3da53a0" exitCode=0
Sep 16 13:07:52.537831 ip-10-0-133-218 kubenswrapper[2580]: I0916 13:07:52.537363    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-7768896644-sm48p" event={"ID":"21450712-a128-4e3d-bf9b-997dcfd74263","Type":"ContainerDied","Data":"21b327d65b50291ac52b63f3664942b1f4b58ee8aa6f207568aeb0abd3da53a0"}
Sep 16 13:07:53.543068 ip-10-0-133-218 kubenswrapper[2580]: I0916 13:07:53.543034    2580 generic.go:356] "Generic (PLEG): container finished" podID="62dc1c6d-fc62-4468-9a5f-3a6f0c86a652" containerID="d74df3d806605baa8a03075eb5b41064c346a77a2c4286c13aa5fcf504e02c6b" exitCode=0
Sep 16 13:07:53.543549 ip-10-0-133-218 kubenswrapper[2580]: I0916 13:07:53.543110    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-8cb968449-w8qp8" event={"ID":"62dc1c6d-fc62-4468-9a5f-3a6f0c86a652","Type":"ContainerDied","Data":"d74df3d806605baa8a03075eb5b41064c346a77a2c4286c13aa5fcf504e02c6b"}
Sep 16 13:07:53.545544 ip-10-0-133-218 kubenswrapper[2580]: I0916 13:07:53.545517    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-7768896644-sm48p" event={"ID":"21450712-a128-4e3d-bf9b-997dcfd74263","Type":"ContainerStarted","Data":"d601ffa8d2c905b8ac812cac28d3b629eabd15d4730503cf9f97f7a83e585440"}
Sep 16 13:07:53.545667 ip-10-0-133-218 kubenswrapper[2580]: I0916 13:07:53.545571    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-7768896644-sm48p" event={"ID":"21450712-a128-4e3d-bf9b-997dcfd74263","Type":"ContainerStarted","Data":"44b96e621b0c76fc2f660ddc0fa0fd1bb77d0a0886f5b65db416991eda9279c7"}
Sep 16 13:07:53.545801 ip-10-0-133-218 kubenswrapper[2580]: I0916 13:07:53.545778    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-7768896644-sm48p"
Sep 16 13:07:53.577907 ip-10-0-133-218 kubenswrapper[2580]: I0916 13:07:53.577865    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-7768896644-sm48p" podStartSLOduration=2.797290464 podStartE2EDuration="6.577850053s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.780559589s" isStatefulPod=false podCreationTimestamp="2026-09-16 13:07:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 13:07:53.57596588 +0000 UTC m=+1212.498838372" watchObservedRunningTime="2026-09-16 13:07:53.577850053 +0000 UTC m=+1212.500722599"
Sep 16 13:07:54.551874 ip-10-0-133-218 kubenswrapper[2580]: I0916 13:07:54.551832    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-8cb968449-w8qp8" event={"ID":"62dc1c6d-fc62-4468-9a5f-3a6f0c86a652","Type":"ContainerStarted","Data":"092ebe3fb3257d1a56e1574b3e8ce5e5d2ab193d4b4be07be023e94d037c3830"}
Sep 16 13:07:54.552295 ip-10-0-133-218 kubenswrapper[2580]: I0916 13:07:54.551932    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-8cb968449-w8qp8" event={"ID":"62dc1c6d-fc62-4468-9a5f-3a6f0c86a652","Type":"ContainerStarted","Data":"fd7e57639480c3ef9cb1790c65feefe21ec07c11281f752198f3a8f1fa984e61"}
Sep 16 13:07:54.552295 ip-10-0-133-218 kubenswrapper[2580]: I0916 13:07:54.552169    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-7768896644-sm48p"
Sep 16 13:07:54.552442 ip-10-0-133-218 kubenswrapper[2580]: I0916 13:07:54.552399    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-8cb968449-w8qp8"
Sep 16 13:07:54.553507 ip-10-0-133-218 kubenswrapper[2580]: I0916 13:07:54.553467    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-7768896644-sm48p" podUID="21450712-a128-4e3d-bf9b-997dcfd74263" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.35:8080: connect: connection refused"
Sep 16 13:07:54.571304 ip-10-0-133-218 kubenswrapper[2580]: I0916 13:07:54.571251    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-8cb968449-w8qp8" podStartSLOduration=3.797347375 podStartE2EDuration="7.571233516s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.773886141s" isStatefulPod=false podCreationTimestamp="2026-09-16 13:07:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 13:07:54.570359312 +0000 UTC m=+1213.493231788" watchObservedRunningTime="2026-09-16 13:07:54.571233516 +0000 UTC m=+1213.494105988"
Sep 16 13:07:55.556534 ip-10-0-133-218 kubenswrapper[2580]: I0916 13:07:55.556476    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-7768896644-sm48p" podUID="21450712-a128-4e3d-bf9b-997dcfd74263" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.35:8080: connect: connection refused"
Sep 16 13:07:55.556931 ip-10-0-133-218 kubenswrapper[2580]: I0916 13:07:55.556673    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-8cb968449-w8qp8"
Sep 16 13:07:55.557456 ip-10-0-133-218 kubenswrapper[2580]: I0916 13:07:55.557432    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-8cb968449-w8qp8" podUID="62dc1c6d-fc62-4468-9a5f-3a6f0c86a652" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.36:8080: connect: connection refused"
Sep 16 13:07:56.561218 ip-10-0-133-218 kubenswrapper[2580]: I0916 13:07:56.561175    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-8cb968449-w8qp8" podUID="62dc1c6d-fc62-4468-9a5f-3a6f0c86a652" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.36:8080: connect: connection refused"
Sep 16 13:08:00.561297 ip-10-0-133-218 kubenswrapper[2580]: I0916 13:08:00.561254    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-7768896644-sm48p"
Sep 16 13:08:00.561958 ip-10-0-133-218 kubenswrapper[2580]: I0916 13:08:00.561926    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-7768896644-sm48p" podUID="21450712-a128-4e3d-bf9b-997dcfd74263" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.35:8080: connect: connection refused"
Sep 16 13:08:01.565381 ip-10-0-133-218 kubenswrapper[2580]: I0916 13:08:01.565346    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-8cb968449-w8qp8"
Sep 16 13:08:01.566108 ip-10-0-133-218 kubenswrapper[2580]: I0916 13:08:01.566073    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-8cb968449-w8qp8" podUID="62dc1c6d-fc62-4468-9a5f-3a6f0c86a652" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.36:8080: connect: connection refused"
Sep 16 13:08:10.562857 ip-10-0-133-218 kubenswrapper[2580]: I0916 13:08:10.562809    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-7768896644-sm48p" podUID="21450712-a128-4e3d-bf9b-997dcfd74263" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.35:8080: connect: connection refused"
Sep 16 13:08:11.566184 ip-10-0-133-218 kubenswrapper[2580]: I0916 13:08:11.566134    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-8cb968449-w8qp8" podUID="62dc1c6d-fc62-4468-9a5f-3a6f0c86a652" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.36:8080: connect: connection refused"
Sep 16 13:08:20.562219 ip-10-0-133-218 kubenswrapper[2580]: I0916 13:08:20.562160    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-7768896644-sm48p" podUID="21450712-a128-4e3d-bf9b-997dcfd74263" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.35:8080: connect: connection refused"
Sep 16 13:08:21.565940 ip-10-0-133-218 kubenswrapper[2580]: I0916 13:08:21.565901    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-8cb968449-w8qp8" podUID="62dc1c6d-fc62-4468-9a5f-3a6f0c86a652" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.36:8080: connect: connection refused"
Sep 16 13:08:30.562206 ip-10-0-133-218 kubenswrapper[2580]: I0916 13:08:30.562166    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-7768896644-sm48p" podUID="21450712-a128-4e3d-bf9b-997dcfd74263" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.35:8080: connect: connection refused"
Sep 16 13:08:31.565959 ip-10-0-133-218 kubenswrapper[2580]: I0916 13:08:31.565914    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-8cb968449-w8qp8" podUID="62dc1c6d-fc62-4468-9a5f-3a6f0c86a652" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.36:8080: connect: connection refused"
Sep 16 13:08:40.562706 ip-10-0-133-218 kubenswrapper[2580]: I0916 13:08:40.562608    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-7768896644-sm48p" podUID="21450712-a128-4e3d-bf9b-997dcfd74263" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.35:8080: connect: connection refused"
Sep 16 13:08:41.566964 ip-10-0-133-218 kubenswrapper[2580]: I0916 13:08:41.566913    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-8cb968449-w8qp8" podUID="62dc1c6d-fc62-4468-9a5f-3a6f0c86a652" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.36:8080: connect: connection refused"
Sep 16 13:08:50.562620 ip-10-0-133-218 kubenswrapper[2580]: I0916 13:08:50.562568    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-7768896644-sm48p" podUID="21450712-a128-4e3d-bf9b-997dcfd74263" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.35:8080: connect: connection refused"
Sep 16 13:08:51.565949 ip-10-0-133-218 kubenswrapper[2580]: I0916 13:08:51.565890    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-8cb968449-w8qp8" podUID="62dc1c6d-fc62-4468-9a5f-3a6f0c86a652" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.36:8080: connect: connection refused"
Sep 16 13:09:00.562807 ip-10-0-133-218 kubenswrapper[2580]: I0916 13:09:00.562760    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-7768896644-sm48p" podUID="21450712-a128-4e3d-bf9b-997dcfd74263" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.35:8080: connect: connection refused"
Sep 16 13:09:01.565930 ip-10-0-133-218 kubenswrapper[2580]: I0916 13:09:01.565889    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-8cb968449-w8qp8" podUID="62dc1c6d-fc62-4468-9a5f-3a6f0c86a652" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.36:8080: connect: connection refused"
Sep 16 13:09:10.563108 ip-10-0-133-218 kubenswrapper[2580]: I0916 13:09:10.563075    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-7768896644-sm48p"
Sep 16 13:09:10.682999 ip-10-0-133-218 kubenswrapper[2580]: I0916 13:09:10.682966    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-predictor-8cb968449-w8qp8"]
Sep 16 13:09:10.683220 ip-10-0-133-218 kubenswrapper[2580]: I0916 13:09:10.683198    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-8cb968449-w8qp8" podUID="62dc1c6d-fc62-4468-9a5f-3a6f0c86a652" containerName="kserve-container" containerID="cri-o://fd7e57639480c3ef9cb1790c65feefe21ec07c11281f752198f3a8f1fa984e61" gracePeriod=30
Sep 16 13:09:10.683346 ip-10-0-133-218 kubenswrapper[2580]: I0916 13:09:10.683280    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-8cb968449-w8qp8" podUID="62dc1c6d-fc62-4468-9a5f-3a6f0c86a652" containerName="kube-rbac-proxy" containerID="cri-o://092ebe3fb3257d1a56e1574b3e8ce5e5d2ab193d4b4be07be023e94d037c3830" gracePeriod=30
Sep 16 13:09:10.684098 ip-10-0-133-218 kubenswrapper[2580]: I0916 13:09:10.683641    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-8cb968449-w8qp8"
Sep 16 13:09:10.858405 ip-10-0-133-218 kubenswrapper[2580]: I0916 13:09:10.858334    2580 generic.go:356] "Generic (PLEG): container finished" podID="62dc1c6d-fc62-4468-9a5f-3a6f0c86a652" containerID="092ebe3fb3257d1a56e1574b3e8ce5e5d2ab193d4b4be07be023e94d037c3830" exitCode=2
Sep 16 13:09:10.858405 ip-10-0-133-218 kubenswrapper[2580]: I0916 13:09:10.858393    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-8cb968449-w8qp8" event={"ID":"62dc1c6d-fc62-4468-9a5f-3a6f0c86a652","Type":"ContainerDied","Data":"092ebe3fb3257d1a56e1574b3e8ce5e5d2ab193d4b4be07be023e94d037c3830"}
Sep 16 13:09:11.562086 ip-10-0-133-218 kubenswrapper[2580]: I0916 13:09:11.562039    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-8cb968449-w8qp8" podUID="62dc1c6d-fc62-4468-9a5f-3a6f0c86a652" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.36:8643/healthz\": dial tcp 10.133.0.36:8643: connect: connection refused"
Sep 16 13:09:15.328368 ip-10-0-133-218 kubenswrapper[2580]: I0916 13:09:15.328340    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-8cb968449-w8qp8"
Sep 16 13:09:15.346982 ip-10-0-133-218 kubenswrapper[2580]: I0916 13:09:15.346955    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-j7qvn\" (UniqueName: \"kubernetes.io/projected/62dc1c6d-fc62-4468-9a5f-3a6f0c86a652-kube-api-access-j7qvn\") pod \"62dc1c6d-fc62-4468-9a5f-3a6f0c86a652\" (UID: \"62dc1c6d-fc62-4468-9a5f-3a6f0c86a652\") "
Sep 16 13:09:15.347114 ip-10-0-133-218 kubenswrapper[2580]: I0916 13:09:15.347026    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/62dc1c6d-fc62-4468-9a5f-3a6f0c86a652-proxy-tls\") pod \"62dc1c6d-fc62-4468-9a5f-3a6f0c86a652\" (UID: \"62dc1c6d-fc62-4468-9a5f-3a6f0c86a652\") "
Sep 16 13:09:15.347114 ip-10-0-133-218 kubenswrapper[2580]: I0916 13:09:15.347068    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/62dc1c6d-fc62-4468-9a5f-3a6f0c86a652-cabundle-cert\") pod \"62dc1c6d-fc62-4468-9a5f-3a6f0c86a652\" (UID: \"62dc1c6d-fc62-4468-9a5f-3a6f0c86a652\") "
Sep 16 13:09:15.347227 ip-10-0-133-218 kubenswrapper[2580]: I0916 13:09:15.347140    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/62dc1c6d-fc62-4468-9a5f-3a6f0c86a652-kube-rbac-proxy-sar-config\") pod \"62dc1c6d-fc62-4468-9a5f-3a6f0c86a652\" (UID: \"62dc1c6d-fc62-4468-9a5f-3a6f0c86a652\") "
Sep 16 13:09:15.347478 ip-10-0-133-218 kubenswrapper[2580]: I0916 13:09:15.347449    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/62dc1c6d-fc62-4468-9a5f-3a6f0c86a652-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "62dc1c6d-fc62-4468-9a5f-3a6f0c86a652" (UID: "62dc1c6d-fc62-4468-9a5f-3a6f0c86a652"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 13:09:15.347701 ip-10-0-133-218 kubenswrapper[2580]: I0916 13:09:15.347678    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/62dc1c6d-fc62-4468-9a5f-3a6f0c86a652-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "62dc1c6d-fc62-4468-9a5f-3a6f0c86a652" (UID: "62dc1c6d-fc62-4468-9a5f-3a6f0c86a652"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 13:09:15.349272 ip-10-0-133-218 kubenswrapper[2580]: I0916 13:09:15.349247    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/62dc1c6d-fc62-4468-9a5f-3a6f0c86a652-kube-api-access-j7qvn" (OuterVolumeSpecName: "kube-api-access-j7qvn") pod "62dc1c6d-fc62-4468-9a5f-3a6f0c86a652" (UID: "62dc1c6d-fc62-4468-9a5f-3a6f0c86a652"). InnerVolumeSpecName "kube-api-access-j7qvn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 13:09:15.349386 ip-10-0-133-218 kubenswrapper[2580]: I0916 13:09:15.349361    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/62dc1c6d-fc62-4468-9a5f-3a6f0c86a652-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "62dc1c6d-fc62-4468-9a5f-3a6f0c86a652" (UID: "62dc1c6d-fc62-4468-9a5f-3a6f0c86a652"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 13:09:15.448207 ip-10-0-133-218 kubenswrapper[2580]: I0916 13:09:15.448150    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/62dc1c6d-fc62-4468-9a5f-3a6f0c86a652-kserve-provision-location\") pod \"62dc1c6d-fc62-4468-9a5f-3a6f0c86a652\" (UID: \"62dc1c6d-fc62-4468-9a5f-3a6f0c86a652\") "
Sep 16 13:09:15.448329 ip-10-0-133-218 kubenswrapper[2580]: I0916 13:09:15.448316    2580 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/62dc1c6d-fc62-4468-9a5f-3a6f0c86a652-kube-rbac-proxy-sar-config\") on node \"ip-10-0-133-218.ec2.internal\" DevicePath \"\""
Sep 16 13:09:15.448372 ip-10-0-133-218 kubenswrapper[2580]: I0916 13:09:15.448333    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j7qvn\" (UniqueName: \"kubernetes.io/projected/62dc1c6d-fc62-4468-9a5f-3a6f0c86a652-kube-api-access-j7qvn\") on node \"ip-10-0-133-218.ec2.internal\" DevicePath \"\""
Sep 16 13:09:15.448372 ip-10-0-133-218 kubenswrapper[2580]: I0916 13:09:15.448342    2580 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/62dc1c6d-fc62-4468-9a5f-3a6f0c86a652-proxy-tls\") on node \"ip-10-0-133-218.ec2.internal\" DevicePath \"\""
Sep 16 13:09:15.448372 ip-10-0-133-218 kubenswrapper[2580]: I0916 13:09:15.448351    2580 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/62dc1c6d-fc62-4468-9a5f-3a6f0c86a652-cabundle-cert\") on node \"ip-10-0-133-218.ec2.internal\" DevicePath \"\""
Sep 16 13:09:15.448467 ip-10-0-133-218 kubenswrapper[2580]: I0916 13:09:15.448398    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/62dc1c6d-fc62-4468-9a5f-3a6f0c86a652-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "62dc1c6d-fc62-4468-9a5f-3a6f0c86a652" (UID: "62dc1c6d-fc62-4468-9a5f-3a6f0c86a652"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 13:09:15.549110 ip-10-0-133-218 kubenswrapper[2580]: I0916 13:09:15.549078    2580 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/62dc1c6d-fc62-4468-9a5f-3a6f0c86a652-kserve-provision-location\") on node \"ip-10-0-133-218.ec2.internal\" DevicePath \"\""
Sep 16 13:09:15.882981 ip-10-0-133-218 kubenswrapper[2580]: I0916 13:09:15.882950    2580 generic.go:356] "Generic (PLEG): container finished" podID="62dc1c6d-fc62-4468-9a5f-3a6f0c86a652" containerID="fd7e57639480c3ef9cb1790c65feefe21ec07c11281f752198f3a8f1fa984e61" exitCode=0
Sep 16 13:09:15.883156 ip-10-0-133-218 kubenswrapper[2580]: I0916 13:09:15.883006    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-8cb968449-w8qp8" event={"ID":"62dc1c6d-fc62-4468-9a5f-3a6f0c86a652","Type":"ContainerDied","Data":"fd7e57639480c3ef9cb1790c65feefe21ec07c11281f752198f3a8f1fa984e61"}
Sep 16 13:09:15.883156 ip-10-0-133-218 kubenswrapper[2580]: I0916 13:09:15.883011    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-8cb968449-w8qp8"
Sep 16 13:09:15.883156 ip-10-0-133-218 kubenswrapper[2580]: I0916 13:09:15.883055    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-8cb968449-w8qp8" event={"ID":"62dc1c6d-fc62-4468-9a5f-3a6f0c86a652","Type":"ContainerDied","Data":"7e371eb728da4877f264dc34926a10ab0c2be1f164be835a42b7373d1a6dae3d"}
Sep 16 13:09:15.883156 ip-10-0-133-218 kubenswrapper[2580]: I0916 13:09:15.883080    2580 scope.go:117] "RemoveContainer" containerID="092ebe3fb3257d1a56e1574b3e8ce5e5d2ab193d4b4be07be023e94d037c3830"
Sep 16 13:09:15.891394 ip-10-0-133-218 kubenswrapper[2580]: I0916 13:09:15.891376    2580 scope.go:117] "RemoveContainer" containerID="fd7e57639480c3ef9cb1790c65feefe21ec07c11281f752198f3a8f1fa984e61"
Sep 16 13:09:15.898206 ip-10-0-133-218 kubenswrapper[2580]: I0916 13:09:15.898189    2580 scope.go:117] "RemoveContainer" containerID="d74df3d806605baa8a03075eb5b41064c346a77a2c4286c13aa5fcf504e02c6b"
Sep 16 13:09:15.902292 ip-10-0-133-218 kubenswrapper[2580]: I0916 13:09:15.902268    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-predictor-8cb968449-w8qp8"]
Sep 16 13:09:15.905211 ip-10-0-133-218 kubenswrapper[2580]: I0916 13:09:15.905189    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-predictor-8cb968449-w8qp8"]
Sep 16 13:09:15.906190 ip-10-0-133-218 kubenswrapper[2580]: I0916 13:09:15.906176    2580 scope.go:117] "RemoveContainer" containerID="092ebe3fb3257d1a56e1574b3e8ce5e5d2ab193d4b4be07be023e94d037c3830"
Sep 16 13:09:15.906459 ip-10-0-133-218 kubenswrapper[2580]: E0916 13:09:15.906439    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"092ebe3fb3257d1a56e1574b3e8ce5e5d2ab193d4b4be07be023e94d037c3830\": container with ID starting with 092ebe3fb3257d1a56e1574b3e8ce5e5d2ab193d4b4be07be023e94d037c3830 not found: ID does not exist" containerID="092ebe3fb3257d1a56e1574b3e8ce5e5d2ab193d4b4be07be023e94d037c3830"
Sep 16 13:09:15.906568 ip-10-0-133-218 kubenswrapper[2580]: I0916 13:09:15.906469    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"092ebe3fb3257d1a56e1574b3e8ce5e5d2ab193d4b4be07be023e94d037c3830"} err="failed to get container status \"092ebe3fb3257d1a56e1574b3e8ce5e5d2ab193d4b4be07be023e94d037c3830\": rpc error: code = NotFound desc = could not find container \"092ebe3fb3257d1a56e1574b3e8ce5e5d2ab193d4b4be07be023e94d037c3830\": container with ID starting with 092ebe3fb3257d1a56e1574b3e8ce5e5d2ab193d4b4be07be023e94d037c3830 not found: ID does not exist"
Sep 16 13:09:15.906568 ip-10-0-133-218 kubenswrapper[2580]: I0916 13:09:15.906514    2580 scope.go:117] "RemoveContainer" containerID="fd7e57639480c3ef9cb1790c65feefe21ec07c11281f752198f3a8f1fa984e61"
Sep 16 13:09:15.906795 ip-10-0-133-218 kubenswrapper[2580]: E0916 13:09:15.906778    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fd7e57639480c3ef9cb1790c65feefe21ec07c11281f752198f3a8f1fa984e61\": container with ID starting with fd7e57639480c3ef9cb1790c65feefe21ec07c11281f752198f3a8f1fa984e61 not found: ID does not exist" containerID="fd7e57639480c3ef9cb1790c65feefe21ec07c11281f752198f3a8f1fa984e61"
Sep 16 13:09:15.906834 ip-10-0-133-218 kubenswrapper[2580]: I0916 13:09:15.906800    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fd7e57639480c3ef9cb1790c65feefe21ec07c11281f752198f3a8f1fa984e61"} err="failed to get container status \"fd7e57639480c3ef9cb1790c65feefe21ec07c11281f752198f3a8f1fa984e61\": rpc error: code = NotFound desc = could not find container \"fd7e57639480c3ef9cb1790c65feefe21ec07c11281f752198f3a8f1fa984e61\": container with ID starting with fd7e57639480c3ef9cb1790c65feefe21ec07c11281f752198f3a8f1fa984e61 not found: ID does not exist"
Sep 16 13:09:15.906834 ip-10-0-133-218 kubenswrapper[2580]: I0916 13:09:15.906815    2580 scope.go:117] "RemoveContainer" containerID="d74df3d806605baa8a03075eb5b41064c346a77a2c4286c13aa5fcf504e02c6b"
Sep 16 13:09:15.907061 ip-10-0-133-218 kubenswrapper[2580]: E0916 13:09:15.907045    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d74df3d806605baa8a03075eb5b41064c346a77a2c4286c13aa5fcf504e02c6b\": container with ID starting with d74df3d806605baa8a03075eb5b41064c346a77a2c4286c13aa5fcf504e02c6b not found: ID does not exist" containerID="d74df3d806605baa8a03075eb5b41064c346a77a2c4286c13aa5fcf504e02c6b"
Sep 16 13:09:15.907106 ip-10-0-133-218 kubenswrapper[2580]: I0916 13:09:15.907062    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d74df3d806605baa8a03075eb5b41064c346a77a2c4286c13aa5fcf504e02c6b"} err="failed to get container status \"d74df3d806605baa8a03075eb5b41064c346a77a2c4286c13aa5fcf504e02c6b\": rpc error: code = NotFound desc = could not find container \"d74df3d806605baa8a03075eb5b41064c346a77a2c4286c13aa5fcf504e02c6b\": container with ID starting with d74df3d806605baa8a03075eb5b41064c346a77a2c4286c13aa5fcf504e02c6b not found: ID does not exist"
Sep 16 13:09:17.599768 ip-10-0-133-218 kubenswrapper[2580]: I0916 13:09:17.599697    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="62dc1c6d-fc62-4468-9a5f-3a6f0c86a652" path="/var/lib/kubelet/pods/62dc1c6d-fc62-4468-9a5f-3a6f0c86a652/volumes"
Sep 16 13:09:17.648237 ip-10-0-133-218 kubenswrapper[2580]: I0916 13:09:17.648204    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-v2-predictor-7768896644-sm48p"]
Sep 16 13:09:17.648504 ip-10-0-133-218 kubenswrapper[2580]: I0916 13:09:17.648460    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-7768896644-sm48p" podUID="21450712-a128-4e3d-bf9b-997dcfd74263" containerName="kserve-container" containerID="cri-o://44b96e621b0c76fc2f660ddc0fa0fd1bb77d0a0886f5b65db416991eda9279c7" gracePeriod=30
Sep 16 13:09:17.648675 ip-10-0-133-218 kubenswrapper[2580]: I0916 13:09:17.648649    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-7768896644-sm48p" podUID="21450712-a128-4e3d-bf9b-997dcfd74263" containerName="kube-rbac-proxy" containerID="cri-o://d601ffa8d2c905b8ac812cac28d3b629eabd15d4730503cf9f97f7a83e585440" gracePeriod=30
Sep 16 13:09:17.894427 ip-10-0-133-218 kubenswrapper[2580]: I0916 13:09:17.894331    2580 generic.go:356] "Generic (PLEG): container finished" podID="21450712-a128-4e3d-bf9b-997dcfd74263" containerID="d601ffa8d2c905b8ac812cac28d3b629eabd15d4730503cf9f97f7a83e585440" exitCode=2
Sep 16 13:09:17.894427 ip-10-0-133-218 kubenswrapper[2580]: I0916 13:09:17.894395    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-7768896644-sm48p" event={"ID":"21450712-a128-4e3d-bf9b-997dcfd74263","Type":"ContainerDied","Data":"d601ffa8d2c905b8ac812cac28d3b629eabd15d4730503cf9f97f7a83e585440"}
Sep 16 13:09:20.557358 ip-10-0-133-218 kubenswrapper[2580]: I0916 13:09:20.557317    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-7768896644-sm48p" podUID="21450712-a128-4e3d-bf9b-997dcfd74263" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.35:8643/healthz\": dial tcp 10.133.0.35:8643: connect: connection refused"
Sep 16 13:09:20.562665 ip-10-0-133-218 kubenswrapper[2580]: I0916 13:09:20.562641    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-7768896644-sm48p" podUID="21450712-a128-4e3d-bf9b-997dcfd74263" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.35:8080: connect: connection refused"
Sep 16 13:09:22.187194 ip-10-0-133-218 kubenswrapper[2580]: I0916 13:09:22.187171    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-7768896644-sm48p"
Sep 16 13:09:22.201651 ip-10-0-133-218 kubenswrapper[2580]: I0916 13:09:22.201631    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/21450712-a128-4e3d-bf9b-997dcfd74263-kube-rbac-proxy-sar-config\") pod \"21450712-a128-4e3d-bf9b-997dcfd74263\" (UID: \"21450712-a128-4e3d-bf9b-997dcfd74263\") "
Sep 16 13:09:22.201721 ip-10-0-133-218 kubenswrapper[2580]: I0916 13:09:22.201657    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/21450712-a128-4e3d-bf9b-997dcfd74263-cabundle-cert\") pod \"21450712-a128-4e3d-bf9b-997dcfd74263\" (UID: \"21450712-a128-4e3d-bf9b-997dcfd74263\") "
Sep 16 13:09:22.201721 ip-10-0-133-218 kubenswrapper[2580]: I0916 13:09:22.201681    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/21450712-a128-4e3d-bf9b-997dcfd74263-proxy-tls\") pod \"21450712-a128-4e3d-bf9b-997dcfd74263\" (UID: \"21450712-a128-4e3d-bf9b-997dcfd74263\") "
Sep 16 13:09:22.201786 ip-10-0-133-218 kubenswrapper[2580]: I0916 13:09:22.201718    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/21450712-a128-4e3d-bf9b-997dcfd74263-kserve-provision-location\") pod \"21450712-a128-4e3d-bf9b-997dcfd74263\" (UID: \"21450712-a128-4e3d-bf9b-997dcfd74263\") "
Sep 16 13:09:22.201829 ip-10-0-133-218 kubenswrapper[2580]: I0916 13:09:22.201815    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-r5zjk\" (UniqueName: \"kubernetes.io/projected/21450712-a128-4e3d-bf9b-997dcfd74263-kube-api-access-r5zjk\") pod \"21450712-a128-4e3d-bf9b-997dcfd74263\" (UID: \"21450712-a128-4e3d-bf9b-997dcfd74263\") "
Sep 16 13:09:22.202084 ip-10-0-133-218 kubenswrapper[2580]: I0916 13:09:22.202059    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/21450712-a128-4e3d-bf9b-997dcfd74263-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "21450712-a128-4e3d-bf9b-997dcfd74263" (UID: "21450712-a128-4e3d-bf9b-997dcfd74263"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 13:09:22.202188 ip-10-0-133-218 kubenswrapper[2580]: I0916 13:09:22.202092    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/21450712-a128-4e3d-bf9b-997dcfd74263-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "21450712-a128-4e3d-bf9b-997dcfd74263" (UID: "21450712-a128-4e3d-bf9b-997dcfd74263"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 13:09:22.202188 ip-10-0-133-218 kubenswrapper[2580]: I0916 13:09:22.202157    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/21450712-a128-4e3d-bf9b-997dcfd74263-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "21450712-a128-4e3d-bf9b-997dcfd74263" (UID: "21450712-a128-4e3d-bf9b-997dcfd74263"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 13:09:22.203718 ip-10-0-133-218 kubenswrapper[2580]: I0916 13:09:22.203694    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/21450712-a128-4e3d-bf9b-997dcfd74263-kube-api-access-r5zjk" (OuterVolumeSpecName: "kube-api-access-r5zjk") pod "21450712-a128-4e3d-bf9b-997dcfd74263" (UID: "21450712-a128-4e3d-bf9b-997dcfd74263"). InnerVolumeSpecName "kube-api-access-r5zjk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 13:09:22.203788 ip-10-0-133-218 kubenswrapper[2580]: I0916 13:09:22.203715    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/21450712-a128-4e3d-bf9b-997dcfd74263-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "21450712-a128-4e3d-bf9b-997dcfd74263" (UID: "21450712-a128-4e3d-bf9b-997dcfd74263"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 13:09:22.303072 ip-10-0-133-218 kubenswrapper[2580]: I0916 13:09:22.303035    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r5zjk\" (UniqueName: \"kubernetes.io/projected/21450712-a128-4e3d-bf9b-997dcfd74263-kube-api-access-r5zjk\") on node \"ip-10-0-133-218.ec2.internal\" DevicePath \"\""
Sep 16 13:09:22.303072 ip-10-0-133-218 kubenswrapper[2580]: I0916 13:09:22.303073    2580 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/21450712-a128-4e3d-bf9b-997dcfd74263-kube-rbac-proxy-sar-config\") on node \"ip-10-0-133-218.ec2.internal\" DevicePath \"\""
Sep 16 13:09:22.303237 ip-10-0-133-218 kubenswrapper[2580]: I0916 13:09:22.303083    2580 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/21450712-a128-4e3d-bf9b-997dcfd74263-cabundle-cert\") on node \"ip-10-0-133-218.ec2.internal\" DevicePath \"\""
Sep 16 13:09:22.303237 ip-10-0-133-218 kubenswrapper[2580]: I0916 13:09:22.303093    2580 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/21450712-a128-4e3d-bf9b-997dcfd74263-proxy-tls\") on node \"ip-10-0-133-218.ec2.internal\" DevicePath \"\""
Sep 16 13:09:22.303237 ip-10-0-133-218 kubenswrapper[2580]: I0916 13:09:22.303101    2580 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/21450712-a128-4e3d-bf9b-997dcfd74263-kserve-provision-location\") on node \"ip-10-0-133-218.ec2.internal\" DevicePath \"\""
Sep 16 13:09:22.888480 ip-10-0-133-218 kubenswrapper[2580]: I0916 13:09:22.888446    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-predictor-8bc5b857-zkjp5"]
Sep 16 13:09:22.888778 ip-10-0-133-218 kubenswrapper[2580]: I0916 13:09:22.888764    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="62dc1c6d-fc62-4468-9a5f-3a6f0c86a652" containerName="kube-rbac-proxy"
Sep 16 13:09:22.888831 ip-10-0-133-218 kubenswrapper[2580]: I0916 13:09:22.888779    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="62dc1c6d-fc62-4468-9a5f-3a6f0c86a652" containerName="kube-rbac-proxy"
Sep 16 13:09:22.888831 ip-10-0-133-218 kubenswrapper[2580]: I0916 13:09:22.888797    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="21450712-a128-4e3d-bf9b-997dcfd74263" containerName="kserve-container"
Sep 16 13:09:22.888831 ip-10-0-133-218 kubenswrapper[2580]: I0916 13:09:22.888803    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="21450712-a128-4e3d-bf9b-997dcfd74263" containerName="kserve-container"
Sep 16 13:09:22.888831 ip-10-0-133-218 kubenswrapper[2580]: I0916 13:09:22.888809    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="21450712-a128-4e3d-bf9b-997dcfd74263" containerName="kube-rbac-proxy"
Sep 16 13:09:22.888831 ip-10-0-133-218 kubenswrapper[2580]: I0916 13:09:22.888814    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="21450712-a128-4e3d-bf9b-997dcfd74263" containerName="kube-rbac-proxy"
Sep 16 13:09:22.888831 ip-10-0-133-218 kubenswrapper[2580]: I0916 13:09:22.888821    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="62dc1c6d-fc62-4468-9a5f-3a6f0c86a652" containerName="kserve-container"
Sep 16 13:09:22.888831 ip-10-0-133-218 kubenswrapper[2580]: I0916 13:09:22.888826    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="62dc1c6d-fc62-4468-9a5f-3a6f0c86a652" containerName="kserve-container"
Sep 16 13:09:22.889057 ip-10-0-133-218 kubenswrapper[2580]: I0916 13:09:22.888840    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="62dc1c6d-fc62-4468-9a5f-3a6f0c86a652" containerName="storage-initializer"
Sep 16 13:09:22.889057 ip-10-0-133-218 kubenswrapper[2580]: I0916 13:09:22.888845    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="62dc1c6d-fc62-4468-9a5f-3a6f0c86a652" containerName="storage-initializer"
Sep 16 13:09:22.889057 ip-10-0-133-218 kubenswrapper[2580]: I0916 13:09:22.888853    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="21450712-a128-4e3d-bf9b-997dcfd74263" containerName="storage-initializer"
Sep 16 13:09:22.889057 ip-10-0-133-218 kubenswrapper[2580]: I0916 13:09:22.888857    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="21450712-a128-4e3d-bf9b-997dcfd74263" containerName="storage-initializer"
Sep 16 13:09:22.889057 ip-10-0-133-218 kubenswrapper[2580]: I0916 13:09:22.888901    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="21450712-a128-4e3d-bf9b-997dcfd74263" containerName="kube-rbac-proxy"
Sep 16 13:09:22.889057 ip-10-0-133-218 kubenswrapper[2580]: I0916 13:09:22.888909    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="62dc1c6d-fc62-4468-9a5f-3a6f0c86a652" containerName="kserve-container"
Sep 16 13:09:22.889057 ip-10-0-133-218 kubenswrapper[2580]: I0916 13:09:22.888915    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="62dc1c6d-fc62-4468-9a5f-3a6f0c86a652" containerName="kube-rbac-proxy"
Sep 16 13:09:22.889057 ip-10-0-133-218 kubenswrapper[2580]: I0916 13:09:22.888924    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="21450712-a128-4e3d-bf9b-997dcfd74263" containerName="kserve-container"
Sep 16 13:09:22.892103 ip-10-0-133-218 kubenswrapper[2580]: I0916 13:09:22.892088    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-8bc5b857-zkjp5"
Sep 16 13:09:22.894534 ip-10-0-133-218 kubenswrapper[2580]: I0916 13:09:22.894479    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-promote-predictor-serving-cert\""
Sep 16 13:09:22.894670 ip-10-0-133-218 kubenswrapper[2580]: I0916 13:09:22.894591    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-promote-kube-rbac-proxy-sar-config\""
Sep 16 13:09:22.903108 ip-10-0-133-218 kubenswrapper[2580]: I0916 13:09:22.903085    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-predictor-8bc5b857-zkjp5"]
Sep 16 13:09:22.907030 ip-10-0-133-218 kubenswrapper[2580]: I0916 13:09:22.907008    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/1f3420c8-2d3f-4958-9bb5-b70b9b63953b-kserve-provision-location\") pod \"isvc-canary-promote-predictor-8bc5b857-zkjp5\" (UID: \"1f3420c8-2d3f-4958-9bb5-b70b9b63953b\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-8bc5b857-zkjp5"
Sep 16 13:09:22.907158 ip-10-0-133-218 kubenswrapper[2580]: I0916 13:09:22.907056    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-52k4w\" (UniqueName: \"kubernetes.io/projected/1f3420c8-2d3f-4958-9bb5-b70b9b63953b-kube-api-access-52k4w\") pod \"isvc-canary-promote-predictor-8bc5b857-zkjp5\" (UID: \"1f3420c8-2d3f-4958-9bb5-b70b9b63953b\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-8bc5b857-zkjp5"
Sep 16 13:09:22.907158 ip-10-0-133-218 kubenswrapper[2580]: I0916 13:09:22.907126    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/1f3420c8-2d3f-4958-9bb5-b70b9b63953b-proxy-tls\") pod \"isvc-canary-promote-predictor-8bc5b857-zkjp5\" (UID: \"1f3420c8-2d3f-4958-9bb5-b70b9b63953b\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-8bc5b857-zkjp5"
Sep 16 13:09:22.907279 ip-10-0-133-218 kubenswrapper[2580]: I0916 13:09:22.907175    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/1f3420c8-2d3f-4958-9bb5-b70b9b63953b-kube-rbac-proxy-sar-config\") pod \"isvc-canary-promote-predictor-8bc5b857-zkjp5\" (UID: \"1f3420c8-2d3f-4958-9bb5-b70b9b63953b\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-8bc5b857-zkjp5"
Sep 16 13:09:22.907279 ip-10-0-133-218 kubenswrapper[2580]: I0916 13:09:22.907214    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/1f3420c8-2d3f-4958-9bb5-b70b9b63953b-cabundle-cert\") pod \"isvc-canary-promote-predictor-8bc5b857-zkjp5\" (UID: \"1f3420c8-2d3f-4958-9bb5-b70b9b63953b\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-8bc5b857-zkjp5"
Sep 16 13:09:22.915432 ip-10-0-133-218 kubenswrapper[2580]: I0916 13:09:22.915410    2580 generic.go:356] "Generic (PLEG): container finished" podID="21450712-a128-4e3d-bf9b-997dcfd74263" containerID="44b96e621b0c76fc2f660ddc0fa0fd1bb77d0a0886f5b65db416991eda9279c7" exitCode=0
Sep 16 13:09:22.915558 ip-10-0-133-218 kubenswrapper[2580]: I0916 13:09:22.915463    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-7768896644-sm48p"
Sep 16 13:09:22.915558 ip-10-0-133-218 kubenswrapper[2580]: I0916 13:09:22.915473    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-7768896644-sm48p" event={"ID":"21450712-a128-4e3d-bf9b-997dcfd74263","Type":"ContainerDied","Data":"44b96e621b0c76fc2f660ddc0fa0fd1bb77d0a0886f5b65db416991eda9279c7"}
Sep 16 13:09:22.915558 ip-10-0-133-218 kubenswrapper[2580]: I0916 13:09:22.915542    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-7768896644-sm48p" event={"ID":"21450712-a128-4e3d-bf9b-997dcfd74263","Type":"ContainerDied","Data":"068717b3db85e6b69d545ba167aa12e54ae1b43d6ae7014b2dbbfa0394b51bf5"}
Sep 16 13:09:22.915688 ip-10-0-133-218 kubenswrapper[2580]: I0916 13:09:22.915568    2580 scope.go:117] "RemoveContainer" containerID="d601ffa8d2c905b8ac812cac28d3b629eabd15d4730503cf9f97f7a83e585440"
Sep 16 13:09:22.924132 ip-10-0-133-218 kubenswrapper[2580]: I0916 13:09:22.924117    2580 scope.go:117] "RemoveContainer" containerID="44b96e621b0c76fc2f660ddc0fa0fd1bb77d0a0886f5b65db416991eda9279c7"
Sep 16 13:09:22.931043 ip-10-0-133-218 kubenswrapper[2580]: I0916 13:09:22.931024    2580 scope.go:117] "RemoveContainer" containerID="21b327d65b50291ac52b63f3664942b1f4b58ee8aa6f207568aeb0abd3da53a0"
Sep 16 13:09:22.938120 ip-10-0-133-218 kubenswrapper[2580]: I0916 13:09:22.938101    2580 scope.go:117] "RemoveContainer" containerID="d601ffa8d2c905b8ac812cac28d3b629eabd15d4730503cf9f97f7a83e585440"
Sep 16 13:09:22.938378 ip-10-0-133-218 kubenswrapper[2580]: E0916 13:09:22.938363    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d601ffa8d2c905b8ac812cac28d3b629eabd15d4730503cf9f97f7a83e585440\": container with ID starting with d601ffa8d2c905b8ac812cac28d3b629eabd15d4730503cf9f97f7a83e585440 not found: ID does not exist" containerID="d601ffa8d2c905b8ac812cac28d3b629eabd15d4730503cf9f97f7a83e585440"
Sep 16 13:09:22.938440 ip-10-0-133-218 kubenswrapper[2580]: I0916 13:09:22.938384    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d601ffa8d2c905b8ac812cac28d3b629eabd15d4730503cf9f97f7a83e585440"} err="failed to get container status \"d601ffa8d2c905b8ac812cac28d3b629eabd15d4730503cf9f97f7a83e585440\": rpc error: code = NotFound desc = could not find container \"d601ffa8d2c905b8ac812cac28d3b629eabd15d4730503cf9f97f7a83e585440\": container with ID starting with d601ffa8d2c905b8ac812cac28d3b629eabd15d4730503cf9f97f7a83e585440 not found: ID does not exist"
Sep 16 13:09:22.938440 ip-10-0-133-218 kubenswrapper[2580]: I0916 13:09:22.938399    2580 scope.go:117] "RemoveContainer" containerID="44b96e621b0c76fc2f660ddc0fa0fd1bb77d0a0886f5b65db416991eda9279c7"
Sep 16 13:09:22.938654 ip-10-0-133-218 kubenswrapper[2580]: E0916 13:09:22.938636    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"44b96e621b0c76fc2f660ddc0fa0fd1bb77d0a0886f5b65db416991eda9279c7\": container with ID starting with 44b96e621b0c76fc2f660ddc0fa0fd1bb77d0a0886f5b65db416991eda9279c7 not found: ID does not exist" containerID="44b96e621b0c76fc2f660ddc0fa0fd1bb77d0a0886f5b65db416991eda9279c7"
Sep 16 13:09:22.938693 ip-10-0-133-218 kubenswrapper[2580]: I0916 13:09:22.938658    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"44b96e621b0c76fc2f660ddc0fa0fd1bb77d0a0886f5b65db416991eda9279c7"} err="failed to get container status \"44b96e621b0c76fc2f660ddc0fa0fd1bb77d0a0886f5b65db416991eda9279c7\": rpc error: code = NotFound desc = could not find container \"44b96e621b0c76fc2f660ddc0fa0fd1bb77d0a0886f5b65db416991eda9279c7\": container with ID starting with 44b96e621b0c76fc2f660ddc0fa0fd1bb77d0a0886f5b65db416991eda9279c7 not found: ID does not exist"
Sep 16 13:09:22.938693 ip-10-0-133-218 kubenswrapper[2580]: I0916 13:09:22.938673    2580 scope.go:117] "RemoveContainer" containerID="21b327d65b50291ac52b63f3664942b1f4b58ee8aa6f207568aeb0abd3da53a0"
Sep 16 13:09:22.938938 ip-10-0-133-218 kubenswrapper[2580]: E0916 13:09:22.938920    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"21b327d65b50291ac52b63f3664942b1f4b58ee8aa6f207568aeb0abd3da53a0\": container with ID starting with 21b327d65b50291ac52b63f3664942b1f4b58ee8aa6f207568aeb0abd3da53a0 not found: ID does not exist" containerID="21b327d65b50291ac52b63f3664942b1f4b58ee8aa6f207568aeb0abd3da53a0"
Sep 16 13:09:22.938976 ip-10-0-133-218 kubenswrapper[2580]: I0916 13:09:22.938945    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"21b327d65b50291ac52b63f3664942b1f4b58ee8aa6f207568aeb0abd3da53a0"} err="failed to get container status \"21b327d65b50291ac52b63f3664942b1f4b58ee8aa6f207568aeb0abd3da53a0\": rpc error: code = NotFound desc = could not find container \"21b327d65b50291ac52b63f3664942b1f4b58ee8aa6f207568aeb0abd3da53a0\": container with ID starting with 21b327d65b50291ac52b63f3664942b1f4b58ee8aa6f207568aeb0abd3da53a0 not found: ID does not exist"
Sep 16 13:09:22.943587 ip-10-0-133-218 kubenswrapper[2580]: I0916 13:09:22.943565    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-v2-predictor-7768896644-sm48p"]
Sep 16 13:09:22.947406 ip-10-0-133-218 kubenswrapper[2580]: I0916 13:09:22.947385    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-v2-predictor-7768896644-sm48p"]
Sep 16 13:09:23.008231 ip-10-0-133-218 kubenswrapper[2580]: I0916 13:09:23.008201    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/1f3420c8-2d3f-4958-9bb5-b70b9b63953b-cabundle-cert\") pod \"isvc-canary-promote-predictor-8bc5b857-zkjp5\" (UID: \"1f3420c8-2d3f-4958-9bb5-b70b9b63953b\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-8bc5b857-zkjp5"
Sep 16 13:09:23.008354 ip-10-0-133-218 kubenswrapper[2580]: I0916 13:09:23.008256    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/1f3420c8-2d3f-4958-9bb5-b70b9b63953b-kserve-provision-location\") pod \"isvc-canary-promote-predictor-8bc5b857-zkjp5\" (UID: \"1f3420c8-2d3f-4958-9bb5-b70b9b63953b\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-8bc5b857-zkjp5"
Sep 16 13:09:23.008354 ip-10-0-133-218 kubenswrapper[2580]: I0916 13:09:23.008285    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-52k4w\" (UniqueName: \"kubernetes.io/projected/1f3420c8-2d3f-4958-9bb5-b70b9b63953b-kube-api-access-52k4w\") pod \"isvc-canary-promote-predictor-8bc5b857-zkjp5\" (UID: \"1f3420c8-2d3f-4958-9bb5-b70b9b63953b\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-8bc5b857-zkjp5"
Sep 16 13:09:23.008354 ip-10-0-133-218 kubenswrapper[2580]: I0916 13:09:23.008313    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/1f3420c8-2d3f-4958-9bb5-b70b9b63953b-proxy-tls\") pod \"isvc-canary-promote-predictor-8bc5b857-zkjp5\" (UID: \"1f3420c8-2d3f-4958-9bb5-b70b9b63953b\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-8bc5b857-zkjp5"
Sep 16 13:09:23.008354 ip-10-0-133-218 kubenswrapper[2580]: I0916 13:09:23.008344    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/1f3420c8-2d3f-4958-9bb5-b70b9b63953b-kube-rbac-proxy-sar-config\") pod \"isvc-canary-promote-predictor-8bc5b857-zkjp5\" (UID: \"1f3420c8-2d3f-4958-9bb5-b70b9b63953b\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-8bc5b857-zkjp5"
Sep 16 13:09:23.008610 ip-10-0-133-218 kubenswrapper[2580]: E0916 13:09:23.008451    2580 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-canary-promote-predictor-serving-cert: secret "isvc-canary-promote-predictor-serving-cert" not found
Sep 16 13:09:23.008610 ip-10-0-133-218 kubenswrapper[2580]: E0916 13:09:23.008547    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1f3420c8-2d3f-4958-9bb5-b70b9b63953b-proxy-tls podName:1f3420c8-2d3f-4958-9bb5-b70b9b63953b nodeName:}" failed. No retries permitted until 2026-09-16 13:09:23.508530472 +0000 UTC m=+1302.431402932 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/1f3420c8-2d3f-4958-9bb5-b70b9b63953b-proxy-tls") pod "isvc-canary-promote-predictor-8bc5b857-zkjp5" (UID: "1f3420c8-2d3f-4958-9bb5-b70b9b63953b") : secret "isvc-canary-promote-predictor-serving-cert" not found
Sep 16 13:09:23.008723 ip-10-0-133-218 kubenswrapper[2580]: I0916 13:09:23.008698    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/1f3420c8-2d3f-4958-9bb5-b70b9b63953b-kserve-provision-location\") pod \"isvc-canary-promote-predictor-8bc5b857-zkjp5\" (UID: \"1f3420c8-2d3f-4958-9bb5-b70b9b63953b\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-8bc5b857-zkjp5"
Sep 16 13:09:23.008982 ip-10-0-133-218 kubenswrapper[2580]: I0916 13:09:23.008960    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/1f3420c8-2d3f-4958-9bb5-b70b9b63953b-kube-rbac-proxy-sar-config\") pod \"isvc-canary-promote-predictor-8bc5b857-zkjp5\" (UID: \"1f3420c8-2d3f-4958-9bb5-b70b9b63953b\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-8bc5b857-zkjp5"
Sep 16 13:09:23.009037 ip-10-0-133-218 kubenswrapper[2580]: I0916 13:09:23.008969    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/1f3420c8-2d3f-4958-9bb5-b70b9b63953b-cabundle-cert\") pod \"isvc-canary-promote-predictor-8bc5b857-zkjp5\" (UID: \"1f3420c8-2d3f-4958-9bb5-b70b9b63953b\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-8bc5b857-zkjp5"
Sep 16 13:09:23.016754 ip-10-0-133-218 kubenswrapper[2580]: I0916 13:09:23.016738    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-52k4w\" (UniqueName: \"kubernetes.io/projected/1f3420c8-2d3f-4958-9bb5-b70b9b63953b-kube-api-access-52k4w\") pod \"isvc-canary-promote-predictor-8bc5b857-zkjp5\" (UID: \"1f3420c8-2d3f-4958-9bb5-b70b9b63953b\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-8bc5b857-zkjp5"
Sep 16 13:09:23.510768 ip-10-0-133-218 kubenswrapper[2580]: I0916 13:09:23.510735    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/1f3420c8-2d3f-4958-9bb5-b70b9b63953b-proxy-tls\") pod \"isvc-canary-promote-predictor-8bc5b857-zkjp5\" (UID: \"1f3420c8-2d3f-4958-9bb5-b70b9b63953b\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-8bc5b857-zkjp5"
Sep 16 13:09:23.513053 ip-10-0-133-218 kubenswrapper[2580]: I0916 13:09:23.513031    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/1f3420c8-2d3f-4958-9bb5-b70b9b63953b-proxy-tls\") pod \"isvc-canary-promote-predictor-8bc5b857-zkjp5\" (UID: \"1f3420c8-2d3f-4958-9bb5-b70b9b63953b\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-8bc5b857-zkjp5"
Sep 16 13:09:23.598754 ip-10-0-133-218 kubenswrapper[2580]: I0916 13:09:23.598725    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="21450712-a128-4e3d-bf9b-997dcfd74263" path="/var/lib/kubelet/pods/21450712-a128-4e3d-bf9b-997dcfd74263/volumes"
Sep 16 13:09:23.802199 ip-10-0-133-218 kubenswrapper[2580]: I0916 13:09:23.802170    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-8bc5b857-zkjp5"
Sep 16 13:09:24.074060 ip-10-0-133-218 kubenswrapper[2580]: I0916 13:09:24.073979    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-predictor-8bc5b857-zkjp5"]
Sep 16 13:09:24.076982 ip-10-0-133-218 kubenswrapper[2580]: W0916 13:09:24.076953    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1f3420c8_2d3f_4958_9bb5_b70b9b63953b.slice/crio-a4d331c9af3e83b838aa1727b8d00ce9b5975282ad0bd009a01243aff03003d3 WatchSource:0}: Error finding container a4d331c9af3e83b838aa1727b8d00ce9b5975282ad0bd009a01243aff03003d3: Status 404 returned error can't find the container with id a4d331c9af3e83b838aa1727b8d00ce9b5975282ad0bd009a01243aff03003d3
Sep 16 13:09:24.925724 ip-10-0-133-218 kubenswrapper[2580]: I0916 13:09:24.925685    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-8bc5b857-zkjp5" event={"ID":"1f3420c8-2d3f-4958-9bb5-b70b9b63953b","Type":"ContainerStarted","Data":"f58eeafe836c249775f2f93a826fd70a7f9833e3b11a82426389004bf251e260"}
Sep 16 13:09:24.926123 ip-10-0-133-218 kubenswrapper[2580]: I0916 13:09:24.925745    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-8bc5b857-zkjp5" event={"ID":"1f3420c8-2d3f-4958-9bb5-b70b9b63953b","Type":"ContainerStarted","Data":"a4d331c9af3e83b838aa1727b8d00ce9b5975282ad0bd009a01243aff03003d3"}
Sep 16 13:09:27.938992 ip-10-0-133-218 kubenswrapper[2580]: I0916 13:09:27.938960    2580 generic.go:356] "Generic (PLEG): container finished" podID="1f3420c8-2d3f-4958-9bb5-b70b9b63953b" containerID="f58eeafe836c249775f2f93a826fd70a7f9833e3b11a82426389004bf251e260" exitCode=0
Sep 16 13:09:27.939350 ip-10-0-133-218 kubenswrapper[2580]: I0916 13:09:27.939034    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-8bc5b857-zkjp5" event={"ID":"1f3420c8-2d3f-4958-9bb5-b70b9b63953b","Type":"ContainerDied","Data":"f58eeafe836c249775f2f93a826fd70a7f9833e3b11a82426389004bf251e260"}
Sep 16 13:09:28.945376 ip-10-0-133-218 kubenswrapper[2580]: I0916 13:09:28.945338    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-8bc5b857-zkjp5" event={"ID":"1f3420c8-2d3f-4958-9bb5-b70b9b63953b","Type":"ContainerStarted","Data":"565ed39389087d5114557f6ae21c36f408909994c93f7d156a1fe966081a9214"}
Sep 16 13:09:28.945879 ip-10-0-133-218 kubenswrapper[2580]: I0916 13:09:28.945413    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-8bc5b857-zkjp5" event={"ID":"1f3420c8-2d3f-4958-9bb5-b70b9b63953b","Type":"ContainerStarted","Data":"1f2a07e10545e7206d274368a83b9b96f9c4cfe2c7521709c4dc137251faa174"}
Sep 16 13:09:28.945879 ip-10-0-133-218 kubenswrapper[2580]: I0916 13:09:28.945652    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-8bc5b857-zkjp5"
Sep 16 13:09:28.963351 ip-10-0-133-218 kubenswrapper[2580]: I0916 13:09:28.963301    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-8bc5b857-zkjp5" podStartSLOduration=3.286822987 podStartE2EDuration="6.963286572s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.676463585s" isStatefulPod=false podCreationTimestamp="2026-09-16 13:09:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 13:09:28.962659588 +0000 UTC m=+1307.885532057" watchObservedRunningTime="2026-09-16 13:09:28.963286572 +0000 UTC m=+1307.886159040"
Sep 16 13:09:29.949805 ip-10-0-133-218 kubenswrapper[2580]: I0916 13:09:29.949771    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-8bc5b857-zkjp5"
Sep 16 13:09:29.950962 ip-10-0-133-218 kubenswrapper[2580]: I0916 13:09:29.950933    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-8bc5b857-zkjp5" podUID="1f3420c8-2d3f-4958-9bb5-b70b9b63953b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.37:8080: connect: connection refused"
Sep 16 13:09:30.953535 ip-10-0-133-218 kubenswrapper[2580]: I0916 13:09:30.953473    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-8bc5b857-zkjp5" podUID="1f3420c8-2d3f-4958-9bb5-b70b9b63953b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.37:8080: connect: connection refused"
Sep 16 13:09:35.959560 ip-10-0-133-218 kubenswrapper[2580]: I0916 13:09:35.959528    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-8bc5b857-zkjp5"
Sep 16 13:09:35.960163 ip-10-0-133-218 kubenswrapper[2580]: I0916 13:09:35.960138    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-8bc5b857-zkjp5" podUID="1f3420c8-2d3f-4958-9bb5-b70b9b63953b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.37:8080: connect: connection refused"
Sep 16 13:09:45.960219 ip-10-0-133-218 kubenswrapper[2580]: I0916 13:09:45.960178    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-8bc5b857-zkjp5" podUID="1f3420c8-2d3f-4958-9bb5-b70b9b63953b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.37:8080: connect: connection refused"
Sep 16 13:09:55.960907 ip-10-0-133-218 kubenswrapper[2580]: I0916 13:09:55.960868    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-8bc5b857-zkjp5" podUID="1f3420c8-2d3f-4958-9bb5-b70b9b63953b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.37:8080: connect: connection refused"
Sep 16 13:10:05.960797 ip-10-0-133-218 kubenswrapper[2580]: I0916 13:10:05.960755    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-8bc5b857-zkjp5" podUID="1f3420c8-2d3f-4958-9bb5-b70b9b63953b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.37:8080: connect: connection refused"
Sep 16 13:10:15.960696 ip-10-0-133-218 kubenswrapper[2580]: I0916 13:10:15.960656    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-8bc5b857-zkjp5" podUID="1f3420c8-2d3f-4958-9bb5-b70b9b63953b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.37:8080: connect: connection refused"
Sep 16 13:10:25.961004 ip-10-0-133-218 kubenswrapper[2580]: I0916 13:10:25.960954    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-8bc5b857-zkjp5" podUID="1f3420c8-2d3f-4958-9bb5-b70b9b63953b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.37:8080: connect: connection refused"
Sep 16 13:10:35.960393 ip-10-0-133-218 kubenswrapper[2580]: I0916 13:10:35.960353    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-8bc5b857-zkjp5" podUID="1f3420c8-2d3f-4958-9bb5-b70b9b63953b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.37:8080: connect: connection refused"
Sep 16 13:10:37.595115 ip-10-0-133-218 kubenswrapper[2580]: I0916 13:10:37.595072    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-8bc5b857-zkjp5" podUID="1f3420c8-2d3f-4958-9bb5-b70b9b63953b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.37:8080: connect: connection refused"
Sep 16 13:10:47.600321 ip-10-0-133-218 kubenswrapper[2580]: I0916 13:10:47.600292    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-8bc5b857-zkjp5"
Sep 16 13:10:52.896061 ip-10-0-133-218 kubenswrapper[2580]: I0916 13:10:52.896026    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-predictor-8bc5b857-zkjp5"]
Sep 16 13:10:52.896549 ip-10-0-133-218 kubenswrapper[2580]: I0916 13:10:52.896261    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-8bc5b857-zkjp5" podUID="1f3420c8-2d3f-4958-9bb5-b70b9b63953b" containerName="kserve-container" containerID="cri-o://1f2a07e10545e7206d274368a83b9b96f9c4cfe2c7521709c4dc137251faa174" gracePeriod=30
Sep 16 13:10:52.896549 ip-10-0-133-218 kubenswrapper[2580]: I0916 13:10:52.896283    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-8bc5b857-zkjp5" podUID="1f3420c8-2d3f-4958-9bb5-b70b9b63953b" containerName="kube-rbac-proxy" containerID="cri-o://565ed39389087d5114557f6ae21c36f408909994c93f7d156a1fe966081a9214" gracePeriod=30
Sep 16 13:10:53.007926 ip-10-0-133-218 kubenswrapper[2580]: I0916 13:10:53.007894    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-65744c97b6-rkjp8"]
Sep 16 13:10:53.011323 ip-10-0-133-218 kubenswrapper[2580]: I0916 13:10:53.011302    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-65744c97b6-rkjp8"
Sep 16 13:10:53.013758 ip-10-0-133-218 kubenswrapper[2580]: I0916 13:10:53.013741    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-promote-v2-predictor-serving-cert\""
Sep 16 13:10:53.020149 ip-10-0-133-218 kubenswrapper[2580]: I0916 13:10:53.020129    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-65744c97b6-rkjp8"]
Sep 16 13:10:53.138719 ip-10-0-133-218 kubenswrapper[2580]: I0916 13:10:53.138684    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-77mxl\" (UniqueName: \"kubernetes.io/projected/c40fd0b4-186c-49e3-a6d9-4fa18505145f-kube-api-access-77mxl\") pod \"isvc-canary-promote-v2-predictor-65744c97b6-rkjp8\" (UID: \"c40fd0b4-186c-49e3-a6d9-4fa18505145f\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-65744c97b6-rkjp8"
Sep 16 13:10:53.138973 ip-10-0-133-218 kubenswrapper[2580]: I0916 13:10:53.138732    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/c40fd0b4-186c-49e3-a6d9-4fa18505145f-cabundle-cert\") pod \"isvc-canary-promote-v2-predictor-65744c97b6-rkjp8\" (UID: \"c40fd0b4-186c-49e3-a6d9-4fa18505145f\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-65744c97b6-rkjp8"
Sep 16 13:10:53.138973 ip-10-0-133-218 kubenswrapper[2580]: I0916 13:10:53.138760    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c40fd0b4-186c-49e3-a6d9-4fa18505145f-proxy-tls\") pod \"isvc-canary-promote-v2-predictor-65744c97b6-rkjp8\" (UID: \"c40fd0b4-186c-49e3-a6d9-4fa18505145f\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-65744c97b6-rkjp8"
Sep 16 13:10:53.138973 ip-10-0-133-218 kubenswrapper[2580]: I0916 13:10:53.138782    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/c40fd0b4-186c-49e3-a6d9-4fa18505145f-kube-rbac-proxy-sar-config\") pod \"isvc-canary-promote-v2-predictor-65744c97b6-rkjp8\" (UID: \"c40fd0b4-186c-49e3-a6d9-4fa18505145f\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-65744c97b6-rkjp8"
Sep 16 13:10:53.138973 ip-10-0-133-218 kubenswrapper[2580]: I0916 13:10:53.138808    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c40fd0b4-186c-49e3-a6d9-4fa18505145f-kserve-provision-location\") pod \"isvc-canary-promote-v2-predictor-65744c97b6-rkjp8\" (UID: \"c40fd0b4-186c-49e3-a6d9-4fa18505145f\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-65744c97b6-rkjp8"
Sep 16 13:10:53.240005 ip-10-0-133-218 kubenswrapper[2580]: I0916 13:10:53.239936    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/c40fd0b4-186c-49e3-a6d9-4fa18505145f-kube-rbac-proxy-sar-config\") pod \"isvc-canary-promote-v2-predictor-65744c97b6-rkjp8\" (UID: \"c40fd0b4-186c-49e3-a6d9-4fa18505145f\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-65744c97b6-rkjp8"
Sep 16 13:10:53.240005 ip-10-0-133-218 kubenswrapper[2580]: I0916 13:10:53.239970    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c40fd0b4-186c-49e3-a6d9-4fa18505145f-kserve-provision-location\") pod \"isvc-canary-promote-v2-predictor-65744c97b6-rkjp8\" (UID: \"c40fd0b4-186c-49e3-a6d9-4fa18505145f\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-65744c97b6-rkjp8"
Sep 16 13:10:53.240005 ip-10-0-133-218 kubenswrapper[2580]: I0916 13:10:53.239998    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-77mxl\" (UniqueName: \"kubernetes.io/projected/c40fd0b4-186c-49e3-a6d9-4fa18505145f-kube-api-access-77mxl\") pod \"isvc-canary-promote-v2-predictor-65744c97b6-rkjp8\" (UID: \"c40fd0b4-186c-49e3-a6d9-4fa18505145f\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-65744c97b6-rkjp8"
Sep 16 13:10:53.240268 ip-10-0-133-218 kubenswrapper[2580]: I0916 13:10:53.240025    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/c40fd0b4-186c-49e3-a6d9-4fa18505145f-cabundle-cert\") pod \"isvc-canary-promote-v2-predictor-65744c97b6-rkjp8\" (UID: \"c40fd0b4-186c-49e3-a6d9-4fa18505145f\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-65744c97b6-rkjp8"
Sep 16 13:10:53.240268 ip-10-0-133-218 kubenswrapper[2580]: I0916 13:10:53.240049    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c40fd0b4-186c-49e3-a6d9-4fa18505145f-proxy-tls\") pod \"isvc-canary-promote-v2-predictor-65744c97b6-rkjp8\" (UID: \"c40fd0b4-186c-49e3-a6d9-4fa18505145f\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-65744c97b6-rkjp8"
Sep 16 13:10:53.240440 ip-10-0-133-218 kubenswrapper[2580]: I0916 13:10:53.240415    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c40fd0b4-186c-49e3-a6d9-4fa18505145f-kserve-provision-location\") pod \"isvc-canary-promote-v2-predictor-65744c97b6-rkjp8\" (UID: \"c40fd0b4-186c-49e3-a6d9-4fa18505145f\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-65744c97b6-rkjp8"
Sep 16 13:10:53.240800 ip-10-0-133-218 kubenswrapper[2580]: I0916 13:10:53.240781    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/c40fd0b4-186c-49e3-a6d9-4fa18505145f-cabundle-cert\") pod \"isvc-canary-promote-v2-predictor-65744c97b6-rkjp8\" (UID: \"c40fd0b4-186c-49e3-a6d9-4fa18505145f\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-65744c97b6-rkjp8"
Sep 16 13:10:53.240850 ip-10-0-133-218 kubenswrapper[2580]: I0916 13:10:53.240818    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/c40fd0b4-186c-49e3-a6d9-4fa18505145f-kube-rbac-proxy-sar-config\") pod \"isvc-canary-promote-v2-predictor-65744c97b6-rkjp8\" (UID: \"c40fd0b4-186c-49e3-a6d9-4fa18505145f\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-65744c97b6-rkjp8"
Sep 16 13:10:53.242435 ip-10-0-133-218 kubenswrapper[2580]: I0916 13:10:53.242418    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c40fd0b4-186c-49e3-a6d9-4fa18505145f-proxy-tls\") pod \"isvc-canary-promote-v2-predictor-65744c97b6-rkjp8\" (UID: \"c40fd0b4-186c-49e3-a6d9-4fa18505145f\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-65744c97b6-rkjp8"
Sep 16 13:10:53.248290 ip-10-0-133-218 kubenswrapper[2580]: I0916 13:10:53.248265    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-77mxl\" (UniqueName: \"kubernetes.io/projected/c40fd0b4-186c-49e3-a6d9-4fa18505145f-kube-api-access-77mxl\") pod \"isvc-canary-promote-v2-predictor-65744c97b6-rkjp8\" (UID: \"c40fd0b4-186c-49e3-a6d9-4fa18505145f\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-65744c97b6-rkjp8"
Sep 16 13:10:53.274543 ip-10-0-133-218 kubenswrapper[2580]: I0916 13:10:53.274506    2580 generic.go:356] "Generic (PLEG): container finished" podID="1f3420c8-2d3f-4958-9bb5-b70b9b63953b" containerID="565ed39389087d5114557f6ae21c36f408909994c93f7d156a1fe966081a9214" exitCode=2
Sep 16 13:10:53.274640 ip-10-0-133-218 kubenswrapper[2580]: I0916 13:10:53.274564    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-8bc5b857-zkjp5" event={"ID":"1f3420c8-2d3f-4958-9bb5-b70b9b63953b","Type":"ContainerDied","Data":"565ed39389087d5114557f6ae21c36f408909994c93f7d156a1fe966081a9214"}
Sep 16 13:10:53.321865 ip-10-0-133-218 kubenswrapper[2580]: I0916 13:10:53.321841    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-65744c97b6-rkjp8"
Sep 16 13:10:53.593060 ip-10-0-133-218 kubenswrapper[2580]: I0916 13:10:53.592916    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-65744c97b6-rkjp8"]
Sep 16 13:10:53.595443 ip-10-0-133-218 kubenswrapper[2580]: W0916 13:10:53.595417    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc40fd0b4_186c_49e3_a6d9_4fa18505145f.slice/crio-ced251ae2cad88d01e9e002afaad6b939ce5ccaab24da805cc1981d132348b47 WatchSource:0}: Error finding container ced251ae2cad88d01e9e002afaad6b939ce5ccaab24da805cc1981d132348b47: Status 404 returned error can't find the container with id ced251ae2cad88d01e9e002afaad6b939ce5ccaab24da805cc1981d132348b47
Sep 16 13:10:54.279910 ip-10-0-133-218 kubenswrapper[2580]: I0916 13:10:54.279874    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-65744c97b6-rkjp8" event={"ID":"c40fd0b4-186c-49e3-a6d9-4fa18505145f","Type":"ContainerStarted","Data":"a4584541bd6e85fd139f68510aac9daeb2a480bee11c74ff5d0e963af8ceb116"}
Sep 16 13:10:54.280307 ip-10-0-133-218 kubenswrapper[2580]: I0916 13:10:54.279938    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-65744c97b6-rkjp8" event={"ID":"c40fd0b4-186c-49e3-a6d9-4fa18505145f","Type":"ContainerStarted","Data":"ced251ae2cad88d01e9e002afaad6b939ce5ccaab24da805cc1981d132348b47"}
Sep 16 13:10:55.954372 ip-10-0-133-218 kubenswrapper[2580]: I0916 13:10:55.954329    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-8bc5b857-zkjp5" podUID="1f3420c8-2d3f-4958-9bb5-b70b9b63953b" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.37:8643/healthz\": dial tcp 10.133.0.37:8643: connect: connection refused"
Sep 16 13:10:57.462181 ip-10-0-133-218 kubenswrapper[2580]: I0916 13:10:57.462153    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-8bc5b857-zkjp5"
Sep 16 13:10:57.576906 ip-10-0-133-218 kubenswrapper[2580]: I0916 13:10:57.576881    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/1f3420c8-2d3f-4958-9bb5-b70b9b63953b-cabundle-cert\") pod \"1f3420c8-2d3f-4958-9bb5-b70b9b63953b\" (UID: \"1f3420c8-2d3f-4958-9bb5-b70b9b63953b\") "
Sep 16 13:10:57.577074 ip-10-0-133-218 kubenswrapper[2580]: I0916 13:10:57.576916    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/1f3420c8-2d3f-4958-9bb5-b70b9b63953b-kserve-provision-location\") pod \"1f3420c8-2d3f-4958-9bb5-b70b9b63953b\" (UID: \"1f3420c8-2d3f-4958-9bb5-b70b9b63953b\") "
Sep 16 13:10:57.577074 ip-10-0-133-218 kubenswrapper[2580]: I0916 13:10:57.576943    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/1f3420c8-2d3f-4958-9bb5-b70b9b63953b-proxy-tls\") pod \"1f3420c8-2d3f-4958-9bb5-b70b9b63953b\" (UID: \"1f3420c8-2d3f-4958-9bb5-b70b9b63953b\") "
Sep 16 13:10:57.577074 ip-10-0-133-218 kubenswrapper[2580]: I0916 13:10:57.577014    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-52k4w\" (UniqueName: \"kubernetes.io/projected/1f3420c8-2d3f-4958-9bb5-b70b9b63953b-kube-api-access-52k4w\") pod \"1f3420c8-2d3f-4958-9bb5-b70b9b63953b\" (UID: \"1f3420c8-2d3f-4958-9bb5-b70b9b63953b\") "
Sep 16 13:10:57.577074 ip-10-0-133-218 kubenswrapper[2580]: I0916 13:10:57.577038    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/1f3420c8-2d3f-4958-9bb5-b70b9b63953b-kube-rbac-proxy-sar-config\") pod \"1f3420c8-2d3f-4958-9bb5-b70b9b63953b\" (UID: \"1f3420c8-2d3f-4958-9bb5-b70b9b63953b\") "
Sep 16 13:10:57.577280 ip-10-0-133-218 kubenswrapper[2580]: I0916 13:10:57.577236    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1f3420c8-2d3f-4958-9bb5-b70b9b63953b-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "1f3420c8-2d3f-4958-9bb5-b70b9b63953b" (UID: "1f3420c8-2d3f-4958-9bb5-b70b9b63953b"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 13:10:57.577280 ip-10-0-133-218 kubenswrapper[2580]: I0916 13:10:57.577248    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1f3420c8-2d3f-4958-9bb5-b70b9b63953b-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "1f3420c8-2d3f-4958-9bb5-b70b9b63953b" (UID: "1f3420c8-2d3f-4958-9bb5-b70b9b63953b"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 13:10:57.577480 ip-10-0-133-218 kubenswrapper[2580]: I0916 13:10:57.577458    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1f3420c8-2d3f-4958-9bb5-b70b9b63953b-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "1f3420c8-2d3f-4958-9bb5-b70b9b63953b" (UID: "1f3420c8-2d3f-4958-9bb5-b70b9b63953b"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 13:10:57.579165 ip-10-0-133-218 kubenswrapper[2580]: I0916 13:10:57.579145    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1f3420c8-2d3f-4958-9bb5-b70b9b63953b-kube-api-access-52k4w" (OuterVolumeSpecName: "kube-api-access-52k4w") pod "1f3420c8-2d3f-4958-9bb5-b70b9b63953b" (UID: "1f3420c8-2d3f-4958-9bb5-b70b9b63953b"). InnerVolumeSpecName "kube-api-access-52k4w". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 13:10:57.579230 ip-10-0-133-218 kubenswrapper[2580]: I0916 13:10:57.579151    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1f3420c8-2d3f-4958-9bb5-b70b9b63953b-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "1f3420c8-2d3f-4958-9bb5-b70b9b63953b" (UID: "1f3420c8-2d3f-4958-9bb5-b70b9b63953b"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 13:10:57.677771 ip-10-0-133-218 kubenswrapper[2580]: I0916 13:10:57.677740    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-52k4w\" (UniqueName: \"kubernetes.io/projected/1f3420c8-2d3f-4958-9bb5-b70b9b63953b-kube-api-access-52k4w\") on node \"ip-10-0-133-218.ec2.internal\" DevicePath \"\""
Sep 16 13:10:57.677771 ip-10-0-133-218 kubenswrapper[2580]: I0916 13:10:57.677767    2580 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/1f3420c8-2d3f-4958-9bb5-b70b9b63953b-kube-rbac-proxy-sar-config\") on node \"ip-10-0-133-218.ec2.internal\" DevicePath \"\""
Sep 16 13:10:57.677939 ip-10-0-133-218 kubenswrapper[2580]: I0916 13:10:57.677782    2580 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/1f3420c8-2d3f-4958-9bb5-b70b9b63953b-cabundle-cert\") on node \"ip-10-0-133-218.ec2.internal\" DevicePath \"\""
Sep 16 13:10:57.677939 ip-10-0-133-218 kubenswrapper[2580]: I0916 13:10:57.677794    2580 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/1f3420c8-2d3f-4958-9bb5-b70b9b63953b-kserve-provision-location\") on node \"ip-10-0-133-218.ec2.internal\" DevicePath \"\""
Sep 16 13:10:57.677939 ip-10-0-133-218 kubenswrapper[2580]: I0916 13:10:57.677806    2580 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/1f3420c8-2d3f-4958-9bb5-b70b9b63953b-proxy-tls\") on node \"ip-10-0-133-218.ec2.internal\" DevicePath \"\""
Sep 16 13:10:58.297203 ip-10-0-133-218 kubenswrapper[2580]: I0916 13:10:58.297170    2580 generic.go:356] "Generic (PLEG): container finished" podID="1f3420c8-2d3f-4958-9bb5-b70b9b63953b" containerID="1f2a07e10545e7206d274368a83b9b96f9c4cfe2c7521709c4dc137251faa174" exitCode=0
Sep 16 13:10:58.297506 ip-10-0-133-218 kubenswrapper[2580]: I0916 13:10:58.297252    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-8bc5b857-zkjp5"
Sep 16 13:10:58.297506 ip-10-0-133-218 kubenswrapper[2580]: I0916 13:10:58.297251    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-8bc5b857-zkjp5" event={"ID":"1f3420c8-2d3f-4958-9bb5-b70b9b63953b","Type":"ContainerDied","Data":"1f2a07e10545e7206d274368a83b9b96f9c4cfe2c7521709c4dc137251faa174"}
Sep 16 13:10:58.297506 ip-10-0-133-218 kubenswrapper[2580]: I0916 13:10:58.297362    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-8bc5b857-zkjp5" event={"ID":"1f3420c8-2d3f-4958-9bb5-b70b9b63953b","Type":"ContainerDied","Data":"a4d331c9af3e83b838aa1727b8d00ce9b5975282ad0bd009a01243aff03003d3"}
Sep 16 13:10:58.297506 ip-10-0-133-218 kubenswrapper[2580]: I0916 13:10:58.297404    2580 scope.go:117] "RemoveContainer" containerID="565ed39389087d5114557f6ae21c36f408909994c93f7d156a1fe966081a9214"
Sep 16 13:10:58.298809 ip-10-0-133-218 kubenswrapper[2580]: I0916 13:10:58.298781    2580 generic.go:356] "Generic (PLEG): container finished" podID="c40fd0b4-186c-49e3-a6d9-4fa18505145f" containerID="a4584541bd6e85fd139f68510aac9daeb2a480bee11c74ff5d0e963af8ceb116" exitCode=0
Sep 16 13:10:58.298934 ip-10-0-133-218 kubenswrapper[2580]: I0916 13:10:58.298850    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-65744c97b6-rkjp8" event={"ID":"c40fd0b4-186c-49e3-a6d9-4fa18505145f","Type":"ContainerDied","Data":"a4584541bd6e85fd139f68510aac9daeb2a480bee11c74ff5d0e963af8ceb116"}
Sep 16 13:10:58.307085 ip-10-0-133-218 kubenswrapper[2580]: I0916 13:10:58.307066    2580 scope.go:117] "RemoveContainer" containerID="1f2a07e10545e7206d274368a83b9b96f9c4cfe2c7521709c4dc137251faa174"
Sep 16 13:10:58.315608 ip-10-0-133-218 kubenswrapper[2580]: I0916 13:10:58.314984    2580 scope.go:117] "RemoveContainer" containerID="f58eeafe836c249775f2f93a826fd70a7f9833e3b11a82426389004bf251e260"
Sep 16 13:10:58.317060 ip-10-0-133-218 kubenswrapper[2580]: I0916 13:10:58.317037    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-predictor-8bc5b857-zkjp5"]
Sep 16 13:10:58.320966 ip-10-0-133-218 kubenswrapper[2580]: I0916 13:10:58.320947    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-predictor-8bc5b857-zkjp5"]
Sep 16 13:10:58.327801 ip-10-0-133-218 kubenswrapper[2580]: I0916 13:10:58.327780    2580 scope.go:117] "RemoveContainer" containerID="565ed39389087d5114557f6ae21c36f408909994c93f7d156a1fe966081a9214"
Sep 16 13:10:58.328092 ip-10-0-133-218 kubenswrapper[2580]: E0916 13:10:58.328073    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"565ed39389087d5114557f6ae21c36f408909994c93f7d156a1fe966081a9214\": container with ID starting with 565ed39389087d5114557f6ae21c36f408909994c93f7d156a1fe966081a9214 not found: ID does not exist" containerID="565ed39389087d5114557f6ae21c36f408909994c93f7d156a1fe966081a9214"
Sep 16 13:10:58.328152 ip-10-0-133-218 kubenswrapper[2580]: I0916 13:10:58.328099    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"565ed39389087d5114557f6ae21c36f408909994c93f7d156a1fe966081a9214"} err="failed to get container status \"565ed39389087d5114557f6ae21c36f408909994c93f7d156a1fe966081a9214\": rpc error: code = NotFound desc = could not find container \"565ed39389087d5114557f6ae21c36f408909994c93f7d156a1fe966081a9214\": container with ID starting with 565ed39389087d5114557f6ae21c36f408909994c93f7d156a1fe966081a9214 not found: ID does not exist"
Sep 16 13:10:58.328152 ip-10-0-133-218 kubenswrapper[2580]: I0916 13:10:58.328116    2580 scope.go:117] "RemoveContainer" containerID="1f2a07e10545e7206d274368a83b9b96f9c4cfe2c7521709c4dc137251faa174"
Sep 16 13:10:58.328335 ip-10-0-133-218 kubenswrapper[2580]: E0916 13:10:58.328315    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1f2a07e10545e7206d274368a83b9b96f9c4cfe2c7521709c4dc137251faa174\": container with ID starting with 1f2a07e10545e7206d274368a83b9b96f9c4cfe2c7521709c4dc137251faa174 not found: ID does not exist" containerID="1f2a07e10545e7206d274368a83b9b96f9c4cfe2c7521709c4dc137251faa174"
Sep 16 13:10:58.328401 ip-10-0-133-218 kubenswrapper[2580]: I0916 13:10:58.328344    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1f2a07e10545e7206d274368a83b9b96f9c4cfe2c7521709c4dc137251faa174"} err="failed to get container status \"1f2a07e10545e7206d274368a83b9b96f9c4cfe2c7521709c4dc137251faa174\": rpc error: code = NotFound desc = could not find container \"1f2a07e10545e7206d274368a83b9b96f9c4cfe2c7521709c4dc137251faa174\": container with ID starting with 1f2a07e10545e7206d274368a83b9b96f9c4cfe2c7521709c4dc137251faa174 not found: ID does not exist"
Sep 16 13:10:58.328401 ip-10-0-133-218 kubenswrapper[2580]: I0916 13:10:58.328366    2580 scope.go:117] "RemoveContainer" containerID="f58eeafe836c249775f2f93a826fd70a7f9833e3b11a82426389004bf251e260"
Sep 16 13:10:58.328599 ip-10-0-133-218 kubenswrapper[2580]: E0916 13:10:58.328583    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f58eeafe836c249775f2f93a826fd70a7f9833e3b11a82426389004bf251e260\": container with ID starting with f58eeafe836c249775f2f93a826fd70a7f9833e3b11a82426389004bf251e260 not found: ID does not exist" containerID="f58eeafe836c249775f2f93a826fd70a7f9833e3b11a82426389004bf251e260"
Sep 16 13:10:58.328645 ip-10-0-133-218 kubenswrapper[2580]: I0916 13:10:58.328602    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f58eeafe836c249775f2f93a826fd70a7f9833e3b11a82426389004bf251e260"} err="failed to get container status \"f58eeafe836c249775f2f93a826fd70a7f9833e3b11a82426389004bf251e260\": rpc error: code = NotFound desc = could not find container \"f58eeafe836c249775f2f93a826fd70a7f9833e3b11a82426389004bf251e260\": container with ID starting with f58eeafe836c249775f2f93a826fd70a7f9833e3b11a82426389004bf251e260 not found: ID does not exist"
Sep 16 13:10:59.305908 ip-10-0-133-218 kubenswrapper[2580]: I0916 13:10:59.305877    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-65744c97b6-rkjp8" event={"ID":"c40fd0b4-186c-49e3-a6d9-4fa18505145f","Type":"ContainerStarted","Data":"3c51a7f222468d110428f05356b775783b61d7e37615236c1d225d0daa1517a9"}
Sep 16 13:10:59.306311 ip-10-0-133-218 kubenswrapper[2580]: I0916 13:10:59.305959    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-65744c97b6-rkjp8" event={"ID":"c40fd0b4-186c-49e3-a6d9-4fa18505145f","Type":"ContainerStarted","Data":"b06c975558ef24a17f3ba5fc94f806882e4bd42d6b105094878e429bb547a481"}
Sep 16 13:10:59.306311 ip-10-0-133-218 kubenswrapper[2580]: I0916 13:10:59.306227    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-65744c97b6-rkjp8"
Sep 16 13:10:59.325095 ip-10-0-133-218 kubenswrapper[2580]: I0916 13:10:59.325045    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-65744c97b6-rkjp8" podStartSLOduration=3.608004328 podStartE2EDuration="7.325031736s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.717027408s" isStatefulPod=false podCreationTimestamp="2026-09-16 13:10:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 13:10:59.324273422 +0000 UTC m=+1398.247146032" watchObservedRunningTime="2026-09-16 13:10:59.325031736 +0000 UTC m=+1398.247904206"
Sep 16 13:10:59.599595 ip-10-0-133-218 kubenswrapper[2580]: I0916 13:10:59.599512    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1f3420c8-2d3f-4958-9bb5-b70b9b63953b" path="/var/lib/kubelet/pods/1f3420c8-2d3f-4958-9bb5-b70b9b63953b/volumes"
Sep 16 13:11:00.309854 ip-10-0-133-218 kubenswrapper[2580]: I0916 13:11:00.309822    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-65744c97b6-rkjp8"
Sep 16 13:11:00.311248 ip-10-0-133-218 kubenswrapper[2580]: I0916 13:11:00.311220    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-65744c97b6-rkjp8" podUID="c40fd0b4-186c-49e3-a6d9-4fa18505145f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.38:8080: connect: connection refused"
Sep 16 13:11:01.313246 ip-10-0-133-218 kubenswrapper[2580]: I0916 13:11:01.313210    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-65744c97b6-rkjp8" podUID="c40fd0b4-186c-49e3-a6d9-4fa18505145f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.38:8080: connect: connection refused"
Sep 16 13:11:06.318473 ip-10-0-133-218 kubenswrapper[2580]: I0916 13:11:06.318444    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-65744c97b6-rkjp8"
Sep 16 13:11:06.318996 ip-10-0-133-218 kubenswrapper[2580]: I0916 13:11:06.318969    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-65744c97b6-rkjp8" podUID="c40fd0b4-186c-49e3-a6d9-4fa18505145f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.38:8080: connect: connection refused"
Sep 16 13:11:16.319102 ip-10-0-133-218 kubenswrapper[2580]: I0916 13:11:16.319058    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-65744c97b6-rkjp8" podUID="c40fd0b4-186c-49e3-a6d9-4fa18505145f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.38:8080: connect: connection refused"
Sep 16 13:11:26.319505 ip-10-0-133-218 kubenswrapper[2580]: I0916 13:11:26.319445    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-65744c97b6-rkjp8" podUID="c40fd0b4-186c-49e3-a6d9-4fa18505145f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.38:8080: connect: connection refused"
Sep 16 13:11:36.319720 ip-10-0-133-218 kubenswrapper[2580]: I0916 13:11:36.319623    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-65744c97b6-rkjp8" podUID="c40fd0b4-186c-49e3-a6d9-4fa18505145f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.38:8080: connect: connection refused"
Sep 16 13:11:46.319597 ip-10-0-133-218 kubenswrapper[2580]: I0916 13:11:46.319553    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-65744c97b6-rkjp8" podUID="c40fd0b4-186c-49e3-a6d9-4fa18505145f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.38:8080: connect: connection refused"
Sep 16 13:11:56.319896 ip-10-0-133-218 kubenswrapper[2580]: I0916 13:11:56.319856    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-65744c97b6-rkjp8" podUID="c40fd0b4-186c-49e3-a6d9-4fa18505145f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.38:8080: connect: connection refused"
Sep 16 13:12:06.319265 ip-10-0-133-218 kubenswrapper[2580]: I0916 13:12:06.319218    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-65744c97b6-rkjp8" podUID="c40fd0b4-186c-49e3-a6d9-4fa18505145f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.38:8080: connect: connection refused"
Sep 16 13:12:16.319668 ip-10-0-133-218 kubenswrapper[2580]: I0916 13:12:16.319637    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-65744c97b6-rkjp8"
Sep 16 13:12:23.148252 ip-10-0-133-218 kubenswrapper[2580]: I0916 13:12:23.148216    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-65744c97b6-rkjp8"]
Sep 16 13:12:23.148818 ip-10-0-133-218 kubenswrapper[2580]: I0916 13:12:23.148547    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-65744c97b6-rkjp8" podUID="c40fd0b4-186c-49e3-a6d9-4fa18505145f" containerName="kserve-container" containerID="cri-o://b06c975558ef24a17f3ba5fc94f806882e4bd42d6b105094878e429bb547a481" gracePeriod=30
Sep 16 13:12:23.148925 ip-10-0-133-218 kubenswrapper[2580]: I0916 13:12:23.148874    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-65744c97b6-rkjp8" podUID="c40fd0b4-186c-49e3-a6d9-4fa18505145f" containerName="kube-rbac-proxy" containerID="cri-o://3c51a7f222468d110428f05356b775783b61d7e37615236c1d225d0daa1517a9" gracePeriod=30
Sep 16 13:12:23.627925 ip-10-0-133-218 kubenswrapper[2580]: I0916 13:12:23.627889    2580 generic.go:356] "Generic (PLEG): container finished" podID="c40fd0b4-186c-49e3-a6d9-4fa18505145f" containerID="3c51a7f222468d110428f05356b775783b61d7e37615236c1d225d0daa1517a9" exitCode=2
Sep 16 13:12:23.628100 ip-10-0-133-218 kubenswrapper[2580]: I0916 13:12:23.627948    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-65744c97b6-rkjp8" event={"ID":"c40fd0b4-186c-49e3-a6d9-4fa18505145f","Type":"ContainerDied","Data":"3c51a7f222468d110428f05356b775783b61d7e37615236c1d225d0daa1517a9"}
Sep 16 13:12:26.314527 ip-10-0-133-218 kubenswrapper[2580]: I0916 13:12:26.314453    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-65744c97b6-rkjp8" podUID="c40fd0b4-186c-49e3-a6d9-4fa18505145f" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.38:8643/healthz\": dial tcp 10.133.0.38:8643: connect: connection refused"
Sep 16 13:12:26.319128 ip-10-0-133-218 kubenswrapper[2580]: I0916 13:12:26.319103    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-65744c97b6-rkjp8" podUID="c40fd0b4-186c-49e3-a6d9-4fa18505145f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.38:8080: connect: connection refused"
Sep 16 13:12:27.785821 ip-10-0-133-218 kubenswrapper[2580]: I0916 13:12:27.785797    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-65744c97b6-rkjp8"
Sep 16 13:12:27.805397 ip-10-0-133-218 kubenswrapper[2580]: I0916 13:12:27.805371    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-77mxl\" (UniqueName: \"kubernetes.io/projected/c40fd0b4-186c-49e3-a6d9-4fa18505145f-kube-api-access-77mxl\") pod \"c40fd0b4-186c-49e3-a6d9-4fa18505145f\" (UID: \"c40fd0b4-186c-49e3-a6d9-4fa18505145f\") "
Sep 16 13:12:27.805557 ip-10-0-133-218 kubenswrapper[2580]: I0916 13:12:27.805417    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c40fd0b4-186c-49e3-a6d9-4fa18505145f-proxy-tls\") pod \"c40fd0b4-186c-49e3-a6d9-4fa18505145f\" (UID: \"c40fd0b4-186c-49e3-a6d9-4fa18505145f\") "
Sep 16 13:12:27.805557 ip-10-0-133-218 kubenswrapper[2580]: I0916 13:12:27.805461    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c40fd0b4-186c-49e3-a6d9-4fa18505145f-kserve-provision-location\") pod \"c40fd0b4-186c-49e3-a6d9-4fa18505145f\" (UID: \"c40fd0b4-186c-49e3-a6d9-4fa18505145f\") "
Sep 16 13:12:27.805557 ip-10-0-133-218 kubenswrapper[2580]: I0916 13:12:27.805527    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/c40fd0b4-186c-49e3-a6d9-4fa18505145f-cabundle-cert\") pod \"c40fd0b4-186c-49e3-a6d9-4fa18505145f\" (UID: \"c40fd0b4-186c-49e3-a6d9-4fa18505145f\") "
Sep 16 13:12:27.805744 ip-10-0-133-218 kubenswrapper[2580]: I0916 13:12:27.805568    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/c40fd0b4-186c-49e3-a6d9-4fa18505145f-kube-rbac-proxy-sar-config\") pod \"c40fd0b4-186c-49e3-a6d9-4fa18505145f\" (UID: \"c40fd0b4-186c-49e3-a6d9-4fa18505145f\") "
Sep 16 13:12:27.806185 ip-10-0-133-218 kubenswrapper[2580]: I0916 13:12:27.806157    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c40fd0b4-186c-49e3-a6d9-4fa18505145f-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "c40fd0b4-186c-49e3-a6d9-4fa18505145f" (UID: "c40fd0b4-186c-49e3-a6d9-4fa18505145f"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 13:12:27.806185 ip-10-0-133-218 kubenswrapper[2580]: I0916 13:12:27.806168    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c40fd0b4-186c-49e3-a6d9-4fa18505145f-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "c40fd0b4-186c-49e3-a6d9-4fa18505145f" (UID: "c40fd0b4-186c-49e3-a6d9-4fa18505145f"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 13:12:27.806330 ip-10-0-133-218 kubenswrapper[2580]: I0916 13:12:27.806303    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c40fd0b4-186c-49e3-a6d9-4fa18505145f-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "c40fd0b4-186c-49e3-a6d9-4fa18505145f" (UID: "c40fd0b4-186c-49e3-a6d9-4fa18505145f"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 13:12:27.807835 ip-10-0-133-218 kubenswrapper[2580]: I0916 13:12:27.807811    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c40fd0b4-186c-49e3-a6d9-4fa18505145f-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "c40fd0b4-186c-49e3-a6d9-4fa18505145f" (UID: "c40fd0b4-186c-49e3-a6d9-4fa18505145f"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 13:12:27.807927 ip-10-0-133-218 kubenswrapper[2580]: I0916 13:12:27.807868    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c40fd0b4-186c-49e3-a6d9-4fa18505145f-kube-api-access-77mxl" (OuterVolumeSpecName: "kube-api-access-77mxl") pod "c40fd0b4-186c-49e3-a6d9-4fa18505145f" (UID: "c40fd0b4-186c-49e3-a6d9-4fa18505145f"). InnerVolumeSpecName "kube-api-access-77mxl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 13:12:27.906445 ip-10-0-133-218 kubenswrapper[2580]: I0916 13:12:27.906369    2580 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/c40fd0b4-186c-49e3-a6d9-4fa18505145f-cabundle-cert\") on node \"ip-10-0-133-218.ec2.internal\" DevicePath \"\""
Sep 16 13:12:27.906445 ip-10-0-133-218 kubenswrapper[2580]: I0916 13:12:27.906396    2580 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/c40fd0b4-186c-49e3-a6d9-4fa18505145f-kube-rbac-proxy-sar-config\") on node \"ip-10-0-133-218.ec2.internal\" DevicePath \"\""
Sep 16 13:12:27.906445 ip-10-0-133-218 kubenswrapper[2580]: I0916 13:12:27.906405    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-77mxl\" (UniqueName: \"kubernetes.io/projected/c40fd0b4-186c-49e3-a6d9-4fa18505145f-kube-api-access-77mxl\") on node \"ip-10-0-133-218.ec2.internal\" DevicePath \"\""
Sep 16 13:12:27.906445 ip-10-0-133-218 kubenswrapper[2580]: I0916 13:12:27.906414    2580 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c40fd0b4-186c-49e3-a6d9-4fa18505145f-proxy-tls\") on node \"ip-10-0-133-218.ec2.internal\" DevicePath \"\""
Sep 16 13:12:27.906445 ip-10-0-133-218 kubenswrapper[2580]: I0916 13:12:27.906423    2580 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c40fd0b4-186c-49e3-a6d9-4fa18505145f-kserve-provision-location\") on node \"ip-10-0-133-218.ec2.internal\" DevicePath \"\""
Sep 16 13:12:28.647800 ip-10-0-133-218 kubenswrapper[2580]: I0916 13:12:28.647761    2580 generic.go:356] "Generic (PLEG): container finished" podID="c40fd0b4-186c-49e3-a6d9-4fa18505145f" containerID="b06c975558ef24a17f3ba5fc94f806882e4bd42d6b105094878e429bb547a481" exitCode=0
Sep 16 13:12:28.648101 ip-10-0-133-218 kubenswrapper[2580]: I0916 13:12:28.647822    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-65744c97b6-rkjp8"
Sep 16 13:12:28.648101 ip-10-0-133-218 kubenswrapper[2580]: I0916 13:12:28.647818    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-65744c97b6-rkjp8" event={"ID":"c40fd0b4-186c-49e3-a6d9-4fa18505145f","Type":"ContainerDied","Data":"b06c975558ef24a17f3ba5fc94f806882e4bd42d6b105094878e429bb547a481"}
Sep 16 13:12:28.648101 ip-10-0-133-218 kubenswrapper[2580]: I0916 13:12:28.647949    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-65744c97b6-rkjp8" event={"ID":"c40fd0b4-186c-49e3-a6d9-4fa18505145f","Type":"ContainerDied","Data":"ced251ae2cad88d01e9e002afaad6b939ce5ccaab24da805cc1981d132348b47"}
Sep 16 13:12:28.648101 ip-10-0-133-218 kubenswrapper[2580]: I0916 13:12:28.647977    2580 scope.go:117] "RemoveContainer" containerID="3c51a7f222468d110428f05356b775783b61d7e37615236c1d225d0daa1517a9"
Sep 16 13:12:28.656345 ip-10-0-133-218 kubenswrapper[2580]: I0916 13:12:28.656329    2580 scope.go:117] "RemoveContainer" containerID="b06c975558ef24a17f3ba5fc94f806882e4bd42d6b105094878e429bb547a481"
Sep 16 13:12:28.663554 ip-10-0-133-218 kubenswrapper[2580]: I0916 13:12:28.663538    2580 scope.go:117] "RemoveContainer" containerID="a4584541bd6e85fd139f68510aac9daeb2a480bee11c74ff5d0e963af8ceb116"
Sep 16 13:12:28.668171 ip-10-0-133-218 kubenswrapper[2580]: I0916 13:12:28.668152    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-65744c97b6-rkjp8"]
Sep 16 13:12:28.671459 ip-10-0-133-218 kubenswrapper[2580]: I0916 13:12:28.671285    2580 scope.go:117] "RemoveContainer" containerID="3c51a7f222468d110428f05356b775783b61d7e37615236c1d225d0daa1517a9"
Sep 16 13:12:28.671617 ip-10-0-133-218 kubenswrapper[2580]: E0916 13:12:28.671599    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3c51a7f222468d110428f05356b775783b61d7e37615236c1d225d0daa1517a9\": container with ID starting with 3c51a7f222468d110428f05356b775783b61d7e37615236c1d225d0daa1517a9 not found: ID does not exist" containerID="3c51a7f222468d110428f05356b775783b61d7e37615236c1d225d0daa1517a9"
Sep 16 13:12:28.671670 ip-10-0-133-218 kubenswrapper[2580]: I0916 13:12:28.671627    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3c51a7f222468d110428f05356b775783b61d7e37615236c1d225d0daa1517a9"} err="failed to get container status \"3c51a7f222468d110428f05356b775783b61d7e37615236c1d225d0daa1517a9\": rpc error: code = NotFound desc = could not find container \"3c51a7f222468d110428f05356b775783b61d7e37615236c1d225d0daa1517a9\": container with ID starting with 3c51a7f222468d110428f05356b775783b61d7e37615236c1d225d0daa1517a9 not found: ID does not exist"
Sep 16 13:12:28.671670 ip-10-0-133-218 kubenswrapper[2580]: I0916 13:12:28.671645    2580 scope.go:117] "RemoveContainer" containerID="b06c975558ef24a17f3ba5fc94f806882e4bd42d6b105094878e429bb547a481"
Sep 16 13:12:28.671749 ip-10-0-133-218 kubenswrapper[2580]: I0916 13:12:28.671710    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-65744c97b6-rkjp8"]
Sep 16 13:12:28.671906 ip-10-0-133-218 kubenswrapper[2580]: E0916 13:12:28.671889    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b06c975558ef24a17f3ba5fc94f806882e4bd42d6b105094878e429bb547a481\": container with ID starting with b06c975558ef24a17f3ba5fc94f806882e4bd42d6b105094878e429bb547a481 not found: ID does not exist" containerID="b06c975558ef24a17f3ba5fc94f806882e4bd42d6b105094878e429bb547a481"
Sep 16 13:12:28.671974 ip-10-0-133-218 kubenswrapper[2580]: I0916 13:12:28.671915    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b06c975558ef24a17f3ba5fc94f806882e4bd42d6b105094878e429bb547a481"} err="failed to get container status \"b06c975558ef24a17f3ba5fc94f806882e4bd42d6b105094878e429bb547a481\": rpc error: code = NotFound desc = could not find container \"b06c975558ef24a17f3ba5fc94f806882e4bd42d6b105094878e429bb547a481\": container with ID starting with b06c975558ef24a17f3ba5fc94f806882e4bd42d6b105094878e429bb547a481 not found: ID does not exist"
Sep 16 13:12:28.671974 ip-10-0-133-218 kubenswrapper[2580]: I0916 13:12:28.671936    2580 scope.go:117] "RemoveContainer" containerID="a4584541bd6e85fd139f68510aac9daeb2a480bee11c74ff5d0e963af8ceb116"
Sep 16 13:12:28.672179 ip-10-0-133-218 kubenswrapper[2580]: E0916 13:12:28.672161    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a4584541bd6e85fd139f68510aac9daeb2a480bee11c74ff5d0e963af8ceb116\": container with ID starting with a4584541bd6e85fd139f68510aac9daeb2a480bee11c74ff5d0e963af8ceb116 not found: ID does not exist" containerID="a4584541bd6e85fd139f68510aac9daeb2a480bee11c74ff5d0e963af8ceb116"
Sep 16 13:12:28.672223 ip-10-0-133-218 kubenswrapper[2580]: I0916 13:12:28.672183    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a4584541bd6e85fd139f68510aac9daeb2a480bee11c74ff5d0e963af8ceb116"} err="failed to get container status \"a4584541bd6e85fd139f68510aac9daeb2a480bee11c74ff5d0e963af8ceb116\": rpc error: code = NotFound desc = could not find container \"a4584541bd6e85fd139f68510aac9daeb2a480bee11c74ff5d0e963af8ceb116\": container with ID starting with a4584541bd6e85fd139f68510aac9daeb2a480bee11c74ff5d0e963af8ceb116 not found: ID does not exist"
Sep 16 13:12:29.599738 ip-10-0-133-218 kubenswrapper[2580]: I0916 13:12:29.599701    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c40fd0b4-186c-49e3-a6d9-4fa18505145f" path="/var/lib/kubelet/pods/c40fd0b4-186c-49e3-a6d9-4fa18505145f/volumes"
Sep 16 13:14:23.934530 ip-10-0-133-218 kubenswrapper[2580]: I0916 13:14:23.934477    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-6c97j/must-gather-42r4r"]
Sep 16 13:14:23.937120 ip-10-0-133-218 kubenswrapper[2580]: I0916 13:14:23.934786    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1f3420c8-2d3f-4958-9bb5-b70b9b63953b" containerName="storage-initializer"
Sep 16 13:14:23.937120 ip-10-0-133-218 kubenswrapper[2580]: I0916 13:14:23.934797    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="1f3420c8-2d3f-4958-9bb5-b70b9b63953b" containerName="storage-initializer"
Sep 16 13:14:23.937120 ip-10-0-133-218 kubenswrapper[2580]: I0916 13:14:23.934806    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1f3420c8-2d3f-4958-9bb5-b70b9b63953b" containerName="kserve-container"
Sep 16 13:14:23.937120 ip-10-0-133-218 kubenswrapper[2580]: I0916 13:14:23.934812    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="1f3420c8-2d3f-4958-9bb5-b70b9b63953b" containerName="kserve-container"
Sep 16 13:14:23.937120 ip-10-0-133-218 kubenswrapper[2580]: I0916 13:14:23.934826    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c40fd0b4-186c-49e3-a6d9-4fa18505145f" containerName="storage-initializer"
Sep 16 13:14:23.937120 ip-10-0-133-218 kubenswrapper[2580]: I0916 13:14:23.934831    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="c40fd0b4-186c-49e3-a6d9-4fa18505145f" containerName="storage-initializer"
Sep 16 13:14:23.937120 ip-10-0-133-218 kubenswrapper[2580]: I0916 13:14:23.934838    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c40fd0b4-186c-49e3-a6d9-4fa18505145f" containerName="kserve-container"
Sep 16 13:14:23.937120 ip-10-0-133-218 kubenswrapper[2580]: I0916 13:14:23.934843    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="c40fd0b4-186c-49e3-a6d9-4fa18505145f" containerName="kserve-container"
Sep 16 13:14:23.937120 ip-10-0-133-218 kubenswrapper[2580]: I0916 13:14:23.934849    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c40fd0b4-186c-49e3-a6d9-4fa18505145f" containerName="kube-rbac-proxy"
Sep 16 13:14:23.937120 ip-10-0-133-218 kubenswrapper[2580]: I0916 13:14:23.934854    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="c40fd0b4-186c-49e3-a6d9-4fa18505145f" containerName="kube-rbac-proxy"
Sep 16 13:14:23.937120 ip-10-0-133-218 kubenswrapper[2580]: I0916 13:14:23.934862    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1f3420c8-2d3f-4958-9bb5-b70b9b63953b" containerName="kube-rbac-proxy"
Sep 16 13:14:23.937120 ip-10-0-133-218 kubenswrapper[2580]: I0916 13:14:23.934868    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="1f3420c8-2d3f-4958-9bb5-b70b9b63953b" containerName="kube-rbac-proxy"
Sep 16 13:14:23.937120 ip-10-0-133-218 kubenswrapper[2580]: I0916 13:14:23.934921    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="1f3420c8-2d3f-4958-9bb5-b70b9b63953b" containerName="kserve-container"
Sep 16 13:14:23.937120 ip-10-0-133-218 kubenswrapper[2580]: I0916 13:14:23.934937    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="c40fd0b4-186c-49e3-a6d9-4fa18505145f" containerName="kserve-container"
Sep 16 13:14:23.937120 ip-10-0-133-218 kubenswrapper[2580]: I0916 13:14:23.934945    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="c40fd0b4-186c-49e3-a6d9-4fa18505145f" containerName="kube-rbac-proxy"
Sep 16 13:14:23.937120 ip-10-0-133-218 kubenswrapper[2580]: I0916 13:14:23.934952    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="1f3420c8-2d3f-4958-9bb5-b70b9b63953b" containerName="kube-rbac-proxy"
Sep 16 13:14:23.938160 ip-10-0-133-218 kubenswrapper[2580]: I0916 13:14:23.938137    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-6c97j/must-gather-42r4r"
Sep 16 13:14:23.940700 ip-10-0-133-218 kubenswrapper[2580]: I0916 13:14:23.940676    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-6c97j\"/\"kube-root-ca.crt\""
Sep 16 13:14:23.940834 ip-10-0-133-218 kubenswrapper[2580]: I0916 13:14:23.940703    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-6c97j\"/\"default-dockercfg-kzfnk\""
Sep 16 13:14:23.940834 ip-10-0-133-218 kubenswrapper[2580]: I0916 13:14:23.940704    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-6c97j\"/\"openshift-service-ca.crt\""
Sep 16 13:14:23.945867 ip-10-0-133-218 kubenswrapper[2580]: I0916 13:14:23.945849    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-6c97j/must-gather-42r4r"]
Sep 16 13:14:24.019041 ip-10-0-133-218 kubenswrapper[2580]: I0916 13:14:24.019006    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/211b47fa-c759-4703-b0ad-e935e5b44cbd-must-gather-output\") pod \"must-gather-42r4r\" (UID: \"211b47fa-c759-4703-b0ad-e935e5b44cbd\") " pod="openshift-must-gather-6c97j/must-gather-42r4r"
Sep 16 13:14:24.019041 ip-10-0-133-218 kubenswrapper[2580]: I0916 13:14:24.019039    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-276mw\" (UniqueName: \"kubernetes.io/projected/211b47fa-c759-4703-b0ad-e935e5b44cbd-kube-api-access-276mw\") pod \"must-gather-42r4r\" (UID: \"211b47fa-c759-4703-b0ad-e935e5b44cbd\") " pod="openshift-must-gather-6c97j/must-gather-42r4r"
Sep 16 13:14:24.120306 ip-10-0-133-218 kubenswrapper[2580]: I0916 13:14:24.120258    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/211b47fa-c759-4703-b0ad-e935e5b44cbd-must-gather-output\") pod \"must-gather-42r4r\" (UID: \"211b47fa-c759-4703-b0ad-e935e5b44cbd\") " pod="openshift-must-gather-6c97j/must-gather-42r4r"
Sep 16 13:14:24.120306 ip-10-0-133-218 kubenswrapper[2580]: I0916 13:14:24.120305    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-276mw\" (UniqueName: \"kubernetes.io/projected/211b47fa-c759-4703-b0ad-e935e5b44cbd-kube-api-access-276mw\") pod \"must-gather-42r4r\" (UID: \"211b47fa-c759-4703-b0ad-e935e5b44cbd\") " pod="openshift-must-gather-6c97j/must-gather-42r4r"
Sep 16 13:14:24.120619 ip-10-0-133-218 kubenswrapper[2580]: I0916 13:14:24.120600    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/211b47fa-c759-4703-b0ad-e935e5b44cbd-must-gather-output\") pod \"must-gather-42r4r\" (UID: \"211b47fa-c759-4703-b0ad-e935e5b44cbd\") " pod="openshift-must-gather-6c97j/must-gather-42r4r"
Sep 16 13:14:24.131816 ip-10-0-133-218 kubenswrapper[2580]: I0916 13:14:24.131785    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-276mw\" (UniqueName: \"kubernetes.io/projected/211b47fa-c759-4703-b0ad-e935e5b44cbd-kube-api-access-276mw\") pod \"must-gather-42r4r\" (UID: \"211b47fa-c759-4703-b0ad-e935e5b44cbd\") " pod="openshift-must-gather-6c97j/must-gather-42r4r"
Sep 16 13:14:24.269267 ip-10-0-133-218 kubenswrapper[2580]: I0916 13:14:24.269188    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-6c97j/must-gather-42r4r"
Sep 16 13:14:24.540338 ip-10-0-133-218 kubenswrapper[2580]: I0916 13:14:24.540316    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-6c97j/must-gather-42r4r"]
Sep 16 13:14:24.542910 ip-10-0-133-218 kubenswrapper[2580]: W0916 13:14:24.542882    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod211b47fa_c759_4703_b0ad_e935e5b44cbd.slice/crio-418ab7c49cbbf6ff045cdc4c6fb898e38cac01c86f5096b95e289daffb2e91b6 WatchSource:0}: Error finding container 418ab7c49cbbf6ff045cdc4c6fb898e38cac01c86f5096b95e289daffb2e91b6: Status 404 returned error can't find the container with id 418ab7c49cbbf6ff045cdc4c6fb898e38cac01c86f5096b95e289daffb2e91b6
Sep 16 13:14:24.544575 ip-10-0-133-218 kubenswrapper[2580]: I0916 13:14:24.544559    2580 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 16 13:14:25.083418 ip-10-0-133-218 kubenswrapper[2580]: I0916 13:14:25.083383    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-6c97j/must-gather-42r4r" event={"ID":"211b47fa-c759-4703-b0ad-e935e5b44cbd","Type":"ContainerStarted","Data":"418ab7c49cbbf6ff045cdc4c6fb898e38cac01c86f5096b95e289daffb2e91b6"}
Sep 16 13:14:31.117686 ip-10-0-133-218 kubenswrapper[2580]: I0916 13:14:31.117648    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-6c97j/must-gather-42r4r" event={"ID":"211b47fa-c759-4703-b0ad-e935e5b44cbd","Type":"ContainerStarted","Data":"5508255d74cc6a312fb93fd00a0e94c74e8e753843aae64bfcb2bb285786e4b5"}
Sep 16 13:14:31.117686 ip-10-0-133-218 kubenswrapper[2580]: I0916 13:14:31.117693    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-6c97j/must-gather-42r4r" event={"ID":"211b47fa-c759-4703-b0ad-e935e5b44cbd","Type":"ContainerStarted","Data":"2625c4f5107fc0c63c54b8efac3c54b98cef4e67f73551df27ad251c08c986f6"}
Sep 16 13:14:31.133527 ip-10-0-133-218 kubenswrapper[2580]: I0916 13:14:31.133464    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-6c97j/must-gather-42r4r" podStartSLOduration=2.078003504 podStartE2EDuration="8.133446906s" totalImagesPullingTime="6.055443402s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 13:14:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 13:14:31.132711916 +0000 UTC m=+1610.055584386" watchObservedRunningTime="2026-09-16 13:14:31.133446906 +0000 UTC m=+1610.056319375"
Sep 16 13:14:54.210648 ip-10-0-133-218 kubenswrapper[2580]: I0916 13:14:54.210612    2580 generic.go:356] "Generic (PLEG): container finished" podID="211b47fa-c759-4703-b0ad-e935e5b44cbd" containerID="2625c4f5107fc0c63c54b8efac3c54b98cef4e67f73551df27ad251c08c986f6" exitCode=0
Sep 16 13:14:54.211117 ip-10-0-133-218 kubenswrapper[2580]: I0916 13:14:54.210666    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-6c97j/must-gather-42r4r" event={"ID":"211b47fa-c759-4703-b0ad-e935e5b44cbd","Type":"ContainerDied","Data":"2625c4f5107fc0c63c54b8efac3c54b98cef4e67f73551df27ad251c08c986f6"}
Sep 16 13:14:54.211117 ip-10-0-133-218 kubenswrapper[2580]: I0916 13:14:54.210978    2580 scope.go:117] "RemoveContainer" containerID="2625c4f5107fc0c63c54b8efac3c54b98cef4e67f73551df27ad251c08c986f6"
Sep 16 13:14:54.744849 ip-10-0-133-218 kubenswrapper[2580]: I0916 13:14:54.744815    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-must-gather-6c97j_must-gather-42r4r_211b47fa-c759-4703-b0ad-e935e5b44cbd/gather/0.log"
Sep 16 13:14:58.132910 ip-10-0-133-218 kubenswrapper[2580]: I0916 13:14:58.132875    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-5jkh4_fea676c8-6e68-4894-99fc-fe8b59886a9a/global-pull-secret-syncer/0.log"
Sep 16 13:14:58.259436 ip-10-0-133-218 kubenswrapper[2580]: I0916 13:14:58.259403    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-p6wht_decaea8f-b0ad-428a-8880-e1ef3d565796/konnectivity-agent/0.log"
Sep 16 13:14:58.328030 ip-10-0-133-218 kubenswrapper[2580]: I0916 13:14:58.328000    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-133-218.ec2.internal_f3d9df87a80a4a167468bfdb12aa0522/haproxy/0.log"
Sep 16 13:15:00.163506 ip-10-0-133-218 kubenswrapper[2580]: I0916 13:15:00.163453    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-must-gather-6c97j/must-gather-42r4r"]
Sep 16 13:15:00.163916 ip-10-0-133-218 kubenswrapper[2580]: I0916 13:15:00.163669    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-must-gather-6c97j/must-gather-42r4r" podUID="211b47fa-c759-4703-b0ad-e935e5b44cbd" containerName="copy" containerID="cri-o://5508255d74cc6a312fb93fd00a0e94c74e8e753843aae64bfcb2bb285786e4b5" gracePeriod=2
Sep 16 13:15:00.171832 ip-10-0-133-218 kubenswrapper[2580]: I0916 13:15:00.171805    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-must-gather-6c97j/must-gather-42r4r"]
Sep 16 13:15:00.394383 ip-10-0-133-218 kubenswrapper[2580]: I0916 13:15:00.394358    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-must-gather-6c97j_must-gather-42r4r_211b47fa-c759-4703-b0ad-e935e5b44cbd/copy/0.log"
Sep 16 13:15:00.395945 ip-10-0-133-218 kubenswrapper[2580]: I0916 13:15:00.395921    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-6c97j/must-gather-42r4r"
Sep 16 13:15:00.397787 ip-10-0-133-218 kubenswrapper[2580]: E0916 13:15:00.397759    2580 status_manager.go:1018] "Failed to get status for pod" err="pods \"must-gather-42r4r\" is forbidden: User \"system:node:ip-10-0-133-218.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"openshift-must-gather-6c97j\": no relationship found between node 'ip-10-0-133-218.ec2.internal' and this object" podUID="211b47fa-c759-4703-b0ad-e935e5b44cbd" pod="openshift-must-gather-6c97j/must-gather-42r4r"
Sep 16 13:15:00.507005 ip-10-0-133-218 kubenswrapper[2580]: I0916 13:15:00.506901    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-276mw\" (UniqueName: \"kubernetes.io/projected/211b47fa-c759-4703-b0ad-e935e5b44cbd-kube-api-access-276mw\") pod \"211b47fa-c759-4703-b0ad-e935e5b44cbd\" (UID: \"211b47fa-c759-4703-b0ad-e935e5b44cbd\") "
Sep 16 13:15:00.507005 ip-10-0-133-218 kubenswrapper[2580]: I0916 13:15:00.506967    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/211b47fa-c759-4703-b0ad-e935e5b44cbd-must-gather-output\") pod \"211b47fa-c759-4703-b0ad-e935e5b44cbd\" (UID: \"211b47fa-c759-4703-b0ad-e935e5b44cbd\") "
Sep 16 13:15:00.509081 ip-10-0-133-218 kubenswrapper[2580]: I0916 13:15:00.509058    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/211b47fa-c759-4703-b0ad-e935e5b44cbd-kube-api-access-276mw" (OuterVolumeSpecName: "kube-api-access-276mw") pod "211b47fa-c759-4703-b0ad-e935e5b44cbd" (UID: "211b47fa-c759-4703-b0ad-e935e5b44cbd"). InnerVolumeSpecName "kube-api-access-276mw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 13:15:00.509150 ip-10-0-133-218 kubenswrapper[2580]: I0916 13:15:00.509100    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/211b47fa-c759-4703-b0ad-e935e5b44cbd-must-gather-output" (OuterVolumeSpecName: "must-gather-output") pod "211b47fa-c759-4703-b0ad-e935e5b44cbd" (UID: "211b47fa-c759-4703-b0ad-e935e5b44cbd"). InnerVolumeSpecName "must-gather-output". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 13:15:00.607560 ip-10-0-133-218 kubenswrapper[2580]: I0916 13:15:00.607525    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-276mw\" (UniqueName: \"kubernetes.io/projected/211b47fa-c759-4703-b0ad-e935e5b44cbd-kube-api-access-276mw\") on node \"ip-10-0-133-218.ec2.internal\" DevicePath \"\""
Sep 16 13:15:00.607560 ip-10-0-133-218 kubenswrapper[2580]: I0916 13:15:00.607554    2580 reconciler_common.go:299] "Volume detached for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/211b47fa-c759-4703-b0ad-e935e5b44cbd-must-gather-output\") on node \"ip-10-0-133-218.ec2.internal\" DevicePath \"\""
Sep 16 13:15:01.246807 ip-10-0-133-218 kubenswrapper[2580]: I0916 13:15:01.246777    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-must-gather-6c97j_must-gather-42r4r_211b47fa-c759-4703-b0ad-e935e5b44cbd/copy/0.log"
Sep 16 13:15:01.247286 ip-10-0-133-218 kubenswrapper[2580]: I0916 13:15:01.247260    2580 generic.go:356] "Generic (PLEG): container finished" podID="211b47fa-c759-4703-b0ad-e935e5b44cbd" containerID="5508255d74cc6a312fb93fd00a0e94c74e8e753843aae64bfcb2bb285786e4b5" exitCode=143
Sep 16 13:15:01.247386 ip-10-0-133-218 kubenswrapper[2580]: I0916 13:15:01.247315    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-6c97j/must-gather-42r4r"
Sep 16 13:15:01.247386 ip-10-0-133-218 kubenswrapper[2580]: I0916 13:15:01.247325    2580 scope.go:117] "RemoveContainer" containerID="5508255d74cc6a312fb93fd00a0e94c74e8e753843aae64bfcb2bb285786e4b5"
Sep 16 13:15:01.254307 ip-10-0-133-218 kubenswrapper[2580]: E0916 13:15:01.254163    2580 status_manager.go:1018] "Failed to get status for pod" err="pods \"must-gather-42r4r\" is forbidden: User \"system:node:ip-10-0-133-218.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"openshift-must-gather-6c97j\": no relationship found between node 'ip-10-0-133-218.ec2.internal' and this object" podUID="211b47fa-c759-4703-b0ad-e935e5b44cbd" pod="openshift-must-gather-6c97j/must-gather-42r4r"
Sep 16 13:15:01.260420 ip-10-0-133-218 kubenswrapper[2580]: E0916 13:15:01.260393    2580 status_manager.go:1018] "Failed to get status for pod" err="pods \"must-gather-42r4r\" is forbidden: User \"system:node:ip-10-0-133-218.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"openshift-must-gather-6c97j\": no relationship found between node 'ip-10-0-133-218.ec2.internal' and this object" podUID="211b47fa-c759-4703-b0ad-e935e5b44cbd" pod="openshift-must-gather-6c97j/must-gather-42r4r"
Sep 16 13:15:01.260775 ip-10-0-133-218 kubenswrapper[2580]: I0916 13:15:01.260756    2580 scope.go:117] "RemoveContainer" containerID="2625c4f5107fc0c63c54b8efac3c54b98cef4e67f73551df27ad251c08c986f6"
Sep 16 13:15:01.271924 ip-10-0-133-218 kubenswrapper[2580]: I0916 13:15:01.271900    2580 scope.go:117] "RemoveContainer" containerID="5508255d74cc6a312fb93fd00a0e94c74e8e753843aae64bfcb2bb285786e4b5"
Sep 16 13:15:01.272185 ip-10-0-133-218 kubenswrapper[2580]: E0916 13:15:01.272169    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5508255d74cc6a312fb93fd00a0e94c74e8e753843aae64bfcb2bb285786e4b5\": container with ID starting with 5508255d74cc6a312fb93fd00a0e94c74e8e753843aae64bfcb2bb285786e4b5 not found: ID does not exist" containerID="5508255d74cc6a312fb93fd00a0e94c74e8e753843aae64bfcb2bb285786e4b5"
Sep 16 13:15:01.272235 ip-10-0-133-218 kubenswrapper[2580]: I0916 13:15:01.272192    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5508255d74cc6a312fb93fd00a0e94c74e8e753843aae64bfcb2bb285786e4b5"} err="failed to get container status \"5508255d74cc6a312fb93fd00a0e94c74e8e753843aae64bfcb2bb285786e4b5\": rpc error: code = NotFound desc = could not find container \"5508255d74cc6a312fb93fd00a0e94c74e8e753843aae64bfcb2bb285786e4b5\": container with ID starting with 5508255d74cc6a312fb93fd00a0e94c74e8e753843aae64bfcb2bb285786e4b5 not found: ID does not exist"
Sep 16 13:15:01.272235 ip-10-0-133-218 kubenswrapper[2580]: I0916 13:15:01.272207    2580 scope.go:117] "RemoveContainer" containerID="2625c4f5107fc0c63c54b8efac3c54b98cef4e67f73551df27ad251c08c986f6"
Sep 16 13:15:01.272473 ip-10-0-133-218 kubenswrapper[2580]: E0916 13:15:01.272455    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2625c4f5107fc0c63c54b8efac3c54b98cef4e67f73551df27ad251c08c986f6\": container with ID starting with 2625c4f5107fc0c63c54b8efac3c54b98cef4e67f73551df27ad251c08c986f6 not found: ID does not exist" containerID="2625c4f5107fc0c63c54b8efac3c54b98cef4e67f73551df27ad251c08c986f6"
Sep 16 13:15:01.272536 ip-10-0-133-218 kubenswrapper[2580]: I0916 13:15:01.272503    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2625c4f5107fc0c63c54b8efac3c54b98cef4e67f73551df27ad251c08c986f6"} err="failed to get container status \"2625c4f5107fc0c63c54b8efac3c54b98cef4e67f73551df27ad251c08c986f6\": rpc error: code = NotFound desc = could not find container \"2625c4f5107fc0c63c54b8efac3c54b98cef4e67f73551df27ad251c08c986f6\": container with ID starting with 2625c4f5107fc0c63c54b8efac3c54b98cef4e67f73551df27ad251c08c986f6 not found: ID does not exist"
Sep 16 13:15:01.598467 ip-10-0-133-218 kubenswrapper[2580]: E0916 13:15:01.598426    2580 status_manager.go:1018] "Failed to get status for pod" err="pods \"must-gather-42r4r\" is forbidden: User \"system:node:ip-10-0-133-218.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"openshift-must-gather-6c97j\": no relationship found between node 'ip-10-0-133-218.ec2.internal' and this object" podUID="211b47fa-c759-4703-b0ad-e935e5b44cbd" pod="openshift-must-gather-6c97j/must-gather-42r4r"
Sep 16 13:15:01.599648 ip-10-0-133-218 kubenswrapper[2580]: I0916 13:15:01.599629    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="211b47fa-c759-4703-b0ad-e935e5b44cbd" path="/var/lib/kubelet/pods/211b47fa-c759-4703-b0ad-e935e5b44cbd/volumes"
Sep 16 13:15:02.503103 ip-10-0-133-218 kubenswrapper[2580]: I0916 13:15:02.503019    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-59fbdd77d-lbkms_cdef4a92-60a3-42f2-a20b-a15bce25691e/kube-state-metrics/0.log"
Sep 16 13:15:02.529844 ip-10-0-133-218 kubenswrapper[2580]: I0916 13:15:02.529818    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-59fbdd77d-lbkms_cdef4a92-60a3-42f2-a20b-a15bce25691e/kube-rbac-proxy-main/0.log"
Sep 16 13:15:02.552797 ip-10-0-133-218 kubenswrapper[2580]: I0916 13:15:02.552768    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-59fbdd77d-lbkms_cdef4a92-60a3-42f2-a20b-a15bce25691e/kube-rbac-proxy-self/0.log"
Sep 16 13:15:02.694126 ip-10-0-133-218 kubenswrapper[2580]: I0916 13:15:02.694085    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-9tn5t_a0c1e96d-918c-423b-b136-d4c3584fd568/node-exporter/0.log"
Sep 16 13:15:02.724269 ip-10-0-133-218 kubenswrapper[2580]: I0916 13:15:02.724234    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-9tn5t_a0c1e96d-918c-423b-b136-d4c3584fd568/kube-rbac-proxy/0.log"
Sep 16 13:15:02.746062 ip-10-0-133-218 kubenswrapper[2580]: I0916 13:15:02.746038    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-9tn5t_a0c1e96d-918c-423b-b136-d4c3584fd568/init-textfile/0.log"
Sep 16 13:15:02.906839 ip-10-0-133-218 kubenswrapper[2580]: I0916 13:15:02.906814    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-67dcf8c979-22bgc_726a71b5-e75b-4313-a1f9-3d3d123b729c/kube-rbac-proxy-main/0.log"
Sep 16 13:15:02.929124 ip-10-0-133-218 kubenswrapper[2580]: I0916 13:15:02.929094    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-67dcf8c979-22bgc_726a71b5-e75b-4313-a1f9-3d3d123b729c/kube-rbac-proxy-self/0.log"
Sep 16 13:15:02.949307 ip-10-0-133-218 kubenswrapper[2580]: I0916 13:15:02.949278    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-67dcf8c979-22bgc_726a71b5-e75b-4313-a1f9-3d3d123b729c/openshift-state-metrics/0.log"
Sep 16 13:15:02.989709 ip-10-0-133-218 kubenswrapper[2580]: I0916 13:15:02.989682    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_81b1531b-258a-4233-b637-b38a3de6d99d/prometheus/0.log"
Sep 16 13:15:03.010555 ip-10-0-133-218 kubenswrapper[2580]: I0916 13:15:03.010469    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_81b1531b-258a-4233-b637-b38a3de6d99d/config-reloader/0.log"
Sep 16 13:15:03.029264 ip-10-0-133-218 kubenswrapper[2580]: I0916 13:15:03.029240    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_81b1531b-258a-4233-b637-b38a3de6d99d/thanos-sidecar/0.log"
Sep 16 13:15:03.049433 ip-10-0-133-218 kubenswrapper[2580]: I0916 13:15:03.049409    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_81b1531b-258a-4233-b637-b38a3de6d99d/kube-rbac-proxy-web/0.log"
Sep 16 13:15:03.070184 ip-10-0-133-218 kubenswrapper[2580]: I0916 13:15:03.070161    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_81b1531b-258a-4233-b637-b38a3de6d99d/kube-rbac-proxy/0.log"
Sep 16 13:15:03.093611 ip-10-0-133-218 kubenswrapper[2580]: I0916 13:15:03.093588    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_81b1531b-258a-4233-b637-b38a3de6d99d/kube-rbac-proxy-thanos/0.log"
Sep 16 13:15:03.113798 ip-10-0-133-218 kubenswrapper[2580]: I0916 13:15:03.113779    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_81b1531b-258a-4233-b637-b38a3de6d99d/init-config-reloader/0.log"
Sep 16 13:15:03.182576 ip-10-0-133-218 kubenswrapper[2580]: I0916 13:15:03.182547    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-admission-webhook-5864cd99d7-bzpbv_882dd5ec-93c0-40fb-a3c0-c463add9c9ca/prometheus-operator-admission-webhook/0.log"
Sep 16 13:15:04.714996 ip-10-0-133-218 kubenswrapper[2580]: I0916 13:15:04.711974    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-console_networking-console-plugin-5dcbc7df6b-jv8fx_14b398fc-37e1-4712-8196-6f1ac6f70967/networking-console-plugin/0.log"
Sep 16 13:15:06.201470 ip-10-0-133-218 kubenswrapper[2580]: I0916 13:15:06.201429    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cluster-storage-operator_volume-data-source-validator-86695b7cc7-zf46w_418cfec8-573d-461e-9a2a-766b421cfbe6/volume-data-source-validator/0.log"
Sep 16 13:15:07.078664 ip-10-0-133-218 kubenswrapper[2580]: I0916 13:15:07.078630    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-hzdmm_b352b0f0-afd0-48c2-965f-7cc63cdf04fd/dns/0.log"
Sep 16 13:15:07.100434 ip-10-0-133-218 kubenswrapper[2580]: I0916 13:15:07.100409    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-hzdmm_b352b0f0-afd0-48c2-965f-7cc63cdf04fd/kube-rbac-proxy/0.log"
Sep 16 13:15:07.143743 ip-10-0-133-218 kubenswrapper[2580]: I0916 13:15:07.143712    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-l458r/perf-node-gather-daemonset-8v6m4"]
Sep 16 13:15:07.144089 ip-10-0-133-218 kubenswrapper[2580]: I0916 13:15:07.144072    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="211b47fa-c759-4703-b0ad-e935e5b44cbd" containerName="gather"
Sep 16 13:15:07.144172 ip-10-0-133-218 kubenswrapper[2580]: I0916 13:15:07.144093    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="211b47fa-c759-4703-b0ad-e935e5b44cbd" containerName="gather"
Sep 16 13:15:07.144172 ip-10-0-133-218 kubenswrapper[2580]: I0916 13:15:07.144116    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="211b47fa-c759-4703-b0ad-e935e5b44cbd" containerName="copy"
Sep 16 13:15:07.144172 ip-10-0-133-218 kubenswrapper[2580]: I0916 13:15:07.144124    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="211b47fa-c759-4703-b0ad-e935e5b44cbd" containerName="copy"
Sep 16 13:15:07.144318 ip-10-0-133-218 kubenswrapper[2580]: I0916 13:15:07.144195    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="211b47fa-c759-4703-b0ad-e935e5b44cbd" containerName="copy"
Sep 16 13:15:07.144318 ip-10-0-133-218 kubenswrapper[2580]: I0916 13:15:07.144209    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="211b47fa-c759-4703-b0ad-e935e5b44cbd" containerName="gather"
Sep 16 13:15:07.148431 ip-10-0-133-218 kubenswrapper[2580]: I0916 13:15:07.148409    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-l458r/perf-node-gather-daemonset-8v6m4"
Sep 16 13:15:07.150714 ip-10-0-133-218 kubenswrapper[2580]: I0916 13:15:07.150695    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-l458r\"/\"openshift-service-ca.crt\""
Sep 16 13:15:07.150830 ip-10-0-133-218 kubenswrapper[2580]: I0916 13:15:07.150787    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-l458r\"/\"kube-root-ca.crt\""
Sep 16 13:15:07.151643 ip-10-0-133-218 kubenswrapper[2580]: I0916 13:15:07.151599    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-l458r\"/\"default-dockercfg-rmglt\""
Sep 16 13:15:07.153284 ip-10-0-133-218 kubenswrapper[2580]: I0916 13:15:07.153263    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-l458r/perf-node-gather-daemonset-8v6m4"]
Sep 16 13:15:07.185064 ip-10-0-133-218 kubenswrapper[2580]: I0916 13:15:07.185036    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-cx649_0b719a93-a161-4199-9aae-5aa99cdf7165/dns-node-resolver/0.log"
Sep 16 13:15:07.261723 ip-10-0-133-218 kubenswrapper[2580]: I0916 13:15:07.261689    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/77e47042-ff03-4247-b231-6a9cfb332875-proc\") pod \"perf-node-gather-daemonset-8v6m4\" (UID: \"77e47042-ff03-4247-b231-6a9cfb332875\") " pod="openshift-must-gather-l458r/perf-node-gather-daemonset-8v6m4"
Sep 16 13:15:07.261723 ip-10-0-133-218 kubenswrapper[2580]: I0916 13:15:07.261721    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/77e47042-ff03-4247-b231-6a9cfb332875-lib-modules\") pod \"perf-node-gather-daemonset-8v6m4\" (UID: \"77e47042-ff03-4247-b231-6a9cfb332875\") " pod="openshift-must-gather-l458r/perf-node-gather-daemonset-8v6m4"
Sep 16 13:15:07.262120 ip-10-0-133-218 kubenswrapper[2580]: I0916 13:15:07.261758    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/77e47042-ff03-4247-b231-6a9cfb332875-sys\") pod \"perf-node-gather-daemonset-8v6m4\" (UID: \"77e47042-ff03-4247-b231-6a9cfb332875\") " pod="openshift-must-gather-l458r/perf-node-gather-daemonset-8v6m4"
Sep 16 13:15:07.262120 ip-10-0-133-218 kubenswrapper[2580]: I0916 13:15:07.261827    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/77e47042-ff03-4247-b231-6a9cfb332875-podres\") pod \"perf-node-gather-daemonset-8v6m4\" (UID: \"77e47042-ff03-4247-b231-6a9cfb332875\") " pod="openshift-must-gather-l458r/perf-node-gather-daemonset-8v6m4"
Sep 16 13:15:07.262120 ip-10-0-133-218 kubenswrapper[2580]: I0916 13:15:07.261878    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dfn5m\" (UniqueName: \"kubernetes.io/projected/77e47042-ff03-4247-b231-6a9cfb332875-kube-api-access-dfn5m\") pod \"perf-node-gather-daemonset-8v6m4\" (UID: \"77e47042-ff03-4247-b231-6a9cfb332875\") " pod="openshift-must-gather-l458r/perf-node-gather-daemonset-8v6m4"
Sep 16 13:15:07.363172 ip-10-0-133-218 kubenswrapper[2580]: I0916 13:15:07.363082    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/77e47042-ff03-4247-b231-6a9cfb332875-sys\") pod \"perf-node-gather-daemonset-8v6m4\" (UID: \"77e47042-ff03-4247-b231-6a9cfb332875\") " pod="openshift-must-gather-l458r/perf-node-gather-daemonset-8v6m4"
Sep 16 13:15:07.363315 ip-10-0-133-218 kubenswrapper[2580]: I0916 13:15:07.363175    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/77e47042-ff03-4247-b231-6a9cfb332875-podres\") pod \"perf-node-gather-daemonset-8v6m4\" (UID: \"77e47042-ff03-4247-b231-6a9cfb332875\") " pod="openshift-must-gather-l458r/perf-node-gather-daemonset-8v6m4"
Sep 16 13:15:07.363315 ip-10-0-133-218 kubenswrapper[2580]: I0916 13:15:07.363190    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/77e47042-ff03-4247-b231-6a9cfb332875-sys\") pod \"perf-node-gather-daemonset-8v6m4\" (UID: \"77e47042-ff03-4247-b231-6a9cfb332875\") " pod="openshift-must-gather-l458r/perf-node-gather-daemonset-8v6m4"
Sep 16 13:15:07.363315 ip-10-0-133-218 kubenswrapper[2580]: I0916 13:15:07.363238    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dfn5m\" (UniqueName: \"kubernetes.io/projected/77e47042-ff03-4247-b231-6a9cfb332875-kube-api-access-dfn5m\") pod \"perf-node-gather-daemonset-8v6m4\" (UID: \"77e47042-ff03-4247-b231-6a9cfb332875\") " pod="openshift-must-gather-l458r/perf-node-gather-daemonset-8v6m4"
Sep 16 13:15:07.363315 ip-10-0-133-218 kubenswrapper[2580]: I0916 13:15:07.363282    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/77e47042-ff03-4247-b231-6a9cfb332875-proc\") pod \"perf-node-gather-daemonset-8v6m4\" (UID: \"77e47042-ff03-4247-b231-6a9cfb332875\") " pod="openshift-must-gather-l458r/perf-node-gather-daemonset-8v6m4"
Sep 16 13:15:07.363315 ip-10-0-133-218 kubenswrapper[2580]: I0916 13:15:07.363303    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/77e47042-ff03-4247-b231-6a9cfb332875-podres\") pod \"perf-node-gather-daemonset-8v6m4\" (UID: \"77e47042-ff03-4247-b231-6a9cfb332875\") " pod="openshift-must-gather-l458r/perf-node-gather-daemonset-8v6m4"
Sep 16 13:15:07.363315 ip-10-0-133-218 kubenswrapper[2580]: I0916 13:15:07.363307    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/77e47042-ff03-4247-b231-6a9cfb332875-lib-modules\") pod \"perf-node-gather-daemonset-8v6m4\" (UID: \"77e47042-ff03-4247-b231-6a9cfb332875\") " pod="openshift-must-gather-l458r/perf-node-gather-daemonset-8v6m4"
Sep 16 13:15:07.363561 ip-10-0-133-218 kubenswrapper[2580]: I0916 13:15:07.363353    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/77e47042-ff03-4247-b231-6a9cfb332875-proc\") pod \"perf-node-gather-daemonset-8v6m4\" (UID: \"77e47042-ff03-4247-b231-6a9cfb332875\") " pod="openshift-must-gather-l458r/perf-node-gather-daemonset-8v6m4"
Sep 16 13:15:07.363561 ip-10-0-133-218 kubenswrapper[2580]: I0916 13:15:07.363381    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/77e47042-ff03-4247-b231-6a9cfb332875-lib-modules\") pod \"perf-node-gather-daemonset-8v6m4\" (UID: \"77e47042-ff03-4247-b231-6a9cfb332875\") " pod="openshift-must-gather-l458r/perf-node-gather-daemonset-8v6m4"
Sep 16 13:15:07.370993 ip-10-0-133-218 kubenswrapper[2580]: I0916 13:15:07.370961    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dfn5m\" (UniqueName: \"kubernetes.io/projected/77e47042-ff03-4247-b231-6a9cfb332875-kube-api-access-dfn5m\") pod \"perf-node-gather-daemonset-8v6m4\" (UID: \"77e47042-ff03-4247-b231-6a9cfb332875\") " pod="openshift-must-gather-l458r/perf-node-gather-daemonset-8v6m4"
Sep 16 13:15:07.459123 ip-10-0-133-218 kubenswrapper[2580]: I0916 13:15:07.459087    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-l458r/perf-node-gather-daemonset-8v6m4"
Sep 16 13:15:07.731580 ip-10-0-133-218 kubenswrapper[2580]: I0916 13:15:07.731429    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-l458r/perf-node-gather-daemonset-8v6m4"]
Sep 16 13:15:07.734295 ip-10-0-133-218 kubenswrapper[2580]: W0916 13:15:07.734269    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod77e47042_ff03_4247_b231_6a9cfb332875.slice/crio-b9b6f01cfb43230aee33dcf1b284a28e910dee297fd0c6e6ccd664143b23e781 WatchSource:0}: Error finding container b9b6f01cfb43230aee33dcf1b284a28e910dee297fd0c6e6ccd664143b23e781: Status 404 returned error can't find the container with id b9b6f01cfb43230aee33dcf1b284a28e910dee297fd0c6e6ccd664143b23e781
Sep 16 13:15:07.785114 ip-10-0-133-218 kubenswrapper[2580]: I0916 13:15:07.785083    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-d9ndr_5d809960-17f1-4e60-9438-dc50a3b54dab/node-ca/0.log"
Sep 16 13:15:08.276713 ip-10-0-133-218 kubenswrapper[2580]: I0916 13:15:08.276672    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-l458r/perf-node-gather-daemonset-8v6m4" event={"ID":"77e47042-ff03-4247-b231-6a9cfb332875","Type":"ContainerStarted","Data":"bd6ea6f667c3c6dfe41b8b69bbd1834d0996ad27cdce4d2d570f87518a3d4032"}
Sep 16 13:15:08.277104 ip-10-0-133-218 kubenswrapper[2580]: I0916 13:15:08.276729    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-l458r/perf-node-gather-daemonset-8v6m4" event={"ID":"77e47042-ff03-4247-b231-6a9cfb332875","Type":"ContainerStarted","Data":"b9b6f01cfb43230aee33dcf1b284a28e910dee297fd0c6e6ccd664143b23e781"}
Sep 16 13:15:08.277104 ip-10-0-133-218 kubenswrapper[2580]: I0916 13:15:08.276770    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-l458r/perf-node-gather-daemonset-8v6m4"
Sep 16 13:15:08.294479 ip-10-0-133-218 kubenswrapper[2580]: I0916 13:15:08.294431    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-l458r/perf-node-gather-daemonset-8v6m4" podStartSLOduration=1.2944161410000001 podStartE2EDuration="1.294416141s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-16 13:15:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 13:15:08.293152039 +0000 UTC m=+1647.216024508" watchObservedRunningTime="2026-09-16 13:15:08.294416141 +0000 UTC m=+1647.217288884"
Sep 16 13:15:08.541244 ip-10-0-133-218 kubenswrapper[2580]: I0916 13:15:08.541216    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-5b4dd555d4-jwzh7_3c9aacb3-1142-4610-b990-a28d15998e71/router/0.log"
Sep 16 13:15:08.859030 ip-10-0-133-218 kubenswrapper[2580]: I0916 13:15:08.858938    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-d4swz_23d2db14-6f1e-4198-8b68-022153a628d5/serve-healthcheck-canary/0.log"
Sep 16 13:15:09.223389 ip-10-0-133-218 kubenswrapper[2580]: I0916 13:15:09.223250    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-2m9hh_e12762e0-2a1d-4bf3-8fca-d22ad8e1eecf/kube-rbac-proxy/0.log"
Sep 16 13:15:09.243019 ip-10-0-133-218 kubenswrapper[2580]: I0916 13:15:09.242994    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-2m9hh_e12762e0-2a1d-4bf3-8fca-d22ad8e1eecf/exporter/0.log"
Sep 16 13:15:09.261731 ip-10-0-133-218 kubenswrapper[2580]: I0916 13:15:09.261708    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-2m9hh_e12762e0-2a1d-4bf3-8fca-d22ad8e1eecf/extractor/0.log"
Sep 16 13:15:11.739588 ip-10-0-133-218 kubenswrapper[2580]: I0916 13:15:11.739549    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_odh-model-controller-7b966d7984-kkx92_988887ee-5cc2-461e-a0e5-5338d5c59ab8/manager/0.log"
Sep 16 13:15:11.769343 ip-10-0-133-218 kubenswrapper[2580]: I0916 13:15:11.769314    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_s3-init-zwqmj_a1bd02a1-01d1-46d7-a28a-5b54fe56c086/s3-init/0.log"
Sep 16 13:15:11.784181 ip-10-0-133-218 kubenswrapper[2580]: I0916 13:15:11.784160    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_s3-init-zwqmj_a1bd02a1-01d1-46d7-a28a-5b54fe56c086/download-hf-model/0.log"
Sep 16 13:15:14.289563 ip-10-0-133-218 kubenswrapper[2580]: I0916 13:15:14.289534    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-l458r/perf-node-gather-daemonset-8v6m4"
Sep 16 13:15:15.572144 ip-10-0-133-218 kubenswrapper[2580]: I0916 13:15:15.572111    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-8b4d697c5-bzww8_46e66faf-5912-4f71-8fe8-57c2c2d64bc4/migrator/0.log"
Sep 16 13:15:15.593322 ip-10-0-133-218 kubenswrapper[2580]: I0916 13:15:15.593299    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-8b4d697c5-bzww8_46e66faf-5912-4f71-8fe8-57c2c2d64bc4/graceful-termination/0.log"
Sep 16 13:15:16.982192 ip-10-0-133-218 kubenswrapper[2580]: I0916 13:15:16.982120    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-544bh_fb1fa9b3-dee8-433f-b3f9-29f24eab1e3a/kube-multus/0.log"
Sep 16 13:15:17.029228 ip-10-0-133-218 kubenswrapper[2580]: I0916 13:15:17.029202    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-hsj4z_7b900f8b-a992-44b3-a6ea-7bacb17d2798/kube-multus-additional-cni-plugins/0.log"
Sep 16 13:15:17.083979 ip-10-0-133-218 kubenswrapper[2580]: I0916 13:15:17.083956    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-hsj4z_7b900f8b-a992-44b3-a6ea-7bacb17d2798/egress-router-binary-copy/0.log"
Sep 16 13:15:17.133357 ip-10-0-133-218 kubenswrapper[2580]: I0916 13:15:17.133331    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-hsj4z_7b900f8b-a992-44b3-a6ea-7bacb17d2798/cni-plugins/0.log"
Sep 16 13:15:17.176689 ip-10-0-133-218 kubenswrapper[2580]: I0916 13:15:17.176664    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-hsj4z_7b900f8b-a992-44b3-a6ea-7bacb17d2798/bond-cni-plugin/0.log"
Sep 16 13:15:17.223718 ip-10-0-133-218 kubenswrapper[2580]: I0916 13:15:17.223696    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-hsj4z_7b900f8b-a992-44b3-a6ea-7bacb17d2798/routeoverride-cni/0.log"
Sep 16 13:15:17.267176 ip-10-0-133-218 kubenswrapper[2580]: I0916 13:15:17.267112    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-hsj4z_7b900f8b-a992-44b3-a6ea-7bacb17d2798/whereabouts-cni-bincopy/0.log"
Sep 16 13:15:17.292397 ip-10-0-133-218 kubenswrapper[2580]: I0916 13:15:17.292371    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-hsj4z_7b900f8b-a992-44b3-a6ea-7bacb17d2798/whereabouts-cni/0.log"
Sep 16 13:15:17.635075 ip-10-0-133-218 kubenswrapper[2580]: I0916 13:15:17.635043    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-62j7x_7d634420-2900-42b3-9609-3fb64baff8f3/network-metrics-daemon/0.log"
Sep 16 13:15:17.654921 ip-10-0-133-218 kubenswrapper[2580]: I0916 13:15:17.654900    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-62j7x_7d634420-2900-42b3-9609-3fb64baff8f3/kube-rbac-proxy/0.log"
Sep 16 13:15:18.526321 ip-10-0-133-218 kubenswrapper[2580]: I0916 13:15:18.526245    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-894fk_ffd98c6e-f973-4bd8-94ba-bd92a16d1acd/ovn-controller/0.log"
Sep 16 13:15:18.542558 ip-10-0-133-218 kubenswrapper[2580]: I0916 13:15:18.542534    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-894fk_ffd98c6e-f973-4bd8-94ba-bd92a16d1acd/ovn-acl-logging/0.log"
Sep 16 13:15:18.563719 ip-10-0-133-218 kubenswrapper[2580]: I0916 13:15:18.563695    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-894fk_ffd98c6e-f973-4bd8-94ba-bd92a16d1acd/kube-rbac-proxy-node/0.log"
Sep 16 13:15:18.588397 ip-10-0-133-218 kubenswrapper[2580]: I0916 13:15:18.588370    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-894fk_ffd98c6e-f973-4bd8-94ba-bd92a16d1acd/kube-rbac-proxy-ovn-metrics/0.log"
Sep 16 13:15:18.606524 ip-10-0-133-218 kubenswrapper[2580]: I0916 13:15:18.606482    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-894fk_ffd98c6e-f973-4bd8-94ba-bd92a16d1acd/northd/0.log"
Sep 16 13:15:18.625273 ip-10-0-133-218 kubenswrapper[2580]: I0916 13:15:18.625252    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-894fk_ffd98c6e-f973-4bd8-94ba-bd92a16d1acd/nbdb/0.log"
Sep 16 13:15:18.644456 ip-10-0-133-218 kubenswrapper[2580]: I0916 13:15:18.644428    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-894fk_ffd98c6e-f973-4bd8-94ba-bd92a16d1acd/sbdb/0.log"
Sep 16 13:15:18.678529 ip-10-0-133-218 kubenswrapper[2580]: I0916 13:15:18.678503    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-894fk_ffd98c6e-f973-4bd8-94ba-bd92a16d1acd/ovnkube-controller/0.log"
Sep 16 13:15:19.996472 ip-10-0-133-218 kubenswrapper[2580]: I0916 13:15:19.996444    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-diagnostics_network-check-source-6f8b768455-jl4wt_f2a2dd43-5c98-4e27-a59a-8ac33a6d9c6e/check-endpoints/0.log"
Sep 16 13:15:20.040216 ip-10-0-133-218 kubenswrapper[2580]: I0916 13:15:20.040190    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-diagnostics_network-check-target-bpv5d_1b26fbcb-3523-4f7c-9e42-5b3653456abc/network-check-target-container/0.log"
Sep 16 13:15:20.987641 ip-10-0-133-218 kubenswrapper[2580]: I0916 13:15:20.987610    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-operator_iptables-alerter-rgxfz_32fe9109-0026-4609-af0c-923a618db835/iptables-alerter/0.log"
Sep 16 13:15:21.608510 ip-10-0-133-218 kubenswrapper[2580]: I0916 13:15:21.608463    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cluster-node-tuning-operator_tuned-9gbm5_8822b539-bda1-46bd-8a2c-27660032a4e5/tuned/0.log"
Sep 16 13:15:23.338403 ip-10-0-133-218 kubenswrapper[2580]: I0916 13:15:23.338368    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cluster-samples-operator_cluster-samples-operator-86499c89c4-xvxlf_5abef85b-1f8d-4f0e-9091-c0f42acf1af1/cluster-samples-operator/0.log"
Sep 16 13:15:23.355856 ip-10-0-133-218 kubenswrapper[2580]: I0916 13:15:23.355827    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cluster-samples-operator_cluster-samples-operator-86499c89c4-xvxlf_5abef85b-1f8d-4f0e-9091-c0f42acf1af1/cluster-samples-operator-watch/0.log"