Sep 16 18:43:57.107833 ip-10-0-132-80 systemd[1]: Starting Kubernetes Kubelet...
Sep 16 18:43:57.534918 ip-10-0-132-80 kubenswrapper[2544]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 16 18:43:57.534918 ip-10-0-132-80 kubenswrapper[2544]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Sep 16 18:43:57.534918 ip-10-0-132-80 kubenswrapper[2544]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 16 18:43:57.534918 ip-10-0-132-80 kubenswrapper[2544]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI.
Sep 16 18:43:57.534918 ip-10-0-132-80 kubenswrapper[2544]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 16 18:43:57.535584 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:43:57.534929    2544 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime"
Sep 16 18:43:57.539079 ip-10-0-132-80 kubenswrapper[2544]: W0916 18:43:57.539062    2544 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 16 18:43:57.539079 ip-10-0-132-80 kubenswrapper[2544]: W0916 18:43:57.539078    2544 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 16 18:43:57.539144 ip-10-0-132-80 kubenswrapper[2544]: W0916 18:43:57.539082    2544 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 16 18:43:57.539144 ip-10-0-132-80 kubenswrapper[2544]: W0916 18:43:57.539085    2544 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 16 18:43:57.539144 ip-10-0-132-80 kubenswrapper[2544]: W0916 18:43:57.539089    2544 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 16 18:43:57.539144 ip-10-0-132-80 kubenswrapper[2544]: W0916 18:43:57.539092    2544 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 16 18:43:57.539144 ip-10-0-132-80 kubenswrapper[2544]: W0916 18:43:57.539094    2544 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 16 18:43:57.539144 ip-10-0-132-80 kubenswrapper[2544]: W0916 18:43:57.539097    2544 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 16 18:43:57.539144 ip-10-0-132-80 kubenswrapper[2544]: W0916 18:43:57.539100    2544 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 16 18:43:57.539144 ip-10-0-132-80 kubenswrapper[2544]: W0916 18:43:57.539107    2544 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 16 18:43:57.539144 ip-10-0-132-80 kubenswrapper[2544]: W0916 18:43:57.539109    2544 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 16 18:43:57.539144 ip-10-0-132-80 kubenswrapper[2544]: W0916 18:43:57.539112    2544 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 16 18:43:57.539144 ip-10-0-132-80 kubenswrapper[2544]: W0916 18:43:57.539115    2544 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 16 18:43:57.539144 ip-10-0-132-80 kubenswrapper[2544]: W0916 18:43:57.539117    2544 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 16 18:43:57.539144 ip-10-0-132-80 kubenswrapper[2544]: W0916 18:43:57.539120    2544 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 16 18:43:57.539144 ip-10-0-132-80 kubenswrapper[2544]: W0916 18:43:57.539123    2544 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 16 18:43:57.539144 ip-10-0-132-80 kubenswrapper[2544]: W0916 18:43:57.539126    2544 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 16 18:43:57.539144 ip-10-0-132-80 kubenswrapper[2544]: W0916 18:43:57.539129    2544 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 16 18:43:57.539144 ip-10-0-132-80 kubenswrapper[2544]: W0916 18:43:57.539132    2544 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 16 18:43:57.539144 ip-10-0-132-80 kubenswrapper[2544]: W0916 18:43:57.539134    2544 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 16 18:43:57.539144 ip-10-0-132-80 kubenswrapper[2544]: W0916 18:43:57.539137    2544 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 16 18:43:57.539144 ip-10-0-132-80 kubenswrapper[2544]: W0916 18:43:57.539140    2544 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 16 18:43:57.539628 ip-10-0-132-80 kubenswrapper[2544]: W0916 18:43:57.539142    2544 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 16 18:43:57.539628 ip-10-0-132-80 kubenswrapper[2544]: W0916 18:43:57.539145    2544 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 16 18:43:57.539628 ip-10-0-132-80 kubenswrapper[2544]: W0916 18:43:57.539148    2544 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 16 18:43:57.539628 ip-10-0-132-80 kubenswrapper[2544]: W0916 18:43:57.539151    2544 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 16 18:43:57.539628 ip-10-0-132-80 kubenswrapper[2544]: W0916 18:43:57.539153    2544 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 16 18:43:57.539628 ip-10-0-132-80 kubenswrapper[2544]: W0916 18:43:57.539156    2544 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 16 18:43:57.539628 ip-10-0-132-80 kubenswrapper[2544]: W0916 18:43:57.539159    2544 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 16 18:43:57.539628 ip-10-0-132-80 kubenswrapper[2544]: W0916 18:43:57.539162    2544 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 16 18:43:57.539628 ip-10-0-132-80 kubenswrapper[2544]: W0916 18:43:57.539164    2544 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 16 18:43:57.539628 ip-10-0-132-80 kubenswrapper[2544]: W0916 18:43:57.539166    2544 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 16 18:43:57.539628 ip-10-0-132-80 kubenswrapper[2544]: W0916 18:43:57.539169    2544 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 16 18:43:57.539628 ip-10-0-132-80 kubenswrapper[2544]: W0916 18:43:57.539172    2544 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 16 18:43:57.539628 ip-10-0-132-80 kubenswrapper[2544]: W0916 18:43:57.539175    2544 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 16 18:43:57.539628 ip-10-0-132-80 kubenswrapper[2544]: W0916 18:43:57.539177    2544 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 16 18:43:57.539628 ip-10-0-132-80 kubenswrapper[2544]: W0916 18:43:57.539180    2544 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 16 18:43:57.539628 ip-10-0-132-80 kubenswrapper[2544]: W0916 18:43:57.539183    2544 feature_gate.go:328] unrecognized feature gate: Example
Sep 16 18:43:57.539628 ip-10-0-132-80 kubenswrapper[2544]: W0916 18:43:57.539185    2544 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 16 18:43:57.539628 ip-10-0-132-80 kubenswrapper[2544]: W0916 18:43:57.539188    2544 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 16 18:43:57.539628 ip-10-0-132-80 kubenswrapper[2544]: W0916 18:43:57.539190    2544 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 16 18:43:57.540086 ip-10-0-132-80 kubenswrapper[2544]: W0916 18:43:57.539193    2544 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 16 18:43:57.540086 ip-10-0-132-80 kubenswrapper[2544]: W0916 18:43:57.539195    2544 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 16 18:43:57.540086 ip-10-0-132-80 kubenswrapper[2544]: W0916 18:43:57.539200    2544 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 16 18:43:57.540086 ip-10-0-132-80 kubenswrapper[2544]: W0916 18:43:57.539204    2544 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 16 18:43:57.540086 ip-10-0-132-80 kubenswrapper[2544]: W0916 18:43:57.539207    2544 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 16 18:43:57.540086 ip-10-0-132-80 kubenswrapper[2544]: W0916 18:43:57.539210    2544 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 16 18:43:57.540086 ip-10-0-132-80 kubenswrapper[2544]: W0916 18:43:57.539213    2544 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 16 18:43:57.540086 ip-10-0-132-80 kubenswrapper[2544]: W0916 18:43:57.539216    2544 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 16 18:43:57.540086 ip-10-0-132-80 kubenswrapper[2544]: W0916 18:43:57.539219    2544 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 16 18:43:57.540086 ip-10-0-132-80 kubenswrapper[2544]: W0916 18:43:57.539222    2544 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 16 18:43:57.540086 ip-10-0-132-80 kubenswrapper[2544]: W0916 18:43:57.539225    2544 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 16 18:43:57.540086 ip-10-0-132-80 kubenswrapper[2544]: W0916 18:43:57.539227    2544 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 16 18:43:57.540086 ip-10-0-132-80 kubenswrapper[2544]: W0916 18:43:57.539230    2544 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 16 18:43:57.540086 ip-10-0-132-80 kubenswrapper[2544]: W0916 18:43:57.539232    2544 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 16 18:43:57.540086 ip-10-0-132-80 kubenswrapper[2544]: W0916 18:43:57.539238    2544 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 16 18:43:57.540086 ip-10-0-132-80 kubenswrapper[2544]: W0916 18:43:57.539241    2544 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 16 18:43:57.540086 ip-10-0-132-80 kubenswrapper[2544]: W0916 18:43:57.539244    2544 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 16 18:43:57.540086 ip-10-0-132-80 kubenswrapper[2544]: W0916 18:43:57.539247    2544 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 16 18:43:57.540086 ip-10-0-132-80 kubenswrapper[2544]: W0916 18:43:57.539250    2544 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 16 18:43:57.540568 ip-10-0-132-80 kubenswrapper[2544]: W0916 18:43:57.539252    2544 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 16 18:43:57.540568 ip-10-0-132-80 kubenswrapper[2544]: W0916 18:43:57.539255    2544 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 16 18:43:57.540568 ip-10-0-132-80 kubenswrapper[2544]: W0916 18:43:57.539257    2544 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 16 18:43:57.540568 ip-10-0-132-80 kubenswrapper[2544]: W0916 18:43:57.539260    2544 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 16 18:43:57.540568 ip-10-0-132-80 kubenswrapper[2544]: W0916 18:43:57.539263    2544 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 16 18:43:57.540568 ip-10-0-132-80 kubenswrapper[2544]: W0916 18:43:57.539265    2544 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 16 18:43:57.540568 ip-10-0-132-80 kubenswrapper[2544]: W0916 18:43:57.539268    2544 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 16 18:43:57.540568 ip-10-0-132-80 kubenswrapper[2544]: W0916 18:43:57.539271    2544 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 16 18:43:57.540568 ip-10-0-132-80 kubenswrapper[2544]: W0916 18:43:57.539274    2544 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 16 18:43:57.540568 ip-10-0-132-80 kubenswrapper[2544]: W0916 18:43:57.539276    2544 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 16 18:43:57.540568 ip-10-0-132-80 kubenswrapper[2544]: W0916 18:43:57.539279    2544 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 16 18:43:57.540568 ip-10-0-132-80 kubenswrapper[2544]: W0916 18:43:57.539281    2544 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 16 18:43:57.540568 ip-10-0-132-80 kubenswrapper[2544]: W0916 18:43:57.539284    2544 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 16 18:43:57.540568 ip-10-0-132-80 kubenswrapper[2544]: W0916 18:43:57.539286    2544 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 16 18:43:57.540568 ip-10-0-132-80 kubenswrapper[2544]: W0916 18:43:57.539289    2544 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 16 18:43:57.540568 ip-10-0-132-80 kubenswrapper[2544]: W0916 18:43:57.539291    2544 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 16 18:43:57.540568 ip-10-0-132-80 kubenswrapper[2544]: W0916 18:43:57.539294    2544 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 16 18:43:57.540568 ip-10-0-132-80 kubenswrapper[2544]: W0916 18:43:57.539296    2544 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 16 18:43:57.540568 ip-10-0-132-80 kubenswrapper[2544]: W0916 18:43:57.539299    2544 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 16 18:43:57.540568 ip-10-0-132-80 kubenswrapper[2544]: W0916 18:43:57.539301    2544 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 16 18:43:57.541388 ip-10-0-132-80 kubenswrapper[2544]: W0916 18:43:57.539303    2544 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 16 18:43:57.541388 ip-10-0-132-80 kubenswrapper[2544]: W0916 18:43:57.539306    2544 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 16 18:43:57.541388 ip-10-0-132-80 kubenswrapper[2544]: W0916 18:43:57.539308    2544 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 16 18:43:57.541388 ip-10-0-132-80 kubenswrapper[2544]: W0916 18:43:57.539311    2544 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 16 18:43:57.541388 ip-10-0-132-80 kubenswrapper[2544]: W0916 18:43:57.539313    2544 feature_gate.go:328] unrecognized feature gate: Example2
Sep 16 18:43:57.541388 ip-10-0-132-80 kubenswrapper[2544]: W0916 18:43:57.539317    2544 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 16 18:43:57.541388 ip-10-0-132-80 kubenswrapper[2544]: W0916 18:43:57.539320    2544 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 16 18:43:57.541388 ip-10-0-132-80 kubenswrapper[2544]: W0916 18:43:57.539324    2544 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 16 18:43:57.541388 ip-10-0-132-80 kubenswrapper[2544]: W0916 18:43:57.539327    2544 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 16 18:43:57.541388 ip-10-0-132-80 kubenswrapper[2544]: W0916 18:43:57.539329    2544 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 16 18:43:57.541388 ip-10-0-132-80 kubenswrapper[2544]: W0916 18:43:57.539332    2544 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 16 18:43:57.541388 ip-10-0-132-80 kubenswrapper[2544]: W0916 18:43:57.539334    2544 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 16 18:43:57.541388 ip-10-0-132-80 kubenswrapper[2544]: W0916 18:43:57.539337    2544 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 16 18:43:57.541388 ip-10-0-132-80 kubenswrapper[2544]: W0916 18:43:57.539340    2544 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 16 18:43:57.541388 ip-10-0-132-80 kubenswrapper[2544]: W0916 18:43:57.539342    2544 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 16 18:43:57.541388 ip-10-0-132-80 kubenswrapper[2544]: W0916 18:43:57.539345    2544 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 16 18:43:57.541388 ip-10-0-132-80 kubenswrapper[2544]: W0916 18:43:57.539348    2544 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 16 18:43:57.541388 ip-10-0-132-80 kubenswrapper[2544]: W0916 18:43:57.539350    2544 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 16 18:43:57.541388 ip-10-0-132-80 kubenswrapper[2544]: W0916 18:43:57.539352    2544 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 16 18:43:57.541388 ip-10-0-132-80 kubenswrapper[2544]: W0916 18:43:57.540104    2544 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 16 18:43:57.542082 ip-10-0-132-80 kubenswrapper[2544]: W0916 18:43:57.540112    2544 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 16 18:43:57.542082 ip-10-0-132-80 kubenswrapper[2544]: W0916 18:43:57.540116    2544 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 16 18:43:57.542082 ip-10-0-132-80 kubenswrapper[2544]: W0916 18:43:57.540119    2544 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 16 18:43:57.542082 ip-10-0-132-80 kubenswrapper[2544]: W0916 18:43:57.540122    2544 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 16 18:43:57.542082 ip-10-0-132-80 kubenswrapper[2544]: W0916 18:43:57.540124    2544 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 16 18:43:57.542082 ip-10-0-132-80 kubenswrapper[2544]: W0916 18:43:57.540127    2544 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 16 18:43:57.542082 ip-10-0-132-80 kubenswrapper[2544]: W0916 18:43:57.540130    2544 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 16 18:43:57.542082 ip-10-0-132-80 kubenswrapper[2544]: W0916 18:43:57.540132    2544 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 16 18:43:57.542082 ip-10-0-132-80 kubenswrapper[2544]: W0916 18:43:57.540135    2544 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 16 18:43:57.542082 ip-10-0-132-80 kubenswrapper[2544]: W0916 18:43:57.540138    2544 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 16 18:43:57.542082 ip-10-0-132-80 kubenswrapper[2544]: W0916 18:43:57.540140    2544 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 16 18:43:57.542082 ip-10-0-132-80 kubenswrapper[2544]: W0916 18:43:57.540143    2544 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 16 18:43:57.542082 ip-10-0-132-80 kubenswrapper[2544]: W0916 18:43:57.540146    2544 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 16 18:43:57.542082 ip-10-0-132-80 kubenswrapper[2544]: W0916 18:43:57.540148    2544 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 16 18:43:57.542082 ip-10-0-132-80 kubenswrapper[2544]: W0916 18:43:57.540151    2544 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 16 18:43:57.542082 ip-10-0-132-80 kubenswrapper[2544]: W0916 18:43:57.540153    2544 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 16 18:43:57.542082 ip-10-0-132-80 kubenswrapper[2544]: W0916 18:43:57.540156    2544 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 16 18:43:57.542082 ip-10-0-132-80 kubenswrapper[2544]: W0916 18:43:57.540159    2544 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 16 18:43:57.542082 ip-10-0-132-80 kubenswrapper[2544]: W0916 18:43:57.540161    2544 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 16 18:43:57.542082 ip-10-0-132-80 kubenswrapper[2544]: W0916 18:43:57.540165    2544 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 16 18:43:57.542636 ip-10-0-132-80 kubenswrapper[2544]: W0916 18:43:57.540168    2544 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 16 18:43:57.542636 ip-10-0-132-80 kubenswrapper[2544]: W0916 18:43:57.540170    2544 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 16 18:43:57.542636 ip-10-0-132-80 kubenswrapper[2544]: W0916 18:43:57.540174    2544 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 16 18:43:57.542636 ip-10-0-132-80 kubenswrapper[2544]: W0916 18:43:57.540178    2544 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 16 18:43:57.542636 ip-10-0-132-80 kubenswrapper[2544]: W0916 18:43:57.540181    2544 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 16 18:43:57.542636 ip-10-0-132-80 kubenswrapper[2544]: W0916 18:43:57.540184    2544 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 16 18:43:57.542636 ip-10-0-132-80 kubenswrapper[2544]: W0916 18:43:57.540186    2544 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 16 18:43:57.542636 ip-10-0-132-80 kubenswrapper[2544]: W0916 18:43:57.540189    2544 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 16 18:43:57.542636 ip-10-0-132-80 kubenswrapper[2544]: W0916 18:43:57.540192    2544 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 16 18:43:57.542636 ip-10-0-132-80 kubenswrapper[2544]: W0916 18:43:57.540194    2544 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 16 18:43:57.542636 ip-10-0-132-80 kubenswrapper[2544]: W0916 18:43:57.540197    2544 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 16 18:43:57.542636 ip-10-0-132-80 kubenswrapper[2544]: W0916 18:43:57.540200    2544 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 16 18:43:57.542636 ip-10-0-132-80 kubenswrapper[2544]: W0916 18:43:57.540203    2544 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 16 18:43:57.542636 ip-10-0-132-80 kubenswrapper[2544]: W0916 18:43:57.540206    2544 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 16 18:43:57.542636 ip-10-0-132-80 kubenswrapper[2544]: W0916 18:43:57.540208    2544 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 16 18:43:57.542636 ip-10-0-132-80 kubenswrapper[2544]: W0916 18:43:57.540211    2544 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 16 18:43:57.542636 ip-10-0-132-80 kubenswrapper[2544]: W0916 18:43:57.540213    2544 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 16 18:43:57.542636 ip-10-0-132-80 kubenswrapper[2544]: W0916 18:43:57.540216    2544 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 16 18:43:57.542636 ip-10-0-132-80 kubenswrapper[2544]: W0916 18:43:57.540219    2544 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 16 18:43:57.542636 ip-10-0-132-80 kubenswrapper[2544]: W0916 18:43:57.540221    2544 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 16 18:43:57.543130 ip-10-0-132-80 kubenswrapper[2544]: W0916 18:43:57.540223    2544 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 16 18:43:57.543130 ip-10-0-132-80 kubenswrapper[2544]: W0916 18:43:57.540226    2544 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 16 18:43:57.543130 ip-10-0-132-80 kubenswrapper[2544]: W0916 18:43:57.540229    2544 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 16 18:43:57.543130 ip-10-0-132-80 kubenswrapper[2544]: W0916 18:43:57.540232    2544 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 16 18:43:57.543130 ip-10-0-132-80 kubenswrapper[2544]: W0916 18:43:57.540235    2544 feature_gate.go:328] unrecognized feature gate: Example
Sep 16 18:43:57.543130 ip-10-0-132-80 kubenswrapper[2544]: W0916 18:43:57.540237    2544 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 16 18:43:57.543130 ip-10-0-132-80 kubenswrapper[2544]: W0916 18:43:57.540240    2544 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 16 18:43:57.543130 ip-10-0-132-80 kubenswrapper[2544]: W0916 18:43:57.540242    2544 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 16 18:43:57.543130 ip-10-0-132-80 kubenswrapper[2544]: W0916 18:43:57.540245    2544 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 16 18:43:57.543130 ip-10-0-132-80 kubenswrapper[2544]: W0916 18:43:57.540248    2544 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 16 18:43:57.543130 ip-10-0-132-80 kubenswrapper[2544]: W0916 18:43:57.540250    2544 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 16 18:43:57.543130 ip-10-0-132-80 kubenswrapper[2544]: W0916 18:43:57.540254    2544 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 16 18:43:57.543130 ip-10-0-132-80 kubenswrapper[2544]: W0916 18:43:57.540256    2544 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 16 18:43:57.543130 ip-10-0-132-80 kubenswrapper[2544]: W0916 18:43:57.540259    2544 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 16 18:43:57.543130 ip-10-0-132-80 kubenswrapper[2544]: W0916 18:43:57.540262    2544 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 16 18:43:57.543130 ip-10-0-132-80 kubenswrapper[2544]: W0916 18:43:57.540264    2544 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 16 18:43:57.543130 ip-10-0-132-80 kubenswrapper[2544]: W0916 18:43:57.540267    2544 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 16 18:43:57.543130 ip-10-0-132-80 kubenswrapper[2544]: W0916 18:43:57.540269    2544 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 16 18:43:57.543130 ip-10-0-132-80 kubenswrapper[2544]: W0916 18:43:57.540272    2544 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 16 18:43:57.543130 ip-10-0-132-80 kubenswrapper[2544]: W0916 18:43:57.540275    2544 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 16 18:43:57.543669 ip-10-0-132-80 kubenswrapper[2544]: W0916 18:43:57.540277    2544 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 16 18:43:57.543669 ip-10-0-132-80 kubenswrapper[2544]: W0916 18:43:57.540280    2544 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 16 18:43:57.543669 ip-10-0-132-80 kubenswrapper[2544]: W0916 18:43:57.540282    2544 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 16 18:43:57.543669 ip-10-0-132-80 kubenswrapper[2544]: W0916 18:43:57.540285    2544 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 16 18:43:57.543669 ip-10-0-132-80 kubenswrapper[2544]: W0916 18:43:57.540287    2544 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 16 18:43:57.543669 ip-10-0-132-80 kubenswrapper[2544]: W0916 18:43:57.540290    2544 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 16 18:43:57.543669 ip-10-0-132-80 kubenswrapper[2544]: W0916 18:43:57.540294    2544 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 16 18:43:57.543669 ip-10-0-132-80 kubenswrapper[2544]: W0916 18:43:57.540297    2544 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 16 18:43:57.543669 ip-10-0-132-80 kubenswrapper[2544]: W0916 18:43:57.540301    2544 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 16 18:43:57.543669 ip-10-0-132-80 kubenswrapper[2544]: W0916 18:43:57.540305    2544 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 16 18:43:57.543669 ip-10-0-132-80 kubenswrapper[2544]: W0916 18:43:57.540308    2544 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 16 18:43:57.543669 ip-10-0-132-80 kubenswrapper[2544]: W0916 18:43:57.540311    2544 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 16 18:43:57.543669 ip-10-0-132-80 kubenswrapper[2544]: W0916 18:43:57.540314    2544 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 16 18:43:57.543669 ip-10-0-132-80 kubenswrapper[2544]: W0916 18:43:57.540316    2544 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 16 18:43:57.543669 ip-10-0-132-80 kubenswrapper[2544]: W0916 18:43:57.540319    2544 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 16 18:43:57.543669 ip-10-0-132-80 kubenswrapper[2544]: W0916 18:43:57.540322    2544 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 16 18:43:57.543669 ip-10-0-132-80 kubenswrapper[2544]: W0916 18:43:57.540324    2544 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 16 18:43:57.543669 ip-10-0-132-80 kubenswrapper[2544]: W0916 18:43:57.540326    2544 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 16 18:43:57.543669 ip-10-0-132-80 kubenswrapper[2544]: W0916 18:43:57.540329    2544 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 16 18:43:57.544132 ip-10-0-132-80 kubenswrapper[2544]: W0916 18:43:57.540332    2544 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 16 18:43:57.544132 ip-10-0-132-80 kubenswrapper[2544]: W0916 18:43:57.540334    2544 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 16 18:43:57.544132 ip-10-0-132-80 kubenswrapper[2544]: W0916 18:43:57.540337    2544 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 16 18:43:57.544132 ip-10-0-132-80 kubenswrapper[2544]: W0916 18:43:57.540340    2544 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 16 18:43:57.544132 ip-10-0-132-80 kubenswrapper[2544]: W0916 18:43:57.540343    2544 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 16 18:43:57.544132 ip-10-0-132-80 kubenswrapper[2544]: W0916 18:43:57.540346    2544 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 16 18:43:57.544132 ip-10-0-132-80 kubenswrapper[2544]: W0916 18:43:57.540349    2544 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 16 18:43:57.544132 ip-10-0-132-80 kubenswrapper[2544]: W0916 18:43:57.540352    2544 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 16 18:43:57.544132 ip-10-0-132-80 kubenswrapper[2544]: W0916 18:43:57.540354    2544 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 16 18:43:57.544132 ip-10-0-132-80 kubenswrapper[2544]: W0916 18:43:57.540357    2544 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 16 18:43:57.544132 ip-10-0-132-80 kubenswrapper[2544]: W0916 18:43:57.540359    2544 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 16 18:43:57.544132 ip-10-0-132-80 kubenswrapper[2544]: W0916 18:43:57.540362    2544 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 16 18:43:57.544132 ip-10-0-132-80 kubenswrapper[2544]: W0916 18:43:57.540365    2544 feature_gate.go:328] unrecognized feature gate: Example2
Sep 16 18:43:57.544132 ip-10-0-132-80 kubenswrapper[2544]: W0916 18:43:57.540367    2544 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 16 18:43:57.544132 ip-10-0-132-80 kubenswrapper[2544]: W0916 18:43:57.540370    2544 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 16 18:43:57.544132 ip-10-0-132-80 kubenswrapper[2544]: W0916 18:43:57.540372    2544 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 16 18:43:57.544132 ip-10-0-132-80 kubenswrapper[2544]: W0916 18:43:57.540375    2544 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 16 18:43:57.544132 ip-10-0-132-80 kubenswrapper[2544]: W0916 18:43:57.540378    2544 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 16 18:43:57.544132 ip-10-0-132-80 kubenswrapper[2544]: W0916 18:43:57.540380    2544 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 16 18:43:57.544132 ip-10-0-132-80 kubenswrapper[2544]: W0916 18:43:57.541804    2544 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 16 18:43:57.544645 ip-10-0-132-80 kubenswrapper[2544]: W0916 18:43:57.541818    2544 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 16 18:43:57.544645 ip-10-0-132-80 kubenswrapper[2544]: W0916 18:43:57.541824    2544 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 16 18:43:57.544645 ip-10-0-132-80 kubenswrapper[2544]: W0916 18:43:57.541832    2544 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 16 18:43:57.544645 ip-10-0-132-80 kubenswrapper[2544]: W0916 18:43:57.541839    2544 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 16 18:43:57.544645 ip-10-0-132-80 kubenswrapper[2544]: W0916 18:43:57.541884    2544 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 16 18:43:57.544645 ip-10-0-132-80 kubenswrapper[2544]: W0916 18:43:57.541890    2544 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 16 18:43:57.544645 ip-10-0-132-80 kubenswrapper[2544]: W0916 18:43:57.541895    2544 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 16 18:43:57.544645 ip-10-0-132-80 kubenswrapper[2544]: W0916 18:43:57.541901    2544 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 16 18:43:57.544645 ip-10-0-132-80 kubenswrapper[2544]: W0916 18:43:57.541906    2544 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 16 18:43:57.544645 ip-10-0-132-80 kubenswrapper[2544]: W0916 18:43:57.541912    2544 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 16 18:43:57.544645 ip-10-0-132-80 kubenswrapper[2544]: W0916 18:43:57.541932    2544 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 16 18:43:57.544645 ip-10-0-132-80 kubenswrapper[2544]: W0916 18:43:57.542073    2544 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 16 18:43:57.544645 ip-10-0-132-80 kubenswrapper[2544]: W0916 18:43:57.542080    2544 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 16 18:43:57.544645 ip-10-0-132-80 kubenswrapper[2544]: W0916 18:43:57.542083    2544 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 16 18:43:57.544645 ip-10-0-132-80 kubenswrapper[2544]: W0916 18:43:57.542087    2544 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 16 18:43:57.544645 ip-10-0-132-80 kubenswrapper[2544]: W0916 18:43:57.542090    2544 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 16 18:43:57.544645 ip-10-0-132-80 kubenswrapper[2544]: W0916 18:43:57.542093    2544 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 16 18:43:57.544645 ip-10-0-132-80 kubenswrapper[2544]: W0916 18:43:57.542096    2544 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 16 18:43:57.544645 ip-10-0-132-80 kubenswrapper[2544]: W0916 18:43:57.542099    2544 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 16 18:43:57.544645 ip-10-0-132-80 kubenswrapper[2544]: W0916 18:43:57.542102    2544 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 16 18:43:57.545210 ip-10-0-132-80 kubenswrapper[2544]: W0916 18:43:57.542105    2544 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 16 18:43:57.545210 ip-10-0-132-80 kubenswrapper[2544]: W0916 18:43:57.542108    2544 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 16 18:43:57.545210 ip-10-0-132-80 kubenswrapper[2544]: W0916 18:43:57.542110    2544 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 16 18:43:57.545210 ip-10-0-132-80 kubenswrapper[2544]: W0916 18:43:57.542113    2544 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 16 18:43:57.545210 ip-10-0-132-80 kubenswrapper[2544]: W0916 18:43:57.542115    2544 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 16 18:43:57.545210 ip-10-0-132-80 kubenswrapper[2544]: W0916 18:43:57.542118    2544 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 16 18:43:57.545210 ip-10-0-132-80 kubenswrapper[2544]: W0916 18:43:57.542121    2544 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 16 18:43:57.545210 ip-10-0-132-80 kubenswrapper[2544]: W0916 18:43:57.542123    2544 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 16 18:43:57.545210 ip-10-0-132-80 kubenswrapper[2544]: W0916 18:43:57.542128    2544 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 16 18:43:57.545210 ip-10-0-132-80 kubenswrapper[2544]: W0916 18:43:57.542133    2544 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 16 18:43:57.545210 ip-10-0-132-80 kubenswrapper[2544]: W0916 18:43:57.542136    2544 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 16 18:43:57.545210 ip-10-0-132-80 kubenswrapper[2544]: W0916 18:43:57.542142    2544 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 16 18:43:57.545210 ip-10-0-132-80 kubenswrapper[2544]: W0916 18:43:57.542146    2544 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 16 18:43:57.545210 ip-10-0-132-80 kubenswrapper[2544]: W0916 18:43:57.542149    2544 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 16 18:43:57.545210 ip-10-0-132-80 kubenswrapper[2544]: W0916 18:43:57.542152    2544 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 16 18:43:57.545210 ip-10-0-132-80 kubenswrapper[2544]: W0916 18:43:57.542155    2544 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 16 18:43:57.545210 ip-10-0-132-80 kubenswrapper[2544]: W0916 18:43:57.542158    2544 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 16 18:43:57.545210 ip-10-0-132-80 kubenswrapper[2544]: W0916 18:43:57.542161    2544 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 16 18:43:57.545210 ip-10-0-132-80 kubenswrapper[2544]: W0916 18:43:57.542163    2544 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 16 18:43:57.545682 ip-10-0-132-80 kubenswrapper[2544]: W0916 18:43:57.542166    2544 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 16 18:43:57.545682 ip-10-0-132-80 kubenswrapper[2544]: W0916 18:43:57.542169    2544 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 16 18:43:57.545682 ip-10-0-132-80 kubenswrapper[2544]: W0916 18:43:57.542172    2544 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 16 18:43:57.545682 ip-10-0-132-80 kubenswrapper[2544]: W0916 18:43:57.542180    2544 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 16 18:43:57.545682 ip-10-0-132-80 kubenswrapper[2544]: W0916 18:43:57.542183    2544 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 16 18:43:57.545682 ip-10-0-132-80 kubenswrapper[2544]: W0916 18:43:57.542186    2544 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 16 18:43:57.545682 ip-10-0-132-80 kubenswrapper[2544]: W0916 18:43:57.542189    2544 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 16 18:43:57.545682 ip-10-0-132-80 kubenswrapper[2544]: W0916 18:43:57.542192    2544 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 16 18:43:57.545682 ip-10-0-132-80 kubenswrapper[2544]: W0916 18:43:57.542195    2544 feature_gate.go:328] unrecognized feature gate: Example
Sep 16 18:43:57.545682 ip-10-0-132-80 kubenswrapper[2544]: W0916 18:43:57.542197    2544 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 16 18:43:57.545682 ip-10-0-132-80 kubenswrapper[2544]: W0916 18:43:57.542200    2544 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 16 18:43:57.545682 ip-10-0-132-80 kubenswrapper[2544]: W0916 18:43:57.542203    2544 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 16 18:43:57.545682 ip-10-0-132-80 kubenswrapper[2544]: W0916 18:43:57.542206    2544 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 16 18:43:57.545682 ip-10-0-132-80 kubenswrapper[2544]: W0916 18:43:57.542208    2544 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 16 18:43:57.545682 ip-10-0-132-80 kubenswrapper[2544]: W0916 18:43:57.542211    2544 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 16 18:43:57.545682 ip-10-0-132-80 kubenswrapper[2544]: W0916 18:43:57.542213    2544 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 16 18:43:57.545682 ip-10-0-132-80 kubenswrapper[2544]: W0916 18:43:57.542216    2544 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 16 18:43:57.545682 ip-10-0-132-80 kubenswrapper[2544]: W0916 18:43:57.542219    2544 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 16 18:43:57.545682 ip-10-0-132-80 kubenswrapper[2544]: W0916 18:43:57.542221    2544 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 16 18:43:57.545682 ip-10-0-132-80 kubenswrapper[2544]: W0916 18:43:57.542224    2544 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 16 18:43:57.546158 ip-10-0-132-80 kubenswrapper[2544]: W0916 18:43:57.542226    2544 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 16 18:43:57.546158 ip-10-0-132-80 kubenswrapper[2544]: W0916 18:43:57.542229    2544 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 16 18:43:57.546158 ip-10-0-132-80 kubenswrapper[2544]: W0916 18:43:57.542232    2544 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 16 18:43:57.546158 ip-10-0-132-80 kubenswrapper[2544]: W0916 18:43:57.542234    2544 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 16 18:43:57.546158 ip-10-0-132-80 kubenswrapper[2544]: W0916 18:43:57.542237    2544 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 16 18:43:57.546158 ip-10-0-132-80 kubenswrapper[2544]: W0916 18:43:57.542239    2544 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 16 18:43:57.546158 ip-10-0-132-80 kubenswrapper[2544]: W0916 18:43:57.542241    2544 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 16 18:43:57.546158 ip-10-0-132-80 kubenswrapper[2544]: W0916 18:43:57.542244    2544 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 16 18:43:57.546158 ip-10-0-132-80 kubenswrapper[2544]: W0916 18:43:57.542246    2544 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 16 18:43:57.546158 ip-10-0-132-80 kubenswrapper[2544]: W0916 18:43:57.542251    2544 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 16 18:43:57.546158 ip-10-0-132-80 kubenswrapper[2544]: W0916 18:43:57.542253    2544 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 16 18:43:57.546158 ip-10-0-132-80 kubenswrapper[2544]: W0916 18:43:57.542256    2544 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 16 18:43:57.546158 ip-10-0-132-80 kubenswrapper[2544]: W0916 18:43:57.542258    2544 feature_gate.go:328] unrecognized feature gate: Example2
Sep 16 18:43:57.546158 ip-10-0-132-80 kubenswrapper[2544]: W0916 18:43:57.542261    2544 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 16 18:43:57.546158 ip-10-0-132-80 kubenswrapper[2544]: W0916 18:43:57.542264    2544 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 16 18:43:57.546158 ip-10-0-132-80 kubenswrapper[2544]: W0916 18:43:57.542266    2544 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 16 18:43:57.546158 ip-10-0-132-80 kubenswrapper[2544]: W0916 18:43:57.542270    2544 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 16 18:43:57.546158 ip-10-0-132-80 kubenswrapper[2544]: W0916 18:43:57.542273    2544 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 16 18:43:57.546158 ip-10-0-132-80 kubenswrapper[2544]: W0916 18:43:57.542276    2544 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 16 18:43:57.546158 ip-10-0-132-80 kubenswrapper[2544]: W0916 18:43:57.542279    2544 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 16 18:43:57.546660 ip-10-0-132-80 kubenswrapper[2544]: W0916 18:43:57.542282    2544 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 16 18:43:57.546660 ip-10-0-132-80 kubenswrapper[2544]: W0916 18:43:57.542284    2544 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 16 18:43:57.546660 ip-10-0-132-80 kubenswrapper[2544]: W0916 18:43:57.542287    2544 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 16 18:43:57.546660 ip-10-0-132-80 kubenswrapper[2544]: W0916 18:43:57.542289    2544 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 16 18:43:57.546660 ip-10-0-132-80 kubenswrapper[2544]: W0916 18:43:57.542292    2544 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 16 18:43:57.546660 ip-10-0-132-80 kubenswrapper[2544]: W0916 18:43:57.542294    2544 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 16 18:43:57.546660 ip-10-0-132-80 kubenswrapper[2544]: W0916 18:43:57.542297    2544 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 16 18:43:57.546660 ip-10-0-132-80 kubenswrapper[2544]: W0916 18:43:57.542299    2544 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 16 18:43:57.546660 ip-10-0-132-80 kubenswrapper[2544]: W0916 18:43:57.542302    2544 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 16 18:43:57.546660 ip-10-0-132-80 kubenswrapper[2544]: W0916 18:43:57.542305    2544 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 16 18:43:57.546660 ip-10-0-132-80 kubenswrapper[2544]: W0916 18:43:57.542307    2544 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 16 18:43:57.546660 ip-10-0-132-80 kubenswrapper[2544]: W0916 18:43:57.542310    2544 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 16 18:43:57.546660 ip-10-0-132-80 kubenswrapper[2544]: W0916 18:43:57.542312    2544 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 16 18:43:57.546660 ip-10-0-132-80 kubenswrapper[2544]: W0916 18:43:57.542315    2544 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 16 18:43:57.546660 ip-10-0-132-80 kubenswrapper[2544]: W0916 18:43:57.542317    2544 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 16 18:43:57.546660 ip-10-0-132-80 kubenswrapper[2544]: W0916 18:43:57.542320    2544 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 16 18:43:57.546660 ip-10-0-132-80 kubenswrapper[2544]: W0916 18:43:57.542322    2544 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 16 18:43:57.546660 ip-10-0-132-80 kubenswrapper[2544]: W0916 18:43:57.542325    2544 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 16 18:43:57.546660 ip-10-0-132-80 kubenswrapper[2544]: W0916 18:43:57.542328    2544 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 16 18:43:57.546660 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:43:57.542400    2544 flags.go:64] FLAG: --address="0.0.0.0"
Sep 16 18:43:57.546660 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:43:57.542434    2544 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Sep 16 18:43:57.547170 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:43:57.542440    2544 flags.go:64] FLAG: --anonymous-auth="true"
Sep 16 18:43:57.547170 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:43:57.542445    2544 flags.go:64] FLAG: --application-metrics-count-limit="100"
Sep 16 18:43:57.547170 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:43:57.542449    2544 flags.go:64] FLAG: --authentication-token-webhook="false"
Sep 16 18:43:57.547170 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:43:57.542453    2544 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Sep 16 18:43:57.547170 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:43:57.542457    2544 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Sep 16 18:43:57.547170 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:43:57.542461    2544 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Sep 16 18:43:57.547170 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:43:57.542465    2544 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Sep 16 18:43:57.547170 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:43:57.542473    2544 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Sep 16 18:43:57.547170 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:43:57.542476    2544 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Sep 16 18:43:57.547170 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:43:57.542480    2544 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Sep 16 18:43:57.547170 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:43:57.542484    2544 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Sep 16 18:43:57.547170 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:43:57.542487    2544 flags.go:64] FLAG: --cgroup-root=""
Sep 16 18:43:57.547170 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:43:57.542490    2544 flags.go:64] FLAG: --cgroups-per-qos="true"
Sep 16 18:43:57.547170 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:43:57.542493    2544 flags.go:64] FLAG: --client-ca-file=""
Sep 16 18:43:57.547170 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:43:57.542496    2544 flags.go:64] FLAG: --cloud-provider="external"
Sep 16 18:43:57.547170 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:43:57.542499    2544 flags.go:64] FLAG: --cluster-dns="[]"
Sep 16 18:43:57.547170 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:43:57.542504    2544 flags.go:64] FLAG: --cluster-domain=""
Sep 16 18:43:57.547170 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:43:57.542507    2544 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Sep 16 18:43:57.547170 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:43:57.542510    2544 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Sep 16 18:43:57.547170 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:43:57.542513    2544 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Sep 16 18:43:57.547170 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:43:57.542517    2544 flags.go:64] FLAG: --container-log-max-files="5"
Sep 16 18:43:57.547170 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:43:57.542523    2544 flags.go:64] FLAG: --container-log-max-size="10Mi"
Sep 16 18:43:57.547170 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:43:57.542526    2544 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Sep 16 18:43:57.547719 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:43:57.542529    2544 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Sep 16 18:43:57.547719 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:43:57.542533    2544 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Sep 16 18:43:57.547719 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:43:57.542568    2544 flags.go:64] FLAG: --contention-profiling="false"
Sep 16 18:43:57.547719 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:43:57.542573    2544 flags.go:64] FLAG: --cpu-cfs-quota="true"
Sep 16 18:43:57.547719 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:43:57.542576    2544 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Sep 16 18:43:57.547719 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:43:57.542580    2544 flags.go:64] FLAG: --cpu-manager-policy="none"
Sep 16 18:43:57.547719 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:43:57.542584    2544 flags.go:64] FLAG: --cpu-manager-policy-options=""
Sep 16 18:43:57.547719 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:43:57.542589    2544 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Sep 16 18:43:57.547719 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:43:57.542592    2544 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Sep 16 18:43:57.547719 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:43:57.542595    2544 flags.go:64] FLAG: --enable-debugging-handlers="true"
Sep 16 18:43:57.547719 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:43:57.542598    2544 flags.go:64] FLAG: --enable-load-reader="false"
Sep 16 18:43:57.547719 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:43:57.542601    2544 flags.go:64] FLAG: --enable-server="true"
Sep 16 18:43:57.547719 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:43:57.542604    2544 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Sep 16 18:43:57.547719 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:43:57.542610    2544 flags.go:64] FLAG: --event-burst="100"
Sep 16 18:43:57.547719 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:43:57.542613    2544 flags.go:64] FLAG: --event-qps="50"
Sep 16 18:43:57.547719 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:43:57.542617    2544 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Sep 16 18:43:57.547719 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:43:57.542622    2544 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Sep 16 18:43:57.547719 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:43:57.542625    2544 flags.go:64] FLAG: --eviction-hard=""
Sep 16 18:43:57.547719 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:43:57.542629    2544 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Sep 16 18:43:57.547719 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:43:57.542632    2544 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Sep 16 18:43:57.547719 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:43:57.542635    2544 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Sep 16 18:43:57.547719 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:43:57.542638    2544 flags.go:64] FLAG: --eviction-soft=""
Sep 16 18:43:57.547719 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:43:57.542641    2544 flags.go:64] FLAG: --eviction-soft-grace-period=""
Sep 16 18:43:57.547719 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:43:57.542644    2544 flags.go:64] FLAG: --exit-on-lock-contention="false"
Sep 16 18:43:57.547719 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:43:57.542647    2544 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Sep 16 18:43:57.548376 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:43:57.542651    2544 flags.go:64] FLAG: --experimental-mounter-path=""
Sep 16 18:43:57.548376 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:43:57.542654    2544 flags.go:64] FLAG: --fail-cgroupv1="false"
Sep 16 18:43:57.548376 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:43:57.542657    2544 flags.go:64] FLAG: --fail-swap-on="true"
Sep 16 18:43:57.548376 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:43:57.542660    2544 flags.go:64] FLAG: --feature-gates=""
Sep 16 18:43:57.548376 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:43:57.542663    2544 flags.go:64] FLAG: --file-check-frequency="20s"
Sep 16 18:43:57.548376 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:43:57.542666    2544 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Sep 16 18:43:57.548376 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:43:57.542669    2544 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Sep 16 18:43:57.548376 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:43:57.542672    2544 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Sep 16 18:43:57.548376 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:43:57.542675    2544 flags.go:64] FLAG: --healthz-port="10248"
Sep 16 18:43:57.548376 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:43:57.542679    2544 flags.go:64] FLAG: --help="false"
Sep 16 18:43:57.548376 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:43:57.542682    2544 flags.go:64] FLAG: --hostname-override="ip-10-0-132-80.ec2.internal"
Sep 16 18:43:57.548376 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:43:57.542685    2544 flags.go:64] FLAG: --housekeeping-interval="10s"
Sep 16 18:43:57.548376 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:43:57.542688    2544 flags.go:64] FLAG: --http-check-frequency="20s"
Sep 16 18:43:57.548376 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:43:57.542691    2544 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Sep 16 18:43:57.548376 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:43:57.542694    2544 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Sep 16 18:43:57.548376 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:43:57.542697    2544 flags.go:64] FLAG: --image-gc-high-threshold="85"
Sep 16 18:43:57.548376 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:43:57.542700    2544 flags.go:64] FLAG: --image-gc-low-threshold="80"
Sep 16 18:43:57.548376 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:43:57.542702    2544 flags.go:64] FLAG: --image-service-endpoint=""
Sep 16 18:43:57.548376 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:43:57.542705    2544 flags.go:64] FLAG: --kernel-memcg-notification="false"
Sep 16 18:43:57.548376 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:43:57.542708    2544 flags.go:64] FLAG: --kube-api-burst="100"
Sep 16 18:43:57.548376 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:43:57.542711    2544 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Sep 16 18:43:57.548376 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:43:57.542714    2544 flags.go:64] FLAG: --kube-api-qps="50"
Sep 16 18:43:57.548376 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:43:57.542717    2544 flags.go:64] FLAG: --kube-reserved=""
Sep 16 18:43:57.548376 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:43:57.542722    2544 flags.go:64] FLAG: --kube-reserved-cgroup=""
Sep 16 18:43:57.549015 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:43:57.542724    2544 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Sep 16 18:43:57.549015 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:43:57.542728    2544 flags.go:64] FLAG: --kubelet-cgroups=""
Sep 16 18:43:57.549015 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:43:57.542731    2544 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Sep 16 18:43:57.549015 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:43:57.542734    2544 flags.go:64] FLAG: --lock-file=""
Sep 16 18:43:57.549015 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:43:57.542736    2544 flags.go:64] FLAG: --log-cadvisor-usage="false"
Sep 16 18:43:57.549015 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:43:57.542739    2544 flags.go:64] FLAG: --log-flush-frequency="5s"
Sep 16 18:43:57.549015 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:43:57.542742    2544 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Sep 16 18:43:57.549015 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:43:57.542748    2544 flags.go:64] FLAG: --log-json-split-stream="false"
Sep 16 18:43:57.549015 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:43:57.542751    2544 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Sep 16 18:43:57.549015 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:43:57.542754    2544 flags.go:64] FLAG: --log-text-split-stream="false"
Sep 16 18:43:57.549015 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:43:57.542765    2544 flags.go:64] FLAG: --logging-format="text"
Sep 16 18:43:57.549015 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:43:57.542768    2544 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Sep 16 18:43:57.549015 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:43:57.542771    2544 flags.go:64] FLAG: --make-iptables-util-chains="true"
Sep 16 18:43:57.549015 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:43:57.542774    2544 flags.go:64] FLAG: --manifest-url=""
Sep 16 18:43:57.549015 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:43:57.542777    2544 flags.go:64] FLAG: --manifest-url-header=""
Sep 16 18:43:57.549015 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:43:57.542782    2544 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Sep 16 18:43:57.549015 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:43:57.542785    2544 flags.go:64] FLAG: --max-open-files="1000000"
Sep 16 18:43:57.549015 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:43:57.542789    2544 flags.go:64] FLAG: --max-pods="110"
Sep 16 18:43:57.549015 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:43:57.542792    2544 flags.go:64] FLAG: --maximum-dead-containers="-1"
Sep 16 18:43:57.549015 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:43:57.542795    2544 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Sep 16 18:43:57.549015 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:43:57.542798    2544 flags.go:64] FLAG: --memory-manager-policy="None"
Sep 16 18:43:57.549015 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:43:57.542801    2544 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Sep 16 18:43:57.549015 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:43:57.542804    2544 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Sep 16 18:43:57.549015 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:43:57.542807    2544 flags.go:64] FLAG: --node-ip="0.0.0.0"
Sep 16 18:43:57.549015 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:43:57.542810    2544 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Sep 16 18:43:57.549626 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:43:57.542817    2544 flags.go:64] FLAG: --node-status-max-images="50"
Sep 16 18:43:57.549626 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:43:57.542820    2544 flags.go:64] FLAG: --node-status-update-frequency="10s"
Sep 16 18:43:57.549626 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:43:57.542823    2544 flags.go:64] FLAG: --oom-score-adj="-999"
Sep 16 18:43:57.549626 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:43:57.542826    2544 flags.go:64] FLAG: --pod-cidr=""
Sep 16 18:43:57.549626 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:43:57.542830    2544 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:eee392dbe3ac2ed4a38bca29ae3bd486f5b7780406c62c600154cd1906871b10"
Sep 16 18:43:57.549626 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:43:57.542835    2544 flags.go:64] FLAG: --pod-manifest-path=""
Sep 16 18:43:57.549626 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:43:57.542840    2544 flags.go:64] FLAG: --pod-max-pids="-1"
Sep 16 18:43:57.549626 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:43:57.542843    2544 flags.go:64] FLAG: --pods-per-core="0"
Sep 16 18:43:57.549626 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:43:57.542846    2544 flags.go:64] FLAG: --port="10250"
Sep 16 18:43:57.549626 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:43:57.542849    2544 flags.go:64] FLAG: --protect-kernel-defaults="false"
Sep 16 18:43:57.549626 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:43:57.542852    2544 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-0eaffd4b0a2d198da"
Sep 16 18:43:57.549626 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:43:57.542856    2544 flags.go:64] FLAG: --qos-reserved=""
Sep 16 18:43:57.549626 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:43:57.542859    2544 flags.go:64] FLAG: --read-only-port="10255"
Sep 16 18:43:57.549626 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:43:57.542862    2544 flags.go:64] FLAG: --register-node="true"
Sep 16 18:43:57.549626 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:43:57.542864    2544 flags.go:64] FLAG: --register-with-taints=""
Sep 16 18:43:57.549626 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:43:57.542868    2544 flags.go:64] FLAG: --registry-burst="10"
Sep 16 18:43:57.549626 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:43:57.542871    2544 flags.go:64] FLAG: --registry-qps="5"
Sep 16 18:43:57.549626 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:43:57.542874    2544 flags.go:64] FLAG: --reserved-cpus=""
Sep 16 18:43:57.549626 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:43:57.542877    2544 flags.go:64] FLAG: --reserved-memory=""
Sep 16 18:43:57.549626 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:43:57.542881    2544 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Sep 16 18:43:57.549626 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:43:57.542884    2544 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Sep 16 18:43:57.549626 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:43:57.542887    2544 flags.go:64] FLAG: --rotate-certificates="false"
Sep 16 18:43:57.549626 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:43:57.542890    2544 flags.go:64] FLAG: --rotate-server-certificates="false"
Sep 16 18:43:57.549626 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:43:57.542893    2544 flags.go:64] FLAG: --runonce="false"
Sep 16 18:43:57.549626 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:43:57.542895    2544 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Sep 16 18:43:57.550230 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:43:57.542898    2544 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Sep 16 18:43:57.550230 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:43:57.542901    2544 flags.go:64] FLAG: --seccomp-default="false"
Sep 16 18:43:57.550230 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:43:57.542904    2544 flags.go:64] FLAG: --serialize-image-pulls="true"
Sep 16 18:43:57.550230 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:43:57.542907    2544 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Sep 16 18:43:57.550230 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:43:57.542910    2544 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Sep 16 18:43:57.550230 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:43:57.542913    2544 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Sep 16 18:43:57.550230 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:43:57.542916    2544 flags.go:64] FLAG: --storage-driver-password="root"
Sep 16 18:43:57.550230 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:43:57.542919    2544 flags.go:64] FLAG: --storage-driver-secure="false"
Sep 16 18:43:57.550230 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:43:57.542922    2544 flags.go:64] FLAG: --storage-driver-table="stats"
Sep 16 18:43:57.550230 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:43:57.542925    2544 flags.go:64] FLAG: --storage-driver-user="root"
Sep 16 18:43:57.550230 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:43:57.542927    2544 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Sep 16 18:43:57.550230 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:43:57.542930    2544 flags.go:64] FLAG: --sync-frequency="1m0s"
Sep 16 18:43:57.550230 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:43:57.542933    2544 flags.go:64] FLAG: --system-cgroups=""
Sep 16 18:43:57.550230 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:43:57.542938    2544 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Sep 16 18:43:57.550230 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:43:57.542945    2544 flags.go:64] FLAG: --system-reserved-cgroup=""
Sep 16 18:43:57.550230 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:43:57.542947    2544 flags.go:64] FLAG: --tls-cert-file=""
Sep 16 18:43:57.550230 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:43:57.542950    2544 flags.go:64] FLAG: --tls-cipher-suites="[]"
Sep 16 18:43:57.550230 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:43:57.542954    2544 flags.go:64] FLAG: --tls-min-version=""
Sep 16 18:43:57.550230 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:43:57.542957    2544 flags.go:64] FLAG: --tls-private-key-file=""
Sep 16 18:43:57.550230 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:43:57.542960    2544 flags.go:64] FLAG: --topology-manager-policy="none"
Sep 16 18:43:57.550230 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:43:57.542963    2544 flags.go:64] FLAG: --topology-manager-policy-options=""
Sep 16 18:43:57.550230 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:43:57.542966    2544 flags.go:64] FLAG: --topology-manager-scope="container"
Sep 16 18:43:57.550230 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:43:57.542969    2544 flags.go:64] FLAG: --v="2"
Sep 16 18:43:57.550230 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:43:57.542974    2544 flags.go:64] FLAG: --version="false"
Sep 16 18:43:57.550230 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:43:57.542978    2544 flags.go:64] FLAG: --vmodule=""
Sep 16 18:43:57.550856 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:43:57.542982    2544 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Sep 16 18:43:57.550856 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:43:57.542985    2544 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Sep 16 18:43:57.550856 ip-10-0-132-80 kubenswrapper[2544]: W0916 18:43:57.543120    2544 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 16 18:43:57.550856 ip-10-0-132-80 kubenswrapper[2544]: W0916 18:43:57.543124    2544 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 16 18:43:57.550856 ip-10-0-132-80 kubenswrapper[2544]: W0916 18:43:57.543127    2544 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 16 18:43:57.550856 ip-10-0-132-80 kubenswrapper[2544]: W0916 18:43:57.543130    2544 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 16 18:43:57.550856 ip-10-0-132-80 kubenswrapper[2544]: W0916 18:43:57.543133    2544 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 16 18:43:57.550856 ip-10-0-132-80 kubenswrapper[2544]: W0916 18:43:57.543136    2544 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 16 18:43:57.550856 ip-10-0-132-80 kubenswrapper[2544]: W0916 18:43:57.543139    2544 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 16 18:43:57.550856 ip-10-0-132-80 kubenswrapper[2544]: W0916 18:43:57.543142    2544 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 16 18:43:57.550856 ip-10-0-132-80 kubenswrapper[2544]: W0916 18:43:57.543145    2544 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 16 18:43:57.550856 ip-10-0-132-80 kubenswrapper[2544]: W0916 18:43:57.543148    2544 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 16 18:43:57.550856 ip-10-0-132-80 kubenswrapper[2544]: W0916 18:43:57.543150    2544 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 16 18:43:57.550856 ip-10-0-132-80 kubenswrapper[2544]: W0916 18:43:57.543153    2544 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 16 18:43:57.550856 ip-10-0-132-80 kubenswrapper[2544]: W0916 18:43:57.543156    2544 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 16 18:43:57.550856 ip-10-0-132-80 kubenswrapper[2544]: W0916 18:43:57.543158    2544 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 16 18:43:57.550856 ip-10-0-132-80 kubenswrapper[2544]: W0916 18:43:57.543161    2544 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 16 18:43:57.550856 ip-10-0-132-80 kubenswrapper[2544]: W0916 18:43:57.543163    2544 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 16 18:43:57.550856 ip-10-0-132-80 kubenswrapper[2544]: W0916 18:43:57.543166    2544 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 16 18:43:57.551323 ip-10-0-132-80 kubenswrapper[2544]: W0916 18:43:57.543169    2544 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 16 18:43:57.551323 ip-10-0-132-80 kubenswrapper[2544]: W0916 18:43:57.543175    2544 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 16 18:43:57.551323 ip-10-0-132-80 kubenswrapper[2544]: W0916 18:43:57.543177    2544 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 16 18:43:57.551323 ip-10-0-132-80 kubenswrapper[2544]: W0916 18:43:57.543180    2544 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 16 18:43:57.551323 ip-10-0-132-80 kubenswrapper[2544]: W0916 18:43:57.543183    2544 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 16 18:43:57.551323 ip-10-0-132-80 kubenswrapper[2544]: W0916 18:43:57.543186    2544 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 16 18:43:57.551323 ip-10-0-132-80 kubenswrapper[2544]: W0916 18:43:57.543188    2544 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 16 18:43:57.551323 ip-10-0-132-80 kubenswrapper[2544]: W0916 18:43:57.543191    2544 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 16 18:43:57.551323 ip-10-0-132-80 kubenswrapper[2544]: W0916 18:43:57.543194    2544 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 16 18:43:57.551323 ip-10-0-132-80 kubenswrapper[2544]: W0916 18:43:57.543196    2544 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 16 18:43:57.551323 ip-10-0-132-80 kubenswrapper[2544]: W0916 18:43:57.543199    2544 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 16 18:43:57.551323 ip-10-0-132-80 kubenswrapper[2544]: W0916 18:43:57.543202    2544 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 16 18:43:57.551323 ip-10-0-132-80 kubenswrapper[2544]: W0916 18:43:57.543205    2544 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 16 18:43:57.551323 ip-10-0-132-80 kubenswrapper[2544]: W0916 18:43:57.543207    2544 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 16 18:43:57.551323 ip-10-0-132-80 kubenswrapper[2544]: W0916 18:43:57.543211    2544 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 16 18:43:57.551323 ip-10-0-132-80 kubenswrapper[2544]: W0916 18:43:57.543213    2544 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 16 18:43:57.551323 ip-10-0-132-80 kubenswrapper[2544]: W0916 18:43:57.543216    2544 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 16 18:43:57.551323 ip-10-0-132-80 kubenswrapper[2544]: W0916 18:43:57.543219    2544 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 16 18:43:57.551323 ip-10-0-132-80 kubenswrapper[2544]: W0916 18:43:57.543221    2544 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 16 18:43:57.551800 ip-10-0-132-80 kubenswrapper[2544]: W0916 18:43:57.543224    2544 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 16 18:43:57.551800 ip-10-0-132-80 kubenswrapper[2544]: W0916 18:43:57.543227    2544 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 16 18:43:57.551800 ip-10-0-132-80 kubenswrapper[2544]: W0916 18:43:57.543229    2544 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 16 18:43:57.551800 ip-10-0-132-80 kubenswrapper[2544]: W0916 18:43:57.543232    2544 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 16 18:43:57.551800 ip-10-0-132-80 kubenswrapper[2544]: W0916 18:43:57.543235    2544 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 16 18:43:57.551800 ip-10-0-132-80 kubenswrapper[2544]: W0916 18:43:57.543237    2544 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 16 18:43:57.551800 ip-10-0-132-80 kubenswrapper[2544]: W0916 18:43:57.543241    2544 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 16 18:43:57.551800 ip-10-0-132-80 kubenswrapper[2544]: W0916 18:43:57.543245    2544 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 16 18:43:57.551800 ip-10-0-132-80 kubenswrapper[2544]: W0916 18:43:57.543249    2544 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 16 18:43:57.551800 ip-10-0-132-80 kubenswrapper[2544]: W0916 18:43:57.543252    2544 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 16 18:43:57.551800 ip-10-0-132-80 kubenswrapper[2544]: W0916 18:43:57.543254    2544 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 16 18:43:57.551800 ip-10-0-132-80 kubenswrapper[2544]: W0916 18:43:57.543257    2544 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 16 18:43:57.551800 ip-10-0-132-80 kubenswrapper[2544]: W0916 18:43:57.543260    2544 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 16 18:43:57.551800 ip-10-0-132-80 kubenswrapper[2544]: W0916 18:43:57.543263    2544 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 16 18:43:57.551800 ip-10-0-132-80 kubenswrapper[2544]: W0916 18:43:57.543267    2544 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 16 18:43:57.551800 ip-10-0-132-80 kubenswrapper[2544]: W0916 18:43:57.543270    2544 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 16 18:43:57.551800 ip-10-0-132-80 kubenswrapper[2544]: W0916 18:43:57.543272    2544 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 16 18:43:57.551800 ip-10-0-132-80 kubenswrapper[2544]: W0916 18:43:57.543275    2544 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 16 18:43:57.551800 ip-10-0-132-80 kubenswrapper[2544]: W0916 18:43:57.543278    2544 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 16 18:43:57.551800 ip-10-0-132-80 kubenswrapper[2544]: W0916 18:43:57.543281    2544 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 16 18:43:57.552297 ip-10-0-132-80 kubenswrapper[2544]: W0916 18:43:57.543283    2544 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 16 18:43:57.552297 ip-10-0-132-80 kubenswrapper[2544]: W0916 18:43:57.543286    2544 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 16 18:43:57.552297 ip-10-0-132-80 kubenswrapper[2544]: W0916 18:43:57.543289    2544 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 16 18:43:57.552297 ip-10-0-132-80 kubenswrapper[2544]: W0916 18:43:57.543291    2544 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 16 18:43:57.552297 ip-10-0-132-80 kubenswrapper[2544]: W0916 18:43:57.543294    2544 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 16 18:43:57.552297 ip-10-0-132-80 kubenswrapper[2544]: W0916 18:43:57.543304    2544 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 16 18:43:57.552297 ip-10-0-132-80 kubenswrapper[2544]: W0916 18:43:57.543306    2544 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 16 18:43:57.552297 ip-10-0-132-80 kubenswrapper[2544]: W0916 18:43:57.543309    2544 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 16 18:43:57.552297 ip-10-0-132-80 kubenswrapper[2544]: W0916 18:43:57.543312    2544 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 16 18:43:57.552297 ip-10-0-132-80 kubenswrapper[2544]: W0916 18:43:57.543315    2544 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 16 18:43:57.552297 ip-10-0-132-80 kubenswrapper[2544]: W0916 18:43:57.543318    2544 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 16 18:43:57.552297 ip-10-0-132-80 kubenswrapper[2544]: W0916 18:43:57.543320    2544 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 16 18:43:57.552297 ip-10-0-132-80 kubenswrapper[2544]: W0916 18:43:57.543324    2544 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 16 18:43:57.552297 ip-10-0-132-80 kubenswrapper[2544]: W0916 18:43:57.543328    2544 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 16 18:43:57.552297 ip-10-0-132-80 kubenswrapper[2544]: W0916 18:43:57.543331    2544 feature_gate.go:328] unrecognized feature gate: Example2
Sep 16 18:43:57.552297 ip-10-0-132-80 kubenswrapper[2544]: W0916 18:43:57.543333    2544 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 16 18:43:57.552297 ip-10-0-132-80 kubenswrapper[2544]: W0916 18:43:57.543336    2544 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 16 18:43:57.552297 ip-10-0-132-80 kubenswrapper[2544]: W0916 18:43:57.543339    2544 feature_gate.go:328] unrecognized feature gate: Example
Sep 16 18:43:57.552297 ip-10-0-132-80 kubenswrapper[2544]: W0916 18:43:57.543342    2544 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 16 18:43:57.552297 ip-10-0-132-80 kubenswrapper[2544]: W0916 18:43:57.543345    2544 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 16 18:43:57.552881 ip-10-0-132-80 kubenswrapper[2544]: W0916 18:43:57.543347    2544 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 16 18:43:57.552881 ip-10-0-132-80 kubenswrapper[2544]: W0916 18:43:57.543350    2544 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 16 18:43:57.552881 ip-10-0-132-80 kubenswrapper[2544]: W0916 18:43:57.543353    2544 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 16 18:43:57.552881 ip-10-0-132-80 kubenswrapper[2544]: W0916 18:43:57.543355    2544 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 16 18:43:57.552881 ip-10-0-132-80 kubenswrapper[2544]: W0916 18:43:57.543358    2544 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 16 18:43:57.552881 ip-10-0-132-80 kubenswrapper[2544]: W0916 18:43:57.543361    2544 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 16 18:43:57.552881 ip-10-0-132-80 kubenswrapper[2544]: W0916 18:43:57.543365    2544 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 16 18:43:57.552881 ip-10-0-132-80 kubenswrapper[2544]: W0916 18:43:57.543367    2544 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 16 18:43:57.552881 ip-10-0-132-80 kubenswrapper[2544]: W0916 18:43:57.543370    2544 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 16 18:43:57.552881 ip-10-0-132-80 kubenswrapper[2544]: W0916 18:43:57.543373    2544 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 16 18:43:57.552881 ip-10-0-132-80 kubenswrapper[2544]: W0916 18:43:57.543386    2544 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 16 18:43:57.552881 ip-10-0-132-80 kubenswrapper[2544]: W0916 18:43:57.543389    2544 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 16 18:43:57.552881 ip-10-0-132-80 kubenswrapper[2544]: W0916 18:43:57.543392    2544 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 16 18:43:57.552881 ip-10-0-132-80 kubenswrapper[2544]: W0916 18:43:57.543395    2544 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 16 18:43:57.552881 ip-10-0-132-80 kubenswrapper[2544]: W0916 18:43:57.543397    2544 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 16 18:43:57.552881 ip-10-0-132-80 kubenswrapper[2544]: W0916 18:43:57.543400    2544 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 16 18:43:57.552881 ip-10-0-132-80 kubenswrapper[2544]: W0916 18:43:57.543403    2544 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 16 18:43:57.552881 ip-10-0-132-80 kubenswrapper[2544]: W0916 18:43:57.543406    2544 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 16 18:43:57.552881 ip-10-0-132-80 kubenswrapper[2544]: W0916 18:43:57.543408    2544 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 16 18:43:57.552881 ip-10-0-132-80 kubenswrapper[2544]: W0916 18:43:57.543411    2544 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 16 18:43:57.553355 ip-10-0-132-80 kubenswrapper[2544]: W0916 18:43:57.543414    2544 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 16 18:43:57.553355 ip-10-0-132-80 kubenswrapper[2544]: W0916 18:43:57.543417    2544 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 16 18:43:57.553355 ip-10-0-132-80 kubenswrapper[2544]: W0916 18:43:57.543421    2544 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 16 18:43:57.553355 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:43:57.544311    2544 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 16 18:43:57.553355 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:43:57.551774    2544 server.go:529] "Kubelet version" kubeletVersion="v1.34.9"
Sep 16 18:43:57.553355 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:43:57.551791    2544 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Sep 16 18:43:57.553355 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:43:57.551810    2544 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Sep 16 18:43:57.553355 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:43:57.551820    2544 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Sep 16 18:43:57.553355 ip-10-0-132-80 kubenswrapper[2544]: W0916 18:43:57.551872    2544 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 16 18:43:57.553355 ip-10-0-132-80 kubenswrapper[2544]: W0916 18:43:57.551877    2544 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 16 18:43:57.553355 ip-10-0-132-80 kubenswrapper[2544]: W0916 18:43:57.551881    2544 feature_gate.go:328] unrecognized feature gate: Example2
Sep 16 18:43:57.553355 ip-10-0-132-80 kubenswrapper[2544]: W0916 18:43:57.551884    2544 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 16 18:43:57.553355 ip-10-0-132-80 kubenswrapper[2544]: W0916 18:43:57.551887    2544 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 16 18:43:57.553355 ip-10-0-132-80 kubenswrapper[2544]: W0916 18:43:57.551890    2544 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 16 18:43:57.553355 ip-10-0-132-80 kubenswrapper[2544]: W0916 18:43:57.551893    2544 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 16 18:43:57.553754 ip-10-0-132-80 kubenswrapper[2544]: W0916 18:43:57.551896    2544 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 16 18:43:57.553754 ip-10-0-132-80 kubenswrapper[2544]: W0916 18:43:57.551899    2544 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 16 18:43:57.553754 ip-10-0-132-80 kubenswrapper[2544]: W0916 18:43:57.551902    2544 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 16 18:43:57.553754 ip-10-0-132-80 kubenswrapper[2544]: W0916 18:43:57.551904    2544 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 16 18:43:57.553754 ip-10-0-132-80 kubenswrapper[2544]: W0916 18:43:57.551907    2544 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 16 18:43:57.553754 ip-10-0-132-80 kubenswrapper[2544]: W0916 18:43:57.551910    2544 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 16 18:43:57.553754 ip-10-0-132-80 kubenswrapper[2544]: W0916 18:43:57.551913    2544 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 16 18:43:57.553754 ip-10-0-132-80 kubenswrapper[2544]: W0916 18:43:57.551915    2544 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 16 18:43:57.553754 ip-10-0-132-80 kubenswrapper[2544]: W0916 18:43:57.551918    2544 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 16 18:43:57.553754 ip-10-0-132-80 kubenswrapper[2544]: W0916 18:43:57.551921    2544 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 16 18:43:57.553754 ip-10-0-132-80 kubenswrapper[2544]: W0916 18:43:57.551923    2544 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 16 18:43:57.553754 ip-10-0-132-80 kubenswrapper[2544]: W0916 18:43:57.551926    2544 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 16 18:43:57.553754 ip-10-0-132-80 kubenswrapper[2544]: W0916 18:43:57.551929    2544 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 16 18:43:57.553754 ip-10-0-132-80 kubenswrapper[2544]: W0916 18:43:57.551932    2544 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 16 18:43:57.553754 ip-10-0-132-80 kubenswrapper[2544]: W0916 18:43:57.551934    2544 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 16 18:43:57.553754 ip-10-0-132-80 kubenswrapper[2544]: W0916 18:43:57.551937    2544 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 16 18:43:57.553754 ip-10-0-132-80 kubenswrapper[2544]: W0916 18:43:57.551939    2544 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 16 18:43:57.553754 ip-10-0-132-80 kubenswrapper[2544]: W0916 18:43:57.551942    2544 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 16 18:43:57.553754 ip-10-0-132-80 kubenswrapper[2544]: W0916 18:43:57.551945    2544 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 16 18:43:57.554200 ip-10-0-132-80 kubenswrapper[2544]: W0916 18:43:57.551947    2544 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 16 18:43:57.554200 ip-10-0-132-80 kubenswrapper[2544]: W0916 18:43:57.551952    2544 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 16 18:43:57.554200 ip-10-0-132-80 kubenswrapper[2544]: W0916 18:43:57.551956    2544 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 16 18:43:57.554200 ip-10-0-132-80 kubenswrapper[2544]: W0916 18:43:57.551961    2544 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 16 18:43:57.554200 ip-10-0-132-80 kubenswrapper[2544]: W0916 18:43:57.551964    2544 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 16 18:43:57.554200 ip-10-0-132-80 kubenswrapper[2544]: W0916 18:43:57.551966    2544 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 16 18:43:57.554200 ip-10-0-132-80 kubenswrapper[2544]: W0916 18:43:57.551969    2544 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 16 18:43:57.554200 ip-10-0-132-80 kubenswrapper[2544]: W0916 18:43:57.551972    2544 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 16 18:43:57.554200 ip-10-0-132-80 kubenswrapper[2544]: W0916 18:43:57.551974    2544 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 16 18:43:57.554200 ip-10-0-132-80 kubenswrapper[2544]: W0916 18:43:57.551977    2544 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 16 18:43:57.554200 ip-10-0-132-80 kubenswrapper[2544]: W0916 18:43:57.551980    2544 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 16 18:43:57.554200 ip-10-0-132-80 kubenswrapper[2544]: W0916 18:43:57.551982    2544 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 16 18:43:57.554200 ip-10-0-132-80 kubenswrapper[2544]: W0916 18:43:57.551985    2544 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 16 18:43:57.554200 ip-10-0-132-80 kubenswrapper[2544]: W0916 18:43:57.551987    2544 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 16 18:43:57.554200 ip-10-0-132-80 kubenswrapper[2544]: W0916 18:43:57.551990    2544 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 16 18:43:57.554200 ip-10-0-132-80 kubenswrapper[2544]: W0916 18:43:57.551993    2544 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 16 18:43:57.554200 ip-10-0-132-80 kubenswrapper[2544]: W0916 18:43:57.551995    2544 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 16 18:43:57.554200 ip-10-0-132-80 kubenswrapper[2544]: W0916 18:43:57.551998    2544 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 16 18:43:57.554200 ip-10-0-132-80 kubenswrapper[2544]: W0916 18:43:57.552000    2544 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 16 18:43:57.554666 ip-10-0-132-80 kubenswrapper[2544]: W0916 18:43:57.552003    2544 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 16 18:43:57.554666 ip-10-0-132-80 kubenswrapper[2544]: W0916 18:43:57.552005    2544 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 16 18:43:57.554666 ip-10-0-132-80 kubenswrapper[2544]: W0916 18:43:57.552008    2544 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 16 18:43:57.554666 ip-10-0-132-80 kubenswrapper[2544]: W0916 18:43:57.552010    2544 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 16 18:43:57.554666 ip-10-0-132-80 kubenswrapper[2544]: W0916 18:43:57.552013    2544 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 16 18:43:57.554666 ip-10-0-132-80 kubenswrapper[2544]: W0916 18:43:57.552015    2544 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 16 18:43:57.554666 ip-10-0-132-80 kubenswrapper[2544]: W0916 18:43:57.552018    2544 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 16 18:43:57.554666 ip-10-0-132-80 kubenswrapper[2544]: W0916 18:43:57.552020    2544 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 16 18:43:57.554666 ip-10-0-132-80 kubenswrapper[2544]: W0916 18:43:57.552023    2544 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 16 18:43:57.554666 ip-10-0-132-80 kubenswrapper[2544]: W0916 18:43:57.552026    2544 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 16 18:43:57.554666 ip-10-0-132-80 kubenswrapper[2544]: W0916 18:43:57.552028    2544 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 16 18:43:57.554666 ip-10-0-132-80 kubenswrapper[2544]: W0916 18:43:57.552030    2544 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 16 18:43:57.554666 ip-10-0-132-80 kubenswrapper[2544]: W0916 18:43:57.552033    2544 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 16 18:43:57.554666 ip-10-0-132-80 kubenswrapper[2544]: W0916 18:43:57.552035    2544 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 16 18:43:57.554666 ip-10-0-132-80 kubenswrapper[2544]: W0916 18:43:57.552038    2544 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 16 18:43:57.554666 ip-10-0-132-80 kubenswrapper[2544]: W0916 18:43:57.552041    2544 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 16 18:43:57.554666 ip-10-0-132-80 kubenswrapper[2544]: W0916 18:43:57.552044    2544 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 16 18:43:57.554666 ip-10-0-132-80 kubenswrapper[2544]: W0916 18:43:57.552047    2544 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 16 18:43:57.554666 ip-10-0-132-80 kubenswrapper[2544]: W0916 18:43:57.552050    2544 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 16 18:43:57.555149 ip-10-0-132-80 kubenswrapper[2544]: W0916 18:43:57.552053    2544 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 16 18:43:57.555149 ip-10-0-132-80 kubenswrapper[2544]: W0916 18:43:57.552055    2544 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 16 18:43:57.555149 ip-10-0-132-80 kubenswrapper[2544]: W0916 18:43:57.552058    2544 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 16 18:43:57.555149 ip-10-0-132-80 kubenswrapper[2544]: W0916 18:43:57.552061    2544 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 16 18:43:57.555149 ip-10-0-132-80 kubenswrapper[2544]: W0916 18:43:57.552063    2544 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 16 18:43:57.555149 ip-10-0-132-80 kubenswrapper[2544]: W0916 18:43:57.552066    2544 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 16 18:43:57.555149 ip-10-0-132-80 kubenswrapper[2544]: W0916 18:43:57.552068    2544 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 16 18:43:57.555149 ip-10-0-132-80 kubenswrapper[2544]: W0916 18:43:57.552071    2544 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 16 18:43:57.555149 ip-10-0-132-80 kubenswrapper[2544]: W0916 18:43:57.552073    2544 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 16 18:43:57.555149 ip-10-0-132-80 kubenswrapper[2544]: W0916 18:43:57.552076    2544 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 16 18:43:57.555149 ip-10-0-132-80 kubenswrapper[2544]: W0916 18:43:57.552078    2544 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 16 18:43:57.555149 ip-10-0-132-80 kubenswrapper[2544]: W0916 18:43:57.552081    2544 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 16 18:43:57.555149 ip-10-0-132-80 kubenswrapper[2544]: W0916 18:43:57.552083    2544 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 16 18:43:57.555149 ip-10-0-132-80 kubenswrapper[2544]: W0916 18:43:57.552086    2544 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 16 18:43:57.555149 ip-10-0-132-80 kubenswrapper[2544]: W0916 18:43:57.552088    2544 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 16 18:43:57.555149 ip-10-0-132-80 kubenswrapper[2544]: W0916 18:43:57.552090    2544 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 16 18:43:57.555149 ip-10-0-132-80 kubenswrapper[2544]: W0916 18:43:57.552093    2544 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 16 18:43:57.555149 ip-10-0-132-80 kubenswrapper[2544]: W0916 18:43:57.552096    2544 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 16 18:43:57.555149 ip-10-0-132-80 kubenswrapper[2544]: W0916 18:43:57.552098    2544 feature_gate.go:328] unrecognized feature gate: Example
Sep 16 18:43:57.555149 ip-10-0-132-80 kubenswrapper[2544]: W0916 18:43:57.552101    2544 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 16 18:43:57.555647 ip-10-0-132-80 kubenswrapper[2544]: W0916 18:43:57.552103    2544 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 16 18:43:57.555647 ip-10-0-132-80 kubenswrapper[2544]: W0916 18:43:57.552106    2544 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 16 18:43:57.555647 ip-10-0-132-80 kubenswrapper[2544]: W0916 18:43:57.552110    2544 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 16 18:43:57.555647 ip-10-0-132-80 kubenswrapper[2544]: W0916 18:43:57.552113    2544 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 16 18:43:57.555647 ip-10-0-132-80 kubenswrapper[2544]: W0916 18:43:57.552116    2544 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 16 18:43:57.555647 ip-10-0-132-80 kubenswrapper[2544]: W0916 18:43:57.552119    2544 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 16 18:43:57.555647 ip-10-0-132-80 kubenswrapper[2544]: W0916 18:43:57.552122    2544 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 16 18:43:57.555647 ip-10-0-132-80 kubenswrapper[2544]: W0916 18:43:57.552125    2544 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 16 18:43:57.555647 ip-10-0-132-80 kubenswrapper[2544]: W0916 18:43:57.552128    2544 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 16 18:43:57.555647 ip-10-0-132-80 kubenswrapper[2544]: W0916 18:43:57.552132    2544 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 16 18:43:57.555647 ip-10-0-132-80 kubenswrapper[2544]: W0916 18:43:57.552136    2544 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 16 18:43:57.555647 ip-10-0-132-80 kubenswrapper[2544]: W0916 18:43:57.552139    2544 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 16 18:43:57.555647 ip-10-0-132-80 kubenswrapper[2544]: W0916 18:43:57.552143    2544 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 16 18:43:57.555647 ip-10-0-132-80 kubenswrapper[2544]: W0916 18:43:57.552145    2544 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 16 18:43:57.555647 ip-10-0-132-80 kubenswrapper[2544]: W0916 18:43:57.552148    2544 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 16 18:43:57.556044 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:43:57.552153    2544 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 16 18:43:57.556044 ip-10-0-132-80 kubenswrapper[2544]: W0916 18:43:57.552259    2544 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 16 18:43:57.556044 ip-10-0-132-80 kubenswrapper[2544]: W0916 18:43:57.552263    2544 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 16 18:43:57.556044 ip-10-0-132-80 kubenswrapper[2544]: W0916 18:43:57.552266    2544 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 16 18:43:57.556044 ip-10-0-132-80 kubenswrapper[2544]: W0916 18:43:57.552269    2544 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 16 18:43:57.556044 ip-10-0-132-80 kubenswrapper[2544]: W0916 18:43:57.552271    2544 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 16 18:43:57.556044 ip-10-0-132-80 kubenswrapper[2544]: W0916 18:43:57.552274    2544 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 16 18:43:57.556044 ip-10-0-132-80 kubenswrapper[2544]: W0916 18:43:57.552277    2544 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 16 18:43:57.556044 ip-10-0-132-80 kubenswrapper[2544]: W0916 18:43:57.552280    2544 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 16 18:43:57.556044 ip-10-0-132-80 kubenswrapper[2544]: W0916 18:43:57.552283    2544 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 16 18:43:57.556044 ip-10-0-132-80 kubenswrapper[2544]: W0916 18:43:57.552285    2544 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 16 18:43:57.556044 ip-10-0-132-80 kubenswrapper[2544]: W0916 18:43:57.552288    2544 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 16 18:43:57.556044 ip-10-0-132-80 kubenswrapper[2544]: W0916 18:43:57.552290    2544 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 16 18:43:57.556044 ip-10-0-132-80 kubenswrapper[2544]: W0916 18:43:57.552293    2544 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 16 18:43:57.556044 ip-10-0-132-80 kubenswrapper[2544]: W0916 18:43:57.552296    2544 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 16 18:43:57.556414 ip-10-0-132-80 kubenswrapper[2544]: W0916 18:43:57.552298    2544 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 16 18:43:57.556414 ip-10-0-132-80 kubenswrapper[2544]: W0916 18:43:57.552301    2544 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 16 18:43:57.556414 ip-10-0-132-80 kubenswrapper[2544]: W0916 18:43:57.552304    2544 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 16 18:43:57.556414 ip-10-0-132-80 kubenswrapper[2544]: W0916 18:43:57.552307    2544 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 16 18:43:57.556414 ip-10-0-132-80 kubenswrapper[2544]: W0916 18:43:57.552309    2544 feature_gate.go:328] unrecognized feature gate: Example
Sep 16 18:43:57.556414 ip-10-0-132-80 kubenswrapper[2544]: W0916 18:43:57.552312    2544 feature_gate.go:328] unrecognized feature gate: Example2
Sep 16 18:43:57.556414 ip-10-0-132-80 kubenswrapper[2544]: W0916 18:43:57.552314    2544 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 16 18:43:57.556414 ip-10-0-132-80 kubenswrapper[2544]: W0916 18:43:57.552317    2544 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 16 18:43:57.556414 ip-10-0-132-80 kubenswrapper[2544]: W0916 18:43:57.552320    2544 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 16 18:43:57.556414 ip-10-0-132-80 kubenswrapper[2544]: W0916 18:43:57.552322    2544 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 16 18:43:57.556414 ip-10-0-132-80 kubenswrapper[2544]: W0916 18:43:57.552325    2544 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 16 18:43:57.556414 ip-10-0-132-80 kubenswrapper[2544]: W0916 18:43:57.552328    2544 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 16 18:43:57.556414 ip-10-0-132-80 kubenswrapper[2544]: W0916 18:43:57.552331    2544 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 16 18:43:57.556414 ip-10-0-132-80 kubenswrapper[2544]: W0916 18:43:57.552333    2544 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 16 18:43:57.556414 ip-10-0-132-80 kubenswrapper[2544]: W0916 18:43:57.552338    2544 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 16 18:43:57.556414 ip-10-0-132-80 kubenswrapper[2544]: W0916 18:43:57.552341    2544 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 16 18:43:57.556414 ip-10-0-132-80 kubenswrapper[2544]: W0916 18:43:57.552344    2544 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 16 18:43:57.556414 ip-10-0-132-80 kubenswrapper[2544]: W0916 18:43:57.552348    2544 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 16 18:43:57.556414 ip-10-0-132-80 kubenswrapper[2544]: W0916 18:43:57.552350    2544 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 16 18:43:57.556894 ip-10-0-132-80 kubenswrapper[2544]: W0916 18:43:57.552354    2544 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 16 18:43:57.556894 ip-10-0-132-80 kubenswrapper[2544]: W0916 18:43:57.552358    2544 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 16 18:43:57.556894 ip-10-0-132-80 kubenswrapper[2544]: W0916 18:43:57.552361    2544 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 16 18:43:57.556894 ip-10-0-132-80 kubenswrapper[2544]: W0916 18:43:57.552364    2544 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 16 18:43:57.556894 ip-10-0-132-80 kubenswrapper[2544]: W0916 18:43:57.552368    2544 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 16 18:43:57.556894 ip-10-0-132-80 kubenswrapper[2544]: W0916 18:43:57.552371    2544 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 16 18:43:57.556894 ip-10-0-132-80 kubenswrapper[2544]: W0916 18:43:57.552374    2544 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 16 18:43:57.556894 ip-10-0-132-80 kubenswrapper[2544]: W0916 18:43:57.552377    2544 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 16 18:43:57.556894 ip-10-0-132-80 kubenswrapper[2544]: W0916 18:43:57.552380    2544 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 16 18:43:57.556894 ip-10-0-132-80 kubenswrapper[2544]: W0916 18:43:57.552382    2544 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 16 18:43:57.556894 ip-10-0-132-80 kubenswrapper[2544]: W0916 18:43:57.552385    2544 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 16 18:43:57.556894 ip-10-0-132-80 kubenswrapper[2544]: W0916 18:43:57.552387    2544 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 16 18:43:57.556894 ip-10-0-132-80 kubenswrapper[2544]: W0916 18:43:57.552390    2544 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 16 18:43:57.556894 ip-10-0-132-80 kubenswrapper[2544]: W0916 18:43:57.552393    2544 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 16 18:43:57.556894 ip-10-0-132-80 kubenswrapper[2544]: W0916 18:43:57.552395    2544 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 16 18:43:57.556894 ip-10-0-132-80 kubenswrapper[2544]: W0916 18:43:57.552398    2544 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 16 18:43:57.556894 ip-10-0-132-80 kubenswrapper[2544]: W0916 18:43:57.552400    2544 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 16 18:43:57.556894 ip-10-0-132-80 kubenswrapper[2544]: W0916 18:43:57.552403    2544 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 16 18:43:57.556894 ip-10-0-132-80 kubenswrapper[2544]: W0916 18:43:57.552406    2544 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 16 18:43:57.557344 ip-10-0-132-80 kubenswrapper[2544]: W0916 18:43:57.552408    2544 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 16 18:43:57.557344 ip-10-0-132-80 kubenswrapper[2544]: W0916 18:43:57.552411    2544 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 16 18:43:57.557344 ip-10-0-132-80 kubenswrapper[2544]: W0916 18:43:57.552413    2544 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 16 18:43:57.557344 ip-10-0-132-80 kubenswrapper[2544]: W0916 18:43:57.552416    2544 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 16 18:43:57.557344 ip-10-0-132-80 kubenswrapper[2544]: W0916 18:43:57.552418    2544 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 16 18:43:57.557344 ip-10-0-132-80 kubenswrapper[2544]: W0916 18:43:57.552421    2544 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 16 18:43:57.557344 ip-10-0-132-80 kubenswrapper[2544]: W0916 18:43:57.552424    2544 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 16 18:43:57.557344 ip-10-0-132-80 kubenswrapper[2544]: W0916 18:43:57.552426    2544 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 16 18:43:57.557344 ip-10-0-132-80 kubenswrapper[2544]: W0916 18:43:57.552429    2544 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 16 18:43:57.557344 ip-10-0-132-80 kubenswrapper[2544]: W0916 18:43:57.552432    2544 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 16 18:43:57.557344 ip-10-0-132-80 kubenswrapper[2544]: W0916 18:43:57.552435    2544 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 16 18:43:57.557344 ip-10-0-132-80 kubenswrapper[2544]: W0916 18:43:57.552437    2544 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 16 18:43:57.557344 ip-10-0-132-80 kubenswrapper[2544]: W0916 18:43:57.552440    2544 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 16 18:43:57.557344 ip-10-0-132-80 kubenswrapper[2544]: W0916 18:43:57.552443    2544 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 16 18:43:57.557344 ip-10-0-132-80 kubenswrapper[2544]: W0916 18:43:57.552445    2544 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 16 18:43:57.557344 ip-10-0-132-80 kubenswrapper[2544]: W0916 18:43:57.552448    2544 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 16 18:43:57.557344 ip-10-0-132-80 kubenswrapper[2544]: W0916 18:43:57.552450    2544 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 16 18:43:57.557344 ip-10-0-132-80 kubenswrapper[2544]: W0916 18:43:57.552453    2544 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 16 18:43:57.557344 ip-10-0-132-80 kubenswrapper[2544]: W0916 18:43:57.552456    2544 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 16 18:43:57.557344 ip-10-0-132-80 kubenswrapper[2544]: W0916 18:43:57.552458    2544 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 16 18:43:57.557871 ip-10-0-132-80 kubenswrapper[2544]: W0916 18:43:57.552461    2544 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 16 18:43:57.557871 ip-10-0-132-80 kubenswrapper[2544]: W0916 18:43:57.552463    2544 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 16 18:43:57.557871 ip-10-0-132-80 kubenswrapper[2544]: W0916 18:43:57.552466    2544 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 16 18:43:57.557871 ip-10-0-132-80 kubenswrapper[2544]: W0916 18:43:57.552468    2544 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 16 18:43:57.557871 ip-10-0-132-80 kubenswrapper[2544]: W0916 18:43:57.552471    2544 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 16 18:43:57.557871 ip-10-0-132-80 kubenswrapper[2544]: W0916 18:43:57.552473    2544 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 16 18:43:57.557871 ip-10-0-132-80 kubenswrapper[2544]: W0916 18:43:57.552476    2544 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 16 18:43:57.557871 ip-10-0-132-80 kubenswrapper[2544]: W0916 18:43:57.552479    2544 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 16 18:43:57.557871 ip-10-0-132-80 kubenswrapper[2544]: W0916 18:43:57.552481    2544 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 16 18:43:57.557871 ip-10-0-132-80 kubenswrapper[2544]: W0916 18:43:57.552484    2544 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 16 18:43:57.557871 ip-10-0-132-80 kubenswrapper[2544]: W0916 18:43:57.552486    2544 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 16 18:43:57.557871 ip-10-0-132-80 kubenswrapper[2544]: W0916 18:43:57.552489    2544 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 16 18:43:57.557871 ip-10-0-132-80 kubenswrapper[2544]: W0916 18:43:57.552491    2544 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 16 18:43:57.557871 ip-10-0-132-80 kubenswrapper[2544]: W0916 18:43:57.552494    2544 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 16 18:43:57.557871 ip-10-0-132-80 kubenswrapper[2544]: W0916 18:43:57.552496    2544 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 16 18:43:57.557871 ip-10-0-132-80 kubenswrapper[2544]: W0916 18:43:57.552499    2544 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 16 18:43:57.557871 ip-10-0-132-80 kubenswrapper[2544]: W0916 18:43:57.552501    2544 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 16 18:43:57.557871 ip-10-0-132-80 kubenswrapper[2544]: W0916 18:43:57.552504    2544 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 16 18:43:57.557871 ip-10-0-132-80 kubenswrapper[2544]: W0916 18:43:57.552506    2544 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 16 18:43:57.557871 ip-10-0-132-80 kubenswrapper[2544]: W0916 18:43:57.552509    2544 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 16 18:43:57.558335 ip-10-0-132-80 kubenswrapper[2544]: W0916 18:43:57.552511    2544 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 16 18:43:57.558335 ip-10-0-132-80 kubenswrapper[2544]: W0916 18:43:57.552514    2544 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 16 18:43:57.558335 ip-10-0-132-80 kubenswrapper[2544]: W0916 18:43:57.552517    2544 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 16 18:43:57.558335 ip-10-0-132-80 kubenswrapper[2544]: W0916 18:43:57.552520    2544 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 16 18:43:57.558335 ip-10-0-132-80 kubenswrapper[2544]: W0916 18:43:57.552522    2544 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 16 18:43:57.558335 ip-10-0-132-80 kubenswrapper[2544]: W0916 18:43:57.552525    2544 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 16 18:43:57.558335 ip-10-0-132-80 kubenswrapper[2544]: W0916 18:43:57.552528    2544 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 16 18:43:57.558335 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:43:57.552532    2544 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 16 18:43:57.558335 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:43:57.552669    2544 server.go:962] "Client rotation is on, will bootstrap in background"
Sep 16 18:43:57.558335 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:43:57.555086    2544 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir"
Sep 16 18:43:57.558335 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:43:57.556048    2544 server.go:1019] "Starting client certificate rotation"
Sep 16 18:43:57.558335 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:43:57.556147    2544 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Sep 16 18:43:57.558335 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:43:57.556186    2544 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet"
Sep 16 18:43:57.559669 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:43:57.559656    2544 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Sep 16 18:43:57.560795 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:43:57.560777    2544 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Sep 16 18:43:57.571281 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:43:57.571267    2544 log.go:25] "Validated CRI v1 runtime API"
Sep 16 18:43:57.575860 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:43:57.575847    2544 log.go:25] "Validated CRI v1 image API"
Sep 16 18:43:57.576741 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:43:57.576720    2544 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Sep 16 18:43:57.582123 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:43:57.582106    2544 fs.go:135] Filesystem UUIDs: map[23e89b6c-c5a5-4d4f-985a-4b0f79dba6a6:/dev/nvme0n1p4 7B77-95E7:/dev/nvme0n1p2 f7b725d9-92d8-4469-a4ee-b99985b663c0:/dev/nvme0n1p3]
Sep 16 18:43:57.582182 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:43:57.582123    2544 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}]
Sep 16 18:43:57.589516 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:43:57.589497    2544 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Sep 16 18:43:57.589663 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:43:57.589530    2544 manager.go:216] Machine: {Timestamp:2026-09-16 18:43:57.588188435 +0000 UTC m=+0.377190972 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3089640 MemoryCapacity:33164468224 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec244d7b02c2a24bfb953beee6bf9ba1 SystemUUID:ec244d7b-02c2-a24b-fb95-3beee6bf9ba1 BootID:594c4c67-cbca-46f2-9a5c-5cac8421ce2e Filesystems:[{Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16582234112 Type:vfs Inodes:4048397 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6632894464 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6111232 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16582234112 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:ff:cd:69:e4:0b Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:ff:cd:69:e4:0b Speed:0 Mtu:9001} {Name:ovs-system MacAddress:4e:1e:04:92:e9:c0 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164468224 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Sep 16 18:43:57.589663 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:43:57.589661    2544 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Sep 16 18:43:57.589779 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:43:57.589768    2544 manager.go:232] Version: {KernelVersion:5.14.0-570.140.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260908-1 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Sep 16 18:43:57.593536 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:43:57.593509    2544 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Sep 16 18:43:57.593683 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:43:57.593539    2544 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-132-80.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Sep 16 18:43:57.595108 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:43:57.595097    2544 topology_manager.go:138] "Creating topology manager with none policy"
Sep 16 18:43:57.595148 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:43:57.595111    2544 container_manager_linux.go:309] "Creating device plugin manager"
Sep 16 18:43:57.595148 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:43:57.595118    2544 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Sep 16 18:43:57.595748 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:43:57.595738    2544 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Sep 16 18:43:57.596968 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:43:57.596959    2544 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager"
Sep 16 18:43:57.601581 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:43:57.601569    2544 state_mem.go:36] "Initialized new in-memory state store"
Sep 16 18:43:57.601695 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:43:57.601686    2544 server.go:1240] "Using root directory" path="/var/lib/kubelet"
Sep 16 18:43:57.601724 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:43:57.601715    2544 kubelet.go:486] "Attempting to sync node with API server"
Sep 16 18:43:57.601758 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:43:57.601725    2544 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests"
Sep 16 18:43:57.601758 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:43:57.601740    2544 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Sep 16 18:43:57.601758 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:43:57.601749    2544 kubelet.go:398] "Adding apiserver pod source"
Sep 16 18:43:57.601758 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:43:57.601757    2544 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Sep 16 18:43:57.602097 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:43:57.602086    2544 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Sep 16 18:43:57.602138 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:43:57.602105    2544 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Sep 16 18:43:57.604429 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:43:57.604412    2544 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.12-5.rhaos4.21.gitd1ec63c.el9" apiVersion="v1"
Sep 16 18:43:57.606742 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:43:57.606728    2544 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Sep 16 18:43:57.606800 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:43:57.606765    2544 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Sep 16 18:43:57.608873 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:43:57.608853    2544 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Sep 16 18:43:57.608873 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:43:57.608872    2544 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Sep 16 18:43:57.608954 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:43:57.608878    2544 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Sep 16 18:43:57.608954 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:43:57.608885    2544 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Sep 16 18:43:57.608954 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:43:57.608894    2544 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Sep 16 18:43:57.608954 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:43:57.608900    2544 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Sep 16 18:43:57.608954 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:43:57.608914    2544 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Sep 16 18:43:57.608954 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:43:57.608919    2544 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Sep 16 18:43:57.608954 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:43:57.608926    2544 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Sep 16 18:43:57.608954 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:43:57.608931    2544 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Sep 16 18:43:57.608954 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:43:57.608940    2544 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Sep 16 18:43:57.608954 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:43:57.608945    2544 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Sep 16 18:43:57.610621 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:43:57.610607    2544 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Sep 16 18:43:57.610657 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:43:57.610622    2544 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Sep 16 18:43:57.611709 ip-10-0-132-80 kubenswrapper[2544]: E0916 18:43:57.611685    2544 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service"
Sep 16 18:43:57.611709 ip-10-0-132-80 kubenswrapper[2544]: E0916 18:43:57.611687    2544 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-132-80.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node"
Sep 16 18:43:57.613930 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:43:57.613913    2544 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-132-80.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Sep 16 18:43:57.614431 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:43:57.614419    2544 server.go:1268] "Started kubelet"
Sep 16 18:43:57.614511 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:43:57.614485    2544 server.go:180] "Starting to listen" address="0.0.0.0" port=10250
Sep 16 18:43:57.614574 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:43:57.614498    2544 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Sep 16 18:43:57.614620 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:43:57.614576    2544 server_v1.go:49] "podresources" method="list" useActivePods=true
Sep 16 18:43:57.615222 ip-10-0-132-80 systemd[1]: Started Kubernetes Kubelet.
Sep 16 18:43:57.616854 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:43:57.616720    2544 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Sep 16 18:43:57.617616 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:43:57.617593    2544 server.go:310] "Adding debug handlers to kubelet server"
Sep 16 18:43:57.622119 ip-10-0-132-80 kubenswrapper[2544]: E0916 18:43:57.622093    2544 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache"
Sep 16 18:43:57.623653 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:43:57.623636    2544 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Sep 16 18:43:57.624088 ip-10-0-132-80 kubenswrapper[2544]: E0916 18:43:57.622872    2544 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-132-80.ec2.internal.18d5e161eaa4e93a  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-132-80.ec2.internal,UID:ip-10-0-132-80.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-132-80.ec2.internal,},FirstTimestamp:2026-09-16 18:43:57.614393658 +0000 UTC m=+0.403396183,LastTimestamp:2026-09-16 18:43:57.614393658 +0000 UTC m=+0.403396183,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-132-80.ec2.internal,}"
Sep 16 18:43:57.624336 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:43:57.624315    2544 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer"
Sep 16 18:43:57.624438 ip-10-0-132-80 kubenswrapper[2544]: E0916 18:43:57.624417    2544 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-132-80.ec2.internal\" not found"
Sep 16 18:43:57.625289 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:43:57.625136    2544 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Sep 16 18:43:57.625289 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:43:57.625158    2544 volume_manager.go:311] "The desired_state_of_world populator starts"
Sep 16 18:43:57.625410 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:43:57.625301    2544 volume_manager.go:313] "Starting Kubelet Volume Manager"
Sep 16 18:43:57.625471 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:43:57.625421    2544 reconstruct.go:97] "Volume reconstruction finished"
Sep 16 18:43:57.625471 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:43:57.625431    2544 reconciler.go:29] "Reconciler: start to sync state"
Sep 16 18:43:57.625756 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:43:57.625740    2544 factory.go:153] Registering CRI-O factory
Sep 16 18:43:57.625821 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:43:57.625778    2544 factory.go:223] Registration of the crio container factory successfully
Sep 16 18:43:57.625870 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:43:57.625824    2544 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Sep 16 18:43:57.625870 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:43:57.625837    2544 factory.go:55] Registering systemd factory
Sep 16 18:43:57.625870 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:43:57.625852    2544 factory.go:223] Registration of the systemd container factory successfully
Sep 16 18:43:57.625870 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:43:57.625869    2544 factory.go:103] Registering Raw factory
Sep 16 18:43:57.626035 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:43:57.625883    2544 manager.go:1196] Started watching for new ooms in manager
Sep 16 18:43:57.626307 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:43:57.626289    2544 manager.go:349] Starting recovery of all containers
Sep 16 18:43:57.632238 ip-10-0-132-80 kubenswrapper[2544]: E0916 18:43:57.632176    2544 reflector.go:205] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver"
Sep 16 18:43:57.632349 ip-10-0-132-80 kubenswrapper[2544]: E0916 18:43:57.632286    2544 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-132-80.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms"
Sep 16 18:43:57.638580 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:43:57.638547    2544 manager.go:354] Recovery completed
Sep 16 18:43:57.643886 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:43:57.643870    2544 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 16 18:43:57.644731 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:43:57.644715    2544 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-6jqlg"
Sep 16 18:43:57.645813 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:43:57.645794    2544 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-80.ec2.internal" event="NodeHasSufficientMemory"
Sep 16 18:43:57.645897 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:43:57.645832    2544 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-80.ec2.internal" event="NodeHasNoDiskPressure"
Sep 16 18:43:57.645897 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:43:57.645849    2544 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-80.ec2.internal" event="NodeHasSufficientPID"
Sep 16 18:43:57.646513 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:43:57.646488    2544 cpu_manager.go:222] "Starting CPU manager" policy="none"
Sep 16 18:43:57.646513 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:43:57.646503    2544 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s"
Sep 16 18:43:57.646677 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:43:57.646521    2544 state_mem.go:36] "Initialized new in-memory state store"
Sep 16 18:43:57.646677 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:43:57.646545    2544 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 16 18:43:57.647483 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:43:57.647459    2544 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-80.ec2.internal" event="NodeHasSufficientMemory"
Sep 16 18:43:57.647541 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:43:57.647501    2544 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-80.ec2.internal" event="NodeHasNoDiskPressure"
Sep 16 18:43:57.647541 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:43:57.647518    2544 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-80.ec2.internal" event="NodeHasSufficientPID"
Sep 16 18:43:57.648259 ip-10-0-132-80 kubenswrapper[2544]: E0916 18:43:57.648183    2544 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-132-80.ec2.internal.18d5e161ec8454d6  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-132-80.ec2.internal,UID:ip-10-0-132-80.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-132-80.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-132-80.ec2.internal,},FirstTimestamp:2026-09-16 18:43:57.64581295 +0000 UTC m=+0.434815484,LastTimestamp:2026-09-16 18:43:57.64581295 +0000 UTC m=+0.434815484,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-132-80.ec2.internal,}"
Sep 16 18:43:57.648868 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:43:57.648853    2544 policy_none.go:49] "None policy: Start"
Sep 16 18:43:57.648932 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:43:57.648875    2544 memory_manager.go:187] "Starting memorymanager" policy="None"
Sep 16 18:43:57.648932 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:43:57.648890    2544 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Sep 16 18:43:57.650993 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:43:57.650980    2544 policy_none.go:47] "Start"
Sep 16 18:43:57.652795 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:43:57.652777    2544 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-6jqlg"
Sep 16 18:43:57.656449 ip-10-0-132-80 kubenswrapper[2544]: E0916 18:43:57.656370    2544 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-132-80.ec2.internal.18d5e161ec84be13  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-132-80.ec2.internal,UID:ip-10-0-132-80.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasNoDiskPressure,Message:Node ip-10-0-132-80.ec2.internal status is now: NodeHasNoDiskPressure,Source:EventSource{Component:kubelet,Host:ip-10-0-132-80.ec2.internal,},FirstTimestamp:2026-09-16 18:43:57.645839891 +0000 UTC m=+0.434842418,LastTimestamp:2026-09-16 18:43:57.645839891 +0000 UTC m=+0.434842418,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-132-80.ec2.internal,}"
Sep 16 18:43:57.660002 ip-10-0-132-80 kubenswrapper[2544]: E0916 18:43:57.659911    2544 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-132-80.ec2.internal.18d5e161ec84ff29  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-132-80.ec2.internal,UID:ip-10-0-132-80.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientPID,Message:Node ip-10-0-132-80.ec2.internal status is now: NodeHasSufficientPID,Source:EventSource{Component:kubelet,Host:ip-10-0-132-80.ec2.internal,},FirstTimestamp:2026-09-16 18:43:57.645856553 +0000 UTC m=+0.434859079,LastTimestamp:2026-09-16 18:43:57.645856553 +0000 UTC m=+0.434859079,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-132-80.ec2.internal,}"
Sep 16 18:43:57.700327 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:43:57.682149    2544 manager.go:337] "Starting Device Plugin manager"
Sep 16 18:43:57.700327 ip-10-0-132-80 kubenswrapper[2544]: E0916 18:43:57.682220    2544 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Sep 16 18:43:57.700327 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:43:57.682234    2544 server.go:87] "Starting device plugin registration server"
Sep 16 18:43:57.700327 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:43:57.682524    2544 eviction_manager.go:189] "Eviction manager: starting control loop"
Sep 16 18:43:57.700327 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:43:57.682535    2544 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Sep 16 18:43:57.700327 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:43:57.682609    2544 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Sep 16 18:43:57.700327 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:43:57.682686    2544 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts"
Sep 16 18:43:57.700327 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:43:57.682694    2544 plugin_manager.go:118] "Starting Kubelet Plugin Manager"
Sep 16 18:43:57.700327 ip-10-0-132-80 kubenswrapper[2544]: E0916 18:43:57.683198    2544 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Sep 16 18:43:57.700327 ip-10-0-132-80 kubenswrapper[2544]: E0916 18:43:57.683250    2544 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-132-80.ec2.internal\" not found"
Sep 16 18:43:57.714660 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:43:57.714629    2544 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Sep 16 18:43:57.715923 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:43:57.715906    2544 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Sep 16 18:43:57.716026 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:43:57.715949    2544 status_manager.go:244] "Starting to sync pod status with apiserver"
Sep 16 18:43:57.716026 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:43:57.715970    2544 kubelet.go:2448] "Starting kubelet main sync loop"
Sep 16 18:43:57.716099 ip-10-0-132-80 kubenswrapper[2544]: E0916 18:43:57.716015    2544 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful"
Sep 16 18:43:57.720638 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:43:57.720618    2544 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 16 18:43:57.783590 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:43:57.783543    2544 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 16 18:43:57.784998 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:43:57.784921    2544 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-80.ec2.internal" event="NodeHasSufficientMemory"
Sep 16 18:43:57.784998 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:43:57.784952    2544 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-80.ec2.internal" event="NodeHasNoDiskPressure"
Sep 16 18:43:57.784998 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:43:57.784962    2544 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-80.ec2.internal" event="NodeHasSufficientPID"
Sep 16 18:43:57.784998 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:43:57.784985    2544 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-132-80.ec2.internal"
Sep 16 18:43:57.794711 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:43:57.794691    2544 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-132-80.ec2.internal"
Sep 16 18:43:57.794711 ip-10-0-132-80 kubenswrapper[2544]: E0916 18:43:57.794713    2544 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-132-80.ec2.internal\": node \"ip-10-0-132-80.ec2.internal\" not found"
Sep 16 18:43:57.816522 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:43:57.816492    2544 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["kube-system/kube-apiserver-proxy-ip-10-0-132-80.ec2.internal","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-80.ec2.internal"]
Sep 16 18:43:57.816636 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:43:57.816574    2544 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 16 18:43:57.817914 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:43:57.817897    2544 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-80.ec2.internal" event="NodeHasSufficientMemory"
Sep 16 18:43:57.817971 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:43:57.817926    2544 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-80.ec2.internal" event="NodeHasNoDiskPressure"
Sep 16 18:43:57.817971 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:43:57.817937    2544 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-80.ec2.internal" event="NodeHasSufficientPID"
Sep 16 18:43:57.820361 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:43:57.820349    2544 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 16 18:43:57.820495 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:43:57.820482    2544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-132-80.ec2.internal"
Sep 16 18:43:57.820536 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:43:57.820509    2544 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 16 18:43:57.821089 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:43:57.821074    2544 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-80.ec2.internal" event="NodeHasSufficientMemory"
Sep 16 18:43:57.821157 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:43:57.821101    2544 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-80.ec2.internal" event="NodeHasNoDiskPressure"
Sep 16 18:43:57.821157 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:43:57.821112    2544 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-80.ec2.internal" event="NodeHasSufficientPID"
Sep 16 18:43:57.821218 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:43:57.821076    2544 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-80.ec2.internal" event="NodeHasSufficientMemory"
Sep 16 18:43:57.821218 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:43:57.821177    2544 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-80.ec2.internal" event="NodeHasNoDiskPressure"
Sep 16 18:43:57.821218 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:43:57.821187    2544 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-80.ec2.internal" event="NodeHasSufficientPID"
Sep 16 18:43:57.825372 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:43:57.825353    2544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-80.ec2.internal"
Sep 16 18:43:57.825466 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:43:57.825389    2544 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 16 18:43:57.826057 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:43:57.826040    2544 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-80.ec2.internal" event="NodeHasSufficientMemory"
Sep 16 18:43:57.826147 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:43:57.826064    2544 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-80.ec2.internal" event="NodeHasNoDiskPressure"
Sep 16 18:43:57.826147 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:43:57.826073    2544 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-80.ec2.internal" event="NodeHasSufficientPID"
Sep 16 18:43:57.827211 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:43:57.827194    2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/752ecfd4a66e13af35021c760a217e52-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-132-80.ec2.internal\" (UID: \"752ecfd4a66e13af35021c760a217e52\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-80.ec2.internal"
Sep 16 18:43:57.827307 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:43:57.827216    2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/752ecfd4a66e13af35021c760a217e52-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-132-80.ec2.internal\" (UID: \"752ecfd4a66e13af35021c760a217e52\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-80.ec2.internal"
Sep 16 18:43:57.827307 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:43:57.827232    2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/3b41cfe1537c43ad1e53b07ff984e93b-config\") pod \"kube-apiserver-proxy-ip-10-0-132-80.ec2.internal\" (UID: \"3b41cfe1537c43ad1e53b07ff984e93b\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-132-80.ec2.internal"
Sep 16 18:43:57.828749 ip-10-0-132-80 kubenswrapper[2544]: E0916 18:43:57.828732    2544 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-132-80.ec2.internal\" not found"
Sep 16 18:43:57.845073 ip-10-0-132-80 kubenswrapper[2544]: E0916 18:43:57.845055    2544 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-132-80.ec2.internal\" not found" node="ip-10-0-132-80.ec2.internal"
Sep 16 18:43:57.849052 ip-10-0-132-80 kubenswrapper[2544]: E0916 18:43:57.849038    2544 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-132-80.ec2.internal\" not found" node="ip-10-0-132-80.ec2.internal"
Sep 16 18:43:57.928059 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:43:57.928025    2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/752ecfd4a66e13af35021c760a217e52-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-132-80.ec2.internal\" (UID: \"752ecfd4a66e13af35021c760a217e52\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-80.ec2.internal"
Sep 16 18:43:57.928219 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:43:57.928068    2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/752ecfd4a66e13af35021c760a217e52-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-132-80.ec2.internal\" (UID: \"752ecfd4a66e13af35021c760a217e52\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-80.ec2.internal"
Sep 16 18:43:57.928219 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:43:57.928077    2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/752ecfd4a66e13af35021c760a217e52-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-132-80.ec2.internal\" (UID: \"752ecfd4a66e13af35021c760a217e52\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-80.ec2.internal"
Sep 16 18:43:57.928219 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:43:57.928103    2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/3b41cfe1537c43ad1e53b07ff984e93b-config\") pod \"kube-apiserver-proxy-ip-10-0-132-80.ec2.internal\" (UID: \"3b41cfe1537c43ad1e53b07ff984e93b\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-132-80.ec2.internal"
Sep 16 18:43:57.928219 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:43:57.928136    2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/752ecfd4a66e13af35021c760a217e52-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-132-80.ec2.internal\" (UID: \"752ecfd4a66e13af35021c760a217e52\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-80.ec2.internal"
Sep 16 18:43:57.928219 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:43:57.928146    2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/3b41cfe1537c43ad1e53b07ff984e93b-config\") pod \"kube-apiserver-proxy-ip-10-0-132-80.ec2.internal\" (UID: \"3b41cfe1537c43ad1e53b07ff984e93b\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-132-80.ec2.internal"
Sep 16 18:43:57.929062 ip-10-0-132-80 kubenswrapper[2544]: E0916 18:43:57.929046    2544 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-132-80.ec2.internal\" not found"
Sep 16 18:43:58.030066 ip-10-0-132-80 kubenswrapper[2544]: E0916 18:43:58.030030    2544 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-132-80.ec2.internal\" not found"
Sep 16 18:43:58.130736 ip-10-0-132-80 kubenswrapper[2544]: E0916 18:43:58.130668    2544 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-132-80.ec2.internal\" not found"
Sep 16 18:43:58.147854 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:43:58.147833    2544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-132-80.ec2.internal"
Sep 16 18:43:58.153011 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:43:58.152709    2544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-80.ec2.internal"
Sep 16 18:43:58.231243 ip-10-0-132-80 kubenswrapper[2544]: E0916 18:43:58.231110    2544 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-132-80.ec2.internal\" not found"
Sep 16 18:43:58.331650 ip-10-0-132-80 kubenswrapper[2544]: E0916 18:43:58.331629    2544 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-132-80.ec2.internal\" not found"
Sep 16 18:43:58.432187 ip-10-0-132-80 kubenswrapper[2544]: E0916 18:43:58.432118    2544 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-132-80.ec2.internal\" not found"
Sep 16 18:43:58.533217 ip-10-0-132-80 kubenswrapper[2544]: E0916 18:43:58.533183    2544 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-132-80.ec2.internal\" not found"
Sep 16 18:43:58.555828 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:43:58.555799    2544 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials"
Sep 16 18:43:58.556360 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:43:58.555941    2544 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Sep 16 18:43:58.603642 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:43:58.603611    2544 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 16 18:43:58.634265 ip-10-0-132-80 kubenswrapper[2544]: E0916 18:43:58.634243    2544 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-132-80.ec2.internal\" not found"
Sep 16 18:43:58.659112 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:43:58.659077    2544 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-09-15 18:38:57 +0000 UTC" deadline="2028-05-01 05:52:30.052385079 +0000 UTC"
Sep 16 18:43:58.659112 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:43:58.659112    2544 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="14219h8m31.393279861s"
Sep 16 18:43:58.733492 ip-10-0-132-80 kubenswrapper[2544]: W0916 18:43:58.733442    2544 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3b41cfe1537c43ad1e53b07ff984e93b.slice/crio-d8441f6bfa2ac6795d7103e38bb2bd510bd162ee84cc70a63b88ef54ee6baf1a WatchSource:0}: Error finding container d8441f6bfa2ac6795d7103e38bb2bd510bd162ee84cc70a63b88ef54ee6baf1a: Status 404 returned error can't find the container with id d8441f6bfa2ac6795d7103e38bb2bd510bd162ee84cc70a63b88ef54ee6baf1a
Sep 16 18:43:58.734612 ip-10-0-132-80 kubenswrapper[2544]: E0916 18:43:58.734588    2544 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-132-80.ec2.internal\" not found"
Sep 16 18:43:58.739545 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:43:58.739529    2544 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 16 18:43:58.747562 ip-10-0-132-80 kubenswrapper[2544]: W0916 18:43:58.747535    2544 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod752ecfd4a66e13af35021c760a217e52.slice/crio-ddd9af6730281ec83461011c13d4cb542a3fb2f0401b5d4e913256b018323b3d WatchSource:0}: Error finding container ddd9af6730281ec83461011c13d4cb542a3fb2f0401b5d4e913256b018323b3d: Status 404 returned error can't find the container with id ddd9af6730281ec83461011c13d4cb542a3fb2f0401b5d4e913256b018323b3d
Sep 16 18:43:58.783174 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:43:58.783147    2544 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 16 18:43:58.784124 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:43:58.784103    2544 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-80.ec2.internal" event="NodeHasSufficientMemory"
Sep 16 18:43:58.784232 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:43:58.784143    2544 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-80.ec2.internal" event="NodeHasNoDiskPressure"
Sep 16 18:43:58.784232 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:43:58.784160    2544 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-80.ec2.internal" event="NodeHasSufficientPID"
Sep 16 18:43:58.835437 ip-10-0-132-80 kubenswrapper[2544]: E0916 18:43:58.835409    2544 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-132-80.ec2.internal\" not found"
Sep 16 18:43:58.935690 ip-10-0-132-80 kubenswrapper[2544]: E0916 18:43:58.935659    2544 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-132-80.ec2.internal\" not found"
Sep 16 18:43:58.942075 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:43:58.942055    2544 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 16 18:43:59.028008 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:43:59.027944    2544 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-132-80.ec2.internal"
Sep 16 18:43:59.038825 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:43:59.038801    2544 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Sep 16 18:43:59.040915 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:43:59.040901    2544 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-80.ec2.internal"
Sep 16 18:43:59.054476 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:43:59.054458    2544 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Sep 16 18:43:59.078968 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:43:59.078942    2544 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 16 18:43:59.603614 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:43:59.603584    2544 apiserver.go:52] "Watching apiserver"
Sep 16 18:43:59.610294 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:43:59.610269    2544 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Sep 16 18:43:59.610778 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:43:59.610756    2544 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ovn-kubernetes/ovnkube-node-nqcpx","kube-system/konnectivity-agent-pzpb9","kube-system/kube-apiserver-proxy-ip-10-0-132-80.ec2.internal","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zv6wl","openshift-cluster-node-tuning-operator/tuned-bbnvm","openshift-multus/multus-dbwbz","openshift-multus/network-metrics-daemon-pdfpq","openshift-network-operator/iptables-alerter-z4wt6","kube-system/global-pull-secret-syncer-pg4kf","openshift-dns/node-resolver-sxbgl","openshift-image-registry/node-ca-2hdh4","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-80.ec2.internal","openshift-multus/multus-additional-cni-plugins-znf6p","openshift-network-diagnostics/network-check-target-t8hx2"]
Sep 16 18:43:59.613421 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:43:59.613397    2544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-pdfpq"
Sep 16 18:43:59.613527 ip-10-0-132-80 kubenswrapper[2544]: E0916 18:43:59.613478    2544 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-pdfpq" podUID="db1e3b32-114c-4bd9-b28c-21a2a9d6efaa"
Sep 16 18:43:59.613527 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:43:59.613511    2544 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 16 18:43:59.615970 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:43:59.615567    2544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-pzpb9"
Sep 16 18:43:59.617661 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:43:59.617639    2544 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Sep 16 18:43:59.617739 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:43:59.617706    2544 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-l7gc7\""
Sep 16 18:43:59.617947 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:43:59.617927    2544 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Sep 16 18:43:59.619081 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:43:59.619044    2544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-znf6p"
Sep 16 18:43:59.621450 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:43:59.621423    2544 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-dtdw6\""
Sep 16 18:43:59.622410 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:43:59.621646    2544 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Sep 16 18:43:59.622410 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:43:59.621740    2544 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Sep 16 18:43:59.622410 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:43:59.621927    2544 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Sep 16 18:43:59.622410 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:43:59.621936    2544 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Sep 16 18:43:59.622410 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:43:59.622130    2544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-z4wt6"
Sep 16 18:43:59.622410 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:43:59.622279    2544 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Sep 16 18:43:59.624398 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:43:59.624150    2544 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving"
Sep 16 18:43:59.624951 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:43:59.624915    2544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zv6wl"
Sep 16 18:43:59.625285 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:43:59.625195    2544 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Sep 16 18:43:59.625385 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:43:59.625333    2544 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Sep 16 18:43:59.625669 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:43:59.625648    2544 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Sep 16 18:43:59.625787 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:43:59.625712    2544 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-xbxtx\""
Sep 16 18:43:59.626420 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:43:59.626401    2544 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Sep 16 18:43:59.626807 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:43:59.626784    2544 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Sep 16 18:43:59.626907 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:43:59.626806    2544 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-dlxff\""
Sep 16 18:43:59.626970 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:43:59.626926    2544 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Sep 16 18:43:59.627452 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:43:59.627293    2544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-dbwbz"
Sep 16 18:43:59.629036 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:43:59.629014    2544 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Sep 16 18:43:59.629245 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:43:59.629226    2544 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-mjbtv\""
Sep 16 18:43:59.629596 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:43:59.629545    2544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-bbnvm"
Sep 16 18:43:59.631595 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:43:59.631399    2544 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Sep 16 18:43:59.631595 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:43:59.631444    2544 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-bklf6\""
Sep 16 18:43:59.631807 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:43:59.631791    2544 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Sep 16 18:43:59.632060 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:43:59.632032    2544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-nqcpx"
Sep 16 18:43:59.634175 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:43:59.634158    2544 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Sep 16 18:43:59.634453 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:43:59.634419    2544 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-2fdsp\""
Sep 16 18:43:59.634667 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:43:59.634652    2544 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Sep 16 18:43:59.634851 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:43:59.634834    2544 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Sep 16 18:43:59.635029 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:43:59.635004    2544 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Sep 16 18:43:59.635111 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:43:59.635049    2544 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Sep 16 18:43:59.635245 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:43:59.635218    2544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-pg4kf"
Sep 16 18:43:59.635319 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:43:59.635252    2544 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Sep 16 18:43:59.635319 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:43:59.635219    2544 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Sep 16 18:43:59.635418 ip-10-0-132-80 kubenswrapper[2544]: E0916 18:43:59.635290    2544 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-pg4kf" podUID="2bbf45e9-df8e-4a67-b777-0c629428fca2"
Sep 16 18:43:59.637077 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:43:59.637055    2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/38dd6a59-da17-47cd-b10f-540f2fff9a89-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-znf6p\" (UID: \"38dd6a59-da17-47cd-b10f-540f2fff9a89\") " pod="openshift-multus/multus-additional-cni-plugins-znf6p"
Sep 16 18:43:59.637172 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:43:59.637088    2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/22e551bf-0ed9-4e8f-9487-8412901fcdcb-host-slash\") pod \"iptables-alerter-z4wt6\" (UID: \"22e551bf-0ed9-4e8f-9487-8412901fcdcb\") " pod="openshift-network-operator/iptables-alerter-z4wt6"
Sep 16 18:43:59.637172 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:43:59.637113    2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/9b277c2d-db88-424b-9366-c89532c4b4f1-sys-fs\") pod \"aws-ebs-csi-driver-node-zv6wl\" (UID: \"9b277c2d-db88-424b-9366-c89532c4b4f1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zv6wl"
Sep 16 18:43:59.637283 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:43:59.637184    2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/bef94c9f-6884-4c22-b616-32f4e38a4772-host-var-lib-cni-multus\") pod \"multus-dbwbz\" (UID: \"bef94c9f-6884-4c22-b616-32f4e38a4772\") " pod="openshift-multus/multus-dbwbz"
Sep 16 18:43:59.637283 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:43:59.637218    2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/db1e3b32-114c-4bd9-b28c-21a2a9d6efaa-metrics-certs\") pod \"network-metrics-daemon-pdfpq\" (UID: \"db1e3b32-114c-4bd9-b28c-21a2a9d6efaa\") " pod="openshift-multus/network-metrics-daemon-pdfpq"
Sep 16 18:43:59.637283 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:43:59.637266    2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/38dd6a59-da17-47cd-b10f-540f2fff9a89-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-znf6p\" (UID: \"38dd6a59-da17-47cd-b10f-540f2fff9a89\") " pod="openshift-multus/multus-additional-cni-plugins-znf6p"
Sep 16 18:43:59.637427 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:43:59.637294    2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hj6jb\" (UniqueName: \"kubernetes.io/projected/38dd6a59-da17-47cd-b10f-540f2fff9a89-kube-api-access-hj6jb\") pod \"multus-additional-cni-plugins-znf6p\" (UID: \"38dd6a59-da17-47cd-b10f-540f2fff9a89\") " pod="openshift-multus/multus-additional-cni-plugins-znf6p"
Sep 16 18:43:59.637427 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:43:59.637333    2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vgwqr\" (UniqueName: \"kubernetes.io/projected/9b277c2d-db88-424b-9366-c89532c4b4f1-kube-api-access-vgwqr\") pod \"aws-ebs-csi-driver-node-zv6wl\" (UID: \"9b277c2d-db88-424b-9366-c89532c4b4f1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zv6wl"
Sep 16 18:43:59.637427 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:43:59.637356    2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/38dd6a59-da17-47cd-b10f-540f2fff9a89-cnibin\") pod \"multus-additional-cni-plugins-znf6p\" (UID: \"38dd6a59-da17-47cd-b10f-540f2fff9a89\") " pod="openshift-multus/multus-additional-cni-plugins-znf6p"
Sep 16 18:43:59.637427 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:43:59.637378    2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/bef94c9f-6884-4c22-b616-32f4e38a4772-cnibin\") pod \"multus-dbwbz\" (UID: \"bef94c9f-6884-4c22-b616-32f4e38a4772\") " pod="openshift-multus/multus-dbwbz"
Sep 16 18:43:59.637427 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:43:59.637400    2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/9b277c2d-db88-424b-9366-c89532c4b4f1-registration-dir\") pod \"aws-ebs-csi-driver-node-zv6wl\" (UID: \"9b277c2d-db88-424b-9366-c89532c4b4f1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zv6wl"
Sep 16 18:43:59.637427 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:43:59.637412    2544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-sxbgl"
Sep 16 18:43:59.637427 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:43:59.637422    2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/9b277c2d-db88-424b-9366-c89532c4b4f1-etc-selinux\") pod \"aws-ebs-csi-driver-node-zv6wl\" (UID: \"9b277c2d-db88-424b-9366-c89532c4b4f1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zv6wl"
Sep 16 18:43:59.637779 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:43:59.637445    2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/bef94c9f-6884-4c22-b616-32f4e38a4772-host-var-lib-kubelet\") pod \"multus-dbwbz\" (UID: \"bef94c9f-6884-4c22-b616-32f4e38a4772\") " pod="openshift-multus/multus-dbwbz"
Sep 16 18:43:59.637779 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:43:59.637469    2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/bef94c9f-6884-4c22-b616-32f4e38a4772-multus-daemon-config\") pod \"multus-dbwbz\" (UID: \"bef94c9f-6884-4c22-b616-32f4e38a4772\") " pod="openshift-multus/multus-dbwbz"
Sep 16 18:43:59.637779 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:43:59.637493    2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/bef94c9f-6884-4c22-b616-32f4e38a4772-etc-kubernetes\") pod \"multus-dbwbz\" (UID: \"bef94c9f-6884-4c22-b616-32f4e38a4772\") " pod="openshift-multus/multus-dbwbz"
Sep 16 18:43:59.637779 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:43:59.637518    2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jwx9f\" (UniqueName: \"kubernetes.io/projected/db1e3b32-114c-4bd9-b28c-21a2a9d6efaa-kube-api-access-jwx9f\") pod \"network-metrics-daemon-pdfpq\" (UID: \"db1e3b32-114c-4bd9-b28c-21a2a9d6efaa\") " pod="openshift-multus/network-metrics-daemon-pdfpq"
Sep 16 18:43:59.637779 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:43:59.637541    2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/38dd6a59-da17-47cd-b10f-540f2fff9a89-cni-binary-copy\") pod \"multus-additional-cni-plugins-znf6p\" (UID: \"38dd6a59-da17-47cd-b10f-540f2fff9a89\") " pod="openshift-multus/multus-additional-cni-plugins-znf6p"
Sep 16 18:43:59.637779 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:43:59.637582    2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/22e551bf-0ed9-4e8f-9487-8412901fcdcb-iptables-alerter-script\") pod \"iptables-alerter-z4wt6\" (UID: \"22e551bf-0ed9-4e8f-9487-8412901fcdcb\") " pod="openshift-network-operator/iptables-alerter-z4wt6"
Sep 16 18:43:59.637779 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:43:59.637605    2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kgbjz\" (UniqueName: \"kubernetes.io/projected/22e551bf-0ed9-4e8f-9487-8412901fcdcb-kube-api-access-kgbjz\") pod \"iptables-alerter-z4wt6\" (UID: \"22e551bf-0ed9-4e8f-9487-8412901fcdcb\") " pod="openshift-network-operator/iptables-alerter-z4wt6"
Sep 16 18:43:59.637779 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:43:59.637647    2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/9b277c2d-db88-424b-9366-c89532c4b4f1-kubelet-dir\") pod \"aws-ebs-csi-driver-node-zv6wl\" (UID: \"9b277c2d-db88-424b-9366-c89532c4b4f1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zv6wl"
Sep 16 18:43:59.637779 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:43:59.637669    2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/bef94c9f-6884-4c22-b616-32f4e38a4772-multus-cni-dir\") pod \"multus-dbwbz\" (UID: \"bef94c9f-6884-4c22-b616-32f4e38a4772\") " pod="openshift-multus/multus-dbwbz"
Sep 16 18:43:59.637779 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:43:59.637716    2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/bef94c9f-6884-4c22-b616-32f4e38a4772-host-run-k8s-cni-cncf-io\") pod \"multus-dbwbz\" (UID: \"bef94c9f-6884-4c22-b616-32f4e38a4772\") " pod="openshift-multus/multus-dbwbz"
Sep 16 18:43:59.637779 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:43:59.637755    2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/bef94c9f-6884-4c22-b616-32f4e38a4772-host-run-multus-certs\") pod \"multus-dbwbz\" (UID: \"bef94c9f-6884-4c22-b616-32f4e38a4772\") " pod="openshift-multus/multus-dbwbz"
Sep 16 18:43:59.638212 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:43:59.637795    2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8jfrb\" (UniqueName: \"kubernetes.io/projected/bef94c9f-6884-4c22-b616-32f4e38a4772-kube-api-access-8jfrb\") pod \"multus-dbwbz\" (UID: \"bef94c9f-6884-4c22-b616-32f4e38a4772\") " pod="openshift-multus/multus-dbwbz"
Sep 16 18:43:59.638212 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:43:59.637818    2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/bef94c9f-6884-4c22-b616-32f4e38a4772-host-var-lib-cni-bin\") pod \"multus-dbwbz\" (UID: \"bef94c9f-6884-4c22-b616-32f4e38a4772\") " pod="openshift-multus/multus-dbwbz"
Sep 16 18:43:59.638212 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:43:59.637846    2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/bef94c9f-6884-4c22-b616-32f4e38a4772-cni-binary-copy\") pod \"multus-dbwbz\" (UID: \"bef94c9f-6884-4c22-b616-32f4e38a4772\") " pod="openshift-multus/multus-dbwbz"
Sep 16 18:43:59.638212 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:43:59.637876    2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/9b277c2d-db88-424b-9366-c89532c4b4f1-socket-dir\") pod \"aws-ebs-csi-driver-node-zv6wl\" (UID: \"9b277c2d-db88-424b-9366-c89532c4b4f1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zv6wl"
Sep 16 18:43:59.638212 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:43:59.637900    2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/bef94c9f-6884-4c22-b616-32f4e38a4772-system-cni-dir\") pod \"multus-dbwbz\" (UID: \"bef94c9f-6884-4c22-b616-32f4e38a4772\") " pod="openshift-multus/multus-dbwbz"
Sep 16 18:43:59.638212 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:43:59.637924    2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/bef94c9f-6884-4c22-b616-32f4e38a4772-hostroot\") pod \"multus-dbwbz\" (UID: \"bef94c9f-6884-4c22-b616-32f4e38a4772\") " pod="openshift-multus/multus-dbwbz"
Sep 16 18:43:59.638212 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:43:59.637947    2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/bef94c9f-6884-4c22-b616-32f4e38a4772-multus-conf-dir\") pod \"multus-dbwbz\" (UID: \"bef94c9f-6884-4c22-b616-32f4e38a4772\") " pod="openshift-multus/multus-dbwbz"
Sep 16 18:43:59.638212 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:43:59.637974    2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/0494be2a-d0e5-4570-b05c-0c595a0b494a-agent-certs\") pod \"konnectivity-agent-pzpb9\" (UID: \"0494be2a-d0e5-4570-b05c-0c595a0b494a\") " pod="kube-system/konnectivity-agent-pzpb9"
Sep 16 18:43:59.638212 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:43:59.638005    2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/38dd6a59-da17-47cd-b10f-540f2fff9a89-system-cni-dir\") pod \"multus-additional-cni-plugins-znf6p\" (UID: \"38dd6a59-da17-47cd-b10f-540f2fff9a89\") " pod="openshift-multus/multus-additional-cni-plugins-znf6p"
Sep 16 18:43:59.638212 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:43:59.638074    2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/38dd6a59-da17-47cd-b10f-540f2fff9a89-tuning-conf-dir\") pod \"multus-additional-cni-plugins-znf6p\" (UID: \"38dd6a59-da17-47cd-b10f-540f2fff9a89\") " pod="openshift-multus/multus-additional-cni-plugins-znf6p"
Sep 16 18:43:59.638212 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:43:59.638112    2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/9b277c2d-db88-424b-9366-c89532c4b4f1-device-dir\") pod \"aws-ebs-csi-driver-node-zv6wl\" (UID: \"9b277c2d-db88-424b-9366-c89532c4b4f1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zv6wl"
Sep 16 18:43:59.638212 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:43:59.638155    2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/bef94c9f-6884-4c22-b616-32f4e38a4772-multus-socket-dir-parent\") pod \"multus-dbwbz\" (UID: \"bef94c9f-6884-4c22-b616-32f4e38a4772\") " pod="openshift-multus/multus-dbwbz"
Sep 16 18:43:59.638212 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:43:59.638181    2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/0494be2a-d0e5-4570-b05c-0c595a0b494a-konnectivity-ca\") pod \"konnectivity-agent-pzpb9\" (UID: \"0494be2a-d0e5-4570-b05c-0c595a0b494a\") " pod="kube-system/konnectivity-agent-pzpb9"
Sep 16 18:43:59.638212 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:43:59.638207    2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/bef94c9f-6884-4c22-b616-32f4e38a4772-host-run-netns\") pod \"multus-dbwbz\" (UID: \"bef94c9f-6884-4c22-b616-32f4e38a4772\") " pod="openshift-multus/multus-dbwbz"
Sep 16 18:43:59.639685 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:43:59.639454    2544 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Sep 16 18:43:59.639685 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:43:59.639601    2544 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Sep 16 18:43:59.639685 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:43:59.639623    2544 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-pcrfn\""
Sep 16 18:43:59.639958 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:43:59.639923    2544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-2hdh4"
Sep 16 18:43:59.641806 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:43:59.641785    2544 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Sep 16 18:43:59.642001 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:43:59.641791    2544 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Sep 16 18:43:59.642061 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:43:59.642033    2544 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Sep 16 18:43:59.642261 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:43:59.642245    2544 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-tlqbc\""
Sep 16 18:43:59.642386 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:43:59.642369    2544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-t8hx2"
Sep 16 18:43:59.642453 ip-10-0-132-80 kubenswrapper[2544]: E0916 18:43:59.642430    2544 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-t8hx2" podUID="3c4f2334-3ac6-47e0-9eb7-4352c53df29a"
Sep 16 18:43:59.658480 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:43:59.658462    2544 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-gdl7j"
Sep 16 18:43:59.669722 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:43:59.669704    2544 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-gdl7j"
Sep 16 18:43:59.721116 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:43:59.721066    2544 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-80.ec2.internal" event={"ID":"752ecfd4a66e13af35021c760a217e52","Type":"ContainerStarted","Data":"ddd9af6730281ec83461011c13d4cb542a3fb2f0401b5d4e913256b018323b3d"}
Sep 16 18:43:59.722179 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:43:59.722157    2544 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-132-80.ec2.internal" event={"ID":"3b41cfe1537c43ad1e53b07ff984e93b","Type":"ContainerStarted","Data":"d8441f6bfa2ac6795d7103e38bb2bd510bd162ee84cc70a63b88ef54ee6baf1a"}
Sep 16 18:43:59.727118 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:43:59.727100    2544 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Sep 16 18:43:59.738869 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:43:59.738843    2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/9b277c2d-db88-424b-9366-c89532c4b4f1-socket-dir\") pod \"aws-ebs-csi-driver-node-zv6wl\" (UID: \"9b277c2d-db88-424b-9366-c89532c4b4f1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zv6wl"
Sep 16 18:43:59.738969 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:43:59.738880    2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/bef94c9f-6884-4c22-b616-32f4e38a4772-multus-conf-dir\") pod \"multus-dbwbz\" (UID: \"bef94c9f-6884-4c22-b616-32f4e38a4772\") " pod="openshift-multus/multus-dbwbz"
Sep 16 18:43:59.738969 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:43:59.738910    2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/524324ca-6ba2-4885-9ed6-570fa75665e7-tmp-dir\") pod \"node-resolver-sxbgl\" (UID: \"524324ca-6ba2-4885-9ed6-570fa75665e7\") " pod="openshift-dns/node-resolver-sxbgl"
Sep 16 18:43:59.738969 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:43:59.738933    2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/7cefdcfe-d218-4ac0-a3d8-811df8c043c2-host-cni-bin\") pod \"ovnkube-node-nqcpx\" (UID: \"7cefdcfe-d218-4ac0-a3d8-811df8c043c2\") " pod="openshift-ovn-kubernetes/ovnkube-node-nqcpx"
Sep 16 18:43:59.738969 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:43:59.738958    2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/38dd6a59-da17-47cd-b10f-540f2fff9a89-system-cni-dir\") pod \"multus-additional-cni-plugins-znf6p\" (UID: \"38dd6a59-da17-47cd-b10f-540f2fff9a89\") " pod="openshift-multus/multus-additional-cni-plugins-znf6p"
Sep 16 18:43:59.739124 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:43:59.738969    2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/bef94c9f-6884-4c22-b616-32f4e38a4772-multus-conf-dir\") pod \"multus-dbwbz\" (UID: \"bef94c9f-6884-4c22-b616-32f4e38a4772\") " pod="openshift-multus/multus-dbwbz"
Sep 16 18:43:59.739124 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:43:59.738984    2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/bef94c9f-6884-4c22-b616-32f4e38a4772-multus-socket-dir-parent\") pod \"multus-dbwbz\" (UID: \"bef94c9f-6884-4c22-b616-32f4e38a4772\") " pod="openshift-multus/multus-dbwbz"
Sep 16 18:43:59.739124 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:43:59.739022    2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/7cefdcfe-d218-4ac0-a3d8-811df8c043c2-host-kubelet\") pod \"ovnkube-node-nqcpx\" (UID: \"7cefdcfe-d218-4ac0-a3d8-811df8c043c2\") " pod="openshift-ovn-kubernetes/ovnkube-node-nqcpx"
Sep 16 18:43:59.739124 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:43:59.739033    2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/9b277c2d-db88-424b-9366-c89532c4b4f1-socket-dir\") pod \"aws-ebs-csi-driver-node-zv6wl\" (UID: \"9b277c2d-db88-424b-9366-c89532c4b4f1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zv6wl"
Sep 16 18:43:59.739124 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:43:59.739050    2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/7cefdcfe-d218-4ac0-a3d8-811df8c043c2-systemd-units\") pod \"ovnkube-node-nqcpx\" (UID: \"7cefdcfe-d218-4ac0-a3d8-811df8c043c2\") " pod="openshift-ovn-kubernetes/ovnkube-node-nqcpx"
Sep 16 18:43:59.739124 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:43:59.739077    2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/22e551bf-0ed9-4e8f-9487-8412901fcdcb-host-slash\") pod \"iptables-alerter-z4wt6\" (UID: \"22e551bf-0ed9-4e8f-9487-8412901fcdcb\") " pod="openshift-network-operator/iptables-alerter-z4wt6"
Sep 16 18:43:59.739124 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:43:59.739102    2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/bef94c9f-6884-4c22-b616-32f4e38a4772-host-var-lib-cni-multus\") pod \"multus-dbwbz\" (UID: \"bef94c9f-6884-4c22-b616-32f4e38a4772\") " pod="openshift-multus/multus-dbwbz"
Sep 16 18:43:59.739124 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:43:59.739112    2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/38dd6a59-da17-47cd-b10f-540f2fff9a89-system-cni-dir\") pod \"multus-additional-cni-plugins-znf6p\" (UID: \"38dd6a59-da17-47cd-b10f-540f2fff9a89\") " pod="openshift-multus/multus-additional-cni-plugins-znf6p"
Sep 16 18:43:59.739124 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:43:59.739115    2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/bef94c9f-6884-4c22-b616-32f4e38a4772-multus-socket-dir-parent\") pod \"multus-dbwbz\" (UID: \"bef94c9f-6884-4c22-b616-32f4e38a4772\") " pod="openshift-multus/multus-dbwbz"
Sep 16 18:43:59.739124 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:43:59.739125    2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/55360bdc-81dd-44da-ad34-12ff0bc70dc8-etc-modprobe-d\") pod \"tuned-bbnvm\" (UID: \"55360bdc-81dd-44da-ad34-12ff0bc70dc8\") " pod="openshift-cluster-node-tuning-operator/tuned-bbnvm"
Sep 16 18:43:59.739514 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:43:59.739155    2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/7cefdcfe-d218-4ac0-a3d8-811df8c043c2-run-systemd\") pod \"ovnkube-node-nqcpx\" (UID: \"7cefdcfe-d218-4ac0-a3d8-811df8c043c2\") " pod="openshift-ovn-kubernetes/ovnkube-node-nqcpx"
Sep 16 18:43:59.739514 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:43:59.739154    2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/22e551bf-0ed9-4e8f-9487-8412901fcdcb-host-slash\") pod \"iptables-alerter-z4wt6\" (UID: \"22e551bf-0ed9-4e8f-9487-8412901fcdcb\") " pod="openshift-network-operator/iptables-alerter-z4wt6"
Sep 16 18:43:59.739514 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:43:59.739173    2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/bef94c9f-6884-4c22-b616-32f4e38a4772-host-var-lib-cni-multus\") pod \"multus-dbwbz\" (UID: \"bef94c9f-6884-4c22-b616-32f4e38a4772\") " pod="openshift-multus/multus-dbwbz"
Sep 16 18:43:59.739514 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:43:59.739185    2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/7cefdcfe-d218-4ac0-a3d8-811df8c043c2-log-socket\") pod \"ovnkube-node-nqcpx\" (UID: \"7cefdcfe-d218-4ac0-a3d8-811df8c043c2\") " pod="openshift-ovn-kubernetes/ovnkube-node-nqcpx"
Sep 16 18:43:59.739514 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:43:59.739209    2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/7cefdcfe-d218-4ac0-a3d8-811df8c043c2-ovn-node-metrics-cert\") pod \"ovnkube-node-nqcpx\" (UID: \"7cefdcfe-d218-4ac0-a3d8-811df8c043c2\") " pod="openshift-ovn-kubernetes/ovnkube-node-nqcpx"
Sep 16 18:43:59.739514 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:43:59.739228    2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hj6jb\" (UniqueName: \"kubernetes.io/projected/38dd6a59-da17-47cd-b10f-540f2fff9a89-kube-api-access-hj6jb\") pod \"multus-additional-cni-plugins-znf6p\" (UID: \"38dd6a59-da17-47cd-b10f-540f2fff9a89\") " pod="openshift-multus/multus-additional-cni-plugins-znf6p"
Sep 16 18:43:59.739514 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:43:59.739248    2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vgwqr\" (UniqueName: \"kubernetes.io/projected/9b277c2d-db88-424b-9366-c89532c4b4f1-kube-api-access-vgwqr\") pod \"aws-ebs-csi-driver-node-zv6wl\" (UID: \"9b277c2d-db88-424b-9366-c89532c4b4f1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zv6wl"
Sep 16 18:43:59.739514 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:43:59.739277    2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/55360bdc-81dd-44da-ad34-12ff0bc70dc8-etc-sysconfig\") pod \"tuned-bbnvm\" (UID: \"55360bdc-81dd-44da-ad34-12ff0bc70dc8\") " pod="openshift-cluster-node-tuning-operator/tuned-bbnvm"
Sep 16 18:43:59.739514 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:43:59.739303    2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/7cefdcfe-d218-4ac0-a3d8-811df8c043c2-host-slash\") pod \"ovnkube-node-nqcpx\" (UID: \"7cefdcfe-d218-4ac0-a3d8-811df8c043c2\") " pod="openshift-ovn-kubernetes/ovnkube-node-nqcpx"
Sep 16 18:43:59.739514 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:43:59.739407    2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rddb6\" (UniqueName: \"kubernetes.io/projected/273fb2b8-0579-4f86-abc5-2801f6f47bcd-kube-api-access-rddb6\") pod \"node-ca-2hdh4\" (UID: \"273fb2b8-0579-4f86-abc5-2801f6f47bcd\") " pod="openshift-image-registry/node-ca-2hdh4"
Sep 16 18:43:59.739514 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:43:59.739441    2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/38dd6a59-da17-47cd-b10f-540f2fff9a89-cnibin\") pod \"multus-additional-cni-plugins-znf6p\" (UID: \"38dd6a59-da17-47cd-b10f-540f2fff9a89\") " pod="openshift-multus/multus-additional-cni-plugins-znf6p"
Sep 16 18:43:59.739514 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:43:59.739466    2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/55360bdc-81dd-44da-ad34-12ff0bc70dc8-etc-systemd\") pod \"tuned-bbnvm\" (UID: \"55360bdc-81dd-44da-ad34-12ff0bc70dc8\") " pod="openshift-cluster-node-tuning-operator/tuned-bbnvm"
Sep 16 18:43:59.739514 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:43:59.739488    2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/55360bdc-81dd-44da-ad34-12ff0bc70dc8-etc-tuned\") pod \"tuned-bbnvm\" (UID: \"55360bdc-81dd-44da-ad34-12ff0bc70dc8\") " pod="openshift-cluster-node-tuning-operator/tuned-bbnvm"
Sep 16 18:43:59.739514 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:43:59.739511    2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/38dd6a59-da17-47cd-b10f-540f2fff9a89-cnibin\") pod \"multus-additional-cni-plugins-znf6p\" (UID: \"38dd6a59-da17-47cd-b10f-540f2fff9a89\") " pod="openshift-multus/multus-additional-cni-plugins-znf6p"
Sep 16 18:43:59.740061 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:43:59.739530    2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/7cefdcfe-d218-4ac0-a3d8-811df8c043c2-ovnkube-config\") pod \"ovnkube-node-nqcpx\" (UID: \"7cefdcfe-d218-4ac0-a3d8-811df8c043c2\") " pod="openshift-ovn-kubernetes/ovnkube-node-nqcpx"
Sep 16 18:43:59.740061 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:43:59.739595    2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/7cefdcfe-d218-4ac0-a3d8-811df8c043c2-ovnkube-script-lib\") pod \"ovnkube-node-nqcpx\" (UID: \"7cefdcfe-d218-4ac0-a3d8-811df8c043c2\") " pod="openshift-ovn-kubernetes/ovnkube-node-nqcpx"
Sep 16 18:43:59.740061 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:43:59.739617    2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/273fb2b8-0579-4f86-abc5-2801f6f47bcd-host\") pod \"node-ca-2hdh4\" (UID: \"273fb2b8-0579-4f86-abc5-2801f6f47bcd\") " pod="openshift-image-registry/node-ca-2hdh4"
Sep 16 18:43:59.740061 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:43:59.739635    2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/9b277c2d-db88-424b-9366-c89532c4b4f1-registration-dir\") pod \"aws-ebs-csi-driver-node-zv6wl\" (UID: \"9b277c2d-db88-424b-9366-c89532c4b4f1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zv6wl"
Sep 16 18:43:59.740061 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:43:59.739667    2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jwx9f\" (UniqueName: \"kubernetes.io/projected/db1e3b32-114c-4bd9-b28c-21a2a9d6efaa-kube-api-access-jwx9f\") pod \"network-metrics-daemon-pdfpq\" (UID: \"db1e3b32-114c-4bd9-b28c-21a2a9d6efaa\") " pod="openshift-multus/network-metrics-daemon-pdfpq"
Sep 16 18:43:59.740061 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:43:59.739692    2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/55360bdc-81dd-44da-ad34-12ff0bc70dc8-etc-sysctl-conf\") pod \"tuned-bbnvm\" (UID: \"55360bdc-81dd-44da-ad34-12ff0bc70dc8\") " pod="openshift-cluster-node-tuning-operator/tuned-bbnvm"
Sep 16 18:43:59.740061 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:43:59.739715    2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/9b277c2d-db88-424b-9366-c89532c4b4f1-registration-dir\") pod \"aws-ebs-csi-driver-node-zv6wl\" (UID: \"9b277c2d-db88-424b-9366-c89532c4b4f1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zv6wl"
Sep 16 18:43:59.740061 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:43:59.739814    2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/55360bdc-81dd-44da-ad34-12ff0bc70dc8-sys\") pod \"tuned-bbnvm\" (UID: \"55360bdc-81dd-44da-ad34-12ff0bc70dc8\") " pod="openshift-cluster-node-tuning-operator/tuned-bbnvm"
Sep 16 18:43:59.740061 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:43:59.739849    2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/38dd6a59-da17-47cd-b10f-540f2fff9a89-cni-binary-copy\") pod \"multus-additional-cni-plugins-znf6p\" (UID: \"38dd6a59-da17-47cd-b10f-540f2fff9a89\") " pod="openshift-multus/multus-additional-cni-plugins-znf6p"
Sep 16 18:43:59.740061 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:43:59.739872    2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/9b277c2d-db88-424b-9366-c89532c4b4f1-kubelet-dir\") pod \"aws-ebs-csi-driver-node-zv6wl\" (UID: \"9b277c2d-db88-424b-9366-c89532c4b4f1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zv6wl"
Sep 16 18:43:59.740061 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:43:59.739899    2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/55360bdc-81dd-44da-ad34-12ff0bc70dc8-run\") pod \"tuned-bbnvm\" (UID: \"55360bdc-81dd-44da-ad34-12ff0bc70dc8\") " pod="openshift-cluster-node-tuning-operator/tuned-bbnvm"
Sep 16 18:43:59.740061 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:43:59.739961    2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/9b277c2d-db88-424b-9366-c89532c4b4f1-kubelet-dir\") pod \"aws-ebs-csi-driver-node-zv6wl\" (UID: \"9b277c2d-db88-424b-9366-c89532c4b4f1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zv6wl"
Sep 16 18:43:59.740061 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:43:59.739999    2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/7cefdcfe-d218-4ac0-a3d8-811df8c043c2-host-run-netns\") pod \"ovnkube-node-nqcpx\" (UID: \"7cefdcfe-d218-4ac0-a3d8-811df8c043c2\") " pod="openshift-ovn-kubernetes/ovnkube-node-nqcpx"
Sep 16 18:43:59.740061 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:43:59.740029    2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/7cefdcfe-d218-4ac0-a3d8-811df8c043c2-run-ovn\") pod \"ovnkube-node-nqcpx\" (UID: \"7cefdcfe-d218-4ac0-a3d8-811df8c043c2\") " pod="openshift-ovn-kubernetes/ovnkube-node-nqcpx"
Sep 16 18:43:59.740061 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:43:59.740052    2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/7cefdcfe-d218-4ac0-a3d8-811df8c043c2-node-log\") pod \"ovnkube-node-nqcpx\" (UID: \"7cefdcfe-d218-4ac0-a3d8-811df8c043c2\") " pod="openshift-ovn-kubernetes/ovnkube-node-nqcpx"
Sep 16 18:43:59.740777 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:43:59.740079    2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/bef94c9f-6884-4c22-b616-32f4e38a4772-cni-binary-copy\") pod \"multus-dbwbz\" (UID: \"bef94c9f-6884-4c22-b616-32f4e38a4772\") " pod="openshift-multus/multus-dbwbz"
Sep 16 18:43:59.740777 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:43:59.740109    2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/2bbf45e9-df8e-4a67-b777-0c629428fca2-kubelet-config\") pod \"global-pull-secret-syncer-pg4kf\" (UID: \"2bbf45e9-df8e-4a67-b777-0c629428fca2\") " pod="kube-system/global-pull-secret-syncer-pg4kf"
Sep 16 18:43:59.740777 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:43:59.740134    2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jfzj7\" (UniqueName: \"kubernetes.io/projected/524324ca-6ba2-4885-9ed6-570fa75665e7-kube-api-access-jfzj7\") pod \"node-resolver-sxbgl\" (UID: \"524324ca-6ba2-4885-9ed6-570fa75665e7\") " pod="openshift-dns/node-resolver-sxbgl"
Sep 16 18:43:59.740777 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:43:59.740157    2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/7cefdcfe-d218-4ac0-a3d8-811df8c043c2-etc-openvswitch\") pod \"ovnkube-node-nqcpx\" (UID: \"7cefdcfe-d218-4ac0-a3d8-811df8c043c2\") " pod="openshift-ovn-kubernetes/ovnkube-node-nqcpx"
Sep 16 18:43:59.740777 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:43:59.740182    2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/bef94c9f-6884-4c22-b616-32f4e38a4772-system-cni-dir\") pod \"multus-dbwbz\" (UID: \"bef94c9f-6884-4c22-b616-32f4e38a4772\") " pod="openshift-multus/multus-dbwbz"
Sep 16 18:43:59.740777 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:43:59.740206    2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/bef94c9f-6884-4c22-b616-32f4e38a4772-hostroot\") pod \"multus-dbwbz\" (UID: \"bef94c9f-6884-4c22-b616-32f4e38a4772\") " pod="openshift-multus/multus-dbwbz"
Sep 16 18:43:59.740777 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:43:59.740242    2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/0494be2a-d0e5-4570-b05c-0c595a0b494a-agent-certs\") pod \"konnectivity-agent-pzpb9\" (UID: \"0494be2a-d0e5-4570-b05c-0c595a0b494a\") " pod="kube-system/konnectivity-agent-pzpb9"
Sep 16 18:43:59.740777 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:43:59.740266    2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/55360bdc-81dd-44da-ad34-12ff0bc70dc8-etc-sysctl-d\") pod \"tuned-bbnvm\" (UID: \"55360bdc-81dd-44da-ad34-12ff0bc70dc8\") " pod="openshift-cluster-node-tuning-operator/tuned-bbnvm"
Sep 16 18:43:59.740777 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:43:59.740289    2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bqrfl\" (UniqueName: \"kubernetes.io/projected/55360bdc-81dd-44da-ad34-12ff0bc70dc8-kube-api-access-bqrfl\") pod \"tuned-bbnvm\" (UID: \"55360bdc-81dd-44da-ad34-12ff0bc70dc8\") " pod="openshift-cluster-node-tuning-operator/tuned-bbnvm"
Sep 16 18:43:59.740777 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:43:59.740297    2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/bef94c9f-6884-4c22-b616-32f4e38a4772-hostroot\") pod \"multus-dbwbz\" (UID: \"bef94c9f-6884-4c22-b616-32f4e38a4772\") " pod="openshift-multus/multus-dbwbz"
Sep 16 18:43:59.740777 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:43:59.740297    2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/bef94c9f-6884-4c22-b616-32f4e38a4772-system-cni-dir\") pod \"multus-dbwbz\" (UID: \"bef94c9f-6884-4c22-b616-32f4e38a4772\") " pod="openshift-multus/multus-dbwbz"
Sep 16 18:43:59.740777 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:43:59.740312    2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/7cefdcfe-d218-4ac0-a3d8-811df8c043c2-var-lib-openvswitch\") pod \"ovnkube-node-nqcpx\" (UID: \"7cefdcfe-d218-4ac0-a3d8-811df8c043c2\") " pod="openshift-ovn-kubernetes/ovnkube-node-nqcpx"
Sep 16 18:43:59.740777 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:43:59.740362    2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/7cefdcfe-d218-4ac0-a3d8-811df8c043c2-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-nqcpx\" (UID: \"7cefdcfe-d218-4ac0-a3d8-811df8c043c2\") " pod="openshift-ovn-kubernetes/ovnkube-node-nqcpx"
Sep 16 18:43:59.740777 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:43:59.740387    2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/7cefdcfe-d218-4ac0-a3d8-811df8c043c2-env-overrides\") pod \"ovnkube-node-nqcpx\" (UID: \"7cefdcfe-d218-4ac0-a3d8-811df8c043c2\") " pod="openshift-ovn-kubernetes/ovnkube-node-nqcpx"
Sep 16 18:43:59.740777 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:43:59.740413    2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/38dd6a59-da17-47cd-b10f-540f2fff9a89-tuning-conf-dir\") pod \"multus-additional-cni-plugins-znf6p\" (UID: \"38dd6a59-da17-47cd-b10f-540f2fff9a89\") " pod="openshift-multus/multus-additional-cni-plugins-znf6p"
Sep 16 18:43:59.740777 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:43:59.740428    2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/38dd6a59-da17-47cd-b10f-540f2fff9a89-cni-binary-copy\") pod \"multus-additional-cni-plugins-znf6p\" (UID: \"38dd6a59-da17-47cd-b10f-540f2fff9a89\") " pod="openshift-multus/multus-additional-cni-plugins-znf6p"
Sep 16 18:43:59.740777 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:43:59.740436    2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/9b277c2d-db88-424b-9366-c89532c4b4f1-device-dir\") pod \"aws-ebs-csi-driver-node-zv6wl\" (UID: \"9b277c2d-db88-424b-9366-c89532c4b4f1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zv6wl"
Sep 16 18:43:59.741592 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:43:59.740474    2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/9b277c2d-db88-424b-9366-c89532c4b4f1-device-dir\") pod \"aws-ebs-csi-driver-node-zv6wl\" (UID: \"9b277c2d-db88-424b-9366-c89532c4b4f1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zv6wl"
Sep 16 18:43:59.741592 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:43:59.740474    2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/0494be2a-d0e5-4570-b05c-0c595a0b494a-konnectivity-ca\") pod \"konnectivity-agent-pzpb9\" (UID: \"0494be2a-d0e5-4570-b05c-0c595a0b494a\") " pod="kube-system/konnectivity-agent-pzpb9"
Sep 16 18:43:59.741592 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:43:59.740542    2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/bef94c9f-6884-4c22-b616-32f4e38a4772-host-run-netns\") pod \"multus-dbwbz\" (UID: \"bef94c9f-6884-4c22-b616-32f4e38a4772\") " pod="openshift-multus/multus-dbwbz"
Sep 16 18:43:59.741592 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:43:59.740591    2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/38dd6a59-da17-47cd-b10f-540f2fff9a89-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-znf6p\" (UID: \"38dd6a59-da17-47cd-b10f-540f2fff9a89\") " pod="openshift-multus/multus-additional-cni-plugins-znf6p"
Sep 16 18:43:59.741592 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:43:59.740601    2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/bef94c9f-6884-4c22-b616-32f4e38a4772-host-run-netns\") pod \"multus-dbwbz\" (UID: \"bef94c9f-6884-4c22-b616-32f4e38a4772\") " pod="openshift-multus/multus-dbwbz"
Sep 16 18:43:59.741592 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:43:59.740615    2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/9b277c2d-db88-424b-9366-c89532c4b4f1-sys-fs\") pod \"aws-ebs-csi-driver-node-zv6wl\" (UID: \"9b277c2d-db88-424b-9366-c89532c4b4f1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zv6wl"
Sep 16 18:43:59.741592 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:43:59.740626    2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/bef94c9f-6884-4c22-b616-32f4e38a4772-cni-binary-copy\") pod \"multus-dbwbz\" (UID: \"bef94c9f-6884-4c22-b616-32f4e38a4772\") " pod="openshift-multus/multus-dbwbz"
Sep 16 18:43:59.741592 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:43:59.740639    2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/db1e3b32-114c-4bd9-b28c-21a2a9d6efaa-metrics-certs\") pod \"network-metrics-daemon-pdfpq\" (UID: \"db1e3b32-114c-4bd9-b28c-21a2a9d6efaa\") " pod="openshift-multus/network-metrics-daemon-pdfpq"
Sep 16 18:43:59.741592 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:43:59.740659    2544 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Sep 16 18:43:59.741592 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:43:59.740634    2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/38dd6a59-da17-47cd-b10f-540f2fff9a89-tuning-conf-dir\") pod \"multus-additional-cni-plugins-znf6p\" (UID: \"38dd6a59-da17-47cd-b10f-540f2fff9a89\") " pod="openshift-multus/multus-additional-cni-plugins-znf6p"
Sep 16 18:43:59.741592 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:43:59.740664    2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/55360bdc-81dd-44da-ad34-12ff0bc70dc8-lib-modules\") pod \"tuned-bbnvm\" (UID: \"55360bdc-81dd-44da-ad34-12ff0bc70dc8\") " pod="openshift-cluster-node-tuning-operator/tuned-bbnvm"
Sep 16 18:43:59.741592 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:43:59.740690    2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/9b277c2d-db88-424b-9366-c89532c4b4f1-sys-fs\") pod \"aws-ebs-csi-driver-node-zv6wl\" (UID: \"9b277c2d-db88-424b-9366-c89532c4b4f1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zv6wl"
Sep 16 18:43:59.741592 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:43:59.740708    2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/7cefdcfe-d218-4ac0-a3d8-811df8c043c2-host-run-ovn-kubernetes\") pod \"ovnkube-node-nqcpx\" (UID: \"7cefdcfe-d218-4ac0-a3d8-811df8c043c2\") " pod="openshift-ovn-kubernetes/ovnkube-node-nqcpx"
Sep 16 18:43:59.741592 ip-10-0-132-80 kubenswrapper[2544]: E0916 18:43:59.740714    2544 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 16 18:43:59.741592 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:43:59.740735    2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-frsll\" (UniqueName: \"kubernetes.io/projected/7cefdcfe-d218-4ac0-a3d8-811df8c043c2-kube-api-access-frsll\") pod \"ovnkube-node-nqcpx\" (UID: \"7cefdcfe-d218-4ac0-a3d8-811df8c043c2\") " pod="openshift-ovn-kubernetes/ovnkube-node-nqcpx"
Sep 16 18:43:59.741592 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:43:59.740758    2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/273fb2b8-0579-4f86-abc5-2801f6f47bcd-serviceca\") pod \"node-ca-2hdh4\" (UID: \"273fb2b8-0579-4f86-abc5-2801f6f47bcd\") " pod="openshift-image-registry/node-ca-2hdh4"
Sep 16 18:43:59.741592 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:43:59.740781    2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/38dd6a59-da17-47cd-b10f-540f2fff9a89-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-znf6p\" (UID: \"38dd6a59-da17-47cd-b10f-540f2fff9a89\") " pod="openshift-multus/multus-additional-cni-plugins-znf6p"
Sep 16 18:43:59.741592 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:43:59.740802    2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/55360bdc-81dd-44da-ad34-12ff0bc70dc8-host\") pod \"tuned-bbnvm\" (UID: \"55360bdc-81dd-44da-ad34-12ff0bc70dc8\") " pod="openshift-cluster-node-tuning-operator/tuned-bbnvm"
Sep 16 18:43:59.742301 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:43:59.740823    2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/2bbf45e9-df8e-4a67-b777-0c629428fca2-dbus\") pod \"global-pull-secret-syncer-pg4kf\" (UID: \"2bbf45e9-df8e-4a67-b777-0c629428fca2\") " pod="kube-system/global-pull-secret-syncer-pg4kf"
Sep 16 18:43:59.742301 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:43:59.740844    2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/524324ca-6ba2-4885-9ed6-570fa75665e7-hosts-file\") pod \"node-resolver-sxbgl\" (UID: \"524324ca-6ba2-4885-9ed6-570fa75665e7\") " pod="openshift-dns/node-resolver-sxbgl"
Sep 16 18:43:59.742301 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:43:59.740868    2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s9f9h\" (UniqueName: \"kubernetes.io/projected/3c4f2334-3ac6-47e0-9eb7-4352c53df29a-kube-api-access-s9f9h\") pod \"network-check-target-t8hx2\" (UID: \"3c4f2334-3ac6-47e0-9eb7-4352c53df29a\") " pod="openshift-network-diagnostics/network-check-target-t8hx2"
Sep 16 18:43:59.742301 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:43:59.740893    2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/bef94c9f-6884-4c22-b616-32f4e38a4772-cnibin\") pod \"multus-dbwbz\" (UID: \"bef94c9f-6884-4c22-b616-32f4e38a4772\") " pod="openshift-multus/multus-dbwbz"
Sep 16 18:43:59.742301 ip-10-0-132-80 kubenswrapper[2544]: E0916 18:43:59.740933    2544 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/db1e3b32-114c-4bd9-b28c-21a2a9d6efaa-metrics-certs podName:db1e3b32-114c-4bd9-b28c-21a2a9d6efaa nodeName:}" failed. No retries permitted until 2026-09-16 18:44:00.240888509 +0000 UTC m=+3.029891034 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/db1e3b32-114c-4bd9-b28c-21a2a9d6efaa-metrics-certs") pod "network-metrics-daemon-pdfpq" (UID: "db1e3b32-114c-4bd9-b28c-21a2a9d6efaa") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 16 18:43:59.742301 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:43:59.740949    2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/bef94c9f-6884-4c22-b616-32f4e38a4772-cnibin\") pod \"multus-dbwbz\" (UID: \"bef94c9f-6884-4c22-b616-32f4e38a4772\") " pod="openshift-multus/multus-dbwbz"
Sep 16 18:43:59.742301 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:43:59.740971    2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/2bbf45e9-df8e-4a67-b777-0c629428fca2-original-pull-secret\") pod \"global-pull-secret-syncer-pg4kf\" (UID: \"2bbf45e9-df8e-4a67-b777-0c629428fca2\") " pod="kube-system/global-pull-secret-syncer-pg4kf"
Sep 16 18:43:59.742301 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:43:59.740988    2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/0494be2a-d0e5-4570-b05c-0c595a0b494a-konnectivity-ca\") pod \"konnectivity-agent-pzpb9\" (UID: \"0494be2a-d0e5-4570-b05c-0c595a0b494a\") " pod="kube-system/konnectivity-agent-pzpb9"
Sep 16 18:43:59.742301 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:43:59.741011    2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/38dd6a59-da17-47cd-b10f-540f2fff9a89-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-znf6p\" (UID: \"38dd6a59-da17-47cd-b10f-540f2fff9a89\") " pod="openshift-multus/multus-additional-cni-plugins-znf6p"
Sep 16 18:43:59.742301 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:43:59.741027    2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/9b277c2d-db88-424b-9366-c89532c4b4f1-etc-selinux\") pod \"aws-ebs-csi-driver-node-zv6wl\" (UID: \"9b277c2d-db88-424b-9366-c89532c4b4f1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zv6wl"
Sep 16 18:43:59.742301 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:43:59.741085    2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/bef94c9f-6884-4c22-b616-32f4e38a4772-host-var-lib-kubelet\") pod \"multus-dbwbz\" (UID: \"bef94c9f-6884-4c22-b616-32f4e38a4772\") " pod="openshift-multus/multus-dbwbz"
Sep 16 18:43:59.742301 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:43:59.741135    2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/9b277c2d-db88-424b-9366-c89532c4b4f1-etc-selinux\") pod \"aws-ebs-csi-driver-node-zv6wl\" (UID: \"9b277c2d-db88-424b-9366-c89532c4b4f1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zv6wl"
Sep 16 18:43:59.742301 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:43:59.741135    2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/bef94c9f-6884-4c22-b616-32f4e38a4772-multus-daemon-config\") pod \"multus-dbwbz\" (UID: \"bef94c9f-6884-4c22-b616-32f4e38a4772\") " pod="openshift-multus/multus-dbwbz"
Sep 16 18:43:59.742301 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:43:59.741166    2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/bef94c9f-6884-4c22-b616-32f4e38a4772-etc-kubernetes\") pod \"multus-dbwbz\" (UID: \"bef94c9f-6884-4c22-b616-32f4e38a4772\") " pod="openshift-multus/multus-dbwbz"
Sep 16 18:43:59.742301 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:43:59.741201    2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/bef94c9f-6884-4c22-b616-32f4e38a4772-etc-kubernetes\") pod \"multus-dbwbz\" (UID: \"bef94c9f-6884-4c22-b616-32f4e38a4772\") " pod="openshift-multus/multus-dbwbz"
Sep 16 18:43:59.742301 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:43:59.741234    2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/bef94c9f-6884-4c22-b616-32f4e38a4772-host-var-lib-kubelet\") pod \"multus-dbwbz\" (UID: \"bef94c9f-6884-4c22-b616-32f4e38a4772\") " pod="openshift-multus/multus-dbwbz"
Sep 16 18:43:59.742927 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:43:59.741241    2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/22e551bf-0ed9-4e8f-9487-8412901fcdcb-iptables-alerter-script\") pod \"iptables-alerter-z4wt6\" (UID: \"22e551bf-0ed9-4e8f-9487-8412901fcdcb\") " pod="openshift-network-operator/iptables-alerter-z4wt6"
Sep 16 18:43:59.742927 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:43:59.741288    2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kgbjz\" (UniqueName: \"kubernetes.io/projected/22e551bf-0ed9-4e8f-9487-8412901fcdcb-kube-api-access-kgbjz\") pod \"iptables-alerter-z4wt6\" (UID: \"22e551bf-0ed9-4e8f-9487-8412901fcdcb\") " pod="openshift-network-operator/iptables-alerter-z4wt6"
Sep 16 18:43:59.742927 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:43:59.741313    2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/bef94c9f-6884-4c22-b616-32f4e38a4772-multus-cni-dir\") pod \"multus-dbwbz\" (UID: \"bef94c9f-6884-4c22-b616-32f4e38a4772\") " pod="openshift-multus/multus-dbwbz"
Sep 16 18:43:59.742927 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:43:59.741337    2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/bef94c9f-6884-4c22-b616-32f4e38a4772-host-run-k8s-cni-cncf-io\") pod \"multus-dbwbz\" (UID: \"bef94c9f-6884-4c22-b616-32f4e38a4772\") " pod="openshift-multus/multus-dbwbz"
Sep 16 18:43:59.742927 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:43:59.741376    2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/bef94c9f-6884-4c22-b616-32f4e38a4772-host-run-multus-certs\") pod \"multus-dbwbz\" (UID: \"bef94c9f-6884-4c22-b616-32f4e38a4772\") " pod="openshift-multus/multus-dbwbz"
Sep 16 18:43:59.742927 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:43:59.741397    2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/38dd6a59-da17-47cd-b10f-540f2fff9a89-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-znf6p\" (UID: \"38dd6a59-da17-47cd-b10f-540f2fff9a89\") " pod="openshift-multus/multus-additional-cni-plugins-znf6p"
Sep 16 18:43:59.742927 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:43:59.741399    2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8jfrb\" (UniqueName: \"kubernetes.io/projected/bef94c9f-6884-4c22-b616-32f4e38a4772-kube-api-access-8jfrb\") pod \"multus-dbwbz\" (UID: \"bef94c9f-6884-4c22-b616-32f4e38a4772\") " pod="openshift-multus/multus-dbwbz"
Sep 16 18:43:59.742927 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:43:59.741450    2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/55360bdc-81dd-44da-ad34-12ff0bc70dc8-var-lib-kubelet\") pod \"tuned-bbnvm\" (UID: \"55360bdc-81dd-44da-ad34-12ff0bc70dc8\") " pod="openshift-cluster-node-tuning-operator/tuned-bbnvm"
Sep 16 18:43:59.742927 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:43:59.741473    2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/7cefdcfe-d218-4ac0-a3d8-811df8c043c2-host-cni-netd\") pod \"ovnkube-node-nqcpx\" (UID: \"7cefdcfe-d218-4ac0-a3d8-811df8c043c2\") " pod="openshift-ovn-kubernetes/ovnkube-node-nqcpx"
Sep 16 18:43:59.742927 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:43:59.741502    2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/bef94c9f-6884-4c22-b616-32f4e38a4772-host-var-lib-cni-bin\") pod \"multus-dbwbz\" (UID: \"bef94c9f-6884-4c22-b616-32f4e38a4772\") " pod="openshift-multus/multus-dbwbz"
Sep 16 18:43:59.742927 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:43:59.741520    2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/55360bdc-81dd-44da-ad34-12ff0bc70dc8-etc-kubernetes\") pod \"tuned-bbnvm\" (UID: \"55360bdc-81dd-44da-ad34-12ff0bc70dc8\") " pod="openshift-cluster-node-tuning-operator/tuned-bbnvm"
Sep 16 18:43:59.742927 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:43:59.741534    2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/55360bdc-81dd-44da-ad34-12ff0bc70dc8-tmp\") pod \"tuned-bbnvm\" (UID: \"55360bdc-81dd-44da-ad34-12ff0bc70dc8\") " pod="openshift-cluster-node-tuning-operator/tuned-bbnvm"
Sep 16 18:43:59.742927 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:43:59.741585    2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/7cefdcfe-d218-4ac0-a3d8-811df8c043c2-run-openvswitch\") pod \"ovnkube-node-nqcpx\" (UID: \"7cefdcfe-d218-4ac0-a3d8-811df8c043c2\") " pod="openshift-ovn-kubernetes/ovnkube-node-nqcpx"
Sep 16 18:43:59.742927 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:43:59.741623    2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/bef94c9f-6884-4c22-b616-32f4e38a4772-host-run-multus-certs\") pod \"multus-dbwbz\" (UID: \"bef94c9f-6884-4c22-b616-32f4e38a4772\") " pod="openshift-multus/multus-dbwbz"
Sep 16 18:43:59.742927 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:43:59.741652    2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/bef94c9f-6884-4c22-b616-32f4e38a4772-multus-daemon-config\") pod \"multus-dbwbz\" (UID: \"bef94c9f-6884-4c22-b616-32f4e38a4772\") " pod="openshift-multus/multus-dbwbz"
Sep 16 18:43:59.742927 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:43:59.741687    2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/bef94c9f-6884-4c22-b616-32f4e38a4772-host-run-k8s-cni-cncf-io\") pod \"multus-dbwbz\" (UID: \"bef94c9f-6884-4c22-b616-32f4e38a4772\") " pod="openshift-multus/multus-dbwbz"
Sep 16 18:43:59.742927 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:43:59.741704    2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/bef94c9f-6884-4c22-b616-32f4e38a4772-multus-cni-dir\") pod \"multus-dbwbz\" (UID: \"bef94c9f-6884-4c22-b616-32f4e38a4772\") " pod="openshift-multus/multus-dbwbz"
Sep 16 18:43:59.743519 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:43:59.741726    2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/bef94c9f-6884-4c22-b616-32f4e38a4772-host-var-lib-cni-bin\") pod \"multus-dbwbz\" (UID: \"bef94c9f-6884-4c22-b616-32f4e38a4772\") " pod="openshift-multus/multus-dbwbz"
Sep 16 18:43:59.743519 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:43:59.741970    2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/22e551bf-0ed9-4e8f-9487-8412901fcdcb-iptables-alerter-script\") pod \"iptables-alerter-z4wt6\" (UID: \"22e551bf-0ed9-4e8f-9487-8412901fcdcb\") " pod="openshift-network-operator/iptables-alerter-z4wt6"
Sep 16 18:43:59.744644 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:43:59.744624    2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/0494be2a-d0e5-4570-b05c-0c595a0b494a-agent-certs\") pod \"konnectivity-agent-pzpb9\" (UID: \"0494be2a-d0e5-4570-b05c-0c595a0b494a\") " pod="kube-system/konnectivity-agent-pzpb9"
Sep 16 18:43:59.747512 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:43:59.747493    2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hj6jb\" (UniqueName: \"kubernetes.io/projected/38dd6a59-da17-47cd-b10f-540f2fff9a89-kube-api-access-hj6jb\") pod \"multus-additional-cni-plugins-znf6p\" (UID: \"38dd6a59-da17-47cd-b10f-540f2fff9a89\") " pod="openshift-multus/multus-additional-cni-plugins-znf6p"
Sep 16 18:43:59.747740 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:43:59.747718    2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vgwqr\" (UniqueName: \"kubernetes.io/projected/9b277c2d-db88-424b-9366-c89532c4b4f1-kube-api-access-vgwqr\") pod \"aws-ebs-csi-driver-node-zv6wl\" (UID: \"9b277c2d-db88-424b-9366-c89532c4b4f1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zv6wl"
Sep 16 18:43:59.748230 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:43:59.748208    2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jwx9f\" (UniqueName: \"kubernetes.io/projected/db1e3b32-114c-4bd9-b28c-21a2a9d6efaa-kube-api-access-jwx9f\") pod \"network-metrics-daemon-pdfpq\" (UID: \"db1e3b32-114c-4bd9-b28c-21a2a9d6efaa\") " pod="openshift-multus/network-metrics-daemon-pdfpq"
Sep 16 18:43:59.749273 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:43:59.749249    2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kgbjz\" (UniqueName: \"kubernetes.io/projected/22e551bf-0ed9-4e8f-9487-8412901fcdcb-kube-api-access-kgbjz\") pod \"iptables-alerter-z4wt6\" (UID: \"22e551bf-0ed9-4e8f-9487-8412901fcdcb\") " pod="openshift-network-operator/iptables-alerter-z4wt6"
Sep 16 18:43:59.749755 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:43:59.749736    2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8jfrb\" (UniqueName: \"kubernetes.io/projected/bef94c9f-6884-4c22-b616-32f4e38a4772-kube-api-access-8jfrb\") pod \"multus-dbwbz\" (UID: \"bef94c9f-6884-4c22-b616-32f4e38a4772\") " pod="openshift-multus/multus-dbwbz"
Sep 16 18:43:59.841993 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:43:59.841956    2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/55360bdc-81dd-44da-ad34-12ff0bc70dc8-lib-modules\") pod \"tuned-bbnvm\" (UID: \"55360bdc-81dd-44da-ad34-12ff0bc70dc8\") " pod="openshift-cluster-node-tuning-operator/tuned-bbnvm"
Sep 16 18:43:59.842166 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:43:59.842001    2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/7cefdcfe-d218-4ac0-a3d8-811df8c043c2-host-run-ovn-kubernetes\") pod \"ovnkube-node-nqcpx\" (UID: \"7cefdcfe-d218-4ac0-a3d8-811df8c043c2\") " pod="openshift-ovn-kubernetes/ovnkube-node-nqcpx"
Sep 16 18:43:59.842166 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:43:59.842028    2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-frsll\" (UniqueName: \"kubernetes.io/projected/7cefdcfe-d218-4ac0-a3d8-811df8c043c2-kube-api-access-frsll\") pod \"ovnkube-node-nqcpx\" (UID: \"7cefdcfe-d218-4ac0-a3d8-811df8c043c2\") " pod="openshift-ovn-kubernetes/ovnkube-node-nqcpx"
Sep 16 18:43:59.842166 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:43:59.842104    2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/7cefdcfe-d218-4ac0-a3d8-811df8c043c2-host-run-ovn-kubernetes\") pod \"ovnkube-node-nqcpx\" (UID: \"7cefdcfe-d218-4ac0-a3d8-811df8c043c2\") " pod="openshift-ovn-kubernetes/ovnkube-node-nqcpx"
Sep 16 18:43:59.842166 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:43:59.842148    2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/273fb2b8-0579-4f86-abc5-2801f6f47bcd-serviceca\") pod \"node-ca-2hdh4\" (UID: \"273fb2b8-0579-4f86-abc5-2801f6f47bcd\") " pod="openshift-image-registry/node-ca-2hdh4"
Sep 16 18:43:59.842364 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:43:59.842164    2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/55360bdc-81dd-44da-ad34-12ff0bc70dc8-lib-modules\") pod \"tuned-bbnvm\" (UID: \"55360bdc-81dd-44da-ad34-12ff0bc70dc8\") " pod="openshift-cluster-node-tuning-operator/tuned-bbnvm"
Sep 16 18:43:59.842364 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:43:59.842182    2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/55360bdc-81dd-44da-ad34-12ff0bc70dc8-host\") pod \"tuned-bbnvm\" (UID: \"55360bdc-81dd-44da-ad34-12ff0bc70dc8\") " pod="openshift-cluster-node-tuning-operator/tuned-bbnvm"
Sep 16 18:43:59.842364 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:43:59.842206    2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/2bbf45e9-df8e-4a67-b777-0c629428fca2-dbus\") pod \"global-pull-secret-syncer-pg4kf\" (UID: \"2bbf45e9-df8e-4a67-b777-0c629428fca2\") " pod="kube-system/global-pull-secret-syncer-pg4kf"
Sep 16 18:43:59.842364 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:43:59.842229    2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/524324ca-6ba2-4885-9ed6-570fa75665e7-hosts-file\") pod \"node-resolver-sxbgl\" (UID: \"524324ca-6ba2-4885-9ed6-570fa75665e7\") " pod="openshift-dns/node-resolver-sxbgl"
Sep 16 18:43:59.842364 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:43:59.842253    2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/55360bdc-81dd-44da-ad34-12ff0bc70dc8-host\") pod \"tuned-bbnvm\" (UID: \"55360bdc-81dd-44da-ad34-12ff0bc70dc8\") " pod="openshift-cluster-node-tuning-operator/tuned-bbnvm"
Sep 16 18:43:59.842364 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:43:59.842254    2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s9f9h\" (UniqueName: \"kubernetes.io/projected/3c4f2334-3ac6-47e0-9eb7-4352c53df29a-kube-api-access-s9f9h\") pod \"network-check-target-t8hx2\" (UID: \"3c4f2334-3ac6-47e0-9eb7-4352c53df29a\") " pod="openshift-network-diagnostics/network-check-target-t8hx2"
Sep 16 18:43:59.842364 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:43:59.842304    2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/2bbf45e9-df8e-4a67-b777-0c629428fca2-original-pull-secret\") pod \"global-pull-secret-syncer-pg4kf\" (UID: \"2bbf45e9-df8e-4a67-b777-0c629428fca2\") " pod="kube-system/global-pull-secret-syncer-pg4kf"
Sep 16 18:43:59.842364 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:43:59.842342    2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/55360bdc-81dd-44da-ad34-12ff0bc70dc8-var-lib-kubelet\") pod \"tuned-bbnvm\" (UID: \"55360bdc-81dd-44da-ad34-12ff0bc70dc8\") " pod="openshift-cluster-node-tuning-operator/tuned-bbnvm"
Sep 16 18:43:59.842364 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:43:59.842346    2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/524324ca-6ba2-4885-9ed6-570fa75665e7-hosts-file\") pod \"node-resolver-sxbgl\" (UID: \"524324ca-6ba2-4885-9ed6-570fa75665e7\") " pod="openshift-dns/node-resolver-sxbgl"
Sep 16 18:43:59.842364 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:43:59.842304    2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/2bbf45e9-df8e-4a67-b777-0c629428fca2-dbus\") pod \"global-pull-secret-syncer-pg4kf\" (UID: \"2bbf45e9-df8e-4a67-b777-0c629428fca2\") " pod="kube-system/global-pull-secret-syncer-pg4kf"
Sep 16 18:43:59.842364 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:43:59.842367    2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/7cefdcfe-d218-4ac0-a3d8-811df8c043c2-host-cni-netd\") pod \"ovnkube-node-nqcpx\" (UID: \"7cefdcfe-d218-4ac0-a3d8-811df8c043c2\") " pod="openshift-ovn-kubernetes/ovnkube-node-nqcpx"
Sep 16 18:43:59.842962 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:43:59.842394    2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/55360bdc-81dd-44da-ad34-12ff0bc70dc8-etc-kubernetes\") pod \"tuned-bbnvm\" (UID: \"55360bdc-81dd-44da-ad34-12ff0bc70dc8\") " pod="openshift-cluster-node-tuning-operator/tuned-bbnvm"
Sep 16 18:43:59.842962 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:43:59.842417    2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/55360bdc-81dd-44da-ad34-12ff0bc70dc8-tmp\") pod \"tuned-bbnvm\" (UID: \"55360bdc-81dd-44da-ad34-12ff0bc70dc8\") " pod="openshift-cluster-node-tuning-operator/tuned-bbnvm"
Sep 16 18:43:59.842962 ip-10-0-132-80 kubenswrapper[2544]: E0916 18:43:59.842435    2544 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 16 18:43:59.842962 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:43:59.842440    2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/7cefdcfe-d218-4ac0-a3d8-811df8c043c2-run-openvswitch\") pod \"ovnkube-node-nqcpx\" (UID: \"7cefdcfe-d218-4ac0-a3d8-811df8c043c2\") " pod="openshift-ovn-kubernetes/ovnkube-node-nqcpx"
Sep 16 18:43:59.842962 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:43:59.842451    2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/7cefdcfe-d218-4ac0-a3d8-811df8c043c2-host-cni-netd\") pod \"ovnkube-node-nqcpx\" (UID: \"7cefdcfe-d218-4ac0-a3d8-811df8c043c2\") " pod="openshift-ovn-kubernetes/ovnkube-node-nqcpx"
Sep 16 18:43:59.842962 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:43:59.842466    2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/524324ca-6ba2-4885-9ed6-570fa75665e7-tmp-dir\") pod \"node-resolver-sxbgl\" (UID: \"524324ca-6ba2-4885-9ed6-570fa75665e7\") " pod="openshift-dns/node-resolver-sxbgl"
Sep 16 18:43:59.842962 ip-10-0-132-80 kubenswrapper[2544]: E0916 18:43:59.842490    2544 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2bbf45e9-df8e-4a67-b777-0c629428fca2-original-pull-secret podName:2bbf45e9-df8e-4a67-b777-0c629428fca2 nodeName:}" failed. No retries permitted until 2026-09-16 18:44:00.342471673 +0000 UTC m=+3.131474188 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/2bbf45e9-df8e-4a67-b777-0c629428fca2-original-pull-secret") pod "global-pull-secret-syncer-pg4kf" (UID: "2bbf45e9-df8e-4a67-b777-0c629428fca2") : object "kube-system"/"original-pull-secret" not registered
Sep 16 18:43:59.842962 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:43:59.842502    2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/55360bdc-81dd-44da-ad34-12ff0bc70dc8-var-lib-kubelet\") pod \"tuned-bbnvm\" (UID: \"55360bdc-81dd-44da-ad34-12ff0bc70dc8\") " pod="openshift-cluster-node-tuning-operator/tuned-bbnvm"
Sep 16 18:43:59.842962 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:43:59.842516    2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/7cefdcfe-d218-4ac0-a3d8-811df8c043c2-host-cni-bin\") pod \"ovnkube-node-nqcpx\" (UID: \"7cefdcfe-d218-4ac0-a3d8-811df8c043c2\") " pod="openshift-ovn-kubernetes/ovnkube-node-nqcpx"
Sep 16 18:43:59.842962 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:43:59.842544    2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/7cefdcfe-d218-4ac0-a3d8-811df8c043c2-host-kubelet\") pod \"ovnkube-node-nqcpx\" (UID: \"7cefdcfe-d218-4ac0-a3d8-811df8c043c2\") " pod="openshift-ovn-kubernetes/ovnkube-node-nqcpx"
Sep 16 18:43:59.842962 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:43:59.842603    2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/7cefdcfe-d218-4ac0-a3d8-811df8c043c2-systemd-units\") pod \"ovnkube-node-nqcpx\" (UID: \"7cefdcfe-d218-4ac0-a3d8-811df8c043c2\") " pod="openshift-ovn-kubernetes/ovnkube-node-nqcpx"
Sep 16 18:43:59.842962 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:43:59.842631    2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/55360bdc-81dd-44da-ad34-12ff0bc70dc8-etc-modprobe-d\") pod \"tuned-bbnvm\" (UID: \"55360bdc-81dd-44da-ad34-12ff0bc70dc8\") " pod="openshift-cluster-node-tuning-operator/tuned-bbnvm"
Sep 16 18:43:59.842962 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:43:59.842655    2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/7cefdcfe-d218-4ac0-a3d8-811df8c043c2-run-systemd\") pod \"ovnkube-node-nqcpx\" (UID: \"7cefdcfe-d218-4ac0-a3d8-811df8c043c2\") " pod="openshift-ovn-kubernetes/ovnkube-node-nqcpx"
Sep 16 18:43:59.842962 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:43:59.842694    2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/7cefdcfe-d218-4ac0-a3d8-811df8c043c2-systemd-units\") pod \"ovnkube-node-nqcpx\" (UID: \"7cefdcfe-d218-4ac0-a3d8-811df8c043c2\") " pod="openshift-ovn-kubernetes/ovnkube-node-nqcpx"
Sep 16 18:43:59.842962 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:43:59.842700    2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/7cefdcfe-d218-4ac0-a3d8-811df8c043c2-host-kubelet\") pod \"ovnkube-node-nqcpx\" (UID: \"7cefdcfe-d218-4ac0-a3d8-811df8c043c2\") " pod="openshift-ovn-kubernetes/ovnkube-node-nqcpx"
Sep 16 18:43:59.842962 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:43:59.842727    2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/7cefdcfe-d218-4ac0-a3d8-811df8c043c2-run-systemd\") pod \"ovnkube-node-nqcpx\" (UID: \"7cefdcfe-d218-4ac0-a3d8-811df8c043c2\") " pod="openshift-ovn-kubernetes/ovnkube-node-nqcpx"
Sep 16 18:43:59.842962 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:43:59.842741    2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/7cefdcfe-d218-4ac0-a3d8-811df8c043c2-log-socket\") pod \"ovnkube-node-nqcpx\" (UID: \"7cefdcfe-d218-4ac0-a3d8-811df8c043c2\") " pod="openshift-ovn-kubernetes/ovnkube-node-nqcpx"
Sep 16 18:43:59.843802 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:43:59.842750    2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/55360bdc-81dd-44da-ad34-12ff0bc70dc8-etc-modprobe-d\") pod \"tuned-bbnvm\" (UID: \"55360bdc-81dd-44da-ad34-12ff0bc70dc8\") " pod="openshift-cluster-node-tuning-operator/tuned-bbnvm"
Sep 16 18:43:59.843802 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:43:59.842762    2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/7cefdcfe-d218-4ac0-a3d8-811df8c043c2-host-cni-bin\") pod \"ovnkube-node-nqcpx\" (UID: \"7cefdcfe-d218-4ac0-a3d8-811df8c043c2\") " pod="openshift-ovn-kubernetes/ovnkube-node-nqcpx"
Sep 16 18:43:59.843802 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:43:59.842762    2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/524324ca-6ba2-4885-9ed6-570fa75665e7-tmp-dir\") pod \"node-resolver-sxbgl\" (UID: \"524324ca-6ba2-4885-9ed6-570fa75665e7\") " pod="openshift-dns/node-resolver-sxbgl"
Sep 16 18:43:59.843802 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:43:59.842775    2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/7cefdcfe-d218-4ac0-a3d8-811df8c043c2-ovn-node-metrics-cert\") pod \"ovnkube-node-nqcpx\" (UID: \"7cefdcfe-d218-4ac0-a3d8-811df8c043c2\") " pod="openshift-ovn-kubernetes/ovnkube-node-nqcpx"
Sep 16 18:43:59.843802 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:43:59.842786    2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/7cefdcfe-d218-4ac0-a3d8-811df8c043c2-log-socket\") pod \"ovnkube-node-nqcpx\" (UID: \"7cefdcfe-d218-4ac0-a3d8-811df8c043c2\") " pod="openshift-ovn-kubernetes/ovnkube-node-nqcpx"
Sep 16 18:43:59.843802 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:43:59.842804    2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/55360bdc-81dd-44da-ad34-12ff0bc70dc8-etc-sysconfig\") pod \"tuned-bbnvm\" (UID: \"55360bdc-81dd-44da-ad34-12ff0bc70dc8\") " pod="openshift-cluster-node-tuning-operator/tuned-bbnvm"
Sep 16 18:43:59.843802 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:43:59.842814    2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/55360bdc-81dd-44da-ad34-12ff0bc70dc8-etc-kubernetes\") pod \"tuned-bbnvm\" (UID: \"55360bdc-81dd-44da-ad34-12ff0bc70dc8\") " pod="openshift-cluster-node-tuning-operator/tuned-bbnvm"
Sep 16 18:43:59.843802 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:43:59.842828    2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/7cefdcfe-d218-4ac0-a3d8-811df8c043c2-host-slash\") pod \"ovnkube-node-nqcpx\" (UID: \"7cefdcfe-d218-4ac0-a3d8-811df8c043c2\") " pod="openshift-ovn-kubernetes/ovnkube-node-nqcpx"
Sep 16 18:43:59.843802 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:43:59.842852    2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rddb6\" (UniqueName: \"kubernetes.io/projected/273fb2b8-0579-4f86-abc5-2801f6f47bcd-kube-api-access-rddb6\") pod \"node-ca-2hdh4\" (UID: \"273fb2b8-0579-4f86-abc5-2801f6f47bcd\") " pod="openshift-image-registry/node-ca-2hdh4"
Sep 16 18:43:59.843802 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:43:59.842857    2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/7cefdcfe-d218-4ac0-a3d8-811df8c043c2-run-openvswitch\") pod \"ovnkube-node-nqcpx\" (UID: \"7cefdcfe-d218-4ac0-a3d8-811df8c043c2\") " pod="openshift-ovn-kubernetes/ovnkube-node-nqcpx"
Sep 16 18:43:59.843802 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:43:59.842877    2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/55360bdc-81dd-44da-ad34-12ff0bc70dc8-etc-systemd\") pod \"tuned-bbnvm\" (UID: \"55360bdc-81dd-44da-ad34-12ff0bc70dc8\") " pod="openshift-cluster-node-tuning-operator/tuned-bbnvm"
Sep 16 18:43:59.843802 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:43:59.842900    2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/55360bdc-81dd-44da-ad34-12ff0bc70dc8-etc-tuned\") pod \"tuned-bbnvm\" (UID: \"55360bdc-81dd-44da-ad34-12ff0bc70dc8\") " pod="openshift-cluster-node-tuning-operator/tuned-bbnvm"
Sep 16 18:43:59.843802 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:43:59.842897    2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/7cefdcfe-d218-4ac0-a3d8-811df8c043c2-host-slash\") pod \"ovnkube-node-nqcpx\" (UID: \"7cefdcfe-d218-4ac0-a3d8-811df8c043c2\") " pod="openshift-ovn-kubernetes/ovnkube-node-nqcpx"
Sep 16 18:43:59.843802 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:43:59.842926    2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/7cefdcfe-d218-4ac0-a3d8-811df8c043c2-ovnkube-config\") pod \"ovnkube-node-nqcpx\" (UID: \"7cefdcfe-d218-4ac0-a3d8-811df8c043c2\") " pod="openshift-ovn-kubernetes/ovnkube-node-nqcpx"
Sep 16 18:43:59.843802 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:43:59.842948    2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/55360bdc-81dd-44da-ad34-12ff0bc70dc8-etc-sysconfig\") pod \"tuned-bbnvm\" (UID: \"55360bdc-81dd-44da-ad34-12ff0bc70dc8\") " pod="openshift-cluster-node-tuning-operator/tuned-bbnvm"
Sep 16 18:43:59.843802 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:43:59.842970    2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/7cefdcfe-d218-4ac0-a3d8-811df8c043c2-ovnkube-script-lib\") pod \"ovnkube-node-nqcpx\" (UID: \"7cefdcfe-d218-4ac0-a3d8-811df8c043c2\") " pod="openshift-ovn-kubernetes/ovnkube-node-nqcpx"
Sep 16 18:43:59.843802 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:43:59.842995    2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/273fb2b8-0579-4f86-abc5-2801f6f47bcd-host\") pod \"node-ca-2hdh4\" (UID: \"273fb2b8-0579-4f86-abc5-2801f6f47bcd\") " pod="openshift-image-registry/node-ca-2hdh4"
Sep 16 18:43:59.843802 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:43:59.843038    2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/55360bdc-81dd-44da-ad34-12ff0bc70dc8-etc-sysctl-conf\") pod \"tuned-bbnvm\" (UID: \"55360bdc-81dd-44da-ad34-12ff0bc70dc8\") " pod="openshift-cluster-node-tuning-operator/tuned-bbnvm"
Sep 16 18:43:59.844694 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:43:59.843063    2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/55360bdc-81dd-44da-ad34-12ff0bc70dc8-sys\") pod \"tuned-bbnvm\" (UID: \"55360bdc-81dd-44da-ad34-12ff0bc70dc8\") " pod="openshift-cluster-node-tuning-operator/tuned-bbnvm"
Sep 16 18:43:59.844694 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:43:59.843087    2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/55360bdc-81dd-44da-ad34-12ff0bc70dc8-run\") pod \"tuned-bbnvm\" (UID: \"55360bdc-81dd-44da-ad34-12ff0bc70dc8\") " pod="openshift-cluster-node-tuning-operator/tuned-bbnvm"
Sep 16 18:43:59.844694 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:43:59.843110    2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/7cefdcfe-d218-4ac0-a3d8-811df8c043c2-host-run-netns\") pod \"ovnkube-node-nqcpx\" (UID: \"7cefdcfe-d218-4ac0-a3d8-811df8c043c2\") " pod="openshift-ovn-kubernetes/ovnkube-node-nqcpx"
Sep 16 18:43:59.844694 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:43:59.843134    2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/7cefdcfe-d218-4ac0-a3d8-811df8c043c2-run-ovn\") pod \"ovnkube-node-nqcpx\" (UID: \"7cefdcfe-d218-4ac0-a3d8-811df8c043c2\") " pod="openshift-ovn-kubernetes/ovnkube-node-nqcpx"
Sep 16 18:43:59.844694 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:43:59.843159    2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/7cefdcfe-d218-4ac0-a3d8-811df8c043c2-node-log\") pod \"ovnkube-node-nqcpx\" (UID: \"7cefdcfe-d218-4ac0-a3d8-811df8c043c2\") " pod="openshift-ovn-kubernetes/ovnkube-node-nqcpx"
Sep 16 18:43:59.844694 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:43:59.843185    2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/2bbf45e9-df8e-4a67-b777-0c629428fca2-kubelet-config\") pod \"global-pull-secret-syncer-pg4kf\" (UID: \"2bbf45e9-df8e-4a67-b777-0c629428fca2\") " pod="kube-system/global-pull-secret-syncer-pg4kf"
Sep 16 18:43:59.844694 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:43:59.843229    2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jfzj7\" (UniqueName: \"kubernetes.io/projected/524324ca-6ba2-4885-9ed6-570fa75665e7-kube-api-access-jfzj7\") pod \"node-resolver-sxbgl\" (UID: \"524324ca-6ba2-4885-9ed6-570fa75665e7\") " pod="openshift-dns/node-resolver-sxbgl"
Sep 16 18:43:59.844694 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:43:59.843247    2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/55360bdc-81dd-44da-ad34-12ff0bc70dc8-etc-sysctl-conf\") pod \"tuned-bbnvm\" (UID: \"55360bdc-81dd-44da-ad34-12ff0bc70dc8\") " pod="openshift-cluster-node-tuning-operator/tuned-bbnvm"
Sep 16 18:43:59.844694 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:43:59.843254    2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/7cefdcfe-d218-4ac0-a3d8-811df8c043c2-etc-openvswitch\") pod \"ovnkube-node-nqcpx\" (UID: \"7cefdcfe-d218-4ac0-a3d8-811df8c043c2\") " pod="openshift-ovn-kubernetes/ovnkube-node-nqcpx"
Sep 16 18:43:59.844694 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:43:59.843291    2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/55360bdc-81dd-44da-ad34-12ff0bc70dc8-etc-sysctl-d\") pod \"tuned-bbnvm\" (UID: \"55360bdc-81dd-44da-ad34-12ff0bc70dc8\") " pod="openshift-cluster-node-tuning-operator/tuned-bbnvm"
Sep 16 18:43:59.844694 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:43:59.843295    2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/273fb2b8-0579-4f86-abc5-2801f6f47bcd-host\") pod \"node-ca-2hdh4\" (UID: \"273fb2b8-0579-4f86-abc5-2801f6f47bcd\") " pod="openshift-image-registry/node-ca-2hdh4"
Sep 16 18:43:59.844694 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:43:59.843341    2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/7cefdcfe-d218-4ac0-a3d8-811df8c043c2-run-ovn\") pod \"ovnkube-node-nqcpx\" (UID: \"7cefdcfe-d218-4ac0-a3d8-811df8c043c2\") " pod="openshift-ovn-kubernetes/ovnkube-node-nqcpx"
Sep 16 18:43:59.844694 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:43:59.843350    2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/2bbf45e9-df8e-4a67-b777-0c629428fca2-kubelet-config\") pod \"global-pull-secret-syncer-pg4kf\" (UID: \"2bbf45e9-df8e-4a67-b777-0c629428fca2\") " pod="kube-system/global-pull-secret-syncer-pg4kf"
Sep 16 18:43:59.844694 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:43:59.843402    2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/7cefdcfe-d218-4ac0-a3d8-811df8c043c2-node-log\") pod \"ovnkube-node-nqcpx\" (UID: \"7cefdcfe-d218-4ac0-a3d8-811df8c043c2\") " pod="openshift-ovn-kubernetes/ovnkube-node-nqcpx"
Sep 16 18:43:59.844694 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:43:59.843405    2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/7cefdcfe-d218-4ac0-a3d8-811df8c043c2-etc-openvswitch\") pod \"ovnkube-node-nqcpx\" (UID: \"7cefdcfe-d218-4ac0-a3d8-811df8c043c2\") " pod="openshift-ovn-kubernetes/ovnkube-node-nqcpx"
Sep 16 18:43:59.844694 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:43:59.843670    2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/7cefdcfe-d218-4ac0-a3d8-811df8c043c2-ovnkube-script-lib\") pod \"ovnkube-node-nqcpx\" (UID: \"7cefdcfe-d218-4ac0-a3d8-811df8c043c2\") " pod="openshift-ovn-kubernetes/ovnkube-node-nqcpx"
Sep 16 18:43:59.844694 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:43:59.843918    2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/7cefdcfe-d218-4ac0-a3d8-811df8c043c2-ovnkube-config\") pod \"ovnkube-node-nqcpx\" (UID: \"7cefdcfe-d218-4ac0-a3d8-811df8c043c2\") " pod="openshift-ovn-kubernetes/ovnkube-node-nqcpx"
Sep 16 18:43:59.844694 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:43:59.843960    2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bqrfl\" (UniqueName: \"kubernetes.io/projected/55360bdc-81dd-44da-ad34-12ff0bc70dc8-kube-api-access-bqrfl\") pod \"tuned-bbnvm\" (UID: \"55360bdc-81dd-44da-ad34-12ff0bc70dc8\") " pod="openshift-cluster-node-tuning-operator/tuned-bbnvm"
Sep 16 18:43:59.845393 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:43:59.843986    2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/7cefdcfe-d218-4ac0-a3d8-811df8c043c2-host-run-netns\") pod \"ovnkube-node-nqcpx\" (UID: \"7cefdcfe-d218-4ac0-a3d8-811df8c043c2\") " pod="openshift-ovn-kubernetes/ovnkube-node-nqcpx"
Sep 16 18:43:59.845393 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:43:59.844007    2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/7cefdcfe-d218-4ac0-a3d8-811df8c043c2-var-lib-openvswitch\") pod \"ovnkube-node-nqcpx\" (UID: \"7cefdcfe-d218-4ac0-a3d8-811df8c043c2\") " pod="openshift-ovn-kubernetes/ovnkube-node-nqcpx"
Sep 16 18:43:59.845393 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:43:59.844040    2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/55360bdc-81dd-44da-ad34-12ff0bc70dc8-sys\") pod \"tuned-bbnvm\" (UID: \"55360bdc-81dd-44da-ad34-12ff0bc70dc8\") " pod="openshift-cluster-node-tuning-operator/tuned-bbnvm"
Sep 16 18:43:59.845393 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:43:59.844042    2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/7cefdcfe-d218-4ac0-a3d8-811df8c043c2-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-nqcpx\" (UID: \"7cefdcfe-d218-4ac0-a3d8-811df8c043c2\") " pod="openshift-ovn-kubernetes/ovnkube-node-nqcpx"
Sep 16 18:43:59.845393 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:43:59.844075    2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/7cefdcfe-d218-4ac0-a3d8-811df8c043c2-env-overrides\") pod \"ovnkube-node-nqcpx\" (UID: \"7cefdcfe-d218-4ac0-a3d8-811df8c043c2\") " pod="openshift-ovn-kubernetes/ovnkube-node-nqcpx"
Sep 16 18:43:59.845393 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:43:59.844088    2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/55360bdc-81dd-44da-ad34-12ff0bc70dc8-run\") pod \"tuned-bbnvm\" (UID: \"55360bdc-81dd-44da-ad34-12ff0bc70dc8\") " pod="openshift-cluster-node-tuning-operator/tuned-bbnvm"
Sep 16 18:43:59.845393 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:43:59.844150    2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/7cefdcfe-d218-4ac0-a3d8-811df8c043c2-var-lib-openvswitch\") pod \"ovnkube-node-nqcpx\" (UID: \"7cefdcfe-d218-4ac0-a3d8-811df8c043c2\") " pod="openshift-ovn-kubernetes/ovnkube-node-nqcpx"
Sep 16 18:43:59.845393 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:43:59.844188    2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/7cefdcfe-d218-4ac0-a3d8-811df8c043c2-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-nqcpx\" (UID: \"7cefdcfe-d218-4ac0-a3d8-811df8c043c2\") " pod="openshift-ovn-kubernetes/ovnkube-node-nqcpx"
Sep 16 18:43:59.845393 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:43:59.844193    2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/55360bdc-81dd-44da-ad34-12ff0bc70dc8-etc-sysctl-d\") pod \"tuned-bbnvm\" (UID: \"55360bdc-81dd-44da-ad34-12ff0bc70dc8\") " pod="openshift-cluster-node-tuning-operator/tuned-bbnvm"
Sep 16 18:43:59.845810 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:43:59.845538    2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/55360bdc-81dd-44da-ad34-12ff0bc70dc8-etc-tuned\") pod \"tuned-bbnvm\" (UID: \"55360bdc-81dd-44da-ad34-12ff0bc70dc8\") " pod="openshift-cluster-node-tuning-operator/tuned-bbnvm"
Sep 16 18:43:59.845810 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:43:59.845663    2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/273fb2b8-0579-4f86-abc5-2801f6f47bcd-serviceca\") pod \"node-ca-2hdh4\" (UID: \"273fb2b8-0579-4f86-abc5-2801f6f47bcd\") " pod="openshift-image-registry/node-ca-2hdh4"
Sep 16 18:43:59.846068 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:43:59.846048    2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/7cefdcfe-d218-4ac0-a3d8-811df8c043c2-ovn-node-metrics-cert\") pod \"ovnkube-node-nqcpx\" (UID: \"7cefdcfe-d218-4ac0-a3d8-811df8c043c2\") " pod="openshift-ovn-kubernetes/ovnkube-node-nqcpx"
Sep 16 18:43:59.846195 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:43:59.846175    2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/55360bdc-81dd-44da-ad34-12ff0bc70dc8-etc-systemd\") pod \"tuned-bbnvm\" (UID: \"55360bdc-81dd-44da-ad34-12ff0bc70dc8\") " pod="openshift-cluster-node-tuning-operator/tuned-bbnvm"
Sep 16 18:43:59.846264 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:43:59.846249    2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/55360bdc-81dd-44da-ad34-12ff0bc70dc8-tmp\") pod \"tuned-bbnvm\" (UID: \"55360bdc-81dd-44da-ad34-12ff0bc70dc8\") " pod="openshift-cluster-node-tuning-operator/tuned-bbnvm"
Sep 16 18:43:59.846450 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:43:59.846429    2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/7cefdcfe-d218-4ac0-a3d8-811df8c043c2-env-overrides\") pod \"ovnkube-node-nqcpx\" (UID: \"7cefdcfe-d218-4ac0-a3d8-811df8c043c2\") " pod="openshift-ovn-kubernetes/ovnkube-node-nqcpx"
Sep 16 18:43:59.851371 ip-10-0-132-80 kubenswrapper[2544]: E0916 18:43:59.851339    2544 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 16 18:43:59.851371 ip-10-0-132-80 kubenswrapper[2544]: E0916 18:43:59.851365    2544 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 16 18:43:59.851533 ip-10-0-132-80 kubenswrapper[2544]: E0916 18:43:59.851379    2544 projected.go:196] Error preparing data for projected volume kube-api-access-s9f9h for pod openshift-network-diagnostics/network-check-target-t8hx2: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 16 18:43:59.851533 ip-10-0-132-80 kubenswrapper[2544]: E0916 18:43:59.851448    2544 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/3c4f2334-3ac6-47e0-9eb7-4352c53df29a-kube-api-access-s9f9h podName:3c4f2334-3ac6-47e0-9eb7-4352c53df29a nodeName:}" failed. No retries permitted until 2026-09-16 18:44:00.3514291 +0000 UTC m=+3.140431617 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-s9f9h" (UniqueName: "kubernetes.io/projected/3c4f2334-3ac6-47e0-9eb7-4352c53df29a-kube-api-access-s9f9h") pod "network-check-target-t8hx2" (UID: "3c4f2334-3ac6-47e0-9eb7-4352c53df29a") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 16 18:43:59.853064 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:43:59.853045    2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-frsll\" (UniqueName: \"kubernetes.io/projected/7cefdcfe-d218-4ac0-a3d8-811df8c043c2-kube-api-access-frsll\") pod \"ovnkube-node-nqcpx\" (UID: \"7cefdcfe-d218-4ac0-a3d8-811df8c043c2\") " pod="openshift-ovn-kubernetes/ovnkube-node-nqcpx"
Sep 16 18:43:59.853750 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:43:59.853702    2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bqrfl\" (UniqueName: \"kubernetes.io/projected/55360bdc-81dd-44da-ad34-12ff0bc70dc8-kube-api-access-bqrfl\") pod \"tuned-bbnvm\" (UID: \"55360bdc-81dd-44da-ad34-12ff0bc70dc8\") " pod="openshift-cluster-node-tuning-operator/tuned-bbnvm"
Sep 16 18:43:59.854615 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:43:59.854594    2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jfzj7\" (UniqueName: \"kubernetes.io/projected/524324ca-6ba2-4885-9ed6-570fa75665e7-kube-api-access-jfzj7\") pod \"node-resolver-sxbgl\" (UID: \"524324ca-6ba2-4885-9ed6-570fa75665e7\") " pod="openshift-dns/node-resolver-sxbgl"
Sep 16 18:43:59.855417 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:43:59.855401    2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rddb6\" (UniqueName: \"kubernetes.io/projected/273fb2b8-0579-4f86-abc5-2801f6f47bcd-kube-api-access-rddb6\") pod \"node-ca-2hdh4\" (UID: \"273fb2b8-0579-4f86-abc5-2801f6f47bcd\") " pod="openshift-image-registry/node-ca-2hdh4"
Sep 16 18:43:59.928719 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:43:59.928670    2544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-pzpb9"
Sep 16 18:43:59.938648 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:43:59.938623    2544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-znf6p"
Sep 16 18:43:59.949418 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:43:59.949392    2544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-z4wt6"
Sep 16 18:43:59.954337 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:43:59.954319    2544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zv6wl"
Sep 16 18:43:59.961125 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:43:59.961102    2544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-dbwbz"
Sep 16 18:43:59.966812 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:43:59.966790    2544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-bbnvm"
Sep 16 18:43:59.973114 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:43:59.973099    2544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-nqcpx"
Sep 16 18:43:59.979080 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:43:59.979063    2544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-sxbgl"
Sep 16 18:43:59.983721 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:43:59.983703    2544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-2hdh4"
Sep 16 18:44:00.246716 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:44:00.246640    2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/db1e3b32-114c-4bd9-b28c-21a2a9d6efaa-metrics-certs\") pod \"network-metrics-daemon-pdfpq\" (UID: \"db1e3b32-114c-4bd9-b28c-21a2a9d6efaa\") " pod="openshift-multus/network-metrics-daemon-pdfpq"
Sep 16 18:44:00.246846 ip-10-0-132-80 kubenswrapper[2544]: E0916 18:44:00.246796    2544 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 16 18:44:00.246893 ip-10-0-132-80 kubenswrapper[2544]: E0916 18:44:00.246863    2544 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/db1e3b32-114c-4bd9-b28c-21a2a9d6efaa-metrics-certs podName:db1e3b32-114c-4bd9-b28c-21a2a9d6efaa nodeName:}" failed. No retries permitted until 2026-09-16 18:44:01.246848514 +0000 UTC m=+4.035851030 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/db1e3b32-114c-4bd9-b28c-21a2a9d6efaa-metrics-certs") pod "network-metrics-daemon-pdfpq" (UID: "db1e3b32-114c-4bd9-b28c-21a2a9d6efaa") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 16 18:44:00.347524 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:44:00.347487    2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/2bbf45e9-df8e-4a67-b777-0c629428fca2-original-pull-secret\") pod \"global-pull-secret-syncer-pg4kf\" (UID: \"2bbf45e9-df8e-4a67-b777-0c629428fca2\") " pod="kube-system/global-pull-secret-syncer-pg4kf"
Sep 16 18:44:00.347711 ip-10-0-132-80 kubenswrapper[2544]: E0916 18:44:00.347629    2544 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 16 18:44:00.347711 ip-10-0-132-80 kubenswrapper[2544]: E0916 18:44:00.347691    2544 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2bbf45e9-df8e-4a67-b777-0c629428fca2-original-pull-secret podName:2bbf45e9-df8e-4a67-b777-0c629428fca2 nodeName:}" failed. No retries permitted until 2026-09-16 18:44:01.347677173 +0000 UTC m=+4.136679702 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/2bbf45e9-df8e-4a67-b777-0c629428fca2-original-pull-secret") pod "global-pull-secret-syncer-pg4kf" (UID: "2bbf45e9-df8e-4a67-b777-0c629428fca2") : object "kube-system"/"original-pull-secret" not registered
Sep 16 18:44:00.448809 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:44:00.448773    2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s9f9h\" (UniqueName: \"kubernetes.io/projected/3c4f2334-3ac6-47e0-9eb7-4352c53df29a-kube-api-access-s9f9h\") pod \"network-check-target-t8hx2\" (UID: \"3c4f2334-3ac6-47e0-9eb7-4352c53df29a\") " pod="openshift-network-diagnostics/network-check-target-t8hx2"
Sep 16 18:44:00.448966 ip-10-0-132-80 kubenswrapper[2544]: E0916 18:44:00.448940    2544 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 16 18:44:00.448966 ip-10-0-132-80 kubenswrapper[2544]: E0916 18:44:00.448964    2544 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 16 18:44:00.449045 ip-10-0-132-80 kubenswrapper[2544]: E0916 18:44:00.448975    2544 projected.go:196] Error preparing data for projected volume kube-api-access-s9f9h for pod openshift-network-diagnostics/network-check-target-t8hx2: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 16 18:44:00.449045 ip-10-0-132-80 kubenswrapper[2544]: E0916 18:44:00.449027    2544 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/3c4f2334-3ac6-47e0-9eb7-4352c53df29a-kube-api-access-s9f9h podName:3c4f2334-3ac6-47e0-9eb7-4352c53df29a nodeName:}" failed. No retries permitted until 2026-09-16 18:44:01.449013682 +0000 UTC m=+4.238016199 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-s9f9h" (UniqueName: "kubernetes.io/projected/3c4f2334-3ac6-47e0-9eb7-4352c53df29a-kube-api-access-s9f9h") pod "network-check-target-t8hx2" (UID: "3c4f2334-3ac6-47e0-9eb7-4352c53df29a") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 16 18:44:00.642099 ip-10-0-132-80 kubenswrapper[2544]: W0916 18:44:00.642075    2544 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbef94c9f_6884_4c22_b616_32f4e38a4772.slice/crio-ed5f05b9168bd346d75c6be2ae9a4e7297a86e42c33ab14c466ae9230561efc8 WatchSource:0}: Error finding container ed5f05b9168bd346d75c6be2ae9a4e7297a86e42c33ab14c466ae9230561efc8: Status 404 returned error can't find the container with id ed5f05b9168bd346d75c6be2ae9a4e7297a86e42c33ab14c466ae9230561efc8
Sep 16 18:44:00.644050 ip-10-0-132-80 kubenswrapper[2544]: W0916 18:44:00.643885    2544 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9b277c2d_db88_424b_9366_c89532c4b4f1.slice/crio-ec11910d4ab627ee3ac0e1fe95137d8d6a9672fdcaae0c18bb2dcf1aab447d90 WatchSource:0}: Error finding container ec11910d4ab627ee3ac0e1fe95137d8d6a9672fdcaae0c18bb2dcf1aab447d90: Status 404 returned error can't find the container with id ec11910d4ab627ee3ac0e1fe95137d8d6a9672fdcaae0c18bb2dcf1aab447d90
Sep 16 18:44:00.647006 ip-10-0-132-80 kubenswrapper[2544]: W0916 18:44:00.646983    2544 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod55360bdc_81dd_44da_ad34_12ff0bc70dc8.slice/crio-86ab6a7f0d55511cad4b2160c135985eb78f9f63d244bec2271b5eb46ba580c3 WatchSource:0}: Error finding container 86ab6a7f0d55511cad4b2160c135985eb78f9f63d244bec2271b5eb46ba580c3: Status 404 returned error can't find the container with id 86ab6a7f0d55511cad4b2160c135985eb78f9f63d244bec2271b5eb46ba580c3
Sep 16 18:44:00.647779 ip-10-0-132-80 kubenswrapper[2544]: W0916 18:44:00.647759    2544 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0494be2a_d0e5_4570_b05c_0c595a0b494a.slice/crio-50232fdd1c23332b674582269edcd71c0e87663e80886fa9af5936b47f20506f WatchSource:0}: Error finding container 50232fdd1c23332b674582269edcd71c0e87663e80886fa9af5936b47f20506f: Status 404 returned error can't find the container with id 50232fdd1c23332b674582269edcd71c0e87663e80886fa9af5936b47f20506f
Sep 16 18:44:00.648610 ip-10-0-132-80 kubenswrapper[2544]: W0916 18:44:00.648577    2544 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7cefdcfe_d218_4ac0_a3d8_811df8c043c2.slice/crio-214f75ea762157aa1894f59be854e587d0e19ad85ec7fd26680a28ee0636fba2 WatchSource:0}: Error finding container 214f75ea762157aa1894f59be854e587d0e19ad85ec7fd26680a28ee0636fba2: Status 404 returned error can't find the container with id 214f75ea762157aa1894f59be854e587d0e19ad85ec7fd26680a28ee0636fba2
Sep 16 18:44:00.649600 ip-10-0-132-80 kubenswrapper[2544]: W0916 18:44:00.649432    2544 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod524324ca_6ba2_4885_9ed6_570fa75665e7.slice/crio-35ed32ba1d0e5dab9898e8ded0912aca982fba9a20c12acd873b83544f21ebe7 WatchSource:0}: Error finding container 35ed32ba1d0e5dab9898e8ded0912aca982fba9a20c12acd873b83544f21ebe7: Status 404 returned error can't find the container with id 35ed32ba1d0e5dab9898e8ded0912aca982fba9a20c12acd873b83544f21ebe7
Sep 16 18:44:00.650310 ip-10-0-132-80 kubenswrapper[2544]: W0916 18:44:00.650226    2544 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod22e551bf_0ed9_4e8f_9487_8412901fcdcb.slice/crio-b7ded8ca25f51950048c581533e43d61561c0ddcbefe55b6136fbd3cee97e69e WatchSource:0}: Error finding container b7ded8ca25f51950048c581533e43d61561c0ddcbefe55b6136fbd3cee97e69e: Status 404 returned error can't find the container with id b7ded8ca25f51950048c581533e43d61561c0ddcbefe55b6136fbd3cee97e69e
Sep 16 18:44:00.652958 ip-10-0-132-80 kubenswrapper[2544]: W0916 18:44:00.652688    2544 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod38dd6a59_da17_47cd_b10f_540f2fff9a89.slice/crio-3e1a2aa4fabb9b06d284c956b698e90c8dd3644ce7af999f76438a8d75e8bc39 WatchSource:0}: Error finding container 3e1a2aa4fabb9b06d284c956b698e90c8dd3644ce7af999f76438a8d75e8bc39: Status 404 returned error can't find the container with id 3e1a2aa4fabb9b06d284c956b698e90c8dd3644ce7af999f76438a8d75e8bc39
Sep 16 18:44:00.652958 ip-10-0-132-80 kubenswrapper[2544]: W0916 18:44:00.652889    2544 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod273fb2b8_0579_4f86_abc5_2801f6f47bcd.slice/crio-65a156f86b17801d0636768ae839cc40d8af3d6693736d8d0bbf02ad38cb4004 WatchSource:0}: Error finding container 65a156f86b17801d0636768ae839cc40d8af3d6693736d8d0bbf02ad38cb4004: Status 404 returned error can't find the container with id 65a156f86b17801d0636768ae839cc40d8af3d6693736d8d0bbf02ad38cb4004
Sep 16 18:44:00.672539 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:44:00.672501    2544 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-09-15 18:38:59 +0000 UTC" deadline="2028-02-14 23:44:00.496375476 +0000 UTC"
Sep 16 18:44:00.672611 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:44:00.672540    2544 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="12388h59m59.823838224s"
Sep 16 18:44:00.716331 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:44:00.716306    2544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-t8hx2"
Sep 16 18:44:00.716429 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:44:00.716346    2544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-pg4kf"
Sep 16 18:44:00.716472 ip-10-0-132-80 kubenswrapper[2544]: E0916 18:44:00.716433    2544 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-t8hx2" podUID="3c4f2334-3ac6-47e0-9eb7-4352c53df29a"
Sep 16 18:44:00.716624 ip-10-0-132-80 kubenswrapper[2544]: E0916 18:44:00.716597    2544 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-pg4kf" podUID="2bbf45e9-df8e-4a67-b777-0c629428fca2"
Sep 16 18:44:00.728889 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:44:00.728799    2544 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-pzpb9" event={"ID":"0494be2a-d0e5-4570-b05c-0c595a0b494a","Type":"ContainerStarted","Data":"50232fdd1c23332b674582269edcd71c0e87663e80886fa9af5936b47f20506f"}
Sep 16 18:44:00.731164 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:44:00.731130    2544 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-2hdh4" event={"ID":"273fb2b8-0579-4f86-abc5-2801f6f47bcd","Type":"ContainerStarted","Data":"65a156f86b17801d0636768ae839cc40d8af3d6693736d8d0bbf02ad38cb4004"}
Sep 16 18:44:00.733493 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:44:00.733472    2544 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-znf6p" event={"ID":"38dd6a59-da17-47cd-b10f-540f2fff9a89","Type":"ContainerStarted","Data":"3e1a2aa4fabb9b06d284c956b698e90c8dd3644ce7af999f76438a8d75e8bc39"}
Sep 16 18:44:00.734604 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:44:00.734584    2544 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-z4wt6" event={"ID":"22e551bf-0ed9-4e8f-9487-8412901fcdcb","Type":"ContainerStarted","Data":"b7ded8ca25f51950048c581533e43d61561c0ddcbefe55b6136fbd3cee97e69e"}
Sep 16 18:44:00.735658 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:44:00.735636    2544 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-sxbgl" event={"ID":"524324ca-6ba2-4885-9ed6-570fa75665e7","Type":"ContainerStarted","Data":"35ed32ba1d0e5dab9898e8ded0912aca982fba9a20c12acd873b83544f21ebe7"}
Sep 16 18:44:00.737908 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:44:00.737889    2544 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-nqcpx" event={"ID":"7cefdcfe-d218-4ac0-a3d8-811df8c043c2","Type":"ContainerStarted","Data":"214f75ea762157aa1894f59be854e587d0e19ad85ec7fd26680a28ee0636fba2"}
Sep 16 18:44:00.743656 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:44:00.743635    2544 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-bbnvm" event={"ID":"55360bdc-81dd-44da-ad34-12ff0bc70dc8","Type":"ContainerStarted","Data":"86ab6a7f0d55511cad4b2160c135985eb78f9f63d244bec2271b5eb46ba580c3"}
Sep 16 18:44:00.746965 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:44:00.746940    2544 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zv6wl" event={"ID":"9b277c2d-db88-424b-9366-c89532c4b4f1","Type":"ContainerStarted","Data":"ec11910d4ab627ee3ac0e1fe95137d8d6a9672fdcaae0c18bb2dcf1aab447d90"}
Sep 16 18:44:00.748320 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:44:00.748293    2544 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-dbwbz" event={"ID":"bef94c9f-6884-4c22-b616-32f4e38a4772","Type":"ContainerStarted","Data":"ed5f05b9168bd346d75c6be2ae9a4e7297a86e42c33ab14c466ae9230561efc8"}
Sep 16 18:44:01.254689 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:44:01.254365    2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/db1e3b32-114c-4bd9-b28c-21a2a9d6efaa-metrics-certs\") pod \"network-metrics-daemon-pdfpq\" (UID: \"db1e3b32-114c-4bd9-b28c-21a2a9d6efaa\") " pod="openshift-multus/network-metrics-daemon-pdfpq"
Sep 16 18:44:01.254689 ip-10-0-132-80 kubenswrapper[2544]: E0916 18:44:01.254618    2544 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 16 18:44:01.254689 ip-10-0-132-80 kubenswrapper[2544]: E0916 18:44:01.254692    2544 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/db1e3b32-114c-4bd9-b28c-21a2a9d6efaa-metrics-certs podName:db1e3b32-114c-4bd9-b28c-21a2a9d6efaa nodeName:}" failed. No retries permitted until 2026-09-16 18:44:03.254672107 +0000 UTC m=+6.043674635 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/db1e3b32-114c-4bd9-b28c-21a2a9d6efaa-metrics-certs") pod "network-metrics-daemon-pdfpq" (UID: "db1e3b32-114c-4bd9-b28c-21a2a9d6efaa") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 16 18:44:01.356392 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:44:01.356356    2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/2bbf45e9-df8e-4a67-b777-0c629428fca2-original-pull-secret\") pod \"global-pull-secret-syncer-pg4kf\" (UID: \"2bbf45e9-df8e-4a67-b777-0c629428fca2\") " pod="kube-system/global-pull-secret-syncer-pg4kf"
Sep 16 18:44:01.356603 ip-10-0-132-80 kubenswrapper[2544]: E0916 18:44:01.356500    2544 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 16 18:44:01.356603 ip-10-0-132-80 kubenswrapper[2544]: E0916 18:44:01.356597    2544 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2bbf45e9-df8e-4a67-b777-0c629428fca2-original-pull-secret podName:2bbf45e9-df8e-4a67-b777-0c629428fca2 nodeName:}" failed. No retries permitted until 2026-09-16 18:44:03.356544579 +0000 UTC m=+6.145547094 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/2bbf45e9-df8e-4a67-b777-0c629428fca2-original-pull-secret") pod "global-pull-secret-syncer-pg4kf" (UID: "2bbf45e9-df8e-4a67-b777-0c629428fca2") : object "kube-system"/"original-pull-secret" not registered
Sep 16 18:44:01.457793 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:44:01.457755    2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s9f9h\" (UniqueName: \"kubernetes.io/projected/3c4f2334-3ac6-47e0-9eb7-4352c53df29a-kube-api-access-s9f9h\") pod \"network-check-target-t8hx2\" (UID: \"3c4f2334-3ac6-47e0-9eb7-4352c53df29a\") " pod="openshift-network-diagnostics/network-check-target-t8hx2"
Sep 16 18:44:01.457965 ip-10-0-132-80 kubenswrapper[2544]: E0916 18:44:01.457914    2544 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 16 18:44:01.457965 ip-10-0-132-80 kubenswrapper[2544]: E0916 18:44:01.457936    2544 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 16 18:44:01.457965 ip-10-0-132-80 kubenswrapper[2544]: E0916 18:44:01.457948    2544 projected.go:196] Error preparing data for projected volume kube-api-access-s9f9h for pod openshift-network-diagnostics/network-check-target-t8hx2: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 16 18:44:01.458119 ip-10-0-132-80 kubenswrapper[2544]: E0916 18:44:01.458008    2544 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/3c4f2334-3ac6-47e0-9eb7-4352c53df29a-kube-api-access-s9f9h podName:3c4f2334-3ac6-47e0-9eb7-4352c53df29a nodeName:}" failed. No retries permitted until 2026-09-16 18:44:03.457989762 +0000 UTC m=+6.246992275 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-s9f9h" (UniqueName: "kubernetes.io/projected/3c4f2334-3ac6-47e0-9eb7-4352c53df29a-kube-api-access-s9f9h") pod "network-check-target-t8hx2" (UID: "3c4f2334-3ac6-47e0-9eb7-4352c53df29a") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 16 18:44:01.673055 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:44:01.672967    2544 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-09-15 18:38:59 +0000 UTC" deadline="2028-05-28 20:19:41.26646575 +0000 UTC"
Sep 16 18:44:01.673055 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:44:01.673008    2544 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14881h35m39.593461989s"
Sep 16 18:44:01.716996 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:44:01.716966    2544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-pdfpq"
Sep 16 18:44:01.717166 ip-10-0-132-80 kubenswrapper[2544]: E0916 18:44:01.717112    2544 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-pdfpq" podUID="db1e3b32-114c-4bd9-b28c-21a2a9d6efaa"
Sep 16 18:44:01.768053 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:44:01.767162    2544 generic.go:356] "Generic (PLEG): container finished" podID="752ecfd4a66e13af35021c760a217e52" containerID="83b3c0c36ba088c224e72dcb084a3b2aa2d524f073365b2df0110574e5aa3922" exitCode=0
Sep 16 18:44:01.768053 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:44:01.767253    2544 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-80.ec2.internal" event={"ID":"752ecfd4a66e13af35021c760a217e52","Type":"ContainerDied","Data":"83b3c0c36ba088c224e72dcb084a3b2aa2d524f073365b2df0110574e5aa3922"}
Sep 16 18:44:01.779738 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:44:01.779706    2544 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-132-80.ec2.internal" event={"ID":"3b41cfe1537c43ad1e53b07ff984e93b","Type":"ContainerStarted","Data":"4090be27c690fa5e8c05525ff026de94336bef6b73256158503ae65acdae764f"}
Sep 16 18:44:01.810596 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:44:01.809282    2544 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-132-80.ec2.internal" podStartSLOduration=2.8092605600000002 podStartE2EDuration="2.80926056s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-16 18:43:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:44:01.808833004 +0000 UTC m=+4.597835539" watchObservedRunningTime="2026-09-16 18:44:01.80926056 +0000 UTC m=+4.598263095"
Sep 16 18:44:02.716715 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:44:02.716679    2544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-pg4kf"
Sep 16 18:44:02.717224 ip-10-0-132-80 kubenswrapper[2544]: E0916 18:44:02.716818    2544 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-pg4kf" podUID="2bbf45e9-df8e-4a67-b777-0c629428fca2"
Sep 16 18:44:02.717224 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:44:02.717193    2544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-t8hx2"
Sep 16 18:44:02.717340 ip-10-0-132-80 kubenswrapper[2544]: E0916 18:44:02.717279    2544 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-t8hx2" podUID="3c4f2334-3ac6-47e0-9eb7-4352c53df29a"
Sep 16 18:44:02.804764 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:44:02.804623    2544 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-80.ec2.internal" event={"ID":"752ecfd4a66e13af35021c760a217e52","Type":"ContainerStarted","Data":"f975f015fe24bc448082efec88944638906fe5f283e42fcec82af2162984ac5b"}
Sep 16 18:44:03.273220 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:44:03.273139    2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/db1e3b32-114c-4bd9-b28c-21a2a9d6efaa-metrics-certs\") pod \"network-metrics-daemon-pdfpq\" (UID: \"db1e3b32-114c-4bd9-b28c-21a2a9d6efaa\") " pod="openshift-multus/network-metrics-daemon-pdfpq"
Sep 16 18:44:03.273479 ip-10-0-132-80 kubenswrapper[2544]: E0916 18:44:03.273301    2544 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 16 18:44:03.273479 ip-10-0-132-80 kubenswrapper[2544]: E0916 18:44:03.273360    2544 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/db1e3b32-114c-4bd9-b28c-21a2a9d6efaa-metrics-certs podName:db1e3b32-114c-4bd9-b28c-21a2a9d6efaa nodeName:}" failed. No retries permitted until 2026-09-16 18:44:07.273341883 +0000 UTC m=+10.062344409 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/db1e3b32-114c-4bd9-b28c-21a2a9d6efaa-metrics-certs") pod "network-metrics-daemon-pdfpq" (UID: "db1e3b32-114c-4bd9-b28c-21a2a9d6efaa") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 16 18:44:03.374418 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:44:03.374385    2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/2bbf45e9-df8e-4a67-b777-0c629428fca2-original-pull-secret\") pod \"global-pull-secret-syncer-pg4kf\" (UID: \"2bbf45e9-df8e-4a67-b777-0c629428fca2\") " pod="kube-system/global-pull-secret-syncer-pg4kf"
Sep 16 18:44:03.374607 ip-10-0-132-80 kubenswrapper[2544]: E0916 18:44:03.374540    2544 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 16 18:44:03.374689 ip-10-0-132-80 kubenswrapper[2544]: E0916 18:44:03.374612    2544 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2bbf45e9-df8e-4a67-b777-0c629428fca2-original-pull-secret podName:2bbf45e9-df8e-4a67-b777-0c629428fca2 nodeName:}" failed. No retries permitted until 2026-09-16 18:44:07.37459484 +0000 UTC m=+10.163597357 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/2bbf45e9-df8e-4a67-b777-0c629428fca2-original-pull-secret") pod "global-pull-secret-syncer-pg4kf" (UID: "2bbf45e9-df8e-4a67-b777-0c629428fca2") : object "kube-system"/"original-pull-secret" not registered
Sep 16 18:44:03.475544 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:44:03.475443    2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s9f9h\" (UniqueName: \"kubernetes.io/projected/3c4f2334-3ac6-47e0-9eb7-4352c53df29a-kube-api-access-s9f9h\") pod \"network-check-target-t8hx2\" (UID: \"3c4f2334-3ac6-47e0-9eb7-4352c53df29a\") " pod="openshift-network-diagnostics/network-check-target-t8hx2"
Sep 16 18:44:03.475734 ip-10-0-132-80 kubenswrapper[2544]: E0916 18:44:03.475650    2544 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 16 18:44:03.475734 ip-10-0-132-80 kubenswrapper[2544]: E0916 18:44:03.475679    2544 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 16 18:44:03.475734 ip-10-0-132-80 kubenswrapper[2544]: E0916 18:44:03.475693    2544 projected.go:196] Error preparing data for projected volume kube-api-access-s9f9h for pod openshift-network-diagnostics/network-check-target-t8hx2: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 16 18:44:03.475883 ip-10-0-132-80 kubenswrapper[2544]: E0916 18:44:03.475755    2544 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/3c4f2334-3ac6-47e0-9eb7-4352c53df29a-kube-api-access-s9f9h podName:3c4f2334-3ac6-47e0-9eb7-4352c53df29a nodeName:}" failed. No retries permitted until 2026-09-16 18:44:07.475734637 +0000 UTC m=+10.264737173 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-s9f9h" (UniqueName: "kubernetes.io/projected/3c4f2334-3ac6-47e0-9eb7-4352c53df29a-kube-api-access-s9f9h") pod "network-check-target-t8hx2" (UID: "3c4f2334-3ac6-47e0-9eb7-4352c53df29a") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 16 18:44:03.716477 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:44:03.716400    2544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-pdfpq"
Sep 16 18:44:03.716644 ip-10-0-132-80 kubenswrapper[2544]: E0916 18:44:03.716540    2544 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-pdfpq" podUID="db1e3b32-114c-4bd9-b28c-21a2a9d6efaa"
Sep 16 18:44:04.716658 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:44:04.716619    2544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-t8hx2"
Sep 16 18:44:04.717108 ip-10-0-132-80 kubenswrapper[2544]: E0916 18:44:04.716753    2544 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-t8hx2" podUID="3c4f2334-3ac6-47e0-9eb7-4352c53df29a"
Sep 16 18:44:04.717180 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:44:04.717123    2544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-pg4kf"
Sep 16 18:44:04.717241 ip-10-0-132-80 kubenswrapper[2544]: E0916 18:44:04.717217    2544 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-pg4kf" podUID="2bbf45e9-df8e-4a67-b777-0c629428fca2"
Sep 16 18:44:05.717142 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:44:05.717108    2544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-pdfpq"
Sep 16 18:44:05.717698 ip-10-0-132-80 kubenswrapper[2544]: E0916 18:44:05.717244    2544 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-pdfpq" podUID="db1e3b32-114c-4bd9-b28c-21a2a9d6efaa"
Sep 16 18:44:06.716751 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:44:06.716209    2544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-t8hx2"
Sep 16 18:44:06.716751 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:44:06.716209    2544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-pg4kf"
Sep 16 18:44:06.716751 ip-10-0-132-80 kubenswrapper[2544]: E0916 18:44:06.716348    2544 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-t8hx2" podUID="3c4f2334-3ac6-47e0-9eb7-4352c53df29a"
Sep 16 18:44:06.716751 ip-10-0-132-80 kubenswrapper[2544]: E0916 18:44:06.716434    2544 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-pg4kf" podUID="2bbf45e9-df8e-4a67-b777-0c629428fca2"
Sep 16 18:44:07.308203 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:44:07.307568    2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/db1e3b32-114c-4bd9-b28c-21a2a9d6efaa-metrics-certs\") pod \"network-metrics-daemon-pdfpq\" (UID: \"db1e3b32-114c-4bd9-b28c-21a2a9d6efaa\") " pod="openshift-multus/network-metrics-daemon-pdfpq"
Sep 16 18:44:07.308203 ip-10-0-132-80 kubenswrapper[2544]: E0916 18:44:07.307807    2544 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 16 18:44:07.308203 ip-10-0-132-80 kubenswrapper[2544]: E0916 18:44:07.307875    2544 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/db1e3b32-114c-4bd9-b28c-21a2a9d6efaa-metrics-certs podName:db1e3b32-114c-4bd9-b28c-21a2a9d6efaa nodeName:}" failed. No retries permitted until 2026-09-16 18:44:15.307855207 +0000 UTC m=+18.096857721 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/db1e3b32-114c-4bd9-b28c-21a2a9d6efaa-metrics-certs") pod "network-metrics-daemon-pdfpq" (UID: "db1e3b32-114c-4bd9-b28c-21a2a9d6efaa") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 16 18:44:07.408284 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:44:07.408217    2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/2bbf45e9-df8e-4a67-b777-0c629428fca2-original-pull-secret\") pod \"global-pull-secret-syncer-pg4kf\" (UID: \"2bbf45e9-df8e-4a67-b777-0c629428fca2\") " pod="kube-system/global-pull-secret-syncer-pg4kf"
Sep 16 18:44:07.408511 ip-10-0-132-80 kubenswrapper[2544]: E0916 18:44:07.408405    2544 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 16 18:44:07.408511 ip-10-0-132-80 kubenswrapper[2544]: E0916 18:44:07.408469    2544 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2bbf45e9-df8e-4a67-b777-0c629428fca2-original-pull-secret podName:2bbf45e9-df8e-4a67-b777-0c629428fca2 nodeName:}" failed. No retries permitted until 2026-09-16 18:44:15.408451295 +0000 UTC m=+18.197453811 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/2bbf45e9-df8e-4a67-b777-0c629428fca2-original-pull-secret") pod "global-pull-secret-syncer-pg4kf" (UID: "2bbf45e9-df8e-4a67-b777-0c629428fca2") : object "kube-system"/"original-pull-secret" not registered
Sep 16 18:44:07.509454 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:44:07.509414    2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s9f9h\" (UniqueName: \"kubernetes.io/projected/3c4f2334-3ac6-47e0-9eb7-4352c53df29a-kube-api-access-s9f9h\") pod \"network-check-target-t8hx2\" (UID: \"3c4f2334-3ac6-47e0-9eb7-4352c53df29a\") " pod="openshift-network-diagnostics/network-check-target-t8hx2"
Sep 16 18:44:07.510108 ip-10-0-132-80 kubenswrapper[2544]: E0916 18:44:07.509636    2544 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 16 18:44:07.510108 ip-10-0-132-80 kubenswrapper[2544]: E0916 18:44:07.509666    2544 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 16 18:44:07.510108 ip-10-0-132-80 kubenswrapper[2544]: E0916 18:44:07.509679    2544 projected.go:196] Error preparing data for projected volume kube-api-access-s9f9h for pod openshift-network-diagnostics/network-check-target-t8hx2: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 16 18:44:07.510108 ip-10-0-132-80 kubenswrapper[2544]: E0916 18:44:07.509743    2544 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/3c4f2334-3ac6-47e0-9eb7-4352c53df29a-kube-api-access-s9f9h podName:3c4f2334-3ac6-47e0-9eb7-4352c53df29a nodeName:}" failed. No retries permitted until 2026-09-16 18:44:15.509722995 +0000 UTC m=+18.298725514 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-s9f9h" (UniqueName: "kubernetes.io/projected/3c4f2334-3ac6-47e0-9eb7-4352c53df29a-kube-api-access-s9f9h") pod "network-check-target-t8hx2" (UID: "3c4f2334-3ac6-47e0-9eb7-4352c53df29a") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 16 18:44:07.717632 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:44:07.717541    2544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-pdfpq"
Sep 16 18:44:07.717787 ip-10-0-132-80 kubenswrapper[2544]: E0916 18:44:07.717667    2544 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-pdfpq" podUID="db1e3b32-114c-4bd9-b28c-21a2a9d6efaa"
Sep 16 18:44:08.716684 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:44:08.716647    2544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-t8hx2"
Sep 16 18:44:08.717128 ip-10-0-132-80 kubenswrapper[2544]: E0916 18:44:08.716757    2544 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-t8hx2" podUID="3c4f2334-3ac6-47e0-9eb7-4352c53df29a"
Sep 16 18:44:08.717128 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:44:08.716971    2544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-pg4kf"
Sep 16 18:44:08.717128 ip-10-0-132-80 kubenswrapper[2544]: E0916 18:44:08.717112    2544 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-pg4kf" podUID="2bbf45e9-df8e-4a67-b777-0c629428fca2"
Sep 16 18:44:09.717005 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:44:09.716964    2544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-pdfpq"
Sep 16 18:44:09.717451 ip-10-0-132-80 kubenswrapper[2544]: E0916 18:44:09.717144    2544 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-pdfpq" podUID="db1e3b32-114c-4bd9-b28c-21a2a9d6efaa"
Sep 16 18:44:10.717171 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:44:10.717127    2544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-pg4kf"
Sep 16 18:44:10.717669 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:44:10.717126    2544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-t8hx2"
Sep 16 18:44:10.717669 ip-10-0-132-80 kubenswrapper[2544]: E0916 18:44:10.717273    2544 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-pg4kf" podUID="2bbf45e9-df8e-4a67-b777-0c629428fca2"
Sep 16 18:44:10.717669 ip-10-0-132-80 kubenswrapper[2544]: E0916 18:44:10.717331    2544 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-t8hx2" podUID="3c4f2334-3ac6-47e0-9eb7-4352c53df29a"
Sep 16 18:44:11.716907 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:44:11.716866    2544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-pdfpq"
Sep 16 18:44:11.717064 ip-10-0-132-80 kubenswrapper[2544]: E0916 18:44:11.716994    2544 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-pdfpq" podUID="db1e3b32-114c-4bd9-b28c-21a2a9d6efaa"
Sep 16 18:44:12.716823 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:44:12.716785    2544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-pg4kf"
Sep 16 18:44:12.717276 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:44:12.716787    2544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-t8hx2"
Sep 16 18:44:12.717276 ip-10-0-132-80 kubenswrapper[2544]: E0916 18:44:12.716909    2544 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-pg4kf" podUID="2bbf45e9-df8e-4a67-b777-0c629428fca2"
Sep 16 18:44:12.717276 ip-10-0-132-80 kubenswrapper[2544]: E0916 18:44:12.717020    2544 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-t8hx2" podUID="3c4f2334-3ac6-47e0-9eb7-4352c53df29a"
Sep 16 18:44:13.716963 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:44:13.716925    2544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-pdfpq"
Sep 16 18:44:13.717344 ip-10-0-132-80 kubenswrapper[2544]: E0916 18:44:13.717073    2544 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-pdfpq" podUID="db1e3b32-114c-4bd9-b28c-21a2a9d6efaa"
Sep 16 18:44:14.717123 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:44:14.717089    2544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-pg4kf"
Sep 16 18:44:14.717580 ip-10-0-132-80 kubenswrapper[2544]: E0916 18:44:14.717205    2544 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-pg4kf" podUID="2bbf45e9-df8e-4a67-b777-0c629428fca2"
Sep 16 18:44:14.717580 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:44:14.717262    2544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-t8hx2"
Sep 16 18:44:14.717580 ip-10-0-132-80 kubenswrapper[2544]: E0916 18:44:14.717377    2544 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-t8hx2" podUID="3c4f2334-3ac6-47e0-9eb7-4352c53df29a"
Sep 16 18:44:15.369815 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:44:15.369760    2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/db1e3b32-114c-4bd9-b28c-21a2a9d6efaa-metrics-certs\") pod \"network-metrics-daemon-pdfpq\" (UID: \"db1e3b32-114c-4bd9-b28c-21a2a9d6efaa\") " pod="openshift-multus/network-metrics-daemon-pdfpq"
Sep 16 18:44:15.370018 ip-10-0-132-80 kubenswrapper[2544]: E0916 18:44:15.369914    2544 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 16 18:44:15.370018 ip-10-0-132-80 kubenswrapper[2544]: E0916 18:44:15.369992    2544 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/db1e3b32-114c-4bd9-b28c-21a2a9d6efaa-metrics-certs podName:db1e3b32-114c-4bd9-b28c-21a2a9d6efaa nodeName:}" failed. No retries permitted until 2026-09-16 18:44:31.369971685 +0000 UTC m=+34.158974201 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/db1e3b32-114c-4bd9-b28c-21a2a9d6efaa-metrics-certs") pod "network-metrics-daemon-pdfpq" (UID: "db1e3b32-114c-4bd9-b28c-21a2a9d6efaa") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 16 18:44:15.470357 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:44:15.470316    2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/2bbf45e9-df8e-4a67-b777-0c629428fca2-original-pull-secret\") pod \"global-pull-secret-syncer-pg4kf\" (UID: \"2bbf45e9-df8e-4a67-b777-0c629428fca2\") " pod="kube-system/global-pull-secret-syncer-pg4kf"
Sep 16 18:44:15.470511 ip-10-0-132-80 kubenswrapper[2544]: E0916 18:44:15.470456    2544 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 16 18:44:15.470579 ip-10-0-132-80 kubenswrapper[2544]: E0916 18:44:15.470522    2544 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2bbf45e9-df8e-4a67-b777-0c629428fca2-original-pull-secret podName:2bbf45e9-df8e-4a67-b777-0c629428fca2 nodeName:}" failed. No retries permitted until 2026-09-16 18:44:31.470507279 +0000 UTC m=+34.259509795 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/2bbf45e9-df8e-4a67-b777-0c629428fca2-original-pull-secret") pod "global-pull-secret-syncer-pg4kf" (UID: "2bbf45e9-df8e-4a67-b777-0c629428fca2") : object "kube-system"/"original-pull-secret" not registered
Sep 16 18:44:15.570848 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:44:15.570808    2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s9f9h\" (UniqueName: \"kubernetes.io/projected/3c4f2334-3ac6-47e0-9eb7-4352c53df29a-kube-api-access-s9f9h\") pod \"network-check-target-t8hx2\" (UID: \"3c4f2334-3ac6-47e0-9eb7-4352c53df29a\") " pod="openshift-network-diagnostics/network-check-target-t8hx2"
Sep 16 18:44:15.571010 ip-10-0-132-80 kubenswrapper[2544]: E0916 18:44:15.570986    2544 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 16 18:44:15.571065 ip-10-0-132-80 kubenswrapper[2544]: E0916 18:44:15.571012    2544 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 16 18:44:15.571065 ip-10-0-132-80 kubenswrapper[2544]: E0916 18:44:15.571025    2544 projected.go:196] Error preparing data for projected volume kube-api-access-s9f9h for pod openshift-network-diagnostics/network-check-target-t8hx2: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 16 18:44:15.571180 ip-10-0-132-80 kubenswrapper[2544]: E0916 18:44:15.571082    2544 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/3c4f2334-3ac6-47e0-9eb7-4352c53df29a-kube-api-access-s9f9h podName:3c4f2334-3ac6-47e0-9eb7-4352c53df29a nodeName:}" failed. No retries permitted until 2026-09-16 18:44:31.571068156 +0000 UTC m=+34.360070673 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-s9f9h" (UniqueName: "kubernetes.io/projected/3c4f2334-3ac6-47e0-9eb7-4352c53df29a-kube-api-access-s9f9h") pod "network-check-target-t8hx2" (UID: "3c4f2334-3ac6-47e0-9eb7-4352c53df29a") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 16 18:44:15.716869 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:44:15.716793    2544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-pdfpq"
Sep 16 18:44:15.717021 ip-10-0-132-80 kubenswrapper[2544]: E0916 18:44:15.716949    2544 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-pdfpq" podUID="db1e3b32-114c-4bd9-b28c-21a2a9d6efaa"
Sep 16 18:44:16.717004 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:44:16.716970    2544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-pg4kf"
Sep 16 18:44:16.717462 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:44:16.716981    2544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-t8hx2"
Sep 16 18:44:16.717462 ip-10-0-132-80 kubenswrapper[2544]: E0916 18:44:16.717102    2544 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-pg4kf" podUID="2bbf45e9-df8e-4a67-b777-0c629428fca2"
Sep 16 18:44:16.717462 ip-10-0-132-80 kubenswrapper[2544]: E0916 18:44:16.717219    2544 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-t8hx2" podUID="3c4f2334-3ac6-47e0-9eb7-4352c53df29a"
Sep 16 18:44:17.717413 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:44:17.717381    2544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-pdfpq"
Sep 16 18:44:17.717892 ip-10-0-132-80 kubenswrapper[2544]: E0916 18:44:17.717485    2544 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-pdfpq" podUID="db1e3b32-114c-4bd9-b28c-21a2a9d6efaa"
Sep 16 18:44:18.716796 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:44:18.716653    2544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-t8hx2"
Sep 16 18:44:18.716902 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:44:18.716656    2544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-pg4kf"
Sep 16 18:44:18.716902 ip-10-0-132-80 kubenswrapper[2544]: E0916 18:44:18.716881    2544 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-t8hx2" podUID="3c4f2334-3ac6-47e0-9eb7-4352c53df29a"
Sep 16 18:44:18.717010 ip-10-0-132-80 kubenswrapper[2544]: E0916 18:44:18.716953    2544 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-pg4kf" podUID="2bbf45e9-df8e-4a67-b777-0c629428fca2"
Sep 16 18:44:18.837638 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:44:18.837609    2544 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-pzpb9" event={"ID":"0494be2a-d0e5-4570-b05c-0c595a0b494a","Type":"ContainerStarted","Data":"59ada2924a0fe0a20cdcbcfebd09823ef17f2134bcefdaed8942d5ef965afcbd"}
Sep 16 18:44:18.839173 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:44:18.839148    2544 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-2hdh4" event={"ID":"273fb2b8-0579-4f86-abc5-2801f6f47bcd","Type":"ContainerStarted","Data":"bd68a5287c7f42e553ceaa1608dba89560fd55bb5a28a19c9fba268756dcb4c0"}
Sep 16 18:44:18.840563 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:44:18.840526    2544 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-znf6p" event={"ID":"38dd6a59-da17-47cd-b10f-540f2fff9a89","Type":"ContainerStarted","Data":"36a3669d1851d73cda961be45973d029fba77a1ba2b6035afcb2ebf28ce65b4a"}
Sep 16 18:44:18.841981 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:44:18.841959    2544 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-sxbgl" event={"ID":"524324ca-6ba2-4885-9ed6-570fa75665e7","Type":"ContainerStarted","Data":"e85a5673b32aaf7489fb93d0ebc2df4d2d1e6c3c2782bde3a9c45434d5dbd244"}
Sep 16 18:44:18.843260 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:44:18.843243    2544 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-bbnvm" event={"ID":"55360bdc-81dd-44da-ad34-12ff0bc70dc8","Type":"ContainerStarted","Data":"e985c639e5c3f1e4794800217c662f046c71d87e0d5e4f47c3b8401e7831a1c0"}
Sep 16 18:44:18.844746 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:44:18.844727    2544 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zv6wl" event={"ID":"9b277c2d-db88-424b-9366-c89532c4b4f1","Type":"ContainerStarted","Data":"3926106bcf83ca5c1da39d46291aa73af62bcbed788e7228bc2f39febef8cddf"}
Sep 16 18:44:18.846046 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:44:18.846025    2544 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-dbwbz" event={"ID":"bef94c9f-6884-4c22-b616-32f4e38a4772","Type":"ContainerStarted","Data":"820bf939e91255a1478a4b3fc360d9b0016bd38f326eca218186a2d0ace8e629"}
Sep 16 18:44:18.852652 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:44:18.852545    2544 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-80.ec2.internal" podStartSLOduration=19.852533397 podStartE2EDuration="19.852533397s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-16 18:43:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:44:02.822376262 +0000 UTC m=+5.611378796" watchObservedRunningTime="2026-09-16 18:44:18.852533397 +0000 UTC m=+21.641535931"
Sep 16 18:44:18.852933 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:44:18.852905    2544 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-pzpb9" podStartSLOduration=7.937825713 podStartE2EDuration="21.852894947s" totalImagesPullingTime="13.915069234s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:43:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:44:18.852186857 +0000 UTC m=+21.641189410" watchObservedRunningTime="2026-09-16 18:44:18.852894947 +0000 UTC m=+21.641897480"
Sep 16 18:44:18.898393 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:44:18.898330    2544 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-sxbgl" podStartSLOduration=3.043629377 podStartE2EDuration="20.898316526s" totalImagesPullingTime="17.854687149s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-16 18:43:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:44:18.883721282 +0000 UTC m=+21.672723849" watchObservedRunningTime="2026-09-16 18:44:18.898316526 +0000 UTC m=+21.687319060"
Sep 16 18:44:18.898485 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:44:18.898411    2544 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-2hdh4" podStartSLOduration=3.078071525 podStartE2EDuration="20.898406452s" totalImagesPullingTime="17.820334927s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-16 18:43:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:44:18.898280476 +0000 UTC m=+21.687283010" watchObservedRunningTime="2026-09-16 18:44:18.898406452 +0000 UTC m=+21.687408985"
Sep 16 18:44:18.915382 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:44:18.915329    2544 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-dbwbz" podStartSLOduration=3.011156279 podStartE2EDuration="20.915308573s" totalImagesPullingTime="17.904152294s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-16 18:43:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:44:18.91503733 +0000 UTC m=+21.704039883" watchObservedRunningTime="2026-09-16 18:44:18.915308573 +0000 UTC m=+21.704311109"
Sep 16 18:44:18.934850 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:44:18.934802    2544 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-bbnvm" podStartSLOduration=3.078599493 podStartE2EDuration="20.934783401s" totalImagesPullingTime="17.856183908s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-16 18:43:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:44:18.934664638 +0000 UTC m=+21.723667198" watchObservedRunningTime="2026-09-16 18:44:18.934783401 +0000 UTC m=+21.723785941"
Sep 16 18:44:19.716751 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:44:19.716569    2544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-pdfpq"
Sep 16 18:44:19.716905 ip-10-0-132-80 kubenswrapper[2544]: E0916 18:44:19.716880    2544 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-pdfpq" podUID="db1e3b32-114c-4bd9-b28c-21a2a9d6efaa"
Sep 16 18:44:19.847119 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:44:19.847092    2544 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Sep 16 18:44:19.851285 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:44:19.851259    2544 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-nqcpx" event={"ID":"7cefdcfe-d218-4ac0-a3d8-811df8c043c2","Type":"ContainerStarted","Data":"0c51983294c63f8cb42f8b05f7360a6c1e78a356e7d0d1f360606825cd2ad6f9"}
Sep 16 18:44:19.851387 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:44:19.851345    2544 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-nqcpx" event={"ID":"7cefdcfe-d218-4ac0-a3d8-811df8c043c2","Type":"ContainerStarted","Data":"6994130b95e4394d8ed8f87edb079c7a97bab9a7dc46cbc37d1a5414f37a3fea"}
Sep 16 18:44:19.851450 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:44:19.851396    2544 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-nqcpx" event={"ID":"7cefdcfe-d218-4ac0-a3d8-811df8c043c2","Type":"ContainerStarted","Data":"e03bd10fb247d21898d3b5e797151ca5ec8ca010199a983375bc82fb2f05768a"}
Sep 16 18:44:19.851507 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:44:19.851461    2544 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-nqcpx" event={"ID":"7cefdcfe-d218-4ac0-a3d8-811df8c043c2","Type":"ContainerStarted","Data":"07405ff0e8a8f3db4348562fc624b0390ffdd2cf23e1102953a36eef6287f09d"}
Sep 16 18:44:19.851574 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:44:19.851511    2544 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-nqcpx" event={"ID":"7cefdcfe-d218-4ac0-a3d8-811df8c043c2","Type":"ContainerStarted","Data":"2cce46cb373b7ea726a3c720161db17c2b147c9a62f9fa07b1fa4d8d20f990a8"}
Sep 16 18:44:19.851640 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:44:19.851577    2544 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-nqcpx" event={"ID":"7cefdcfe-d218-4ac0-a3d8-811df8c043c2","Type":"ContainerStarted","Data":"a6c365752efd22676441f91579e80b6f70ec9ddf97c349ea0fe735ad810584cc"}
Sep 16 18:44:19.852943 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:44:19.852918    2544 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zv6wl" event={"ID":"9b277c2d-db88-424b-9366-c89532c4b4f1","Type":"ContainerStarted","Data":"1f8860c4857994ca388ff55b76957a75ff007c8ee2b6917b4dffedc60d7c8519"}
Sep 16 18:44:19.854181 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:44:19.854158    2544 generic.go:356] "Generic (PLEG): container finished" podID="38dd6a59-da17-47cd-b10f-540f2fff9a89" containerID="36a3669d1851d73cda961be45973d029fba77a1ba2b6035afcb2ebf28ce65b4a" exitCode=0
Sep 16 18:44:19.854258 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:44:19.854233    2544 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-znf6p" event={"ID":"38dd6a59-da17-47cd-b10f-540f2fff9a89","Type":"ContainerDied","Data":"36a3669d1851d73cda961be45973d029fba77a1ba2b6035afcb2ebf28ce65b4a"}
Sep 16 18:44:19.855674 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:44:19.855651    2544 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-z4wt6" event={"ID":"22e551bf-0ed9-4e8f-9487-8412901fcdcb","Type":"ContainerStarted","Data":"357fd1e80ad6add31560940f0e6edb8cd535db5f88e4533ee83ab03079b27106"}
Sep 16 18:44:19.891020 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:44:19.890984    2544 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-z4wt6" podStartSLOduration=5.069573039 podStartE2EDuration="22.89097186s" totalImagesPullingTime="17.821398821s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-16 18:43:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:44:19.890648845 +0000 UTC m=+22.679651390" watchObservedRunningTime="2026-09-16 18:44:19.89097186 +0000 UTC m=+22.679974393"
Sep 16 18:44:20.693996 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:44:20.693898    2544 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-09-16T18:44:19.847116524Z","UUID":"52442847-c87c-4b03-80c9-4982ecc8aa66","Handler":null,"Name":"","Endpoint":""}
Sep 16 18:44:20.695961 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:44:20.695802    2544 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Sep 16 18:44:20.695961 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:44:20.695836    2544 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Sep 16 18:44:20.717252 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:44:20.717139    2544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-pg4kf"
Sep 16 18:44:20.717372 ip-10-0-132-80 kubenswrapper[2544]: E0916 18:44:20.717254    2544 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-pg4kf" podUID="2bbf45e9-df8e-4a67-b777-0c629428fca2"
Sep 16 18:44:20.717372 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:44:20.717315    2544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-t8hx2"
Sep 16 18:44:20.717484 ip-10-0-132-80 kubenswrapper[2544]: E0916 18:44:20.717423    2544 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-t8hx2" podUID="3c4f2334-3ac6-47e0-9eb7-4352c53df29a"
Sep 16 18:44:20.733442 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:44:20.733378    2544 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-pzpb9"
Sep 16 18:44:20.734103 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:44:20.733976    2544 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-pzpb9"
Sep 16 18:44:20.860616 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:44:20.860582    2544 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zv6wl" event={"ID":"9b277c2d-db88-424b-9366-c89532c4b4f1","Type":"ContainerStarted","Data":"2bd16650e2a9b5cee8ff2188cd85a1a0b1750fd5715e0d82b3eeba887915b77c"}
Sep 16 18:44:20.861080 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:44:20.861037    2544 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-pzpb9"
Sep 16 18:44:20.861299 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:44:20.861281    2544 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-pzpb9"
Sep 16 18:44:20.881212 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:44:20.881165    2544 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zv6wl" podStartSLOduration=3.18356753 podStartE2EDuration="22.881148162s" totalImagesPullingTime="19.697580632s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-16 18:43:58 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:44:20.880598056 +0000 UTC m=+23.669600608" watchObservedRunningTime="2026-09-16 18:44:20.881148162 +0000 UTC m=+23.670150697"
Sep 16 18:44:21.716452 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:44:21.716427    2544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-pdfpq"
Sep 16 18:44:21.716599 ip-10-0-132-80 kubenswrapper[2544]: E0916 18:44:21.716579    2544 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-pdfpq" podUID="db1e3b32-114c-4bd9-b28c-21a2a9d6efaa"
Sep 16 18:44:21.866394 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:44:21.866349    2544 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-nqcpx" event={"ID":"7cefdcfe-d218-4ac0-a3d8-811df8c043c2","Type":"ContainerStarted","Data":"9794548f969302d8021ec073f45e2db4dc81eb0415f4b32b8a46fc1fdf31dffd"}
Sep 16 18:44:22.716941 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:44:22.716780    2544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-t8hx2"
Sep 16 18:44:22.717115 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:44:22.716801    2544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-pg4kf"
Sep 16 18:44:22.717115 ip-10-0-132-80 kubenswrapper[2544]: E0916 18:44:22.717033    2544 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-t8hx2" podUID="3c4f2334-3ac6-47e0-9eb7-4352c53df29a"
Sep 16 18:44:22.717115 ip-10-0-132-80 kubenswrapper[2544]: E0916 18:44:22.717089    2544 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-pg4kf" podUID="2bbf45e9-df8e-4a67-b777-0c629428fca2"
Sep 16 18:44:23.716970 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:44:23.716928    2544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-pdfpq"
Sep 16 18:44:23.717407 ip-10-0-132-80 kubenswrapper[2544]: E0916 18:44:23.717072    2544 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-pdfpq" podUID="db1e3b32-114c-4bd9-b28c-21a2a9d6efaa"
Sep 16 18:44:23.874380 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:44:23.874339    2544 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-znf6p" event={"ID":"38dd6a59-da17-47cd-b10f-540f2fff9a89","Type":"ContainerStarted","Data":"cf40897ac4e694cc05c662eaebba49d20eafc0d02b537dcae3ebbf09ea629255"}
Sep 16 18:44:23.879233 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:44:23.879184    2544 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-nqcpx" event={"ID":"7cefdcfe-d218-4ac0-a3d8-811df8c043c2","Type":"ContainerStarted","Data":"c08230a00f940a0f40aba65c4e9f6552b1c5ed09a1fcf66549f17a4cff6d02ee"}
Sep 16 18:44:23.879747 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:44:23.879428    2544 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-nqcpx"
Sep 16 18:44:23.879747 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:44:23.879498    2544 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-nqcpx"
Sep 16 18:44:23.879747 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:44:23.879530    2544 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-nqcpx"
Sep 16 18:44:23.898986 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:44:23.898684    2544 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-nqcpx"
Sep 16 18:44:23.899936 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:44:23.899225    2544 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-nqcpx"
Sep 16 18:44:23.929302 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:44:23.929261    2544 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-nqcpx" podStartSLOduration=7.688392582 podStartE2EDuration="25.929248317s" totalImagesPullingTime="18.240855735s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-16 18:43:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:44:23.929164888 +0000 UTC m=+26.718167449" watchObservedRunningTime="2026-09-16 18:44:23.929248317 +0000 UTC m=+26.718250887"
Sep 16 18:44:24.716396 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:44:24.716362    2544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-t8hx2"
Sep 16 18:44:24.716547 ip-10-0-132-80 kubenswrapper[2544]: E0916 18:44:24.716476    2544 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-t8hx2" podUID="3c4f2334-3ac6-47e0-9eb7-4352c53df29a"
Sep 16 18:44:24.716547 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:44:24.716531    2544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-pg4kf"
Sep 16 18:44:24.716666 ip-10-0-132-80 kubenswrapper[2544]: E0916 18:44:24.716634    2544 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-pg4kf" podUID="2bbf45e9-df8e-4a67-b777-0c629428fca2"
Sep 16 18:44:24.882989 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:44:24.882960    2544 generic.go:356] "Generic (PLEG): container finished" podID="38dd6a59-da17-47cd-b10f-540f2fff9a89" containerID="cf40897ac4e694cc05c662eaebba49d20eafc0d02b537dcae3ebbf09ea629255" exitCode=0
Sep 16 18:44:24.883747 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:44:24.883052    2544 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-znf6p" event={"ID":"38dd6a59-da17-47cd-b10f-540f2fff9a89","Type":"ContainerDied","Data":"cf40897ac4e694cc05c662eaebba49d20eafc0d02b537dcae3ebbf09ea629255"}
Sep 16 18:44:25.694336 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:44:25.694102    2544 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-pg4kf"]
Sep 16 18:44:25.694505 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:44:25.694414    2544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-pg4kf"
Sep 16 18:44:25.694548 ip-10-0-132-80 kubenswrapper[2544]: E0916 18:44:25.694504    2544 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-pg4kf" podUID="2bbf45e9-df8e-4a67-b777-0c629428fca2"
Sep 16 18:44:25.696948 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:44:25.696924    2544 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-t8hx2"]
Sep 16 18:44:25.697065 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:44:25.697016    2544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-t8hx2"
Sep 16 18:44:25.697128 ip-10-0-132-80 kubenswrapper[2544]: E0916 18:44:25.697108    2544 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-t8hx2" podUID="3c4f2334-3ac6-47e0-9eb7-4352c53df29a"
Sep 16 18:44:25.700942 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:44:25.700919    2544 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-pdfpq"]
Sep 16 18:44:25.701036 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:44:25.701018    2544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-pdfpq"
Sep 16 18:44:25.701139 ip-10-0-132-80 kubenswrapper[2544]: E0916 18:44:25.701121    2544 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-pdfpq" podUID="db1e3b32-114c-4bd9-b28c-21a2a9d6efaa"
Sep 16 18:44:25.887449 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:44:25.887415    2544 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-znf6p" event={"ID":"38dd6a59-da17-47cd-b10f-540f2fff9a89","Type":"ContainerStarted","Data":"528ca83dba6a93dea89e3f44328d9a1b1e35e1a350f9bb752600b92ddafac289"}
Sep 16 18:44:26.892013 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:44:26.891978    2544 generic.go:356] "Generic (PLEG): container finished" podID="38dd6a59-da17-47cd-b10f-540f2fff9a89" containerID="528ca83dba6a93dea89e3f44328d9a1b1e35e1a350f9bb752600b92ddafac289" exitCode=0
Sep 16 18:44:26.892374 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:44:26.892045    2544 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-znf6p" event={"ID":"38dd6a59-da17-47cd-b10f-540f2fff9a89","Type":"ContainerDied","Data":"528ca83dba6a93dea89e3f44328d9a1b1e35e1a350f9bb752600b92ddafac289"}
Sep 16 18:44:27.716968 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:44:27.716937    2544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-pg4kf"
Sep 16 18:44:27.717093 ip-10-0-132-80 kubenswrapper[2544]: E0916 18:44:27.717017    2544 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-pg4kf" podUID="2bbf45e9-df8e-4a67-b777-0c629428fca2"
Sep 16 18:44:27.717159 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:44:27.717120    2544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-pdfpq"
Sep 16 18:44:27.717281 ip-10-0-132-80 kubenswrapper[2544]: E0916 18:44:27.717257    2544 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-pdfpq" podUID="db1e3b32-114c-4bd9-b28c-21a2a9d6efaa"
Sep 16 18:44:27.717326 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:44:27.717290    2544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-t8hx2"
Sep 16 18:44:27.717364 ip-10-0-132-80 kubenswrapper[2544]: E0916 18:44:27.717347    2544 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-t8hx2" podUID="3c4f2334-3ac6-47e0-9eb7-4352c53df29a"
Sep 16 18:44:27.897080 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:44:27.897050    2544 generic.go:356] "Generic (PLEG): container finished" podID="38dd6a59-da17-47cd-b10f-540f2fff9a89" containerID="18c0bfff103e612865e09794f55f31a745173876f7bf0d2c8667a50f0bbdf123" exitCode=0
Sep 16 18:44:27.897501 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:44:27.897112    2544 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-znf6p" event={"ID":"38dd6a59-da17-47cd-b10f-540f2fff9a89","Type":"ContainerDied","Data":"18c0bfff103e612865e09794f55f31a745173876f7bf0d2c8667a50f0bbdf123"}
Sep 16 18:44:29.719904 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:44:29.719870    2544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-pg4kf"
Sep 16 18:44:29.720353 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:44:29.719870    2544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-pdfpq"
Sep 16 18:44:29.720353 ip-10-0-132-80 kubenswrapper[2544]: E0916 18:44:29.719969    2544 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-pg4kf" podUID="2bbf45e9-df8e-4a67-b777-0c629428fca2"
Sep 16 18:44:29.720353 ip-10-0-132-80 kubenswrapper[2544]: E0916 18:44:29.720068    2544 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-pdfpq" podUID="db1e3b32-114c-4bd9-b28c-21a2a9d6efaa"
Sep 16 18:44:29.720353 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:44:29.719871    2544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-t8hx2"
Sep 16 18:44:29.720353 ip-10-0-132-80 kubenswrapper[2544]: E0916 18:44:29.720146    2544 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-t8hx2" podUID="3c4f2334-3ac6-47e0-9eb7-4352c53df29a"
Sep 16 18:44:31.382900 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:44:31.382669    2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/db1e3b32-114c-4bd9-b28c-21a2a9d6efaa-metrics-certs\") pod \"network-metrics-daemon-pdfpq\" (UID: \"db1e3b32-114c-4bd9-b28c-21a2a9d6efaa\") " pod="openshift-multus/network-metrics-daemon-pdfpq"
Sep 16 18:44:31.383392 ip-10-0-132-80 kubenswrapper[2544]: E0916 18:44:31.382818    2544 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 16 18:44:31.383392 ip-10-0-132-80 kubenswrapper[2544]: E0916 18:44:31.383042    2544 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/db1e3b32-114c-4bd9-b28c-21a2a9d6efaa-metrics-certs podName:db1e3b32-114c-4bd9-b28c-21a2a9d6efaa nodeName:}" failed. No retries permitted until 2026-09-16 18:45:03.383018826 +0000 UTC m=+66.172021388 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/db1e3b32-114c-4bd9-b28c-21a2a9d6efaa-metrics-certs") pod "network-metrics-daemon-pdfpq" (UID: "db1e3b32-114c-4bd9-b28c-21a2a9d6efaa") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 16 18:44:31.483619 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:44:31.483506    2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/2bbf45e9-df8e-4a67-b777-0c629428fca2-original-pull-secret\") pod \"global-pull-secret-syncer-pg4kf\" (UID: \"2bbf45e9-df8e-4a67-b777-0c629428fca2\") " pod="kube-system/global-pull-secret-syncer-pg4kf"
Sep 16 18:44:31.483782 ip-10-0-132-80 kubenswrapper[2544]: E0916 18:44:31.483662    2544 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 16 18:44:31.483782 ip-10-0-132-80 kubenswrapper[2544]: E0916 18:44:31.483727    2544 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2bbf45e9-df8e-4a67-b777-0c629428fca2-original-pull-secret podName:2bbf45e9-df8e-4a67-b777-0c629428fca2 nodeName:}" failed. No retries permitted until 2026-09-16 18:45:03.483713403 +0000 UTC m=+66.272715919 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/2bbf45e9-df8e-4a67-b777-0c629428fca2-original-pull-secret") pod "global-pull-secret-syncer-pg4kf" (UID: "2bbf45e9-df8e-4a67-b777-0c629428fca2") : object "kube-system"/"original-pull-secret" not registered
Sep 16 18:44:31.499420 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:44:31.499392    2544 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-80.ec2.internal" event="NodeReady"
Sep 16 18:44:31.499582 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:44:31.499546    2544 kubelet_node_status.go:489] "Fast updating node status as it just became ready"
Sep 16 18:44:31.557340 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:44:31.557307    2544 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-76874c7675-4dhws"]
Sep 16 18:44:31.584537 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:44:31.584496    2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s9f9h\" (UniqueName: \"kubernetes.io/projected/3c4f2334-3ac6-47e0-9eb7-4352c53df29a-kube-api-access-s9f9h\") pod \"network-check-target-t8hx2\" (UID: \"3c4f2334-3ac6-47e0-9eb7-4352c53df29a\") " pod="openshift-network-diagnostics/network-check-target-t8hx2"
Sep 16 18:44:31.584712 ip-10-0-132-80 kubenswrapper[2544]: E0916 18:44:31.584693    2544 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 16 18:44:31.584764 ip-10-0-132-80 kubenswrapper[2544]: E0916 18:44:31.584719    2544 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 16 18:44:31.584764 ip-10-0-132-80 kubenswrapper[2544]: E0916 18:44:31.584733    2544 projected.go:196] Error preparing data for projected volume kube-api-access-s9f9h for pod openshift-network-diagnostics/network-check-target-t8hx2: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 16 18:44:31.584837 ip-10-0-132-80 kubenswrapper[2544]: E0916 18:44:31.584795    2544 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/3c4f2334-3ac6-47e0-9eb7-4352c53df29a-kube-api-access-s9f9h podName:3c4f2334-3ac6-47e0-9eb7-4352c53df29a nodeName:}" failed. No retries permitted until 2026-09-16 18:45:03.584777575 +0000 UTC m=+66.373780102 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-s9f9h" (UniqueName: "kubernetes.io/projected/3c4f2334-3ac6-47e0-9eb7-4352c53df29a-kube-api-access-s9f9h") pod "network-check-target-t8hx2" (UID: "3c4f2334-3ac6-47e0-9eb7-4352c53df29a") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 16 18:44:31.604219 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:44:31.604195    2544 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-n8djq"]
Sep 16 18:44:31.604375 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:44:31.604319    2544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-76874c7675-4dhws"
Sep 16 18:44:31.607102 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:44:31.607081    2544 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\""
Sep 16 18:44:31.607329 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:44:31.607220    2544 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-rd9rd\""
Sep 16 18:44:31.607982 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:44:31.607864    2544 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\""
Sep 16 18:44:31.608251 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:44:31.608231    2544 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\""
Sep 16 18:44:31.611544 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:44:31.611523    2544 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\""
Sep 16 18:44:31.619196 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:44:31.619176    2544 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-jmnkv"]
Sep 16 18:44:31.619299 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:44:31.619287    2544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-n8djq"
Sep 16 18:44:31.623762 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:44:31.623743    2544 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Sep 16 18:44:31.623864 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:44:31.623804    2544 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Sep 16 18:44:31.624092 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:44:31.624077    2544 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-h7kfw\""
Sep 16 18:44:31.635312 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:44:31.635293    2544 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-76874c7675-4dhws"]
Sep 16 18:44:31.635420 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:44:31.635325    2544 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-jmnkv"]
Sep 16 18:44:31.635420 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:44:31.635342    2544 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-n8djq"]
Sep 16 18:44:31.635532 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:44:31.635445    2544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-jmnkv"
Sep 16 18:44:31.637730 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:44:31.637487    2544 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-wx8jg\""
Sep 16 18:44:31.637730 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:44:31.637588    2544 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Sep 16 18:44:31.637730 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:44:31.637620    2544 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Sep 16 18:44:31.637983 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:44:31.637968    2544 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Sep 16 18:44:31.685383 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:44:31.685359    2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/b03d56d8-b081-41fd-bac9-ba7da38f733a-image-registry-private-configuration\") pod \"image-registry-76874c7675-4dhws\" (UID: \"b03d56d8-b081-41fd-bac9-ba7da38f733a\") " pod="openshift-image-registry/image-registry-76874c7675-4dhws"
Sep 16 18:44:31.685484 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:44:31.685394    2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/b03d56d8-b081-41fd-bac9-ba7da38f733a-bound-sa-token\") pod \"image-registry-76874c7675-4dhws\" (UID: \"b03d56d8-b081-41fd-bac9-ba7da38f733a\") " pod="openshift-image-registry/image-registry-76874c7675-4dhws"
Sep 16 18:44:31.685484 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:44:31.685427    2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/b03d56d8-b081-41fd-bac9-ba7da38f733a-trusted-ca\") pod \"image-registry-76874c7675-4dhws\" (UID: \"b03d56d8-b081-41fd-bac9-ba7da38f733a\") " pod="openshift-image-registry/image-registry-76874c7675-4dhws"
Sep 16 18:44:31.685629 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:44:31.685523    2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/b03d56d8-b081-41fd-bac9-ba7da38f733a-registry-certificates\") pod \"image-registry-76874c7675-4dhws\" (UID: \"b03d56d8-b081-41fd-bac9-ba7da38f733a\") " pod="openshift-image-registry/image-registry-76874c7675-4dhws"
Sep 16 18:44:31.685629 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:44:31.685617    2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/b03d56d8-b081-41fd-bac9-ba7da38f733a-installation-pull-secrets\") pod \"image-registry-76874c7675-4dhws\" (UID: \"b03d56d8-b081-41fd-bac9-ba7da38f733a\") " pod="openshift-image-registry/image-registry-76874c7675-4dhws"
Sep 16 18:44:31.685735 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:44:31.685647    2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zbbk8\" (UniqueName: \"kubernetes.io/projected/b03d56d8-b081-41fd-bac9-ba7da38f733a-kube-api-access-zbbk8\") pod \"image-registry-76874c7675-4dhws\" (UID: \"b03d56d8-b081-41fd-bac9-ba7da38f733a\") " pod="openshift-image-registry/image-registry-76874c7675-4dhws"
Sep 16 18:44:31.685735 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:44:31.685675    2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/b03d56d8-b081-41fd-bac9-ba7da38f733a-registry-tls\") pod \"image-registry-76874c7675-4dhws\" (UID: \"b03d56d8-b081-41fd-bac9-ba7da38f733a\") " pod="openshift-image-registry/image-registry-76874c7675-4dhws"
Sep 16 18:44:31.685822 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:44:31.685771    2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/b03d56d8-b081-41fd-bac9-ba7da38f733a-ca-trust-extracted\") pod \"image-registry-76874c7675-4dhws\" (UID: \"b03d56d8-b081-41fd-bac9-ba7da38f733a\") " pod="openshift-image-registry/image-registry-76874c7675-4dhws"
Sep 16 18:44:31.716544 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:44:31.716518    2544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-pg4kf"
Sep 16 18:44:31.716544 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:44:31.716533    2544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-pdfpq"
Sep 16 18:44:31.716827 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:44:31.716809    2544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-t8hx2"
Sep 16 18:44:31.722354 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:44:31.722332    2544 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Sep 16 18:44:31.722455 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:44:31.722375    2544 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-rk9xq\""
Sep 16 18:44:31.722847 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:44:31.722828    2544 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Sep 16 18:44:31.722933 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:44:31.722853    2544 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-hvlxk\""
Sep 16 18:44:31.722933 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:44:31.722883    2544 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Sep 16 18:44:31.723090 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:44:31.723065    2544 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Sep 16 18:44:31.787902 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:44:31.786719    2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/b291cc26-ff6e-4adb-b219-8bd70a9aa3b2-config-volume\") pod \"dns-default-n8djq\" (UID: \"b291cc26-ff6e-4adb-b219-8bd70a9aa3b2\") " pod="openshift-dns/dns-default-n8djq"
Sep 16 18:44:31.787902 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:44:31.786777    2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/b03d56d8-b081-41fd-bac9-ba7da38f733a-image-registry-private-configuration\") pod \"image-registry-76874c7675-4dhws\" (UID: \"b03d56d8-b081-41fd-bac9-ba7da38f733a\") " pod="openshift-image-registry/image-registry-76874c7675-4dhws"
Sep 16 18:44:31.787902 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:44:31.786816    2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/b03d56d8-b081-41fd-bac9-ba7da38f733a-bound-sa-token\") pod \"image-registry-76874c7675-4dhws\" (UID: \"b03d56d8-b081-41fd-bac9-ba7da38f733a\") " pod="openshift-image-registry/image-registry-76874c7675-4dhws"
Sep 16 18:44:31.787902 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:44:31.786841    2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/ee8d40ee-7c8a-496a-a4e4-56cdc3e0aaa2-cert\") pod \"ingress-canary-jmnkv\" (UID: \"ee8d40ee-7c8a-496a-a4e4-56cdc3e0aaa2\") " pod="openshift-ingress-canary/ingress-canary-jmnkv"
Sep 16 18:44:31.787902 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:44:31.786878    2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/b291cc26-ff6e-4adb-b219-8bd70a9aa3b2-metrics-tls\") pod \"dns-default-n8djq\" (UID: \"b291cc26-ff6e-4adb-b219-8bd70a9aa3b2\") " pod="openshift-dns/dns-default-n8djq"
Sep 16 18:44:31.787902 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:44:31.786912    2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zz7k9\" (UniqueName: \"kubernetes.io/projected/b291cc26-ff6e-4adb-b219-8bd70a9aa3b2-kube-api-access-zz7k9\") pod \"dns-default-n8djq\" (UID: \"b291cc26-ff6e-4adb-b219-8bd70a9aa3b2\") " pod="openshift-dns/dns-default-n8djq"
Sep 16 18:44:31.787902 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:44:31.786948    2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/b03d56d8-b081-41fd-bac9-ba7da38f733a-trusted-ca\") pod \"image-registry-76874c7675-4dhws\" (UID: \"b03d56d8-b081-41fd-bac9-ba7da38f733a\") " pod="openshift-image-registry/image-registry-76874c7675-4dhws"
Sep 16 18:44:31.787902 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:44:31.786987    2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/b03d56d8-b081-41fd-bac9-ba7da38f733a-registry-certificates\") pod \"image-registry-76874c7675-4dhws\" (UID: \"b03d56d8-b081-41fd-bac9-ba7da38f733a\") " pod="openshift-image-registry/image-registry-76874c7675-4dhws"
Sep 16 18:44:31.787902 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:44:31.787011    2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b291cc26-ff6e-4adb-b219-8bd70a9aa3b2-tmp-dir\") pod \"dns-default-n8djq\" (UID: \"b291cc26-ff6e-4adb-b219-8bd70a9aa3b2\") " pod="openshift-dns/dns-default-n8djq"
Sep 16 18:44:31.787902 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:44:31.787099    2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/b03d56d8-b081-41fd-bac9-ba7da38f733a-installation-pull-secrets\") pod \"image-registry-76874c7675-4dhws\" (UID: \"b03d56d8-b081-41fd-bac9-ba7da38f733a\") " pod="openshift-image-registry/image-registry-76874c7675-4dhws"
Sep 16 18:44:31.787902 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:44:31.787129    2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zbbk8\" (UniqueName: \"kubernetes.io/projected/b03d56d8-b081-41fd-bac9-ba7da38f733a-kube-api-access-zbbk8\") pod \"image-registry-76874c7675-4dhws\" (UID: \"b03d56d8-b081-41fd-bac9-ba7da38f733a\") " pod="openshift-image-registry/image-registry-76874c7675-4dhws"
Sep 16 18:44:31.787902 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:44:31.787161    2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/b03d56d8-b081-41fd-bac9-ba7da38f733a-registry-tls\") pod \"image-registry-76874c7675-4dhws\" (UID: \"b03d56d8-b081-41fd-bac9-ba7da38f733a\") " pod="openshift-image-registry/image-registry-76874c7675-4dhws"
Sep 16 18:44:31.787902 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:44:31.787196    2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/b03d56d8-b081-41fd-bac9-ba7da38f733a-ca-trust-extracted\") pod \"image-registry-76874c7675-4dhws\" (UID: \"b03d56d8-b081-41fd-bac9-ba7da38f733a\") " pod="openshift-image-registry/image-registry-76874c7675-4dhws"
Sep 16 18:44:31.787902 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:44:31.787224    2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wmqrk\" (UniqueName: \"kubernetes.io/projected/ee8d40ee-7c8a-496a-a4e4-56cdc3e0aaa2-kube-api-access-wmqrk\") pod \"ingress-canary-jmnkv\" (UID: \"ee8d40ee-7c8a-496a-a4e4-56cdc3e0aaa2\") " pod="openshift-ingress-canary/ingress-canary-jmnkv"
Sep 16 18:44:31.789672 ip-10-0-132-80 kubenswrapper[2544]: E0916 18:44:31.789242    2544 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 16 18:44:31.789672 ip-10-0-132-80 kubenswrapper[2544]: E0916 18:44:31.789263    2544 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-76874c7675-4dhws: secret "image-registry-tls" not found
Sep 16 18:44:31.789672 ip-10-0-132-80 kubenswrapper[2544]: E0916 18:44:31.789330    2544 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/b03d56d8-b081-41fd-bac9-ba7da38f733a-registry-tls podName:b03d56d8-b081-41fd-bac9-ba7da38f733a nodeName:}" failed. No retries permitted until 2026-09-16 18:44:32.289306387 +0000 UTC m=+35.078308903 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/b03d56d8-b081-41fd-bac9-ba7da38f733a-registry-tls") pod "image-registry-76874c7675-4dhws" (UID: "b03d56d8-b081-41fd-bac9-ba7da38f733a") : secret "image-registry-tls" not found
Sep 16 18:44:31.790017 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:44:31.789997    2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/b03d56d8-b081-41fd-bac9-ba7da38f733a-ca-trust-extracted\") pod \"image-registry-76874c7675-4dhws\" (UID: \"b03d56d8-b081-41fd-bac9-ba7da38f733a\") " pod="openshift-image-registry/image-registry-76874c7675-4dhws"
Sep 16 18:44:31.790165 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:44:31.790141    2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/b03d56d8-b081-41fd-bac9-ba7da38f733a-trusted-ca\") pod \"image-registry-76874c7675-4dhws\" (UID: \"b03d56d8-b081-41fd-bac9-ba7da38f733a\") " pod="openshift-image-registry/image-registry-76874c7675-4dhws"
Sep 16 18:44:31.790165 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:44:31.790157    2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/b03d56d8-b081-41fd-bac9-ba7da38f733a-registry-certificates\") pod \"image-registry-76874c7675-4dhws\" (UID: \"b03d56d8-b081-41fd-bac9-ba7da38f733a\") " pod="openshift-image-registry/image-registry-76874c7675-4dhws"
Sep 16 18:44:31.792483 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:44:31.792464    2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/b03d56d8-b081-41fd-bac9-ba7da38f733a-installation-pull-secrets\") pod \"image-registry-76874c7675-4dhws\" (UID: \"b03d56d8-b081-41fd-bac9-ba7da38f733a\") " pod="openshift-image-registry/image-registry-76874c7675-4dhws"
Sep 16 18:44:31.792590 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:44:31.792507    2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/b03d56d8-b081-41fd-bac9-ba7da38f733a-image-registry-private-configuration\") pod \"image-registry-76874c7675-4dhws\" (UID: \"b03d56d8-b081-41fd-bac9-ba7da38f733a\") " pod="openshift-image-registry/image-registry-76874c7675-4dhws"
Sep 16 18:44:31.808106 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:44:31.808084    2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zbbk8\" (UniqueName: \"kubernetes.io/projected/b03d56d8-b081-41fd-bac9-ba7da38f733a-kube-api-access-zbbk8\") pod \"image-registry-76874c7675-4dhws\" (UID: \"b03d56d8-b081-41fd-bac9-ba7da38f733a\") " pod="openshift-image-registry/image-registry-76874c7675-4dhws"
Sep 16 18:44:31.809562 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:44:31.809538    2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/b03d56d8-b081-41fd-bac9-ba7da38f733a-bound-sa-token\") pod \"image-registry-76874c7675-4dhws\" (UID: \"b03d56d8-b081-41fd-bac9-ba7da38f733a\") " pod="openshift-image-registry/image-registry-76874c7675-4dhws"
Sep 16 18:44:31.870920 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:44:31.870886    2544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pzpb9_0494be2a-d0e5-4570-b05c-0c595a0b494a/konnectivity-agent/0.log" limit=1024
Sep 16 18:44:31.887805 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:44:31.887780    2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/b291cc26-ff6e-4adb-b219-8bd70a9aa3b2-config-volume\") pod \"dns-default-n8djq\" (UID: \"b291cc26-ff6e-4adb-b219-8bd70a9aa3b2\") " pod="openshift-dns/dns-default-n8djq"
Sep 16 18:44:31.887900 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:44:31.887810    2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/ee8d40ee-7c8a-496a-a4e4-56cdc3e0aaa2-cert\") pod \"ingress-canary-jmnkv\" (UID: \"ee8d40ee-7c8a-496a-a4e4-56cdc3e0aaa2\") " pod="openshift-ingress-canary/ingress-canary-jmnkv"
Sep 16 18:44:31.887900 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:44:31.887838    2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/b291cc26-ff6e-4adb-b219-8bd70a9aa3b2-metrics-tls\") pod \"dns-default-n8djq\" (UID: \"b291cc26-ff6e-4adb-b219-8bd70a9aa3b2\") " pod="openshift-dns/dns-default-n8djq"
Sep 16 18:44:31.887900 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:44:31.887854    2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zz7k9\" (UniqueName: \"kubernetes.io/projected/b291cc26-ff6e-4adb-b219-8bd70a9aa3b2-kube-api-access-zz7k9\") pod \"dns-default-n8djq\" (UID: \"b291cc26-ff6e-4adb-b219-8bd70a9aa3b2\") " pod="openshift-dns/dns-default-n8djq"
Sep 16 18:44:31.888045 ip-10-0-132-80 kubenswrapper[2544]: E0916 18:44:31.887989    2544 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 16 18:44:31.888045 ip-10-0-132-80 kubenswrapper[2544]: E0916 18:44:31.888037    2544 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b291cc26-ff6e-4adb-b219-8bd70a9aa3b2-metrics-tls podName:b291cc26-ff6e-4adb-b219-8bd70a9aa3b2 nodeName:}" failed. No retries permitted until 2026-09-16 18:44:32.38802359 +0000 UTC m=+35.177026102 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/b291cc26-ff6e-4adb-b219-8bd70a9aa3b2-metrics-tls") pod "dns-default-n8djq" (UID: "b291cc26-ff6e-4adb-b219-8bd70a9aa3b2") : secret "dns-default-metrics-tls" not found
Sep 16 18:44:31.888149 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:44:31.888055    2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b291cc26-ff6e-4adb-b219-8bd70a9aa3b2-tmp-dir\") pod \"dns-default-n8djq\" (UID: \"b291cc26-ff6e-4adb-b219-8bd70a9aa3b2\") " pod="openshift-dns/dns-default-n8djq"
Sep 16 18:44:31.888149 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:44:31.888125    2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wmqrk\" (UniqueName: \"kubernetes.io/projected/ee8d40ee-7c8a-496a-a4e4-56cdc3e0aaa2-kube-api-access-wmqrk\") pod \"ingress-canary-jmnkv\" (UID: \"ee8d40ee-7c8a-496a-a4e4-56cdc3e0aaa2\") " pod="openshift-ingress-canary/ingress-canary-jmnkv"
Sep 16 18:44:31.888335 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:44:31.888316    2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/b291cc26-ff6e-4adb-b219-8bd70a9aa3b2-config-volume\") pod \"dns-default-n8djq\" (UID: \"b291cc26-ff6e-4adb-b219-8bd70a9aa3b2\") " pod="openshift-dns/dns-default-n8djq"
Sep 16 18:44:31.888389 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:44:31.888367    2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b291cc26-ff6e-4adb-b219-8bd70a9aa3b2-tmp-dir\") pod \"dns-default-n8djq\" (UID: \"b291cc26-ff6e-4adb-b219-8bd70a9aa3b2\") " pod="openshift-dns/dns-default-n8djq"
Sep 16 18:44:31.890077 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:44:31.890059    2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/ee8d40ee-7c8a-496a-a4e4-56cdc3e0aaa2-cert\") pod \"ingress-canary-jmnkv\" (UID: \"ee8d40ee-7c8a-496a-a4e4-56cdc3e0aaa2\") " pod="openshift-ingress-canary/ingress-canary-jmnkv"
Sep 16 18:44:31.900147 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:44:31.900129    2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wmqrk\" (UniqueName: \"kubernetes.io/projected/ee8d40ee-7c8a-496a-a4e4-56cdc3e0aaa2-kube-api-access-wmqrk\") pod \"ingress-canary-jmnkv\" (UID: \"ee8d40ee-7c8a-496a-a4e4-56cdc3e0aaa2\") " pod="openshift-ingress-canary/ingress-canary-jmnkv"
Sep 16 18:44:31.900215 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:44:31.900130    2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zz7k9\" (UniqueName: \"kubernetes.io/projected/b291cc26-ff6e-4adb-b219-8bd70a9aa3b2-kube-api-access-zz7k9\") pod \"dns-default-n8djq\" (UID: \"b291cc26-ff6e-4adb-b219-8bd70a9aa3b2\") " pod="openshift-dns/dns-default-n8djq"
Sep 16 18:44:31.944799 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:44:31.944780    2544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-jmnkv"
Sep 16 18:44:32.030568 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:44:32.030491    2544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pzpb9_0494be2a-d0e5-4570-b05c-0c595a0b494a/konnectivity-agent/0.log" limit=1024
Sep 16 18:44:32.241695 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:44:32.241670    2544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pzpb9_0494be2a-d0e5-4570-b05c-0c595a0b494a/konnectivity-agent/0.log" limit=1024
Sep 16 18:44:32.291654 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:44:32.291587    2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/b03d56d8-b081-41fd-bac9-ba7da38f733a-registry-tls\") pod \"image-registry-76874c7675-4dhws\" (UID: \"b03d56d8-b081-41fd-bac9-ba7da38f733a\") " pod="openshift-image-registry/image-registry-76874c7675-4dhws"
Sep 16 18:44:32.291754 ip-10-0-132-80 kubenswrapper[2544]: E0916 18:44:32.291729    2544 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 16 18:44:32.291754 ip-10-0-132-80 kubenswrapper[2544]: E0916 18:44:32.291747    2544 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-76874c7675-4dhws: secret "image-registry-tls" not found
Sep 16 18:44:32.291823 ip-10-0-132-80 kubenswrapper[2544]: E0916 18:44:32.291797    2544 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/b03d56d8-b081-41fd-bac9-ba7da38f733a-registry-tls podName:b03d56d8-b081-41fd-bac9-ba7da38f733a nodeName:}" failed. No retries permitted until 2026-09-16 18:44:33.291781494 +0000 UTC m=+36.080784023 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/b03d56d8-b081-41fd-bac9-ba7da38f733a-registry-tls") pod "image-registry-76874c7675-4dhws" (UID: "b03d56d8-b081-41fd-bac9-ba7da38f733a") : secret "image-registry-tls" not found
Sep 16 18:44:32.352476 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:44:32.352449    2544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pzpb9_0494be2a-d0e5-4570-b05c-0c595a0b494a/konnectivity-agent/0.log" limit=1024
Sep 16 18:44:32.392568 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:44:32.392529    2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/b291cc26-ff6e-4adb-b219-8bd70a9aa3b2-metrics-tls\") pod \"dns-default-n8djq\" (UID: \"b291cc26-ff6e-4adb-b219-8bd70a9aa3b2\") " pod="openshift-dns/dns-default-n8djq"
Sep 16 18:44:32.393189 ip-10-0-132-80 kubenswrapper[2544]: E0916 18:44:32.392678    2544 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 16 18:44:32.393189 ip-10-0-132-80 kubenswrapper[2544]: E0916 18:44:32.392738    2544 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b291cc26-ff6e-4adb-b219-8bd70a9aa3b2-metrics-tls podName:b291cc26-ff6e-4adb-b219-8bd70a9aa3b2 nodeName:}" failed. No retries permitted until 2026-09-16 18:44:33.392721729 +0000 UTC m=+36.181724242 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/b291cc26-ff6e-4adb-b219-8bd70a9aa3b2-metrics-tls") pod "dns-default-n8djq" (UID: "b291cc26-ff6e-4adb-b219-8bd70a9aa3b2") : secret "dns-default-metrics-tls" not found
Sep 16 18:44:32.402174 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:44:32.402150    2544 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-jmnkv"]
Sep 16 18:44:32.422356 ip-10-0-132-80 kubenswrapper[2544]: W0916 18:44:32.422329    2544 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podee8d40ee_7c8a_496a_a4e4_56cdc3e0aaa2.slice/crio-76d9e0afbaffe6fff64dc3bee507caf37aabc59911ece766d279f0bbc4447094 WatchSource:0}: Error finding container 76d9e0afbaffe6fff64dc3bee507caf37aabc59911ece766d279f0bbc4447094: Status 404 returned error can't find the container with id 76d9e0afbaffe6fff64dc3bee507caf37aabc59911ece766d279f0bbc4447094
Sep 16 18:44:32.911260 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:44:32.911095    2544 generic.go:356] "Generic (PLEG): container finished" podID="38dd6a59-da17-47cd-b10f-540f2fff9a89" containerID="f12bde71d09a4ae1b8a687d1600ef96664efe85c170dba9342b2e8639d46673f" exitCode=0
Sep 16 18:44:32.911414 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:44:32.911178    2544 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-znf6p" event={"ID":"38dd6a59-da17-47cd-b10f-540f2fff9a89","Type":"ContainerDied","Data":"f12bde71d09a4ae1b8a687d1600ef96664efe85c170dba9342b2e8639d46673f"}
Sep 16 18:44:32.912438 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:44:32.912412    2544 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-jmnkv" event={"ID":"ee8d40ee-7c8a-496a-a4e4-56cdc3e0aaa2","Type":"ContainerStarted","Data":"76d9e0afbaffe6fff64dc3bee507caf37aabc59911ece766d279f0bbc4447094"}
Sep 16 18:44:33.300082 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:44:33.300007    2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/b03d56d8-b081-41fd-bac9-ba7da38f733a-registry-tls\") pod \"image-registry-76874c7675-4dhws\" (UID: \"b03d56d8-b081-41fd-bac9-ba7da38f733a\") " pod="openshift-image-registry/image-registry-76874c7675-4dhws"
Sep 16 18:44:33.300243 ip-10-0-132-80 kubenswrapper[2544]: E0916 18:44:33.300188    2544 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 16 18:44:33.300243 ip-10-0-132-80 kubenswrapper[2544]: E0916 18:44:33.300212    2544 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-76874c7675-4dhws: secret "image-registry-tls" not found
Sep 16 18:44:33.300320 ip-10-0-132-80 kubenswrapper[2544]: E0916 18:44:33.300282    2544 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/b03d56d8-b081-41fd-bac9-ba7da38f733a-registry-tls podName:b03d56d8-b081-41fd-bac9-ba7da38f733a nodeName:}" failed. No retries permitted until 2026-09-16 18:44:35.300261275 +0000 UTC m=+38.089263793 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/b03d56d8-b081-41fd-bac9-ba7da38f733a-registry-tls") pod "image-registry-76874c7675-4dhws" (UID: "b03d56d8-b081-41fd-bac9-ba7da38f733a") : secret "image-registry-tls" not found
Sep 16 18:44:33.400845 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:44:33.400805    2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/b291cc26-ff6e-4adb-b219-8bd70a9aa3b2-metrics-tls\") pod \"dns-default-n8djq\" (UID: \"b291cc26-ff6e-4adb-b219-8bd70a9aa3b2\") " pod="openshift-dns/dns-default-n8djq"
Sep 16 18:44:33.401277 ip-10-0-132-80 kubenswrapper[2544]: E0916 18:44:33.400955    2544 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 16 18:44:33.401277 ip-10-0-132-80 kubenswrapper[2544]: E0916 18:44:33.401017    2544 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b291cc26-ff6e-4adb-b219-8bd70a9aa3b2-metrics-tls podName:b291cc26-ff6e-4adb-b219-8bd70a9aa3b2 nodeName:}" failed. No retries permitted until 2026-09-16 18:44:35.400998814 +0000 UTC m=+38.190001336 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/b291cc26-ff6e-4adb-b219-8bd70a9aa3b2-metrics-tls") pod "dns-default-n8djq" (UID: "b291cc26-ff6e-4adb-b219-8bd70a9aa3b2") : secret "dns-default-metrics-tls" not found
Sep 16 18:44:33.917361 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:44:33.917331    2544 generic.go:356] "Generic (PLEG): container finished" podID="38dd6a59-da17-47cd-b10f-540f2fff9a89" containerID="7dfe77344f7ba356b84f35b6f298e88880b7f0af8c83c01ecbd7b6116104c3db" exitCode=0
Sep 16 18:44:33.917516 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:44:33.917393    2544 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-znf6p" event={"ID":"38dd6a59-da17-47cd-b10f-540f2fff9a89","Type":"ContainerDied","Data":"7dfe77344f7ba356b84f35b6f298e88880b7f0af8c83c01ecbd7b6116104c3db"}
Sep 16 18:44:34.922668 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:44:34.922633    2544 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-znf6p" event={"ID":"38dd6a59-da17-47cd-b10f-540f2fff9a89","Type":"ContainerStarted","Data":"9e72f0b184d2bdee310dc61311e388721fb809b9dfd5ec5b54c9dc0e968b4aeb"}
Sep 16 18:44:34.942103 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:44:34.942063    2544 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-znf6p" podStartSLOduration=10.394755293 podStartE2EDuration="37.942049791s" totalImagesPullingTime="27.547294498s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-16 18:43:57 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:44:34.941366829 +0000 UTC m=+37.730369384" watchObservedRunningTime="2026-09-16 18:44:34.942049791 +0000 UTC m=+37.731052326"
Sep 16 18:44:35.064099 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:44:35.064073    2544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pzpb9_0494be2a-d0e5-4570-b05c-0c595a0b494a/konnectivity-agent/0.log" limit=1024
Sep 16 18:44:35.166842 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:44:35.166793    2544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pzpb9_0494be2a-d0e5-4570-b05c-0c595a0b494a/konnectivity-agent/0.log" limit=1024
Sep 16 18:44:35.317132 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:44:35.317101    2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/b03d56d8-b081-41fd-bac9-ba7da38f733a-registry-tls\") pod \"image-registry-76874c7675-4dhws\" (UID: \"b03d56d8-b081-41fd-bac9-ba7da38f733a\") " pod="openshift-image-registry/image-registry-76874c7675-4dhws"
Sep 16 18:44:35.317309 ip-10-0-132-80 kubenswrapper[2544]: E0916 18:44:35.317245    2544 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 16 18:44:35.317309 ip-10-0-132-80 kubenswrapper[2544]: E0916 18:44:35.317263    2544 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-76874c7675-4dhws: secret "image-registry-tls" not found
Sep 16 18:44:35.317412 ip-10-0-132-80 kubenswrapper[2544]: E0916 18:44:35.317328    2544 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/b03d56d8-b081-41fd-bac9-ba7da38f733a-registry-tls podName:b03d56d8-b081-41fd-bac9-ba7da38f733a nodeName:}" failed. No retries permitted until 2026-09-16 18:44:39.317309814 +0000 UTC m=+42.106312326 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/b03d56d8-b081-41fd-bac9-ba7da38f733a-registry-tls") pod "image-registry-76874c7675-4dhws" (UID: "b03d56d8-b081-41fd-bac9-ba7da38f733a") : secret "image-registry-tls" not found
Sep 16 18:44:35.417815 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:44:35.417774    2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/b291cc26-ff6e-4adb-b219-8bd70a9aa3b2-metrics-tls\") pod \"dns-default-n8djq\" (UID: \"b291cc26-ff6e-4adb-b219-8bd70a9aa3b2\") " pod="openshift-dns/dns-default-n8djq"
Sep 16 18:44:35.417983 ip-10-0-132-80 kubenswrapper[2544]: E0916 18:44:35.417927    2544 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 16 18:44:35.418026 ip-10-0-132-80 kubenswrapper[2544]: E0916 18:44:35.417995    2544 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b291cc26-ff6e-4adb-b219-8bd70a9aa3b2-metrics-tls podName:b291cc26-ff6e-4adb-b219-8bd70a9aa3b2 nodeName:}" failed. No retries permitted until 2026-09-16 18:44:39.41797709 +0000 UTC m=+42.206979621 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/b291cc26-ff6e-4adb-b219-8bd70a9aa3b2-metrics-tls") pod "dns-default-n8djq" (UID: "b291cc26-ff6e-4adb-b219-8bd70a9aa3b2") : secret "dns-default-metrics-tls" not found
Sep 16 18:44:35.926033 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:44:35.925996    2544 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-jmnkv" event={"ID":"ee8d40ee-7c8a-496a-a4e4-56cdc3e0aaa2","Type":"ContainerStarted","Data":"d42067ec3e843c4375cab489f178bbe5860b741f92312b0e0c64a03a7f8db517"}
Sep 16 18:44:35.947327 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:44:35.947285    2544 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-jmnkv" podStartSLOduration=2.378708916 podStartE2EDuration="4.947267974s" totalImagesPullingTime="2.568559058s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:44:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:44:35.946890373 +0000 UTC m=+38.735892930" watchObservedRunningTime="2026-09-16 18:44:35.947267974 +0000 UTC m=+38.736270507"
Sep 16 18:44:36.663011 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:44:36.662977    2544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pzpb9_0494be2a-d0e5-4570-b05c-0c595a0b494a/konnectivity-agent/0.log" limit=1024
Sep 16 18:44:38.058991 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:44:38.058951    2544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pzpb9_0494be2a-d0e5-4570-b05c-0c595a0b494a/konnectivity-agent/0.log" limit=1024
Sep 16 18:44:39.346052 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:44:39.346006    2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/b03d56d8-b081-41fd-bac9-ba7da38f733a-registry-tls\") pod \"image-registry-76874c7675-4dhws\" (UID: \"b03d56d8-b081-41fd-bac9-ba7da38f733a\") " pod="openshift-image-registry/image-registry-76874c7675-4dhws"
Sep 16 18:44:39.346514 ip-10-0-132-80 kubenswrapper[2544]: E0916 18:44:39.346187    2544 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 16 18:44:39.346514 ip-10-0-132-80 kubenswrapper[2544]: E0916 18:44:39.346213    2544 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-76874c7675-4dhws: secret "image-registry-tls" not found
Sep 16 18:44:39.346514 ip-10-0-132-80 kubenswrapper[2544]: E0916 18:44:39.346287    2544 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/b03d56d8-b081-41fd-bac9-ba7da38f733a-registry-tls podName:b03d56d8-b081-41fd-bac9-ba7da38f733a nodeName:}" failed. No retries permitted until 2026-09-16 18:44:47.346264327 +0000 UTC m=+50.135266842 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/b03d56d8-b081-41fd-bac9-ba7da38f733a-registry-tls") pod "image-registry-76874c7675-4dhws" (UID: "b03d56d8-b081-41fd-bac9-ba7da38f733a") : secret "image-registry-tls" not found
Sep 16 18:44:39.447155 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:44:39.447118    2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/b291cc26-ff6e-4adb-b219-8bd70a9aa3b2-metrics-tls\") pod \"dns-default-n8djq\" (UID: \"b291cc26-ff6e-4adb-b219-8bd70a9aa3b2\") " pod="openshift-dns/dns-default-n8djq"
Sep 16 18:44:39.447302 ip-10-0-132-80 kubenswrapper[2544]: E0916 18:44:39.447269    2544 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 16 18:44:39.447344 ip-10-0-132-80 kubenswrapper[2544]: E0916 18:44:39.447330    2544 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b291cc26-ff6e-4adb-b219-8bd70a9aa3b2-metrics-tls podName:b291cc26-ff6e-4adb-b219-8bd70a9aa3b2 nodeName:}" failed. No retries permitted until 2026-09-16 18:44:47.44731574 +0000 UTC m=+50.236318252 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/b291cc26-ff6e-4adb-b219-8bd70a9aa3b2-metrics-tls") pod "dns-default-n8djq" (UID: "b291cc26-ff6e-4adb-b219-8bd70a9aa3b2") : secret "dns-default-metrics-tls" not found
Sep 16 18:44:41.848225 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:44:41.848182    2544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pzpb9_0494be2a-d0e5-4570-b05c-0c595a0b494a/konnectivity-agent/0.log" limit=1024
Sep 16 18:44:41.998831 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:44:41.998797    2544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pzpb9_0494be2a-d0e5-4570-b05c-0c595a0b494a/konnectivity-agent/0.log" limit=1024
Sep 16 18:44:42.785296 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:44:42.785258    2544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pzpb9_0494be2a-d0e5-4570-b05c-0c595a0b494a/konnectivity-agent/0.log" limit=1024
Sep 16 18:44:42.855311 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:44:42.855269    2544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pzpb9_0494be2a-d0e5-4570-b05c-0c595a0b494a/konnectivity-agent/0.log" limit=1024
Sep 16 18:44:43.686216 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:44:43.686187    2544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pzpb9_0494be2a-d0e5-4570-b05c-0c595a0b494a/konnectivity-agent/0.log" limit=1024
Sep 16 18:44:47.405643 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:44:47.405614    2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/b03d56d8-b081-41fd-bac9-ba7da38f733a-registry-tls\") pod \"image-registry-76874c7675-4dhws\" (UID: \"b03d56d8-b081-41fd-bac9-ba7da38f733a\") " pod="openshift-image-registry/image-registry-76874c7675-4dhws"
Sep 16 18:44:47.405986 ip-10-0-132-80 kubenswrapper[2544]: E0916 18:44:47.405731    2544 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 16 18:44:47.405986 ip-10-0-132-80 kubenswrapper[2544]: E0916 18:44:47.405745    2544 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-76874c7675-4dhws: secret "image-registry-tls" not found
Sep 16 18:44:47.405986 ip-10-0-132-80 kubenswrapper[2544]: E0916 18:44:47.405832    2544 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/b03d56d8-b081-41fd-bac9-ba7da38f733a-registry-tls podName:b03d56d8-b081-41fd-bac9-ba7da38f733a nodeName:}" failed. No retries permitted until 2026-09-16 18:45:03.405816321 +0000 UTC m=+66.194818853 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/b03d56d8-b081-41fd-bac9-ba7da38f733a-registry-tls") pod "image-registry-76874c7675-4dhws" (UID: "b03d56d8-b081-41fd-bac9-ba7da38f733a") : secret "image-registry-tls" not found
Sep 16 18:44:47.506227 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:44:47.506177    2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/b291cc26-ff6e-4adb-b219-8bd70a9aa3b2-metrics-tls\") pod \"dns-default-n8djq\" (UID: \"b291cc26-ff6e-4adb-b219-8bd70a9aa3b2\") " pod="openshift-dns/dns-default-n8djq"
Sep 16 18:44:47.506354 ip-10-0-132-80 kubenswrapper[2544]: E0916 18:44:47.506333    2544 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 16 18:44:47.506409 ip-10-0-132-80 kubenswrapper[2544]: E0916 18:44:47.506399    2544 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b291cc26-ff6e-4adb-b219-8bd70a9aa3b2-metrics-tls podName:b291cc26-ff6e-4adb-b219-8bd70a9aa3b2 nodeName:}" failed. No retries permitted until 2026-09-16 18:45:03.506382456 +0000 UTC m=+66.295384967 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/b291cc26-ff6e-4adb-b219-8bd70a9aa3b2-metrics-tls") pod "dns-default-n8djq" (UID: "b291cc26-ff6e-4adb-b219-8bd70a9aa3b2") : secret "dns-default-metrics-tls" not found
Sep 16 18:44:52.023986 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:44:52.023937    2544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pzpb9_0494be2a-d0e5-4570-b05c-0c595a0b494a/konnectivity-agent/0.log" limit=1024
Sep 16 18:44:55.897884 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:44:55.897853    2544 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-nqcpx"
Sep 16 18:45:03.413927 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:45:03.413876    2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/db1e3b32-114c-4bd9-b28c-21a2a9d6efaa-metrics-certs\") pod \"network-metrics-daemon-pdfpq\" (UID: \"db1e3b32-114c-4bd9-b28c-21a2a9d6efaa\") " pod="openshift-multus/network-metrics-daemon-pdfpq"
Sep 16 18:45:03.414354 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:45:03.413960    2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/b03d56d8-b081-41fd-bac9-ba7da38f733a-registry-tls\") pod \"image-registry-76874c7675-4dhws\" (UID: \"b03d56d8-b081-41fd-bac9-ba7da38f733a\") " pod="openshift-image-registry/image-registry-76874c7675-4dhws"
Sep 16 18:45:03.414354 ip-10-0-132-80 kubenswrapper[2544]: E0916 18:45:03.414105    2544 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 16 18:45:03.414354 ip-10-0-132-80 kubenswrapper[2544]: E0916 18:45:03.414125    2544 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-76874c7675-4dhws: secret "image-registry-tls" not found
Sep 16 18:45:03.414354 ip-10-0-132-80 kubenswrapper[2544]: E0916 18:45:03.414179    2544 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/b03d56d8-b081-41fd-bac9-ba7da38f733a-registry-tls podName:b03d56d8-b081-41fd-bac9-ba7da38f733a nodeName:}" failed. No retries permitted until 2026-09-16 18:45:35.414164876 +0000 UTC m=+98.203167388 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/b03d56d8-b081-41fd-bac9-ba7da38f733a-registry-tls") pod "image-registry-76874c7675-4dhws" (UID: "b03d56d8-b081-41fd-bac9-ba7da38f733a") : secret "image-registry-tls" not found
Sep 16 18:45:03.416460 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:45:03.416442    2544 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Sep 16 18:45:03.424782 ip-10-0-132-80 kubenswrapper[2544]: E0916 18:45:03.424764    2544 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
Sep 16 18:45:03.424882 ip-10-0-132-80 kubenswrapper[2544]: E0916 18:45:03.424829    2544 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/db1e3b32-114c-4bd9-b28c-21a2a9d6efaa-metrics-certs podName:db1e3b32-114c-4bd9-b28c-21a2a9d6efaa nodeName:}" failed. No retries permitted until 2026-09-16 18:46:07.424808178 +0000 UTC m=+130.213810710 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/db1e3b32-114c-4bd9-b28c-21a2a9d6efaa-metrics-certs") pod "network-metrics-daemon-pdfpq" (UID: "db1e3b32-114c-4bd9-b28c-21a2a9d6efaa") : secret "metrics-daemon-secret" not found
Sep 16 18:45:03.515271 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:45:03.515223    2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/b291cc26-ff6e-4adb-b219-8bd70a9aa3b2-metrics-tls\") pod \"dns-default-n8djq\" (UID: \"b291cc26-ff6e-4adb-b219-8bd70a9aa3b2\") " pod="openshift-dns/dns-default-n8djq"
Sep 16 18:45:03.515429 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:45:03.515294    2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/2bbf45e9-df8e-4a67-b777-0c629428fca2-original-pull-secret\") pod \"global-pull-secret-syncer-pg4kf\" (UID: \"2bbf45e9-df8e-4a67-b777-0c629428fca2\") " pod="kube-system/global-pull-secret-syncer-pg4kf"
Sep 16 18:45:03.515429 ip-10-0-132-80 kubenswrapper[2544]: E0916 18:45:03.515366    2544 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 16 18:45:03.515429 ip-10-0-132-80 kubenswrapper[2544]: E0916 18:45:03.515421    2544 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b291cc26-ff6e-4adb-b219-8bd70a9aa3b2-metrics-tls podName:b291cc26-ff6e-4adb-b219-8bd70a9aa3b2 nodeName:}" failed. No retries permitted until 2026-09-16 18:45:35.51540696 +0000 UTC m=+98.304409472 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/b291cc26-ff6e-4adb-b219-8bd70a9aa3b2-metrics-tls") pod "dns-default-n8djq" (UID: "b291cc26-ff6e-4adb-b219-8bd70a9aa3b2") : secret "dns-default-metrics-tls" not found
Sep 16 18:45:03.517482 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:45:03.517467    2544 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Sep 16 18:45:03.529146 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:45:03.529126    2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/2bbf45e9-df8e-4a67-b777-0c629428fca2-original-pull-secret\") pod \"global-pull-secret-syncer-pg4kf\" (UID: \"2bbf45e9-df8e-4a67-b777-0c629428fca2\") " pod="kube-system/global-pull-secret-syncer-pg4kf"
Sep 16 18:45:03.615598 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:45:03.615541    2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s9f9h\" (UniqueName: \"kubernetes.io/projected/3c4f2334-3ac6-47e0-9eb7-4352c53df29a-kube-api-access-s9f9h\") pod \"network-check-target-t8hx2\" (UID: \"3c4f2334-3ac6-47e0-9eb7-4352c53df29a\") " pod="openshift-network-diagnostics/network-check-target-t8hx2"
Sep 16 18:45:03.619219 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:45:03.619201    2544 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Sep 16 18:45:03.628921 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:45:03.628902    2544 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Sep 16 18:45:03.639059 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:45:03.639035    2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-s9f9h\" (UniqueName: \"kubernetes.io/projected/3c4f2334-3ac6-47e0-9eb7-4352c53df29a-kube-api-access-s9f9h\") pod \"network-check-target-t8hx2\" (UID: \"3c4f2334-3ac6-47e0-9eb7-4352c53df29a\") " pod="openshift-network-diagnostics/network-check-target-t8hx2"
Sep 16 18:45:03.828682 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:45:03.828654    2544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-pg4kf"
Sep 16 18:45:03.841413 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:45:03.841385    2544 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-rk9xq\""
Sep 16 18:45:03.849124 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:45:03.849098    2544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-t8hx2"
Sep 16 18:45:04.108969 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:45:04.108895    2544 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-pg4kf"]
Sep 16 18:45:04.112908 ip-10-0-132-80 kubenswrapper[2544]: W0916 18:45:04.112860    2544 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2bbf45e9_df8e_4a67_b777_0c629428fca2.slice/crio-1bb7e1a40cb68ac3cbee95fd387d3a61f76005e760574760fa6e1f35c7e4d732 WatchSource:0}: Error finding container 1bb7e1a40cb68ac3cbee95fd387d3a61f76005e760574760fa6e1f35c7e4d732: Status 404 returned error can't find the container with id 1bb7e1a40cb68ac3cbee95fd387d3a61f76005e760574760fa6e1f35c7e4d732
Sep 16 18:45:04.129191 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:45:04.129167    2544 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-t8hx2"]
Sep 16 18:45:04.131932 ip-10-0-132-80 kubenswrapper[2544]: W0916 18:45:04.131909    2544 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3c4f2334_3ac6_47e0_9eb7_4352c53df29a.slice/crio-14eef1097bdbf98da2279baa7777f5f90c20302b45f62f8d6f8417cc1f291ab3 WatchSource:0}: Error finding container 14eef1097bdbf98da2279baa7777f5f90c20302b45f62f8d6f8417cc1f291ab3: Status 404 returned error can't find the container with id 14eef1097bdbf98da2279baa7777f5f90c20302b45f62f8d6f8417cc1f291ab3
Sep 16 18:45:04.994951 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:45:04.994906    2544 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-t8hx2" event={"ID":"3c4f2334-3ac6-47e0-9eb7-4352c53df29a","Type":"ContainerStarted","Data":"14eef1097bdbf98da2279baa7777f5f90c20302b45f62f8d6f8417cc1f291ab3"}
Sep 16 18:45:04.996222 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:45:04.996186    2544 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-pg4kf" event={"ID":"2bbf45e9-df8e-4a67-b777-0c629428fca2","Type":"ContainerStarted","Data":"1bb7e1a40cb68ac3cbee95fd387d3a61f76005e760574760fa6e1f35c7e4d732"}
Sep 16 18:45:05.789702 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:45:05.789664    2544 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console-operator/console-operator-696884fb9c-pfcb7"]
Sep 16 18:45:05.792398 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:45:05.792371    2544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-696884fb9c-pfcb7"
Sep 16 18:45:05.796173 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:45:05.795792    2544 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"console-operator-config\""
Sep 16 18:45:05.796173 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:45:05.795899    2544 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"openshift-service-ca.crt\""
Sep 16 18:45:05.796173 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:45:05.796018    2544 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"console-operator-dockercfg-g4btp\""
Sep 16 18:45:05.796173 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:45:05.796112    2544 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"kube-root-ca.crt\""
Sep 16 18:45:05.796502 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:45:05.796482    2544 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"serving-cert\""
Sep 16 18:45:05.804248 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:45:05.804208    2544 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-696884fb9c-pfcb7"]
Sep 16 18:45:05.806387 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:45:05.806363    2544 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"trusted-ca\""
Sep 16 18:45:05.831695 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:45:05.831657    2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/9e3f9e34-420e-43ef-96b4-c7cad6b2eaa9-trusted-ca\") pod \"console-operator-696884fb9c-pfcb7\" (UID: \"9e3f9e34-420e-43ef-96b4-c7cad6b2eaa9\") " pod="openshift-console-operator/console-operator-696884fb9c-pfcb7"
Sep 16 18:45:05.831885 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:45:05.831765    2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-crpt7\" (UniqueName: \"kubernetes.io/projected/9e3f9e34-420e-43ef-96b4-c7cad6b2eaa9-kube-api-access-crpt7\") pod \"console-operator-696884fb9c-pfcb7\" (UID: \"9e3f9e34-420e-43ef-96b4-c7cad6b2eaa9\") " pod="openshift-console-operator/console-operator-696884fb9c-pfcb7"
Sep 16 18:45:05.831885 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:45:05.831800    2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/9e3f9e34-420e-43ef-96b4-c7cad6b2eaa9-serving-cert\") pod \"console-operator-696884fb9c-pfcb7\" (UID: \"9e3f9e34-420e-43ef-96b4-c7cad6b2eaa9\") " pod="openshift-console-operator/console-operator-696884fb9c-pfcb7"
Sep 16 18:45:05.831885 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:45:05.831855    2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/9e3f9e34-420e-43ef-96b4-c7cad6b2eaa9-config\") pod \"console-operator-696884fb9c-pfcb7\" (UID: \"9e3f9e34-420e-43ef-96b4-c7cad6b2eaa9\") " pod="openshift-console-operator/console-operator-696884fb9c-pfcb7"
Sep 16 18:45:05.933072 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:45:05.933034    2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-crpt7\" (UniqueName: \"kubernetes.io/projected/9e3f9e34-420e-43ef-96b4-c7cad6b2eaa9-kube-api-access-crpt7\") pod \"console-operator-696884fb9c-pfcb7\" (UID: \"9e3f9e34-420e-43ef-96b4-c7cad6b2eaa9\") " pod="openshift-console-operator/console-operator-696884fb9c-pfcb7"
Sep 16 18:45:05.933257 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:45:05.933087    2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/9e3f9e34-420e-43ef-96b4-c7cad6b2eaa9-serving-cert\") pod \"console-operator-696884fb9c-pfcb7\" (UID: \"9e3f9e34-420e-43ef-96b4-c7cad6b2eaa9\") " pod="openshift-console-operator/console-operator-696884fb9c-pfcb7"
Sep 16 18:45:05.933257 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:45:05.933138    2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/9e3f9e34-420e-43ef-96b4-c7cad6b2eaa9-config\") pod \"console-operator-696884fb9c-pfcb7\" (UID: \"9e3f9e34-420e-43ef-96b4-c7cad6b2eaa9\") " pod="openshift-console-operator/console-operator-696884fb9c-pfcb7"
Sep 16 18:45:05.933257 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:45:05.933169    2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/9e3f9e34-420e-43ef-96b4-c7cad6b2eaa9-trusted-ca\") pod \"console-operator-696884fb9c-pfcb7\" (UID: \"9e3f9e34-420e-43ef-96b4-c7cad6b2eaa9\") " pod="openshift-console-operator/console-operator-696884fb9c-pfcb7"
Sep 16 18:45:05.934155 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:45:05.934130    2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/9e3f9e34-420e-43ef-96b4-c7cad6b2eaa9-config\") pod \"console-operator-696884fb9c-pfcb7\" (UID: \"9e3f9e34-420e-43ef-96b4-c7cad6b2eaa9\") " pod="openshift-console-operator/console-operator-696884fb9c-pfcb7"
Sep 16 18:45:05.934865 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:45:05.934823    2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/9e3f9e34-420e-43ef-96b4-c7cad6b2eaa9-trusted-ca\") pod \"console-operator-696884fb9c-pfcb7\" (UID: \"9e3f9e34-420e-43ef-96b4-c7cad6b2eaa9\") " pod="openshift-console-operator/console-operator-696884fb9c-pfcb7"
Sep 16 18:45:05.936937 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:45:05.936915    2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/9e3f9e34-420e-43ef-96b4-c7cad6b2eaa9-serving-cert\") pod \"console-operator-696884fb9c-pfcb7\" (UID: \"9e3f9e34-420e-43ef-96b4-c7cad6b2eaa9\") " pod="openshift-console-operator/console-operator-696884fb9c-pfcb7"
Sep 16 18:45:05.942535 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:45:05.942460    2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-crpt7\" (UniqueName: \"kubernetes.io/projected/9e3f9e34-420e-43ef-96b4-c7cad6b2eaa9-kube-api-access-crpt7\") pod \"console-operator-696884fb9c-pfcb7\" (UID: \"9e3f9e34-420e-43ef-96b4-c7cad6b2eaa9\") " pod="openshift-console-operator/console-operator-696884fb9c-pfcb7"
Sep 16 18:45:05.962329 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:45:05.962249    2544 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-network-console/networking-console-plugin-5dcbc7df6b-q4dbv"]
Sep 16 18:45:05.965755 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:45:05.965731    2544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-q4dbv"
Sep 16 18:45:05.968222 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:45:05.967973    2544 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"default-dockercfg-vtnbz\""
Sep 16 18:45:05.968222 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:45:05.967990    2544 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-console\"/\"networking-console-plugin\""
Sep 16 18:45:05.968222 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:45:05.968049    2544 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"networking-console-plugin-cert\""
Sep 16 18:45:05.982799 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:45:05.982766    2544 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-5dcbc7df6b-q4dbv"]
Sep 16 18:45:06.033732 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:45:06.033692    2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/aa574cca-15b0-43dd-b30c-58d124d977d2-nginx-conf\") pod \"networking-console-plugin-5dcbc7df6b-q4dbv\" (UID: \"aa574cca-15b0-43dd-b30c-58d124d977d2\") " pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-q4dbv"
Sep 16 18:45:06.034274 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:45:06.033757    2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/aa574cca-15b0-43dd-b30c-58d124d977d2-networking-console-plugin-cert\") pod \"networking-console-plugin-5dcbc7df6b-q4dbv\" (UID: \"aa574cca-15b0-43dd-b30c-58d124d977d2\") " pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-q4dbv"
Sep 16 18:45:06.105126 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:45:06.105073    2544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-696884fb9c-pfcb7"
Sep 16 18:45:06.148938 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:45:06.148896    2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/aa574cca-15b0-43dd-b30c-58d124d977d2-nginx-conf\") pod \"networking-console-plugin-5dcbc7df6b-q4dbv\" (UID: \"aa574cca-15b0-43dd-b30c-58d124d977d2\") " pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-q4dbv"
Sep 16 18:45:06.149123 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:45:06.148961    2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/aa574cca-15b0-43dd-b30c-58d124d977d2-networking-console-plugin-cert\") pod \"networking-console-plugin-5dcbc7df6b-q4dbv\" (UID: \"aa574cca-15b0-43dd-b30c-58d124d977d2\") " pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-q4dbv"
Sep 16 18:45:06.149123 ip-10-0-132-80 kubenswrapper[2544]: E0916 18:45:06.149109    2544 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Sep 16 18:45:06.149415 ip-10-0-132-80 kubenswrapper[2544]: E0916 18:45:06.149180    2544 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/aa574cca-15b0-43dd-b30c-58d124d977d2-networking-console-plugin-cert podName:aa574cca-15b0-43dd-b30c-58d124d977d2 nodeName:}" failed. No retries permitted until 2026-09-16 18:45:06.649157903 +0000 UTC m=+69.438160418 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/aa574cca-15b0-43dd-b30c-58d124d977d2-networking-console-plugin-cert") pod "networking-console-plugin-5dcbc7df6b-q4dbv" (UID: "aa574cca-15b0-43dd-b30c-58d124d977d2") : secret "networking-console-plugin-cert" not found
Sep 16 18:45:06.150106 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:45:06.150081    2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/aa574cca-15b0-43dd-b30c-58d124d977d2-nginx-conf\") pod \"networking-console-plugin-5dcbc7df6b-q4dbv\" (UID: \"aa574cca-15b0-43dd-b30c-58d124d977d2\") " pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-q4dbv"
Sep 16 18:45:06.411864 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:45:06.411778    2544 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-696884fb9c-pfcb7"]
Sep 16 18:45:06.416021 ip-10-0-132-80 kubenswrapper[2544]: W0916 18:45:06.415988    2544 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9e3f9e34_420e_43ef_96b4_c7cad6b2eaa9.slice/crio-60851400a97940d230fe02d59fa3e58a7c2ebe137402da0ffa25d5357a2c564d WatchSource:0}: Error finding container 60851400a97940d230fe02d59fa3e58a7c2ebe137402da0ffa25d5357a2c564d: Status 404 returned error can't find the container with id 60851400a97940d230fe02d59fa3e58a7c2ebe137402da0ffa25d5357a2c564d
Sep 16 18:45:06.653214 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:45:06.653127    2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/aa574cca-15b0-43dd-b30c-58d124d977d2-networking-console-plugin-cert\") pod \"networking-console-plugin-5dcbc7df6b-q4dbv\" (UID: \"aa574cca-15b0-43dd-b30c-58d124d977d2\") " pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-q4dbv"
Sep 16 18:45:06.653529 ip-10-0-132-80 kubenswrapper[2544]: E0916 18:45:06.653319    2544 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Sep 16 18:45:06.653529 ip-10-0-132-80 kubenswrapper[2544]: E0916 18:45:06.653408    2544 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/aa574cca-15b0-43dd-b30c-58d124d977d2-networking-console-plugin-cert podName:aa574cca-15b0-43dd-b30c-58d124d977d2 nodeName:}" failed. No retries permitted until 2026-09-16 18:45:07.653385686 +0000 UTC m=+70.442388202 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/aa574cca-15b0-43dd-b30c-58d124d977d2-networking-console-plugin-cert") pod "networking-console-plugin-5dcbc7df6b-q4dbv" (UID: "aa574cca-15b0-43dd-b30c-58d124d977d2") : secret "networking-console-plugin-cert" not found
Sep 16 18:45:07.004143 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:45:07.004109    2544 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-696884fb9c-pfcb7" event={"ID":"9e3f9e34-420e-43ef-96b4-c7cad6b2eaa9","Type":"ContainerStarted","Data":"60851400a97940d230fe02d59fa3e58a7c2ebe137402da0ffa25d5357a2c564d"}
Sep 16 18:45:07.660616 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:45:07.660574    2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/aa574cca-15b0-43dd-b30c-58d124d977d2-networking-console-plugin-cert\") pod \"networking-console-plugin-5dcbc7df6b-q4dbv\" (UID: \"aa574cca-15b0-43dd-b30c-58d124d977d2\") " pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-q4dbv"
Sep 16 18:45:07.661059 ip-10-0-132-80 kubenswrapper[2544]: E0916 18:45:07.660728    2544 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Sep 16 18:45:07.661059 ip-10-0-132-80 kubenswrapper[2544]: E0916 18:45:07.660810    2544 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/aa574cca-15b0-43dd-b30c-58d124d977d2-networking-console-plugin-cert podName:aa574cca-15b0-43dd-b30c-58d124d977d2 nodeName:}" failed. No retries permitted until 2026-09-16 18:45:09.660788258 +0000 UTC m=+72.449790797 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/aa574cca-15b0-43dd-b30c-58d124d977d2-networking-console-plugin-cert") pod "networking-console-plugin-5dcbc7df6b-q4dbv" (UID: "aa574cca-15b0-43dd-b30c-58d124d977d2") : secret "networking-console-plugin-cert" not found
Sep 16 18:45:09.676614 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:45:09.676562    2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/aa574cca-15b0-43dd-b30c-58d124d977d2-networking-console-plugin-cert\") pod \"networking-console-plugin-5dcbc7df6b-q4dbv\" (UID: \"aa574cca-15b0-43dd-b30c-58d124d977d2\") " pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-q4dbv"
Sep 16 18:45:09.677038 ip-10-0-132-80 kubenswrapper[2544]: E0916 18:45:09.676636    2544 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Sep 16 18:45:09.677038 ip-10-0-132-80 kubenswrapper[2544]: E0916 18:45:09.676711    2544 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/aa574cca-15b0-43dd-b30c-58d124d977d2-networking-console-plugin-cert podName:aa574cca-15b0-43dd-b30c-58d124d977d2 nodeName:}" failed. No retries permitted until 2026-09-16 18:45:13.67669165 +0000 UTC m=+76.465694185 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/aa574cca-15b0-43dd-b30c-58d124d977d2-networking-console-plugin-cert") pod "networking-console-plugin-5dcbc7df6b-q4dbv" (UID: "aa574cca-15b0-43dd-b30c-58d124d977d2") : secret "networking-console-plugin-cert" not found
Sep 16 18:45:10.013467 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:45:10.013385    2544 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-pg4kf" event={"ID":"2bbf45e9-df8e-4a67-b777-0c629428fca2","Type":"ContainerStarted","Data":"b059fae140340d4804dc23933615757df40f6b1376ed87fa3e78f22d693ce05c"}
Sep 16 18:45:10.015139 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:45:10.015109    2544 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-t8hx2" event={"ID":"3c4f2334-3ac6-47e0-9eb7-4352c53df29a","Type":"ContainerStarted","Data":"53f7ec8ce91b28ddeb0ea8be0de53d72d89bd884ed42cadc3fc025d1fb0eaddc"}
Sep 16 18:45:10.015265 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:45:10.015247    2544 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-t8hx2"
Sep 16 18:45:10.032159 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:45:10.031994    2544 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-pg4kf" podStartSLOduration=67.025632598 podStartE2EDuration="1m12.03197299s" totalImagesPullingTime="5.006340392s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-16 18:43:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:45:10.031100491 +0000 UTC m=+72.820103050" watchObservedRunningTime="2026-09-16 18:45:10.03197299 +0000 UTC m=+72.820975518"
Sep 16 18:45:10.050852 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:45:10.050807    2544 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-t8hx2" podStartSLOduration=67.062914461 podStartE2EDuration="1m12.050792075s" totalImagesPullingTime="4.987877614s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:43:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:45:10.0499712 +0000 UTC m=+72.838973759" watchObservedRunningTime="2026-09-16 18:45:10.050792075 +0000 UTC m=+72.839794609"
Sep 16 18:45:11.019520 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:45:11.019492    2544 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-pfcb7_9e3f9e34-420e-43ef-96b4-c7cad6b2eaa9/console-operator/0.log"
Sep 16 18:45:11.019954 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:45:11.019532    2544 generic.go:356] "Generic (PLEG): container finished" podID="9e3f9e34-420e-43ef-96b4-c7cad6b2eaa9" containerID="5fb58a076c8d3f88d3c1b233d6a94c3b9c18459970dd3f4584d7b7ff374713c4" exitCode=255
Sep 16 18:45:11.019954 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:45:11.019648    2544 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-696884fb9c-pfcb7" event={"ID":"9e3f9e34-420e-43ef-96b4-c7cad6b2eaa9","Type":"ContainerDied","Data":"5fb58a076c8d3f88d3c1b233d6a94c3b9c18459970dd3f4584d7b7ff374713c4"}
Sep 16 18:45:11.019954 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:45:11.019839    2544 scope.go:117] "RemoveContainer" containerID="5fb58a076c8d3f88d3c1b233d6a94c3b9c18459970dd3f4584d7b7ff374713c4"
Sep 16 18:45:12.023463 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:45:12.023436    2544 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-pfcb7_9e3f9e34-420e-43ef-96b4-c7cad6b2eaa9/console-operator/1.log"
Sep 16 18:45:12.023900 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:45:12.023886    2544 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-pfcb7_9e3f9e34-420e-43ef-96b4-c7cad6b2eaa9/console-operator/0.log"
Sep 16 18:45:12.023947 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:45:12.023920    2544 generic.go:356] "Generic (PLEG): container finished" podID="9e3f9e34-420e-43ef-96b4-c7cad6b2eaa9" containerID="ac95d772785093117506c3e1602208fe902bd9ec46f5646b88f8c5d47c1d8e5a" exitCode=255
Sep 16 18:45:12.023985 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:45:12.023946    2544 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-696884fb9c-pfcb7" event={"ID":"9e3f9e34-420e-43ef-96b4-c7cad6b2eaa9","Type":"ContainerDied","Data":"ac95d772785093117506c3e1602208fe902bd9ec46f5646b88f8c5d47c1d8e5a"}
Sep 16 18:45:12.024015 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:45:12.023987    2544 scope.go:117] "RemoveContainer" containerID="5fb58a076c8d3f88d3c1b233d6a94c3b9c18459970dd3f4584d7b7ff374713c4"
Sep 16 18:45:12.024220 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:45:12.024196    2544 scope.go:117] "RemoveContainer" containerID="ac95d772785093117506c3e1602208fe902bd9ec46f5646b88f8c5d47c1d8e5a"
Sep 16 18:45:12.024399 ip-10-0-132-80 kubenswrapper[2544]: E0916 18:45:12.024381    2544 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-696884fb9c-pfcb7_openshift-console-operator(9e3f9e34-420e-43ef-96b4-c7cad6b2eaa9)\"" pod="openshift-console-operator/console-operator-696884fb9c-pfcb7" podUID="9e3f9e34-420e-43ef-96b4-c7cad6b2eaa9"
Sep 16 18:45:12.816916 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:45:12.816880    2544 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-xhj2z"]
Sep 16 18:45:12.820976 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:45:12.820960    2544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-xhj2z"
Sep 16 18:45:12.822970 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:45:12.822948    2544 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Sep 16 18:45:12.823308 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:45:12.823288    2544 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-k8svg\""
Sep 16 18:45:12.823518 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:45:12.823494    2544 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Sep 16 18:45:12.823518 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:45:12.823514    2544 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Sep 16 18:45:12.824264 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:45:12.824249    2544 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Sep 16 18:45:12.836206 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:45:12.836186    2544 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-xhj2z"]
Sep 16 18:45:12.903334 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:45:12.903302    2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/3e46e7aa-10f9-4c7d-bfa5-62504dc45cfd-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-xhj2z\" (UID: \"3e46e7aa-10f9-4c7d-bfa5-62504dc45cfd\") " pod="openshift-insights/insights-runtime-extractor-xhj2z"
Sep 16 18:45:12.903334 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:45:12.903336    2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/3e46e7aa-10f9-4c7d-bfa5-62504dc45cfd-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-xhj2z\" (UID: \"3e46e7aa-10f9-4c7d-bfa5-62504dc45cfd\") " pod="openshift-insights/insights-runtime-extractor-xhj2z"
Sep 16 18:45:12.903491 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:45:12.903375    2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/3e46e7aa-10f9-4c7d-bfa5-62504dc45cfd-crio-socket\") pod \"insights-runtime-extractor-xhj2z\" (UID: \"3e46e7aa-10f9-4c7d-bfa5-62504dc45cfd\") " pod="openshift-insights/insights-runtime-extractor-xhj2z"
Sep 16 18:45:12.903491 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:45:12.903427    2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7v8gp\" (UniqueName: \"kubernetes.io/projected/3e46e7aa-10f9-4c7d-bfa5-62504dc45cfd-kube-api-access-7v8gp\") pod \"insights-runtime-extractor-xhj2z\" (UID: \"3e46e7aa-10f9-4c7d-bfa5-62504dc45cfd\") " pod="openshift-insights/insights-runtime-extractor-xhj2z"
Sep 16 18:45:12.903491 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:45:12.903454    2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/3e46e7aa-10f9-4c7d-bfa5-62504dc45cfd-data-volume\") pod \"insights-runtime-extractor-xhj2z\" (UID: \"3e46e7aa-10f9-4c7d-bfa5-62504dc45cfd\") " pod="openshift-insights/insights-runtime-extractor-xhj2z"
Sep 16 18:45:13.004663 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:45:13.004633    2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/3e46e7aa-10f9-4c7d-bfa5-62504dc45cfd-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-xhj2z\" (UID: \"3e46e7aa-10f9-4c7d-bfa5-62504dc45cfd\") " pod="openshift-insights/insights-runtime-extractor-xhj2z"
Sep 16 18:45:13.004720 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:45:13.004671    2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/3e46e7aa-10f9-4c7d-bfa5-62504dc45cfd-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-xhj2z\" (UID: \"3e46e7aa-10f9-4c7d-bfa5-62504dc45cfd\") " pod="openshift-insights/insights-runtime-extractor-xhj2z"
Sep 16 18:45:13.004770 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:45:13.004721    2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/3e46e7aa-10f9-4c7d-bfa5-62504dc45cfd-crio-socket\") pod \"insights-runtime-extractor-xhj2z\" (UID: \"3e46e7aa-10f9-4c7d-bfa5-62504dc45cfd\") " pod="openshift-insights/insights-runtime-extractor-xhj2z"
Sep 16 18:45:13.004770 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:45:13.004742    2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7v8gp\" (UniqueName: \"kubernetes.io/projected/3e46e7aa-10f9-4c7d-bfa5-62504dc45cfd-kube-api-access-7v8gp\") pod \"insights-runtime-extractor-xhj2z\" (UID: \"3e46e7aa-10f9-4c7d-bfa5-62504dc45cfd\") " pod="openshift-insights/insights-runtime-extractor-xhj2z"
Sep 16 18:45:13.004770 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:45:13.004760    2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/3e46e7aa-10f9-4c7d-bfa5-62504dc45cfd-data-volume\") pod \"insights-runtime-extractor-xhj2z\" (UID: \"3e46e7aa-10f9-4c7d-bfa5-62504dc45cfd\") " pod="openshift-insights/insights-runtime-extractor-xhj2z"
Sep 16 18:45:13.004862 ip-10-0-132-80 kubenswrapper[2544]: E0916 18:45:13.004769    2544 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 16 18:45:13.004862 ip-10-0-132-80 kubenswrapper[2544]: E0916 18:45:13.004823    2544 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 16 18:45:13.004862 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:45:13.004840    2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/3e46e7aa-10f9-4c7d-bfa5-62504dc45cfd-crio-socket\") pod \"insights-runtime-extractor-xhj2z\" (UID: \"3e46e7aa-10f9-4c7d-bfa5-62504dc45cfd\") " pod="openshift-insights/insights-runtime-extractor-xhj2z"
Sep 16 18:45:13.004862 ip-10-0-132-80 kubenswrapper[2544]: E0916 18:45:13.004855    2544 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3e46e7aa-10f9-4c7d-bfa5-62504dc45cfd-insights-runtime-extractor-tls podName:3e46e7aa-10f9-4c7d-bfa5-62504dc45cfd nodeName:}" failed. No retries permitted until 2026-09-16 18:45:13.504836165 +0000 UTC m=+76.293838689 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/3e46e7aa-10f9-4c7d-bfa5-62504dc45cfd-insights-runtime-extractor-tls") pod "insights-runtime-extractor-xhj2z" (UID: "3e46e7aa-10f9-4c7d-bfa5-62504dc45cfd") : secret "insights-runtime-extractor-tls" not found
Sep 16 18:45:13.004991 ip-10-0-132-80 kubenswrapper[2544]: E0916 18:45:13.004924    2544 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/3e46e7aa-10f9-4c7d-bfa5-62504dc45cfd-kube-rbac-proxy-cm podName:3e46e7aa-10f9-4c7d-bfa5-62504dc45cfd nodeName:}" failed. No retries permitted until 2026-09-16 18:45:13.504912257 +0000 UTC m=+76.293914794 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/3e46e7aa-10f9-4c7d-bfa5-62504dc45cfd-kube-rbac-proxy-cm") pod "insights-runtime-extractor-xhj2z" (UID: "3e46e7aa-10f9-4c7d-bfa5-62504dc45cfd") : configmap "kube-rbac-proxy" not found
Sep 16 18:45:13.005067 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:45:13.005054    2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/3e46e7aa-10f9-4c7d-bfa5-62504dc45cfd-data-volume\") pod \"insights-runtime-extractor-xhj2z\" (UID: \"3e46e7aa-10f9-4c7d-bfa5-62504dc45cfd\") " pod="openshift-insights/insights-runtime-extractor-xhj2z"
Sep 16 18:45:13.013300 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:45:13.013279    2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7v8gp\" (UniqueName: \"kubernetes.io/projected/3e46e7aa-10f9-4c7d-bfa5-62504dc45cfd-kube-api-access-7v8gp\") pod \"insights-runtime-extractor-xhj2z\" (UID: \"3e46e7aa-10f9-4c7d-bfa5-62504dc45cfd\") " pod="openshift-insights/insights-runtime-extractor-xhj2z"
Sep 16 18:45:13.027850 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:45:13.027827    2544 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-pfcb7_9e3f9e34-420e-43ef-96b4-c7cad6b2eaa9/console-operator/1.log"
Sep 16 18:45:13.028151 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:45:13.028137    2544 scope.go:117] "RemoveContainer" containerID="ac95d772785093117506c3e1602208fe902bd9ec46f5646b88f8c5d47c1d8e5a"
Sep 16 18:45:13.028302 ip-10-0-132-80 kubenswrapper[2544]: E0916 18:45:13.028277    2544 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-696884fb9c-pfcb7_openshift-console-operator(9e3f9e34-420e-43ef-96b4-c7cad6b2eaa9)\"" pod="openshift-console-operator/console-operator-696884fb9c-pfcb7" podUID="9e3f9e34-420e-43ef-96b4-c7cad6b2eaa9"
Sep 16 18:45:13.162322 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:45:13.162251    2544 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator/migrator-8b4d697c5-mm4qq"]
Sep 16 18:45:13.165245 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:45:13.165229    2544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-8b4d697c5-mm4qq"
Sep 16 18:45:13.167482 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:45:13.167456    2544 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"openshift-service-ca.crt\""
Sep 16 18:45:13.167482 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:45:13.167456    2544 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-storage-version-migrator-sa-dockercfg-prpl4\""
Sep 16 18:45:13.167593 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:45:13.167490    2544 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-root-ca.crt\""
Sep 16 18:45:13.174387 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:45:13.174364    2544 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-8b4d697c5-mm4qq"]
Sep 16 18:45:13.206991 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:45:13.206962    2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-84mml\" (UniqueName: \"kubernetes.io/projected/358d96b5-9e80-4d99-9478-81be76c4db64-kube-api-access-84mml\") pod \"migrator-8b4d697c5-mm4qq\" (UID: \"358d96b5-9e80-4d99-9478-81be76c4db64\") " pod="openshift-kube-storage-version-migrator/migrator-8b4d697c5-mm4qq"
Sep 16 18:45:13.308236 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:45:13.308213    2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-84mml\" (UniqueName: \"kubernetes.io/projected/358d96b5-9e80-4d99-9478-81be76c4db64-kube-api-access-84mml\") pod \"migrator-8b4d697c5-mm4qq\" (UID: \"358d96b5-9e80-4d99-9478-81be76c4db64\") " pod="openshift-kube-storage-version-migrator/migrator-8b4d697c5-mm4qq"
Sep 16 18:45:13.316879 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:45:13.316857    2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-84mml\" (UniqueName: \"kubernetes.io/projected/358d96b5-9e80-4d99-9478-81be76c4db64-kube-api-access-84mml\") pod \"migrator-8b4d697c5-mm4qq\" (UID: \"358d96b5-9e80-4d99-9478-81be76c4db64\") " pod="openshift-kube-storage-version-migrator/migrator-8b4d697c5-mm4qq"
Sep 16 18:45:13.474583 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:45:13.474481    2544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-8b4d697c5-mm4qq"
Sep 16 18:45:13.509739 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:45:13.509706    2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/3e46e7aa-10f9-4c7d-bfa5-62504dc45cfd-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-xhj2z\" (UID: \"3e46e7aa-10f9-4c7d-bfa5-62504dc45cfd\") " pod="openshift-insights/insights-runtime-extractor-xhj2z"
Sep 16 18:45:13.509849 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:45:13.509747    2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/3e46e7aa-10f9-4c7d-bfa5-62504dc45cfd-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-xhj2z\" (UID: \"3e46e7aa-10f9-4c7d-bfa5-62504dc45cfd\") " pod="openshift-insights/insights-runtime-extractor-xhj2z"
Sep 16 18:45:13.509849 ip-10-0-132-80 kubenswrapper[2544]: E0916 18:45:13.509824    2544 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 16 18:45:13.509932 ip-10-0-132-80 kubenswrapper[2544]: E0916 18:45:13.509859    2544 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 16 18:45:13.509932 ip-10-0-132-80 kubenswrapper[2544]: E0916 18:45:13.509886    2544 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/3e46e7aa-10f9-4c7d-bfa5-62504dc45cfd-kube-rbac-proxy-cm podName:3e46e7aa-10f9-4c7d-bfa5-62504dc45cfd nodeName:}" failed. No retries permitted until 2026-09-16 18:45:14.509869293 +0000 UTC m=+77.298871825 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/3e46e7aa-10f9-4c7d-bfa5-62504dc45cfd-kube-rbac-proxy-cm") pod "insights-runtime-extractor-xhj2z" (UID: "3e46e7aa-10f9-4c7d-bfa5-62504dc45cfd") : configmap "kube-rbac-proxy" not found
Sep 16 18:45:13.509932 ip-10-0-132-80 kubenswrapper[2544]: E0916 18:45:13.509914    2544 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3e46e7aa-10f9-4c7d-bfa5-62504dc45cfd-insights-runtime-extractor-tls podName:3e46e7aa-10f9-4c7d-bfa5-62504dc45cfd nodeName:}" failed. No retries permitted until 2026-09-16 18:45:14.509900126 +0000 UTC m=+77.298902638 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/3e46e7aa-10f9-4c7d-bfa5-62504dc45cfd-insights-runtime-extractor-tls") pod "insights-runtime-extractor-xhj2z" (UID: "3e46e7aa-10f9-4c7d-bfa5-62504dc45cfd") : secret "insights-runtime-extractor-tls" not found
Sep 16 18:45:13.711427 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:45:13.711391    2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/aa574cca-15b0-43dd-b30c-58d124d977d2-networking-console-plugin-cert\") pod \"networking-console-plugin-5dcbc7df6b-q4dbv\" (UID: \"aa574cca-15b0-43dd-b30c-58d124d977d2\") " pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-q4dbv"
Sep 16 18:45:13.711710 ip-10-0-132-80 kubenswrapper[2544]: E0916 18:45:13.711578    2544 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Sep 16 18:45:13.711710 ip-10-0-132-80 kubenswrapper[2544]: E0916 18:45:13.711694    2544 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/aa574cca-15b0-43dd-b30c-58d124d977d2-networking-console-plugin-cert podName:aa574cca-15b0-43dd-b30c-58d124d977d2 nodeName:}" failed. No retries permitted until 2026-09-16 18:45:21.711670759 +0000 UTC m=+84.500673287 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/aa574cca-15b0-43dd-b30c-58d124d977d2-networking-console-plugin-cert") pod "networking-console-plugin-5dcbc7df6b-q4dbv" (UID: "aa574cca-15b0-43dd-b30c-58d124d977d2") : secret "networking-console-plugin-cert" not found
Sep 16 18:45:13.750475 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:45:13.750456    2544 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-8b4d697c5-mm4qq"]
Sep 16 18:45:13.752137 ip-10-0-132-80 kubenswrapper[2544]: W0916 18:45:13.752110    2544 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod358d96b5_9e80_4d99_9478_81be76c4db64.slice/crio-e4b408a2dc491f5bb27fccaf9a78ba8ad4592a2894a7e312388433b8f2ea5561 WatchSource:0}: Error finding container e4b408a2dc491f5bb27fccaf9a78ba8ad4592a2894a7e312388433b8f2ea5561: Status 404 returned error can't find the container with id e4b408a2dc491f5bb27fccaf9a78ba8ad4592a2894a7e312388433b8f2ea5561
Sep 16 18:45:14.031492 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:45:14.031416    2544 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-8b4d697c5-mm4qq" event={"ID":"358d96b5-9e80-4d99-9478-81be76c4db64","Type":"ContainerStarted","Data":"e4b408a2dc491f5bb27fccaf9a78ba8ad4592a2894a7e312388433b8f2ea5561"}
Sep 16 18:45:14.517646 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:45:14.517613    2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/3e46e7aa-10f9-4c7d-bfa5-62504dc45cfd-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-xhj2z\" (UID: \"3e46e7aa-10f9-4c7d-bfa5-62504dc45cfd\") " pod="openshift-insights/insights-runtime-extractor-xhj2z"
Sep 16 18:45:14.517838 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:45:14.517658    2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/3e46e7aa-10f9-4c7d-bfa5-62504dc45cfd-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-xhj2z\" (UID: \"3e46e7aa-10f9-4c7d-bfa5-62504dc45cfd\") " pod="openshift-insights/insights-runtime-extractor-xhj2z"
Sep 16 18:45:14.517838 ip-10-0-132-80 kubenswrapper[2544]: E0916 18:45:14.517773    2544 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 16 18:45:14.517838 ip-10-0-132-80 kubenswrapper[2544]: E0916 18:45:14.517797    2544 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 16 18:45:14.517965 ip-10-0-132-80 kubenswrapper[2544]: E0916 18:45:14.517848    2544 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3e46e7aa-10f9-4c7d-bfa5-62504dc45cfd-insights-runtime-extractor-tls podName:3e46e7aa-10f9-4c7d-bfa5-62504dc45cfd nodeName:}" failed. No retries permitted until 2026-09-16 18:45:16.51783036 +0000 UTC m=+79.306832894 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/3e46e7aa-10f9-4c7d-bfa5-62504dc45cfd-insights-runtime-extractor-tls") pod "insights-runtime-extractor-xhj2z" (UID: "3e46e7aa-10f9-4c7d-bfa5-62504dc45cfd") : secret "insights-runtime-extractor-tls" not found
Sep 16 18:45:14.517965 ip-10-0-132-80 kubenswrapper[2544]: E0916 18:45:14.517863    2544 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/3e46e7aa-10f9-4c7d-bfa5-62504dc45cfd-kube-rbac-proxy-cm podName:3e46e7aa-10f9-4c7d-bfa5-62504dc45cfd nodeName:}" failed. No retries permitted until 2026-09-16 18:45:16.517856349 +0000 UTC m=+79.306858861 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/3e46e7aa-10f9-4c7d-bfa5-62504dc45cfd-kube-rbac-proxy-cm") pod "insights-runtime-extractor-xhj2z" (UID: "3e46e7aa-10f9-4c7d-bfa5-62504dc45cfd") : configmap "kube-rbac-proxy" not found
Sep 16 18:45:14.943727 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:45:14.943702    2544 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-sxbgl_524324ca-6ba2-4885-9ed6-570fa75665e7/dns-node-resolver/0.log"
Sep 16 18:45:15.542384 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:45:15.542362    2544 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-2hdh4_273fb2b8-0579-4f86-abc5-2801f6f47bcd/node-ca/0.log"
Sep 16 18:45:16.041685 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:45:16.041647    2544 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-8b4d697c5-mm4qq" event={"ID":"358d96b5-9e80-4d99-9478-81be76c4db64","Type":"ContainerStarted","Data":"c4d8096d8e0337bb01b7e147dea81b03ea00ed796a9deabde31cc03949240612"}
Sep 16 18:45:16.041954 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:45:16.041713    2544 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-8b4d697c5-mm4qq" event={"ID":"358d96b5-9e80-4d99-9478-81be76c4db64","Type":"ContainerStarted","Data":"4a702d92f8ab95ed6c268a8c9eefa9012310adb187b4dbf0bd9844981f72ff02"}
Sep 16 18:45:16.063265 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:45:16.063219    2544 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator/migrator-8b4d697c5-mm4qq" podStartSLOduration=1.8354007669999999 podStartE2EDuration="3.063205562s" totalImagesPullingTime="1.227804795s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:45:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:45:16.062092378 +0000 UTC m=+78.851094906" watchObservedRunningTime="2026-09-16 18:45:16.063205562 +0000 UTC m=+78.852208096"
Sep 16 18:45:16.105563 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:45:16.105531    2544 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="openshift-console-operator/console-operator-696884fb9c-pfcb7"
Sep 16 18:45:16.105665 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:45:16.105588    2544 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-696884fb9c-pfcb7"
Sep 16 18:45:16.105897 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:45:16.105880    2544 scope.go:117] "RemoveContainer" containerID="ac95d772785093117506c3e1602208fe902bd9ec46f5646b88f8c5d47c1d8e5a"
Sep 16 18:45:16.106035 ip-10-0-132-80 kubenswrapper[2544]: E0916 18:45:16.106018    2544 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-696884fb9c-pfcb7_openshift-console-operator(9e3f9e34-420e-43ef-96b4-c7cad6b2eaa9)\"" pod="openshift-console-operator/console-operator-696884fb9c-pfcb7" podUID="9e3f9e34-420e-43ef-96b4-c7cad6b2eaa9"
Sep 16 18:45:16.533966 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:45:16.533924    2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/3e46e7aa-10f9-4c7d-bfa5-62504dc45cfd-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-xhj2z\" (UID: \"3e46e7aa-10f9-4c7d-bfa5-62504dc45cfd\") " pod="openshift-insights/insights-runtime-extractor-xhj2z"
Sep 16 18:45:16.534237 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:45:16.533977    2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/3e46e7aa-10f9-4c7d-bfa5-62504dc45cfd-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-xhj2z\" (UID: \"3e46e7aa-10f9-4c7d-bfa5-62504dc45cfd\") " pod="openshift-insights/insights-runtime-extractor-xhj2z"
Sep 16 18:45:16.534237 ip-10-0-132-80 kubenswrapper[2544]: E0916 18:45:16.534059    2544 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 16 18:45:16.534237 ip-10-0-132-80 kubenswrapper[2544]: E0916 18:45:16.534082    2544 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 16 18:45:16.534237 ip-10-0-132-80 kubenswrapper[2544]: E0916 18:45:16.534126    2544 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/3e46e7aa-10f9-4c7d-bfa5-62504dc45cfd-kube-rbac-proxy-cm podName:3e46e7aa-10f9-4c7d-bfa5-62504dc45cfd nodeName:}" failed. No retries permitted until 2026-09-16 18:45:20.534107144 +0000 UTC m=+83.323109672 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/3e46e7aa-10f9-4c7d-bfa5-62504dc45cfd-kube-rbac-proxy-cm") pod "insights-runtime-extractor-xhj2z" (UID: "3e46e7aa-10f9-4c7d-bfa5-62504dc45cfd") : configmap "kube-rbac-proxy" not found
Sep 16 18:45:16.534237 ip-10-0-132-80 kubenswrapper[2544]: E0916 18:45:16.534220    2544 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3e46e7aa-10f9-4c7d-bfa5-62504dc45cfd-insights-runtime-extractor-tls podName:3e46e7aa-10f9-4c7d-bfa5-62504dc45cfd nodeName:}" failed. No retries permitted until 2026-09-16 18:45:20.534210869 +0000 UTC m=+83.323213380 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/3e46e7aa-10f9-4c7d-bfa5-62504dc45cfd-insights-runtime-extractor-tls") pod "insights-runtime-extractor-xhj2z" (UID: "3e46e7aa-10f9-4c7d-bfa5-62504dc45cfd") : secret "insights-runtime-extractor-tls" not found
Sep 16 18:45:16.542615 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:45:16.542591    2544 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-jmnkv_ee8d40ee-7c8a-496a-a4e4-56cdc3e0aaa2/serve-healthcheck-canary/0.log"
Sep 16 18:45:20.565363 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:45:20.565243    2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/3e46e7aa-10f9-4c7d-bfa5-62504dc45cfd-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-xhj2z\" (UID: \"3e46e7aa-10f9-4c7d-bfa5-62504dc45cfd\") " pod="openshift-insights/insights-runtime-extractor-xhj2z"
Sep 16 18:45:20.565363 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:45:20.565291    2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/3e46e7aa-10f9-4c7d-bfa5-62504dc45cfd-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-xhj2z\" (UID: \"3e46e7aa-10f9-4c7d-bfa5-62504dc45cfd\") " pod="openshift-insights/insights-runtime-extractor-xhj2z"
Sep 16 18:45:20.565830 ip-10-0-132-80 kubenswrapper[2544]: E0916 18:45:20.565379    2544 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 16 18:45:20.565830 ip-10-0-132-80 kubenswrapper[2544]: E0916 18:45:20.565399    2544 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 16 18:45:20.565830 ip-10-0-132-80 kubenswrapper[2544]: E0916 18:45:20.565464    2544 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3e46e7aa-10f9-4c7d-bfa5-62504dc45cfd-insights-runtime-extractor-tls podName:3e46e7aa-10f9-4c7d-bfa5-62504dc45cfd nodeName:}" failed. No retries permitted until 2026-09-16 18:45:28.5654499 +0000 UTC m=+91.354452428 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/3e46e7aa-10f9-4c7d-bfa5-62504dc45cfd-insights-runtime-extractor-tls") pod "insights-runtime-extractor-xhj2z" (UID: "3e46e7aa-10f9-4c7d-bfa5-62504dc45cfd") : secret "insights-runtime-extractor-tls" not found
Sep 16 18:45:20.565830 ip-10-0-132-80 kubenswrapper[2544]: E0916 18:45:20.565480    2544 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/3e46e7aa-10f9-4c7d-bfa5-62504dc45cfd-kube-rbac-proxy-cm podName:3e46e7aa-10f9-4c7d-bfa5-62504dc45cfd nodeName:}" failed. No retries permitted until 2026-09-16 18:45:28.565473218 +0000 UTC m=+91.354475730 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/3e46e7aa-10f9-4c7d-bfa5-62504dc45cfd-kube-rbac-proxy-cm") pod "insights-runtime-extractor-xhj2z" (UID: "3e46e7aa-10f9-4c7d-bfa5-62504dc45cfd") : configmap "kube-rbac-proxy" not found
Sep 16 18:45:21.776981 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:45:21.776944    2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/aa574cca-15b0-43dd-b30c-58d124d977d2-networking-console-plugin-cert\") pod \"networking-console-plugin-5dcbc7df6b-q4dbv\" (UID: \"aa574cca-15b0-43dd-b30c-58d124d977d2\") " pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-q4dbv"
Sep 16 18:45:21.777341 ip-10-0-132-80 kubenswrapper[2544]: E0916 18:45:21.777098    2544 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Sep 16 18:45:21.777341 ip-10-0-132-80 kubenswrapper[2544]: E0916 18:45:21.777165    2544 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/aa574cca-15b0-43dd-b30c-58d124d977d2-networking-console-plugin-cert podName:aa574cca-15b0-43dd-b30c-58d124d977d2 nodeName:}" failed. No retries permitted until 2026-09-16 18:45:37.777148677 +0000 UTC m=+100.566151205 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/aa574cca-15b0-43dd-b30c-58d124d977d2-networking-console-plugin-cert") pod "networking-console-plugin-5dcbc7df6b-q4dbv" (UID: "aa574cca-15b0-43dd-b30c-58d124d977d2") : secret "networking-console-plugin-cert" not found
Sep 16 18:45:28.632197 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:45:28.632151    2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/3e46e7aa-10f9-4c7d-bfa5-62504dc45cfd-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-xhj2z\" (UID: \"3e46e7aa-10f9-4c7d-bfa5-62504dc45cfd\") " pod="openshift-insights/insights-runtime-extractor-xhj2z"
Sep 16 18:45:28.632197 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:45:28.632202    2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/3e46e7aa-10f9-4c7d-bfa5-62504dc45cfd-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-xhj2z\" (UID: \"3e46e7aa-10f9-4c7d-bfa5-62504dc45cfd\") " pod="openshift-insights/insights-runtime-extractor-xhj2z"
Sep 16 18:45:28.632648 ip-10-0-132-80 kubenswrapper[2544]: E0916 18:45:28.632301    2544 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 16 18:45:28.632648 ip-10-0-132-80 kubenswrapper[2544]: E0916 18:45:28.632379    2544 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3e46e7aa-10f9-4c7d-bfa5-62504dc45cfd-insights-runtime-extractor-tls podName:3e46e7aa-10f9-4c7d-bfa5-62504dc45cfd nodeName:}" failed. No retries permitted until 2026-09-16 18:45:44.632363085 +0000 UTC m=+107.421365596 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/3e46e7aa-10f9-4c7d-bfa5-62504dc45cfd-insights-runtime-extractor-tls") pod "insights-runtime-extractor-xhj2z" (UID: "3e46e7aa-10f9-4c7d-bfa5-62504dc45cfd") : secret "insights-runtime-extractor-tls" not found
Sep 16 18:45:28.632722 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:45:28.632689    2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/3e46e7aa-10f9-4c7d-bfa5-62504dc45cfd-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-xhj2z\" (UID: \"3e46e7aa-10f9-4c7d-bfa5-62504dc45cfd\") " pod="openshift-insights/insights-runtime-extractor-xhj2z"
Sep 16 18:45:28.716627 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:45:28.716605    2544 scope.go:117] "RemoveContainer" containerID="ac95d772785093117506c3e1602208fe902bd9ec46f5646b88f8c5d47c1d8e5a"
Sep 16 18:45:29.082739 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:45:29.082711    2544 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-pfcb7_9e3f9e34-420e-43ef-96b4-c7cad6b2eaa9/console-operator/2.log"
Sep 16 18:45:29.083163 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:45:29.083149    2544 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-pfcb7_9e3f9e34-420e-43ef-96b4-c7cad6b2eaa9/console-operator/1.log"
Sep 16 18:45:29.083214 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:45:29.083183    2544 generic.go:356] "Generic (PLEG): container finished" podID="9e3f9e34-420e-43ef-96b4-c7cad6b2eaa9" containerID="0e9d7fd39602558562e958aad0b61e348b64f1e8bfb010cc289c924a0f49e5e9" exitCode=255
Sep 16 18:45:29.083248 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:45:29.083229    2544 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-696884fb9c-pfcb7" event={"ID":"9e3f9e34-420e-43ef-96b4-c7cad6b2eaa9","Type":"ContainerDied","Data":"0e9d7fd39602558562e958aad0b61e348b64f1e8bfb010cc289c924a0f49e5e9"}
Sep 16 18:45:29.083280 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:45:29.083272    2544 scope.go:117] "RemoveContainer" containerID="ac95d772785093117506c3e1602208fe902bd9ec46f5646b88f8c5d47c1d8e5a"
Sep 16 18:45:29.083612 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:45:29.083589    2544 scope.go:117] "RemoveContainer" containerID="0e9d7fd39602558562e958aad0b61e348b64f1e8bfb010cc289c924a0f49e5e9"
Sep 16 18:45:29.083810 ip-10-0-132-80 kubenswrapper[2544]: E0916 18:45:29.083785    2544 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 20s restarting failed container=console-operator pod=console-operator-696884fb9c-pfcb7_openshift-console-operator(9e3f9e34-420e-43ef-96b4-c7cad6b2eaa9)\"" pod="openshift-console-operator/console-operator-696884fb9c-pfcb7" podUID="9e3f9e34-420e-43ef-96b4-c7cad6b2eaa9"
Sep 16 18:45:30.087715 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:45:30.087686    2544 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-pfcb7_9e3f9e34-420e-43ef-96b4-c7cad6b2eaa9/console-operator/2.log"
Sep 16 18:45:35.480763 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:45:35.480666    2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/b03d56d8-b081-41fd-bac9-ba7da38f733a-registry-tls\") pod \"image-registry-76874c7675-4dhws\" (UID: \"b03d56d8-b081-41fd-bac9-ba7da38f733a\") " pod="openshift-image-registry/image-registry-76874c7675-4dhws"
Sep 16 18:45:35.482983 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:45:35.482960    2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/b03d56d8-b081-41fd-bac9-ba7da38f733a-registry-tls\") pod \"image-registry-76874c7675-4dhws\" (UID: \"b03d56d8-b081-41fd-bac9-ba7da38f733a\") " pod="openshift-image-registry/image-registry-76874c7675-4dhws"
Sep 16 18:45:35.518968 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:45:35.518934    2544 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-rd9rd\""
Sep 16 18:45:35.527108 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:45:35.527084    2544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-76874c7675-4dhws"
Sep 16 18:45:35.582255 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:45:35.582169    2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/b291cc26-ff6e-4adb-b219-8bd70a9aa3b2-metrics-tls\") pod \"dns-default-n8djq\" (UID: \"b291cc26-ff6e-4adb-b219-8bd70a9aa3b2\") " pod="openshift-dns/dns-default-n8djq"
Sep 16 18:45:35.584453 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:45:35.584433    2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/b291cc26-ff6e-4adb-b219-8bd70a9aa3b2-metrics-tls\") pod \"dns-default-n8djq\" (UID: \"b291cc26-ff6e-4adb-b219-8bd70a9aa3b2\") " pod="openshift-dns/dns-default-n8djq"
Sep 16 18:45:35.799821 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:45:35.799740    2544 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-76874c7675-4dhws"]
Sep 16 18:45:35.802525 ip-10-0-132-80 kubenswrapper[2544]: W0916 18:45:35.802497    2544 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb03d56d8_b081_41fd_bac9_ba7da38f733a.slice/crio-1ea2e77a794610d6355a62d061729ba524f0678192386319965ccdd3fd5698ad WatchSource:0}: Error finding container 1ea2e77a794610d6355a62d061729ba524f0678192386319965ccdd3fd5698ad: Status 404 returned error can't find the container with id 1ea2e77a794610d6355a62d061729ba524f0678192386319965ccdd3fd5698ad
Sep 16 18:45:35.832028 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:45:35.832003    2544 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-h7kfw\""
Sep 16 18:45:35.840111 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:45:35.840090    2544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-n8djq"
Sep 16 18:45:36.105239 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:45:36.105200    2544 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="openshift-console-operator/console-operator-696884fb9c-pfcb7"
Sep 16 18:45:36.105406 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:45:36.105248    2544 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-696884fb9c-pfcb7"
Sep 16 18:45:36.105600 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:45:36.105586    2544 scope.go:117] "RemoveContainer" containerID="0e9d7fd39602558562e958aad0b61e348b64f1e8bfb010cc289c924a0f49e5e9"
Sep 16 18:45:36.105672 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:45:36.105648    2544 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-76874c7675-4dhws" event={"ID":"b03d56d8-b081-41fd-bac9-ba7da38f733a","Type":"ContainerStarted","Data":"4e010777b617ab2a76883143bbebcea44bdb5943d989d9f1a06eb4635b380a0e"}
Sep 16 18:45:36.105712 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:45:36.105690    2544 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-76874c7675-4dhws" event={"ID":"b03d56d8-b081-41fd-bac9-ba7da38f733a","Type":"ContainerStarted","Data":"1ea2e77a794610d6355a62d061729ba524f0678192386319965ccdd3fd5698ad"}
Sep 16 18:45:36.105761 ip-10-0-132-80 kubenswrapper[2544]: E0916 18:45:36.105742    2544 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 20s restarting failed container=console-operator pod=console-operator-696884fb9c-pfcb7_openshift-console-operator(9e3f9e34-420e-43ef-96b4-c7cad6b2eaa9)\"" pod="openshift-console-operator/console-operator-696884fb9c-pfcb7" podUID="9e3f9e34-420e-43ef-96b4-c7cad6b2eaa9"
Sep 16 18:45:36.105796 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:45:36.105787    2544 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-76874c7675-4dhws"
Sep 16 18:45:36.108912 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:45:36.108881    2544 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-n8djq"]
Sep 16 18:45:36.112911 ip-10-0-132-80 kubenswrapper[2544]: W0916 18:45:36.112884    2544 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb291cc26_ff6e_4adb_b219_8bd70a9aa3b2.slice/crio-9a5f99b466c4166177038194159deeb83cc70b67ab5ab523ad67a77a87a869ba WatchSource:0}: Error finding container 9a5f99b466c4166177038194159deeb83cc70b67ab5ab523ad67a77a87a869ba: Status 404 returned error can't find the container with id 9a5f99b466c4166177038194159deeb83cc70b67ab5ab523ad67a77a87a869ba
Sep 16 18:45:36.150357 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:45:36.150308    2544 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/image-registry-76874c7675-4dhws" podStartSLOduration=98.15029295 podStartE2EDuration="1m38.15029295s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:43:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:45:36.149884603 +0000 UTC m=+98.938887163" watchObservedRunningTime="2026-09-16 18:45:36.15029295 +0000 UTC m=+98.939295483"
Sep 16 18:45:37.109833 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:45:37.109789    2544 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-n8djq" event={"ID":"b291cc26-ff6e-4adb-b219-8bd70a9aa3b2","Type":"ContainerStarted","Data":"9a5f99b466c4166177038194159deeb83cc70b67ab5ab523ad67a77a87a869ba"}
Sep 16 18:45:37.802523 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:45:37.801608    2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/aa574cca-15b0-43dd-b30c-58d124d977d2-networking-console-plugin-cert\") pod \"networking-console-plugin-5dcbc7df6b-q4dbv\" (UID: \"aa574cca-15b0-43dd-b30c-58d124d977d2\") " pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-q4dbv"
Sep 16 18:45:37.804548 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:45:37.804524    2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/aa574cca-15b0-43dd-b30c-58d124d977d2-networking-console-plugin-cert\") pod \"networking-console-plugin-5dcbc7df6b-q4dbv\" (UID: \"aa574cca-15b0-43dd-b30c-58d124d977d2\") " pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-q4dbv"
Sep 16 18:45:38.078990 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:45:38.078946    2544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-q4dbv"
Sep 16 18:45:38.115476 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:45:38.115441    2544 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-n8djq" event={"ID":"b291cc26-ff6e-4adb-b219-8bd70a9aa3b2","Type":"ContainerStarted","Data":"aa99aa7e7c61be249b99d222b0dedfd0f9270c53f9752ef8b7669cb8223ce5f2"}
Sep 16 18:45:38.115983 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:45:38.115502    2544 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-n8djq" event={"ID":"b291cc26-ff6e-4adb-b219-8bd70a9aa3b2","Type":"ContainerStarted","Data":"c21e32f6c8bee2d3a386ddd1b89a92ffdccc97e1de2d2c67d4df2a84ff5688ab"}
Sep 16 18:45:38.115983 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:45:38.115615    2544 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-n8djq"
Sep 16 18:45:38.136044 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:45:38.135998    2544 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-n8djq" podStartSLOduration=65.548357682 podStartE2EDuration="1m7.135981886s" totalImagesPullingTime="1.587624204s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:44:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:45:38.134723257 +0000 UTC m=+100.923725808" watchObservedRunningTime="2026-09-16 18:45:38.135981886 +0000 UTC m=+100.924984419"
Sep 16 18:45:38.347543 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:45:38.347464    2544 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-5dcbc7df6b-q4dbv"]
Sep 16 18:45:38.350834 ip-10-0-132-80 kubenswrapper[2544]: W0916 18:45:38.350803    2544 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaa574cca_15b0_43dd_b30c_58d124d977d2.slice/crio-b0c63110ea9d41d40623263b5a7bcb3bd88ee52b059f035b9f251039babed7af WatchSource:0}: Error finding container b0c63110ea9d41d40623263b5a7bcb3bd88ee52b059f035b9f251039babed7af: Status 404 returned error can't find the container with id b0c63110ea9d41d40623263b5a7bcb3bd88ee52b059f035b9f251039babed7af
Sep 16 18:45:39.120488 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:45:39.120455    2544 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-q4dbv" event={"ID":"aa574cca-15b0-43dd-b30c-58d124d977d2","Type":"ContainerStarted","Data":"b0c63110ea9d41d40623263b5a7bcb3bd88ee52b059f035b9f251039babed7af"}
Sep 16 18:45:39.145960 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:45:39.145913    2544 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-9bbd67cbd-8gfd9"]
Sep 16 18:45:39.150905 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:45:39.150879    2544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-9bbd67cbd-8gfd9"
Sep 16 18:45:39.153182 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:45:39.153160    2544 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"managed-serviceaccount-hub-kubeconfig\""
Sep 16 18:45:39.153895 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:45:39.153872    2544 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"managed-serviceaccount-dockercfg-tzhtd\""
Sep 16 18:45:39.154018 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:45:39.153933    2544 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"kube-root-ca.crt\""
Sep 16 18:45:39.157333 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:45:39.156995    2544 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"openshift-service-ca.crt\""
Sep 16 18:45:39.158879 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:45:39.158848    2544 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"open-cluster-management-image-pull-credentials\""
Sep 16 18:45:39.169299 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:45:39.169278    2544 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-9bbd67cbd-8gfd9"]
Sep 16 18:45:39.315088 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:45:39.315048    2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/5d7b6c59-bc28-447b-82d3-efeda9927161-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-9bbd67cbd-8gfd9\" (UID: \"5d7b6c59-bc28-447b-82d3-efeda9927161\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-9bbd67cbd-8gfd9"
Sep 16 18:45:39.315274 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:45:39.315152    2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f7rtf\" (UniqueName: \"kubernetes.io/projected/5d7b6c59-bc28-447b-82d3-efeda9927161-kube-api-access-f7rtf\") pod \"managed-serviceaccount-addon-agent-9bbd67cbd-8gfd9\" (UID: \"5d7b6c59-bc28-447b-82d3-efeda9927161\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-9bbd67cbd-8gfd9"
Sep 16 18:45:39.416056 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:45:39.416029    2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f7rtf\" (UniqueName: \"kubernetes.io/projected/5d7b6c59-bc28-447b-82d3-efeda9927161-kube-api-access-f7rtf\") pod \"managed-serviceaccount-addon-agent-9bbd67cbd-8gfd9\" (UID: \"5d7b6c59-bc28-447b-82d3-efeda9927161\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-9bbd67cbd-8gfd9"
Sep 16 18:45:39.416153 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:45:39.416084    2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/5d7b6c59-bc28-447b-82d3-efeda9927161-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-9bbd67cbd-8gfd9\" (UID: \"5d7b6c59-bc28-447b-82d3-efeda9927161\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-9bbd67cbd-8gfd9"
Sep 16 18:45:39.418212 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:45:39.418190    2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/5d7b6c59-bc28-447b-82d3-efeda9927161-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-9bbd67cbd-8gfd9\" (UID: \"5d7b6c59-bc28-447b-82d3-efeda9927161\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-9bbd67cbd-8gfd9"
Sep 16 18:45:39.428637 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:45:39.428608    2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-f7rtf\" (UniqueName: \"kubernetes.io/projected/5d7b6c59-bc28-447b-82d3-efeda9927161-kube-api-access-f7rtf\") pod \"managed-serviceaccount-addon-agent-9bbd67cbd-8gfd9\" (UID: \"5d7b6c59-bc28-447b-82d3-efeda9927161\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-9bbd67cbd-8gfd9"
Sep 16 18:45:39.473817 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:45:39.473794    2544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-9bbd67cbd-8gfd9"
Sep 16 18:45:39.743925 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:45:39.743847    2544 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-9bbd67cbd-8gfd9"]
Sep 16 18:45:39.747523 ip-10-0-132-80 kubenswrapper[2544]: W0916 18:45:39.747490    2544 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod5d7b6c59_bc28_447b_82d3_efeda9927161.slice/crio-42cea491d4cbfe931c52a922896ab0ae3f8e156546d99e3306a3c8ea2ce5895c WatchSource:0}: Error finding container 42cea491d4cbfe931c52a922896ab0ae3f8e156546d99e3306a3c8ea2ce5895c: Status 404 returned error can't find the container with id 42cea491d4cbfe931c52a922896ab0ae3f8e156546d99e3306a3c8ea2ce5895c
Sep 16 18:45:40.125066 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:45:40.125030    2544 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-q4dbv" event={"ID":"aa574cca-15b0-43dd-b30c-58d124d977d2","Type":"ContainerStarted","Data":"32e8eb6c50f0c5dd7e26d6caa71f71f998cb183086edd6b0b450fbdfae9ec43f"}
Sep 16 18:45:40.126224 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:45:40.126197    2544 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-9bbd67cbd-8gfd9" event={"ID":"5d7b6c59-bc28-447b-82d3-efeda9927161","Type":"ContainerStarted","Data":"42cea491d4cbfe931c52a922896ab0ae3f8e156546d99e3306a3c8ea2ce5895c"}
Sep 16 18:45:40.142807 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:45:40.142749    2544 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-q4dbv" podStartSLOduration=34.11050598 podStartE2EDuration="35.142729758s" totalImagesPullingTime="1.032223778s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:45:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:45:40.142643832 +0000 UTC m=+102.931646376" watchObservedRunningTime="2026-09-16 18:45:40.142729758 +0000 UTC m=+102.931732293"
Sep 16 18:45:41.023214 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:45:41.023182    2544 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-t8hx2"
Sep 16 18:45:43.137088 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:45:43.137012    2544 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-9bbd67cbd-8gfd9" event={"ID":"5d7b6c59-bc28-447b-82d3-efeda9927161","Type":"ContainerStarted","Data":"c142700e8b9dde15bbd473690e1451f4d4fdcf241e996567579595f52a42e301"}
Sep 16 18:45:43.152225 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:45:43.152179    2544 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-9bbd67cbd-8gfd9" podStartSLOduration=1.050444729 podStartE2EDuration="4.152163891s" totalImagesPullingTime="3.101719162s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:45:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:45:43.150667017 +0000 UTC m=+105.939669551" watchObservedRunningTime="2026-09-16 18:45:43.152163891 +0000 UTC m=+105.941166427"
Sep 16 18:45:44.657279 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:45:44.657222    2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/3e46e7aa-10f9-4c7d-bfa5-62504dc45cfd-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-xhj2z\" (UID: \"3e46e7aa-10f9-4c7d-bfa5-62504dc45cfd\") " pod="openshift-insights/insights-runtime-extractor-xhj2z"
Sep 16 18:45:44.659435 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:45:44.659415    2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/3e46e7aa-10f9-4c7d-bfa5-62504dc45cfd-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-xhj2z\" (UID: \"3e46e7aa-10f9-4c7d-bfa5-62504dc45cfd\") " pod="openshift-insights/insights-runtime-extractor-xhj2z"
Sep 16 18:45:44.929367 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:45:44.929283    2544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-xhj2z"
Sep 16 18:45:45.199862 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:45:45.198166    2544 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-xhj2z"]
Sep 16 18:45:45.203330 ip-10-0-132-80 kubenswrapper[2544]: W0916 18:45:45.203300    2544 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3e46e7aa_10f9_4c7d_bfa5_62504dc45cfd.slice/crio-46215e3a36138a2a64035a1530d3a4364aa4b90f703e4b9b1b4c404fb727a83f WatchSource:0}: Error finding container 46215e3a36138a2a64035a1530d3a4364aa4b90f703e4b9b1b4c404fb727a83f: Status 404 returned error can't find the container with id 46215e3a36138a2a64035a1530d3a4364aa4b90f703e4b9b1b4c404fb727a83f
Sep 16 18:45:45.920738 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:45:45.920698    2544 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-pk89k"]
Sep 16 18:45:45.923614 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:45:45.923591    2544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-pk89k"
Sep 16 18:45:45.925609 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:45:45.925584    2544 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-tls\""
Sep 16 18:45:45.925718 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:45:45.925653    2544 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-dockercfg-4f9tq\""
Sep 16 18:45:45.934181 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:45:45.934156    2544 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-pk89k"]
Sep 16 18:45:45.968177 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:45:45.968141    2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/837cae96-cf48-488d-8a96-e698f41de5df-tls-certificates\") pod \"prometheus-operator-admission-webhook-5864cd99d7-pk89k\" (UID: \"837cae96-cf48-488d-8a96-e698f41de5df\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-pk89k"
Sep 16 18:45:46.068493 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:45:46.068449    2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/837cae96-cf48-488d-8a96-e698f41de5df-tls-certificates\") pod \"prometheus-operator-admission-webhook-5864cd99d7-pk89k\" (UID: \"837cae96-cf48-488d-8a96-e698f41de5df\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-pk89k"
Sep 16 18:45:46.071536 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:45:46.071502    2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/837cae96-cf48-488d-8a96-e698f41de5df-tls-certificates\") pod \"prometheus-operator-admission-webhook-5864cd99d7-pk89k\" (UID: \"837cae96-cf48-488d-8a96-e698f41de5df\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-pk89k"
Sep 16 18:45:46.149384 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:45:46.149345    2544 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-xhj2z" event={"ID":"3e46e7aa-10f9-4c7d-bfa5-62504dc45cfd","Type":"ContainerStarted","Data":"5424c76c2c20c154e7732d805de4a4e8d15326e01146d13ad1ef4b79494a9fcd"}
Sep 16 18:45:46.149573 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:45:46.149417    2544 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-xhj2z" event={"ID":"3e46e7aa-10f9-4c7d-bfa5-62504dc45cfd","Type":"ContainerStarted","Data":"46215e3a36138a2a64035a1530d3a4364aa4b90f703e4b9b1b4c404fb727a83f"}
Sep 16 18:45:46.232605 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:45:46.232573    2544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-pk89k"
Sep 16 18:45:46.502678 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:45:46.502607    2544 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-pk89k"]
Sep 16 18:45:46.505518 ip-10-0-132-80 kubenswrapper[2544]: W0916 18:45:46.505492    2544 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod837cae96_cf48_488d_8a96_e698f41de5df.slice/crio-ae7db3d3cf09ae854999a88b51ef5ee4ad047acbbb558818b16274f3ce22dc2e WatchSource:0}: Error finding container ae7db3d3cf09ae854999a88b51ef5ee4ad047acbbb558818b16274f3ce22dc2e: Status 404 returned error can't find the container with id ae7db3d3cf09ae854999a88b51ef5ee4ad047acbbb558818b16274f3ce22dc2e
Sep 16 18:45:46.717001 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:45:46.716970    2544 scope.go:117] "RemoveContainer" containerID="0e9d7fd39602558562e958aad0b61e348b64f1e8bfb010cc289c924a0f49e5e9"
Sep 16 18:45:46.717142 ip-10-0-132-80 kubenswrapper[2544]: E0916 18:45:46.717118    2544 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 20s restarting failed container=console-operator pod=console-operator-696884fb9c-pfcb7_openshift-console-operator(9e3f9e34-420e-43ef-96b4-c7cad6b2eaa9)\"" pod="openshift-console-operator/console-operator-696884fb9c-pfcb7" podUID="9e3f9e34-420e-43ef-96b4-c7cad6b2eaa9"
Sep 16 18:45:47.155719 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:45:47.155660    2544 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-xhj2z" event={"ID":"3e46e7aa-10f9-4c7d-bfa5-62504dc45cfd","Type":"ContainerStarted","Data":"c8228945ab5b184afc0c7282c982d41ec8b03e4b2b7889097e1178ef43f461d2"}
Sep 16 18:45:47.157143 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:45:47.157117    2544 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-pk89k" event={"ID":"837cae96-cf48-488d-8a96-e698f41de5df","Type":"ContainerStarted","Data":"ae7db3d3cf09ae854999a88b51ef5ee4ad047acbbb558818b16274f3ce22dc2e"}
Sep 16 18:45:48.123283 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:45:48.123248    2544 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-n8djq"
Sep 16 18:45:48.164007 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:45:48.163966    2544 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-xhj2z" event={"ID":"3e46e7aa-10f9-4c7d-bfa5-62504dc45cfd","Type":"ContainerStarted","Data":"323d129b968bc2d59413e6d1474f3b05a9b846da3322f37f63f093dde354f406"}
Sep 16 18:45:48.182824 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:45:48.182536    2544 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-xhj2z" podStartSLOduration=33.898678003 podStartE2EDuration="36.182517416s" totalImagesPullingTime="2.283839413s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-16 18:45:12 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:45:48.181846724 +0000 UTC m=+110.970849258" watchObservedRunningTime="2026-09-16 18:45:48.182517416 +0000 UTC m=+110.971519951"
Sep 16 18:45:49.172678 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:45:49.172640    2544 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-pk89k" event={"ID":"837cae96-cf48-488d-8a96-e698f41de5df","Type":"ContainerStarted","Data":"e04b7669f3faef3ef951b275962ad2cc927e18be37e585956859790c8400d6c1"}
Sep 16 18:45:49.173065 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:45:49.172872    2544 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-pk89k"
Sep 16 18:45:49.177390 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:45:49.177360    2544 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-pk89k"
Sep 16 18:45:49.190325 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:45:49.190284    2544 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-pk89k" podStartSLOduration=2.408666485 podStartE2EDuration="4.190271968s" totalImagesPullingTime="1.781605483s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:45:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:45:49.188950276 +0000 UTC m=+111.977952829" watchObservedRunningTime="2026-09-16 18:45:49.190271968 +0000 UTC m=+111.979274503"
Sep 16 18:45:54.308087 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:45:54.308056    2544 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/openshift-state-metrics-67dcf8c979-694l8"]
Sep 16 18:45:54.312316 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:45:54.312299    2544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-694l8"
Sep 16 18:45:54.314906 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:45:54.314879    2544 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
Sep 16 18:45:54.314906 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:45:54.314896    2544 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-kube-rbac-proxy-config\""
Sep 16 18:45:54.315144 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:45:54.315121    2544 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-dockercfg-w8cfp\""
Sep 16 18:45:54.315188 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:45:54.315147    2544 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
Sep 16 18:45:54.315823 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:45:54.315803    2544 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-tls\""
Sep 16 18:45:54.315915 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:45:54.315875    2544 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\""
Sep 16 18:45:54.320723 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:45:54.320700    2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/ce1ddaa9-0566-4f7a-bc19-e7a7b958ae22-metrics-client-ca\") pod \"openshift-state-metrics-67dcf8c979-694l8\" (UID: \"ce1ddaa9-0566-4f7a-bc19-e7a7b958ae22\") " pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-694l8"
Sep 16 18:45:54.320817 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:45:54.320731    2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/ce1ddaa9-0566-4f7a-bc19-e7a7b958ae22-openshift-state-metrics-tls\") pod \"openshift-state-metrics-67dcf8c979-694l8\" (UID: \"ce1ddaa9-0566-4f7a-bc19-e7a7b958ae22\") " pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-694l8"
Sep 16 18:45:54.320817 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:45:54.320754    2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/ce1ddaa9-0566-4f7a-bc19-e7a7b958ae22-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-67dcf8c979-694l8\" (UID: \"ce1ddaa9-0566-4f7a-bc19-e7a7b958ae22\") " pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-694l8"
Sep 16 18:45:54.320904 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:45:54.320813    2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qcgpw\" (UniqueName: \"kubernetes.io/projected/ce1ddaa9-0566-4f7a-bc19-e7a7b958ae22-kube-api-access-qcgpw\") pod \"openshift-state-metrics-67dcf8c979-694l8\" (UID: \"ce1ddaa9-0566-4f7a-bc19-e7a7b958ae22\") " pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-694l8"
Sep 16 18:45:54.324116 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:45:54.324094    2544 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-67dcf8c979-694l8"]
Sep 16 18:45:54.335951 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:45:54.335931    2544 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/kube-state-metrics-59fbdd77d-2fwf6"]
Sep 16 18:45:54.339726 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:45:54.339710    2544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-59fbdd77d-2fwf6"
Sep 16 18:45:54.342322 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:45:54.342301    2544 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-custom-resource-state-configmap\""
Sep 16 18:45:54.342420 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:45:54.342389    2544 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-tls\""
Sep 16 18:45:54.342742 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:45:54.342728    2544 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-kube-rbac-proxy-config\""
Sep 16 18:45:54.344418 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:45:54.344401    2544 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-dockercfg-zkh7l\""
Sep 16 18:45:54.352044 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:45:54.352025    2544 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-sljjd"]
Sep 16 18:45:54.355737 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:45:54.355718    2544 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-59fbdd77d-2fwf6"]
Sep 16 18:45:54.355846 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:45:54.355832    2544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-sljjd"
Sep 16 18:45:54.357919 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:45:54.357898    2544 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\""
Sep 16 18:45:54.358025 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:45:54.357929    2544 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\""
Sep 16 18:45:54.358025 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:45:54.357956    2544 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\""
Sep 16 18:45:54.358174 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:45:54.358159    2544 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-9k2g7\""
Sep 16 18:45:54.421489 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:45:54.421464    2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/8b245d92-ed58-4240-8b0d-7dab57750690-root\") pod \"node-exporter-sljjd\" (UID: \"8b245d92-ed58-4240-8b0d-7dab57750690\") " pod="openshift-monitoring/node-exporter-sljjd"
Sep 16 18:45:54.421602 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:45:54.421491    2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7vgtg\" (UniqueName: \"kubernetes.io/projected/8b245d92-ed58-4240-8b0d-7dab57750690-kube-api-access-7vgtg\") pod \"node-exporter-sljjd\" (UID: \"8b245d92-ed58-4240-8b0d-7dab57750690\") " pod="openshift-monitoring/node-exporter-sljjd"
Sep 16 18:45:54.421891 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:45:54.421869    2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/8b245d92-ed58-4240-8b0d-7dab57750690-node-exporter-tls\") pod \"node-exporter-sljjd\" (UID: \"8b245d92-ed58-4240-8b0d-7dab57750690\") " pod="openshift-monitoring/node-exporter-sljjd"
Sep 16 18:45:54.421972 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:45:54.421924    2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/ce1ddaa9-0566-4f7a-bc19-e7a7b958ae22-metrics-client-ca\") pod \"openshift-state-metrics-67dcf8c979-694l8\" (UID: \"ce1ddaa9-0566-4f7a-bc19-e7a7b958ae22\") " pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-694l8"
Sep 16 18:45:54.421972 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:45:54.421960    2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/8b245d92-ed58-4240-8b0d-7dab57750690-sys\") pod \"node-exporter-sljjd\" (UID: \"8b245d92-ed58-4240-8b0d-7dab57750690\") " pod="openshift-monitoring/node-exporter-sljjd"
Sep 16 18:45:54.422073 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:45:54.422034    2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/8b245d92-ed58-4240-8b0d-7dab57750690-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-sljjd\" (UID: \"8b245d92-ed58-4240-8b0d-7dab57750690\") " pod="openshift-monitoring/node-exporter-sljjd"
Sep 16 18:45:54.422127 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:45:54.422084    2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/2b942120-fb59-49f7-a96f-3f7165b19110-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-59fbdd77d-2fwf6\" (UID: \"2b942120-fb59-49f7-a96f-3f7165b19110\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-2fwf6"
Sep 16 18:45:54.422127 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:45:54.422118    2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/ce1ddaa9-0566-4f7a-bc19-e7a7b958ae22-openshift-state-metrics-tls\") pod \"openshift-state-metrics-67dcf8c979-694l8\" (UID: \"ce1ddaa9-0566-4f7a-bc19-e7a7b958ae22\") " pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-694l8"
Sep 16 18:45:54.422219 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:45:54.422156    2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/8b245d92-ed58-4240-8b0d-7dab57750690-node-exporter-accelerators-collector-config\") pod \"node-exporter-sljjd\" (UID: \"8b245d92-ed58-4240-8b0d-7dab57750690\") " pod="openshift-monitoring/node-exporter-sljjd"
Sep 16 18:45:54.422219 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:45:54.422211    2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/ce1ddaa9-0566-4f7a-bc19-e7a7b958ae22-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-67dcf8c979-694l8\" (UID: \"ce1ddaa9-0566-4f7a-bc19-e7a7b958ae22\") " pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-694l8"
Sep 16 18:45:54.422315 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:45:54.422269    2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/2b942120-fb59-49f7-a96f-3f7165b19110-kube-state-metrics-tls\") pod \"kube-state-metrics-59fbdd77d-2fwf6\" (UID: \"2b942120-fb59-49f7-a96f-3f7165b19110\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-2fwf6"
Sep 16 18:45:54.422315 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:45:54.422302    2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/2b942120-fb59-49f7-a96f-3f7165b19110-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-59fbdd77d-2fwf6\" (UID: \"2b942120-fb59-49f7-a96f-3f7165b19110\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-2fwf6"
Sep 16 18:45:54.422403 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:45:54.422343    2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/8b245d92-ed58-4240-8b0d-7dab57750690-node-exporter-wtmp\") pod \"node-exporter-sljjd\" (UID: \"8b245d92-ed58-4240-8b0d-7dab57750690\") " pod="openshift-monitoring/node-exporter-sljjd"
Sep 16 18:45:54.422403 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:45:54.422388    2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/2b942120-fb59-49f7-a96f-3f7165b19110-volume-directive-shadow\") pod \"kube-state-metrics-59fbdd77d-2fwf6\" (UID: \"2b942120-fb59-49f7-a96f-3f7165b19110\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-2fwf6"
Sep 16 18:45:54.422493 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:45:54.422457    2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qcgpw\" (UniqueName: \"kubernetes.io/projected/ce1ddaa9-0566-4f7a-bc19-e7a7b958ae22-kube-api-access-qcgpw\") pod \"openshift-state-metrics-67dcf8c979-694l8\" (UID: \"ce1ddaa9-0566-4f7a-bc19-e7a7b958ae22\") " pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-694l8"
Sep 16 18:45:54.422539 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:45:54.422495    2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/2b942120-fb59-49f7-a96f-3f7165b19110-metrics-client-ca\") pod \"kube-state-metrics-59fbdd77d-2fwf6\" (UID: \"2b942120-fb59-49f7-a96f-3f7165b19110\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-2fwf6"
Sep 16 18:45:54.422604 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:45:54.422582    2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/8b245d92-ed58-4240-8b0d-7dab57750690-metrics-client-ca\") pod \"node-exporter-sljjd\" (UID: \"8b245d92-ed58-4240-8b0d-7dab57750690\") " pod="openshift-monitoring/node-exporter-sljjd"
Sep 16 18:45:54.422661 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:45:54.422622    2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k2kws\" (UniqueName: \"kubernetes.io/projected/2b942120-fb59-49f7-a96f-3f7165b19110-kube-api-access-k2kws\") pod \"kube-state-metrics-59fbdd77d-2fwf6\" (UID: \"2b942120-fb59-49f7-a96f-3f7165b19110\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-2fwf6"
Sep 16 18:45:54.422711 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:45:54.422658    2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/8b245d92-ed58-4240-8b0d-7dab57750690-node-exporter-textfile\") pod \"node-exporter-sljjd\" (UID: \"8b245d92-ed58-4240-8b0d-7dab57750690\") " pod="openshift-monitoring/node-exporter-sljjd"
Sep 16 18:45:54.422884 ip-10-0-132-80 kubenswrapper[2544]: E0916 18:45:54.422864    2544 secret.go:189] Couldn't get secret openshift-monitoring/openshift-state-metrics-tls: secret "openshift-state-metrics-tls" not found
Sep 16 18:45:54.423032 ip-10-0-132-80 kubenswrapper[2544]: E0916 18:45:54.423021    2544 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ce1ddaa9-0566-4f7a-bc19-e7a7b958ae22-openshift-state-metrics-tls podName:ce1ddaa9-0566-4f7a-bc19-e7a7b958ae22 nodeName:}" failed. No retries permitted until 2026-09-16 18:45:54.922995346 +0000 UTC m=+117.711997862 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "openshift-state-metrics-tls" (UniqueName: "kubernetes.io/secret/ce1ddaa9-0566-4f7a-bc19-e7a7b958ae22-openshift-state-metrics-tls") pod "openshift-state-metrics-67dcf8c979-694l8" (UID: "ce1ddaa9-0566-4f7a-bc19-e7a7b958ae22") : secret "openshift-state-metrics-tls" not found
Sep 16 18:45:54.423225 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:45:54.423199    2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/ce1ddaa9-0566-4f7a-bc19-e7a7b958ae22-metrics-client-ca\") pod \"openshift-state-metrics-67dcf8c979-694l8\" (UID: \"ce1ddaa9-0566-4f7a-bc19-e7a7b958ae22\") " pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-694l8"
Sep 16 18:45:54.426061 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:45:54.426036    2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/ce1ddaa9-0566-4f7a-bc19-e7a7b958ae22-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-67dcf8c979-694l8\" (UID: \"ce1ddaa9-0566-4f7a-bc19-e7a7b958ae22\") " pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-694l8"
Sep 16 18:45:54.434336 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:45:54.434316    2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qcgpw\" (UniqueName: \"kubernetes.io/projected/ce1ddaa9-0566-4f7a-bc19-e7a7b958ae22-kube-api-access-qcgpw\") pod \"openshift-state-metrics-67dcf8c979-694l8\" (UID: \"ce1ddaa9-0566-4f7a-bc19-e7a7b958ae22\") " pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-694l8"
Sep 16 18:45:54.523066 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:45:54.523035    2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/8b245d92-ed58-4240-8b0d-7dab57750690-node-exporter-tls\") pod \"node-exporter-sljjd\" (UID: \"8b245d92-ed58-4240-8b0d-7dab57750690\") " pod="openshift-monitoring/node-exporter-sljjd"
Sep 16 18:45:54.523226 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:45:54.523074    2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/8b245d92-ed58-4240-8b0d-7dab57750690-sys\") pod \"node-exporter-sljjd\" (UID: \"8b245d92-ed58-4240-8b0d-7dab57750690\") " pod="openshift-monitoring/node-exporter-sljjd"
Sep 16 18:45:54.523226 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:45:54.523097    2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/8b245d92-ed58-4240-8b0d-7dab57750690-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-sljjd\" (UID: \"8b245d92-ed58-4240-8b0d-7dab57750690\") " pod="openshift-monitoring/node-exporter-sljjd"
Sep 16 18:45:54.523226 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:45:54.523127    2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/2b942120-fb59-49f7-a96f-3f7165b19110-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-59fbdd77d-2fwf6\" (UID: \"2b942120-fb59-49f7-a96f-3f7165b19110\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-2fwf6"
Sep 16 18:45:54.523405 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:45:54.523216    2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/8b245d92-ed58-4240-8b0d-7dab57750690-node-exporter-accelerators-collector-config\") pod \"node-exporter-sljjd\" (UID: \"8b245d92-ed58-4240-8b0d-7dab57750690\") " pod="openshift-monitoring/node-exporter-sljjd"
Sep 16 18:45:54.523405 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:45:54.523319    2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/2b942120-fb59-49f7-a96f-3f7165b19110-kube-state-metrics-tls\") pod \"kube-state-metrics-59fbdd77d-2fwf6\" (UID: \"2b942120-fb59-49f7-a96f-3f7165b19110\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-2fwf6"
Sep 16 18:45:54.523405 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:45:54.523319    2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/8b245d92-ed58-4240-8b0d-7dab57750690-sys\") pod \"node-exporter-sljjd\" (UID: \"8b245d92-ed58-4240-8b0d-7dab57750690\") " pod="openshift-monitoring/node-exporter-sljjd"
Sep 16 18:45:54.523405 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:45:54.523346    2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/2b942120-fb59-49f7-a96f-3f7165b19110-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-59fbdd77d-2fwf6\" (UID: \"2b942120-fb59-49f7-a96f-3f7165b19110\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-2fwf6"
Sep 16 18:45:54.523405 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:45:54.523393    2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/8b245d92-ed58-4240-8b0d-7dab57750690-node-exporter-wtmp\") pod \"node-exporter-sljjd\" (UID: \"8b245d92-ed58-4240-8b0d-7dab57750690\") " pod="openshift-monitoring/node-exporter-sljjd"
Sep 16 18:45:54.523667 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:45:54.523423    2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/2b942120-fb59-49f7-a96f-3f7165b19110-volume-directive-shadow\") pod \"kube-state-metrics-59fbdd77d-2fwf6\" (UID: \"2b942120-fb59-49f7-a96f-3f7165b19110\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-2fwf6"
Sep 16 18:45:54.523667 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:45:54.523470    2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/2b942120-fb59-49f7-a96f-3f7165b19110-metrics-client-ca\") pod \"kube-state-metrics-59fbdd77d-2fwf6\" (UID: \"2b942120-fb59-49f7-a96f-3f7165b19110\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-2fwf6"
Sep 16 18:45:54.523667 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:45:54.523509    2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/8b245d92-ed58-4240-8b0d-7dab57750690-metrics-client-ca\") pod \"node-exporter-sljjd\" (UID: \"8b245d92-ed58-4240-8b0d-7dab57750690\") " pod="openshift-monitoring/node-exporter-sljjd"
Sep 16 18:45:54.523667 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:45:54.523541    2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k2kws\" (UniqueName: \"kubernetes.io/projected/2b942120-fb59-49f7-a96f-3f7165b19110-kube-api-access-k2kws\") pod \"kube-state-metrics-59fbdd77d-2fwf6\" (UID: \"2b942120-fb59-49f7-a96f-3f7165b19110\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-2fwf6"
Sep 16 18:45:54.523667 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:45:54.523591    2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/8b245d92-ed58-4240-8b0d-7dab57750690-node-exporter-textfile\") pod \"node-exporter-sljjd\" (UID: \"8b245d92-ed58-4240-8b0d-7dab57750690\") " pod="openshift-monitoring/node-exporter-sljjd"
Sep 16 18:45:54.523667 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:45:54.523632    2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/8b245d92-ed58-4240-8b0d-7dab57750690-root\") pod \"node-exporter-sljjd\" (UID: \"8b245d92-ed58-4240-8b0d-7dab57750690\") " pod="openshift-monitoring/node-exporter-sljjd"
Sep 16 18:45:54.523667 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:45:54.523657    2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7vgtg\" (UniqueName: \"kubernetes.io/projected/8b245d92-ed58-4240-8b0d-7dab57750690-kube-api-access-7vgtg\") pod \"node-exporter-sljjd\" (UID: \"8b245d92-ed58-4240-8b0d-7dab57750690\") " pod="openshift-monitoring/node-exporter-sljjd"
Sep 16 18:45:54.524019 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:45:54.523995    2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/8b245d92-ed58-4240-8b0d-7dab57750690-node-exporter-accelerators-collector-config\") pod \"node-exporter-sljjd\" (UID: \"8b245d92-ed58-4240-8b0d-7dab57750690\") " pod="openshift-monitoring/node-exporter-sljjd"
Sep 16 18:45:54.524123 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:45:54.524096    2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/2b942120-fb59-49f7-a96f-3f7165b19110-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-59fbdd77d-2fwf6\" (UID: \"2b942120-fb59-49f7-a96f-3f7165b19110\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-2fwf6"
Sep 16 18:45:54.524263 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:45:54.524240    2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/8b245d92-ed58-4240-8b0d-7dab57750690-metrics-client-ca\") pod \"node-exporter-sljjd\" (UID: \"8b245d92-ed58-4240-8b0d-7dab57750690\") " pod="openshift-monitoring/node-exporter-sljjd"
Sep 16 18:45:54.524326 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:45:54.524274    2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/8b245d92-ed58-4240-8b0d-7dab57750690-root\") pod \"node-exporter-sljjd\" (UID: \"8b245d92-ed58-4240-8b0d-7dab57750690\") " pod="openshift-monitoring/node-exporter-sljjd"
Sep 16 18:45:54.524387 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:45:54.524330    2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/2b942120-fb59-49f7-a96f-3f7165b19110-volume-directive-shadow\") pod \"kube-state-metrics-59fbdd77d-2fwf6\" (UID: \"2b942120-fb59-49f7-a96f-3f7165b19110\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-2fwf6"
Sep 16 18:45:54.524602 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:45:54.524580    2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/8b245d92-ed58-4240-8b0d-7dab57750690-node-exporter-wtmp\") pod \"node-exporter-sljjd\" (UID: \"8b245d92-ed58-4240-8b0d-7dab57750690\") " pod="openshift-monitoring/node-exporter-sljjd"
Sep 16 18:45:54.524763 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:45:54.524739    2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/8b245d92-ed58-4240-8b0d-7dab57750690-node-exporter-textfile\") pod \"node-exporter-sljjd\" (UID: \"8b245d92-ed58-4240-8b0d-7dab57750690\") " pod="openshift-monitoring/node-exporter-sljjd"
Sep 16 18:45:54.525264 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:45:54.525242    2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/2b942120-fb59-49f7-a96f-3f7165b19110-metrics-client-ca\") pod \"kube-state-metrics-59fbdd77d-2fwf6\" (UID: \"2b942120-fb59-49f7-a96f-3f7165b19110\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-2fwf6"
Sep 16 18:45:54.526070 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:45:54.526048    2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/8b245d92-ed58-4240-8b0d-7dab57750690-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-sljjd\" (UID: \"8b245d92-ed58-4240-8b0d-7dab57750690\") " pod="openshift-monitoring/node-exporter-sljjd"
Sep 16 18:45:54.526165 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:45:54.526122    2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/8b245d92-ed58-4240-8b0d-7dab57750690-node-exporter-tls\") pod \"node-exporter-sljjd\" (UID: \"8b245d92-ed58-4240-8b0d-7dab57750690\") " pod="openshift-monitoring/node-exporter-sljjd"
Sep 16 18:45:54.526290 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:45:54.526272    2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/2b942120-fb59-49f7-a96f-3f7165b19110-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-59fbdd77d-2fwf6\" (UID: \"2b942120-fb59-49f7-a96f-3f7165b19110\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-2fwf6"
Sep 16 18:45:54.526352 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:45:54.526317    2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/2b942120-fb59-49f7-a96f-3f7165b19110-kube-state-metrics-tls\") pod \"kube-state-metrics-59fbdd77d-2fwf6\" (UID: \"2b942120-fb59-49f7-a96f-3f7165b19110\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-2fwf6"
Sep 16 18:45:54.537291 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:45:54.537271    2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7vgtg\" (UniqueName: \"kubernetes.io/projected/8b245d92-ed58-4240-8b0d-7dab57750690-kube-api-access-7vgtg\") pod \"node-exporter-sljjd\" (UID: \"8b245d92-ed58-4240-8b0d-7dab57750690\") " pod="openshift-monitoring/node-exporter-sljjd"
Sep 16 18:45:54.537700 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:45:54.537680    2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-k2kws\" (UniqueName: \"kubernetes.io/projected/2b942120-fb59-49f7-a96f-3f7165b19110-kube-api-access-k2kws\") pod \"kube-state-metrics-59fbdd77d-2fwf6\" (UID: \"2b942120-fb59-49f7-a96f-3f7165b19110\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-2fwf6"
Sep 16 18:45:54.647874 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:45:54.647846    2544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-59fbdd77d-2fwf6"
Sep 16 18:45:54.663819 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:45:54.663799    2544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-sljjd"
Sep 16 18:45:54.675455 ip-10-0-132-80 kubenswrapper[2544]: W0916 18:45:54.675426    2544 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8b245d92_ed58_4240_8b0d_7dab57750690.slice/crio-fe7013700782fda7359415b41d38800e06fa9496cdff1dbdafeb52b0286afe24 WatchSource:0}: Error finding container fe7013700782fda7359415b41d38800e06fa9496cdff1dbdafeb52b0286afe24: Status 404 returned error can't find the container with id fe7013700782fda7359415b41d38800e06fa9496cdff1dbdafeb52b0286afe24
Sep 16 18:45:54.926623 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:45:54.926513    2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/ce1ddaa9-0566-4f7a-bc19-e7a7b958ae22-openshift-state-metrics-tls\") pod \"openshift-state-metrics-67dcf8c979-694l8\" (UID: \"ce1ddaa9-0566-4f7a-bc19-e7a7b958ae22\") " pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-694l8"
Sep 16 18:45:54.928851 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:45:54.928830    2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/ce1ddaa9-0566-4f7a-bc19-e7a7b958ae22-openshift-state-metrics-tls\") pod \"openshift-state-metrics-67dcf8c979-694l8\" (UID: \"ce1ddaa9-0566-4f7a-bc19-e7a7b958ae22\") " pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-694l8"
Sep 16 18:45:54.934891 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:45:54.934862    2544 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-59fbdd77d-2fwf6"]
Sep 16 18:45:54.937598 ip-10-0-132-80 kubenswrapper[2544]: W0916 18:45:54.937573    2544 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2b942120_fb59_49f7_a96f_3f7165b19110.slice/crio-6a33c592f0283e16523de518971628ff0bfa078d24e53f1593f0e22c7874750b WatchSource:0}: Error finding container 6a33c592f0283e16523de518971628ff0bfa078d24e53f1593f0e22c7874750b: Status 404 returned error can't find the container with id 6a33c592f0283e16523de518971628ff0bfa078d24e53f1593f0e22c7874750b
Sep 16 18:45:55.192764 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:45:55.192670    2544 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-sljjd" event={"ID":"8b245d92-ed58-4240-8b0d-7dab57750690","Type":"ContainerStarted","Data":"fe7013700782fda7359415b41d38800e06fa9496cdff1dbdafeb52b0286afe24"}
Sep 16 18:45:55.194228 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:45:55.194199    2544 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-59fbdd77d-2fwf6" event={"ID":"2b942120-fb59-49f7-a96f-3f7165b19110","Type":"ContainerStarted","Data":"6a33c592f0283e16523de518971628ff0bfa078d24e53f1593f0e22c7874750b"}
Sep 16 18:45:55.221267 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:45:55.221243    2544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-694l8"
Sep 16 18:45:55.500181 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:45:55.500105    2544 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-67dcf8c979-694l8"]
Sep 16 18:45:55.526394 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:45:55.526362    2544 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Sep 16 18:45:55.531990 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:45:55.531964    2544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Sep 16 18:45:55.534170 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:45:55.534146    2544 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\""
Sep 16 18:45:55.534287 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:45:55.534267    2544 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\""
Sep 16 18:45:55.534408 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:45:55.534390    2544 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\""
Sep 16 18:45:55.534644 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:45:55.534626    2544 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\""
Sep 16 18:45:55.534725 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:45:55.534666    2544 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\""
Sep 16 18:45:55.534725 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:45:55.534711    2544 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\""
Sep 16 18:45:55.534998 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:45:55.534969    2544 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\""
Sep 16 18:45:55.534998 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:45:55.534987    2544 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\""
Sep 16 18:45:55.535148 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:45:55.535109    2544 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-nlphx\""
Sep 16 18:45:55.535288 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:45:55.535264    2544 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\""
Sep 16 18:45:55.546625 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:45:55.546522    2544 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Sep 16 18:45:55.594527 ip-10-0-132-80 kubenswrapper[2544]: W0916 18:45:55.594493    2544 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podce1ddaa9_0566_4f7a_bc19_e7a7b958ae22.slice/crio-115f61bb62ef0d48410a0a53d2a131ba0a5fa655b560ca99918e8c2a62a9c6d0 WatchSource:0}: Error finding container 115f61bb62ef0d48410a0a53d2a131ba0a5fa655b560ca99918e8c2a62a9c6d0: Status 404 returned error can't find the container with id 115f61bb62ef0d48410a0a53d2a131ba0a5fa655b560ca99918e8c2a62a9c6d0
Sep 16 18:45:55.631922 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:45:55.631895    2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bgjh4\" (UniqueName: \"kubernetes.io/projected/7a642ca2-3fee-4a4c-8c92-3f3069a99ac5-kube-api-access-bgjh4\") pod \"alertmanager-main-0\" (UID: \"7a642ca2-3fee-4a4c-8c92-3f3069a99ac5\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 18:45:55.632053 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:45:55.631934    2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/7a642ca2-3fee-4a4c-8c92-3f3069a99ac5-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"7a642ca2-3fee-4a4c-8c92-3f3069a99ac5\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 18:45:55.632053 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:45:55.631960    2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/7a642ca2-3fee-4a4c-8c92-3f3069a99ac5-config-volume\") pod \"alertmanager-main-0\" (UID: \"7a642ca2-3fee-4a4c-8c92-3f3069a99ac5\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 18:45:55.632053 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:45:55.631986    2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/7a642ca2-3fee-4a4c-8c92-3f3069a99ac5-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"7a642ca2-3fee-4a4c-8c92-3f3069a99ac5\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 18:45:55.632053 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:45:55.632013    2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/7a642ca2-3fee-4a4c-8c92-3f3069a99ac5-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"7a642ca2-3fee-4a4c-8c92-3f3069a99ac5\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 18:45:55.632053 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:45:55.632041    2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/7a642ca2-3fee-4a4c-8c92-3f3069a99ac5-tls-assets\") pod \"alertmanager-main-0\" (UID: \"7a642ca2-3fee-4a4c-8c92-3f3069a99ac5\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 18:45:55.632329 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:45:55.632107    2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/7a642ca2-3fee-4a4c-8c92-3f3069a99ac5-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"7a642ca2-3fee-4a4c-8c92-3f3069a99ac5\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 18:45:55.632329 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:45:55.632133    2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7a642ca2-3fee-4a4c-8c92-3f3069a99ac5-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"7a642ca2-3fee-4a4c-8c92-3f3069a99ac5\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 18:45:55.632329 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:45:55.632223    2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/7a642ca2-3fee-4a4c-8c92-3f3069a99ac5-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"7a642ca2-3fee-4a4c-8c92-3f3069a99ac5\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 18:45:55.632329 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:45:55.632265    2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/7a642ca2-3fee-4a4c-8c92-3f3069a99ac5-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"7a642ca2-3fee-4a4c-8c92-3f3069a99ac5\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 18:45:55.632329 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:45:55.632310    2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/7a642ca2-3fee-4a4c-8c92-3f3069a99ac5-web-config\") pod \"alertmanager-main-0\" (UID: \"7a642ca2-3fee-4a4c-8c92-3f3069a99ac5\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 18:45:55.632597 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:45:55.632338    2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/7a642ca2-3fee-4a4c-8c92-3f3069a99ac5-config-out\") pod \"alertmanager-main-0\" (UID: \"7a642ca2-3fee-4a4c-8c92-3f3069a99ac5\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 18:45:55.632597 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:45:55.632362    2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/7a642ca2-3fee-4a4c-8c92-3f3069a99ac5-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"7a642ca2-3fee-4a4c-8c92-3f3069a99ac5\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 18:45:55.733270 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:45:55.733241    2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/7a642ca2-3fee-4a4c-8c92-3f3069a99ac5-web-config\") pod \"alertmanager-main-0\" (UID: \"7a642ca2-3fee-4a4c-8c92-3f3069a99ac5\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 18:45:55.733391 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:45:55.733287    2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/7a642ca2-3fee-4a4c-8c92-3f3069a99ac5-config-out\") pod \"alertmanager-main-0\" (UID: \"7a642ca2-3fee-4a4c-8c92-3f3069a99ac5\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 18:45:55.733391 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:45:55.733312    2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/7a642ca2-3fee-4a4c-8c92-3f3069a99ac5-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"7a642ca2-3fee-4a4c-8c92-3f3069a99ac5\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 18:45:55.733391 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:45:55.733365    2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bgjh4\" (UniqueName: \"kubernetes.io/projected/7a642ca2-3fee-4a4c-8c92-3f3069a99ac5-kube-api-access-bgjh4\") pod \"alertmanager-main-0\" (UID: \"7a642ca2-3fee-4a4c-8c92-3f3069a99ac5\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 18:45:55.733525 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:45:55.733390    2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/7a642ca2-3fee-4a4c-8c92-3f3069a99ac5-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"7a642ca2-3fee-4a4c-8c92-3f3069a99ac5\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 18:45:55.733525 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:45:55.733415    2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/7a642ca2-3fee-4a4c-8c92-3f3069a99ac5-config-volume\") pod \"alertmanager-main-0\" (UID: \"7a642ca2-3fee-4a4c-8c92-3f3069a99ac5\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 18:45:55.733525 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:45:55.733440    2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/7a642ca2-3fee-4a4c-8c92-3f3069a99ac5-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"7a642ca2-3fee-4a4c-8c92-3f3069a99ac5\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 18:45:55.733525 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:45:55.733515    2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/7a642ca2-3fee-4a4c-8c92-3f3069a99ac5-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"7a642ca2-3fee-4a4c-8c92-3f3069a99ac5\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 18:45:55.733757 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:45:55.733632    2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/7a642ca2-3fee-4a4c-8c92-3f3069a99ac5-tls-assets\") pod \"alertmanager-main-0\" (UID: \"7a642ca2-3fee-4a4c-8c92-3f3069a99ac5\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 18:45:55.733757 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:45:55.733720    2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/7a642ca2-3fee-4a4c-8c92-3f3069a99ac5-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"7a642ca2-3fee-4a4c-8c92-3f3069a99ac5\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 18:45:55.733757 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:45:55.733748    2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7a642ca2-3fee-4a4c-8c92-3f3069a99ac5-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"7a642ca2-3fee-4a4c-8c92-3f3069a99ac5\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 18:45:55.733901 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:45:55.733776    2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/7a642ca2-3fee-4a4c-8c92-3f3069a99ac5-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"7a642ca2-3fee-4a4c-8c92-3f3069a99ac5\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 18:45:55.733901 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:45:55.733804    2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/7a642ca2-3fee-4a4c-8c92-3f3069a99ac5-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"7a642ca2-3fee-4a4c-8c92-3f3069a99ac5\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 18:45:55.734795 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:45:55.734165    2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/7a642ca2-3fee-4a4c-8c92-3f3069a99ac5-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"7a642ca2-3fee-4a4c-8c92-3f3069a99ac5\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 18:45:55.734795 ip-10-0-132-80 kubenswrapper[2544]: E0916 18:45:55.734278    2544 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/7a642ca2-3fee-4a4c-8c92-3f3069a99ac5-alertmanager-trusted-ca-bundle podName:7a642ca2-3fee-4a4c-8c92-3f3069a99ac5 nodeName:}" failed. No retries permitted until 2026-09-16 18:45:56.234257296 +0000 UTC m=+119.023259815 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "alertmanager-trusted-ca-bundle" (UniqueName: "kubernetes.io/configmap/7a642ca2-3fee-4a4c-8c92-3f3069a99ac5-alertmanager-trusted-ca-bundle") pod "alertmanager-main-0" (UID: "7a642ca2-3fee-4a4c-8c92-3f3069a99ac5") : configmap references non-existent config key: ca-bundle.crt
Sep 16 18:45:55.736614 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:45:55.736159    2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/7a642ca2-3fee-4a4c-8c92-3f3069a99ac5-web-config\") pod \"alertmanager-main-0\" (UID: \"7a642ca2-3fee-4a4c-8c92-3f3069a99ac5\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 18:45:55.736614 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:45:55.736382    2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/7a642ca2-3fee-4a4c-8c92-3f3069a99ac5-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"7a642ca2-3fee-4a4c-8c92-3f3069a99ac5\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 18:45:55.737680 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:45:55.737658    2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/7a642ca2-3fee-4a4c-8c92-3f3069a99ac5-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"7a642ca2-3fee-4a4c-8c92-3f3069a99ac5\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 18:45:55.738490 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:45:55.738254    2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/7a642ca2-3fee-4a4c-8c92-3f3069a99ac5-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"7a642ca2-3fee-4a4c-8c92-3f3069a99ac5\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 18:45:55.739130 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:45:55.739085    2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/7a642ca2-3fee-4a4c-8c92-3f3069a99ac5-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"7a642ca2-3fee-4a4c-8c92-3f3069a99ac5\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 18:45:55.739253 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:45:55.739225    2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/7a642ca2-3fee-4a4c-8c92-3f3069a99ac5-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"7a642ca2-3fee-4a4c-8c92-3f3069a99ac5\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 18:45:55.739413 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:45:55.739390    2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/7a642ca2-3fee-4a4c-8c92-3f3069a99ac5-tls-assets\") pod \"alertmanager-main-0\" (UID: \"7a642ca2-3fee-4a4c-8c92-3f3069a99ac5\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 18:45:55.740525 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:45:55.740502    2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/7a642ca2-3fee-4a4c-8c92-3f3069a99ac5-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"7a642ca2-3fee-4a4c-8c92-3f3069a99ac5\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 18:45:55.741027 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:45:55.740868    2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/7a642ca2-3fee-4a4c-8c92-3f3069a99ac5-config-volume\") pod \"alertmanager-main-0\" (UID: \"7a642ca2-3fee-4a4c-8c92-3f3069a99ac5\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 18:45:55.741117 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:45:55.741074    2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/7a642ca2-3fee-4a4c-8c92-3f3069a99ac5-config-out\") pod \"alertmanager-main-0\" (UID: \"7a642ca2-3fee-4a4c-8c92-3f3069a99ac5\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 18:45:55.746749 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:45:55.746701    2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bgjh4\" (UniqueName: \"kubernetes.io/projected/7a642ca2-3fee-4a4c-8c92-3f3069a99ac5-kube-api-access-bgjh4\") pod \"alertmanager-main-0\" (UID: \"7a642ca2-3fee-4a4c-8c92-3f3069a99ac5\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 18:45:56.198945 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:45:56.198916    2544 generic.go:356] "Generic (PLEG): container finished" podID="8b245d92-ed58-4240-8b0d-7dab57750690" containerID="c480db776ba4c060fe4b3b126f8ec20e0deef24d486fe86779e797ecd4b0035f" exitCode=0
Sep 16 18:45:56.199039 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:45:56.198996    2544 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-sljjd" event={"ID":"8b245d92-ed58-4240-8b0d-7dab57750690","Type":"ContainerDied","Data":"c480db776ba4c060fe4b3b126f8ec20e0deef24d486fe86779e797ecd4b0035f"}
Sep 16 18:45:56.200820 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:45:56.200799    2544 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-694l8" event={"ID":"ce1ddaa9-0566-4f7a-bc19-e7a7b958ae22","Type":"ContainerStarted","Data":"a6d066378548d79c734ec1f7cfdb57fa0c71d9fc096d074671c27bb9d7a40b6b"}
Sep 16 18:45:56.200889 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:45:56.200841    2544 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-694l8" event={"ID":"ce1ddaa9-0566-4f7a-bc19-e7a7b958ae22","Type":"ContainerStarted","Data":"30afd756a2ac358076135bedec2a30c88b2f51dc32d27ee6cb1185b831e01128"}
Sep 16 18:45:56.200889 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:45:56.200860    2544 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-694l8" event={"ID":"ce1ddaa9-0566-4f7a-bc19-e7a7b958ae22","Type":"ContainerStarted","Data":"115f61bb62ef0d48410a0a53d2a131ba0a5fa655b560ca99918e8c2a62a9c6d0"}
Sep 16 18:45:56.239713 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:45:56.239677    2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7a642ca2-3fee-4a4c-8c92-3f3069a99ac5-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"7a642ca2-3fee-4a4c-8c92-3f3069a99ac5\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 18:45:56.240494 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:45:56.240467    2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7a642ca2-3fee-4a4c-8c92-3f3069a99ac5-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"7a642ca2-3fee-4a4c-8c92-3f3069a99ac5\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 18:45:56.444638 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:45:56.444600    2544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Sep 16 18:45:56.747165 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:45:56.746230    2544 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Sep 16 18:45:56.749280 ip-10-0-132-80 kubenswrapper[2544]: W0916 18:45:56.749245    2544 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7a642ca2_3fee_4a4c_8c92_3f3069a99ac5.slice/crio-ed3f704d98d544da2c3c5ee1cb84c96ba83a2c96e97341908f366fa44b5f9020 WatchSource:0}: Error finding container ed3f704d98d544da2c3c5ee1cb84c96ba83a2c96e97341908f366fa44b5f9020: Status 404 returned error can't find the container with id ed3f704d98d544da2c3c5ee1cb84c96ba83a2c96e97341908f366fa44b5f9020
Sep 16 18:45:57.114361 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:45:57.114333    2544 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-76874c7675-4dhws"
Sep 16 18:45:57.206853 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:45:57.206814    2544 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-sljjd" event={"ID":"8b245d92-ed58-4240-8b0d-7dab57750690","Type":"ContainerStarted","Data":"2f560df049008c406664f427203d36a327d105707c3055e8178373f7411f9c5b"}
Sep 16 18:45:57.207031 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:45:57.206879    2544 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-sljjd" event={"ID":"8b245d92-ed58-4240-8b0d-7dab57750690","Type":"ContainerStarted","Data":"af3b7c724c555e18648cf617f88b211539e1c0e0d759d5698a2e1eceb3604a24"}
Sep 16 18:45:57.208968 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:45:57.208940    2544 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-59fbdd77d-2fwf6" event={"ID":"2b942120-fb59-49f7-a96f-3f7165b19110","Type":"ContainerStarted","Data":"d4433e20f9f830501abe76be0274ffabdff59ecc34c6f802c370d2ca03671313"}
Sep 16 18:45:57.209084 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:45:57.208998    2544 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-59fbdd77d-2fwf6" event={"ID":"2b942120-fb59-49f7-a96f-3f7165b19110","Type":"ContainerStarted","Data":"a0973c895d0f40e53fbe9b63cec72d009a0f8efb95bc231912f4122f15466b27"}
Sep 16 18:45:57.209084 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:45:57.209023    2544 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-59fbdd77d-2fwf6" event={"ID":"2b942120-fb59-49f7-a96f-3f7165b19110","Type":"ContainerStarted","Data":"b84932eff7cd00fec968f9dd56bb79477bcb455e34b4ea47525835a8bdd6b322"}
Sep 16 18:45:57.210008 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:45:57.209988    2544 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"7a642ca2-3fee-4a4c-8c92-3f3069a99ac5","Type":"ContainerStarted","Data":"ed3f704d98d544da2c3c5ee1cb84c96ba83a2c96e97341908f366fa44b5f9020"}
Sep 16 18:45:57.211907 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:45:57.211886    2544 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-694l8" event={"ID":"ce1ddaa9-0566-4f7a-bc19-e7a7b958ae22","Type":"ContainerStarted","Data":"8b383ad60b4f539fb6f7c02f82e7173c53ed121f9a600ee802d69ce1f71c4198"}
Sep 16 18:45:57.232441 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:45:57.232397    2544 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-sljjd" podStartSLOduration=2.195700175 podStartE2EDuration="3.232382781s" totalImagesPullingTime="965.221537ms" totalInitContainerRuntime="71.461069ms" isStatefulPod=true podCreationTimestamp="2026-09-16 18:45:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:45:57.230925525 +0000 UTC m=+120.019928070" watchObservedRunningTime="2026-09-16 18:45:57.232382781 +0000 UTC m=+120.021385306"
Sep 16 18:45:57.292852 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:45:57.292745    2544 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-694l8" podStartSLOduration=2.005689968 podStartE2EDuration="3.292728478s" totalImagesPullingTime="1.28703851s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:45:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:45:57.259817115 +0000 UTC m=+120.048819648" watchObservedRunningTime="2026-09-16 18:45:57.292728478 +0000 UTC m=+120.081731011"
Sep 16 18:45:58.216882 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:45:58.216850    2544 generic.go:356] "Generic (PLEG): container finished" podID="7a642ca2-3fee-4a4c-8c92-3f3069a99ac5" containerID="fa8b3480b4cd7b3ae46246e3d6cab25c95a961ca441be61b82514a692b4ec3cf" exitCode=0
Sep 16 18:45:58.217267 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:45:58.216957    2544 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"7a642ca2-3fee-4a4c-8c92-3f3069a99ac5","Type":"ContainerDied","Data":"fa8b3480b4cd7b3ae46246e3d6cab25c95a961ca441be61b82514a692b4ec3cf"}
Sep 16 18:45:58.241618 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:45:58.241567    2544 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/kube-state-metrics-59fbdd77d-2fwf6" podStartSLOduration=3.006952949 podStartE2EDuration="4.241537969s" totalImagesPullingTime="1.23458502s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:45:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:45:57.297349264 +0000 UTC m=+120.086351811" watchObservedRunningTime="2026-09-16 18:45:58.241537969 +0000 UTC m=+121.030540503"
Sep 16 18:45:58.642445 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:45:58.642411    2544 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/metrics-server-7544b7b8b8-fk7dc"]
Sep 16 18:45:58.645528 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:45:58.645506    2544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-7544b7b8b8-fk7dc"
Sep 16 18:45:58.647678 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:45:58.647651    2544 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-client-certs\""
Sep 16 18:45:58.647678 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:45:58.647664    2544 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-server-audit-profiles\""
Sep 16 18:45:58.647678 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:45:58.647672    2544 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-dockercfg-56p2q\""
Sep 16 18:45:58.647893 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:45:58.647706    2544 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-tls\""
Sep 16 18:45:58.647893 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:45:58.647664    2544 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\""
Sep 16 18:45:58.647893 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:45:58.647741    2544 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-ec8t7jo910632\""
Sep 16 18:45:58.655323 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:45:58.655304    2544 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-7544b7b8b8-fk7dc"]
Sep 16 18:45:58.662170 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:45:58.662147    2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/e96b6d09-3c97-41ed-860d-02ffed52f284-metrics-server-audit-profiles\") pod \"metrics-server-7544b7b8b8-fk7dc\" (UID: \"e96b6d09-3c97-41ed-860d-02ffed52f284\") " pod="openshift-monitoring/metrics-server-7544b7b8b8-fk7dc"
Sep 16 18:45:58.662250 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:45:58.662174    2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ptts9\" (UniqueName: \"kubernetes.io/projected/e96b6d09-3c97-41ed-860d-02ffed52f284-kube-api-access-ptts9\") pod \"metrics-server-7544b7b8b8-fk7dc\" (UID: \"e96b6d09-3c97-41ed-860d-02ffed52f284\") " pod="openshift-monitoring/metrics-server-7544b7b8b8-fk7dc"
Sep 16 18:45:58.662250 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:45:58.662199    2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/e96b6d09-3c97-41ed-860d-02ffed52f284-client-ca-bundle\") pod \"metrics-server-7544b7b8b8-fk7dc\" (UID: \"e96b6d09-3c97-41ed-860d-02ffed52f284\") " pod="openshift-monitoring/metrics-server-7544b7b8b8-fk7dc"
Sep 16 18:45:58.662330 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:45:58.662296    2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/e96b6d09-3c97-41ed-860d-02ffed52f284-secret-metrics-server-tls\") pod \"metrics-server-7544b7b8b8-fk7dc\" (UID: \"e96b6d09-3c97-41ed-860d-02ffed52f284\") " pod="openshift-monitoring/metrics-server-7544b7b8b8-fk7dc"
Sep 16 18:45:58.662330 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:45:58.662326    2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/e96b6d09-3c97-41ed-860d-02ffed52f284-secret-metrics-server-client-certs\") pod \"metrics-server-7544b7b8b8-fk7dc\" (UID: \"e96b6d09-3c97-41ed-860d-02ffed52f284\") " pod="openshift-monitoring/metrics-server-7544b7b8b8-fk7dc"
Sep 16 18:45:58.662393 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:45:58.662372    2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e96b6d09-3c97-41ed-860d-02ffed52f284-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-7544b7b8b8-fk7dc\" (UID: \"e96b6d09-3c97-41ed-860d-02ffed52f284\") " pod="openshift-monitoring/metrics-server-7544b7b8b8-fk7dc"
Sep 16 18:45:58.662422 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:45:58.662394    2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/e96b6d09-3c97-41ed-860d-02ffed52f284-audit-log\") pod \"metrics-server-7544b7b8b8-fk7dc\" (UID: \"e96b6d09-3c97-41ed-860d-02ffed52f284\") " pod="openshift-monitoring/metrics-server-7544b7b8b8-fk7dc"
Sep 16 18:45:58.716992 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:45:58.716971    2544 scope.go:117] "RemoveContainer" containerID="0e9d7fd39602558562e958aad0b61e348b64f1e8bfb010cc289c924a0f49e5e9"
Sep 16 18:45:58.762820 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:45:58.762794    2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/e96b6d09-3c97-41ed-860d-02ffed52f284-secret-metrics-server-tls\") pod \"metrics-server-7544b7b8b8-fk7dc\" (UID: \"e96b6d09-3c97-41ed-860d-02ffed52f284\") " pod="openshift-monitoring/metrics-server-7544b7b8b8-fk7dc"
Sep 16 18:45:58.762922 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:45:58.762846    2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/e96b6d09-3c97-41ed-860d-02ffed52f284-secret-metrics-server-client-certs\") pod \"metrics-server-7544b7b8b8-fk7dc\" (UID: \"e96b6d09-3c97-41ed-860d-02ffed52f284\") " pod="openshift-monitoring/metrics-server-7544b7b8b8-fk7dc"
Sep 16 18:45:58.762922 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:45:58.762900    2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e96b6d09-3c97-41ed-860d-02ffed52f284-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-7544b7b8b8-fk7dc\" (UID: \"e96b6d09-3c97-41ed-860d-02ffed52f284\") " pod="openshift-monitoring/metrics-server-7544b7b8b8-fk7dc"
Sep 16 18:45:58.763011 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:45:58.762947    2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/e96b6d09-3c97-41ed-860d-02ffed52f284-audit-log\") pod \"metrics-server-7544b7b8b8-fk7dc\" (UID: \"e96b6d09-3c97-41ed-860d-02ffed52f284\") " pod="openshift-monitoring/metrics-server-7544b7b8b8-fk7dc"
Sep 16 18:45:58.763234 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:45:58.763163    2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/e96b6d09-3c97-41ed-860d-02ffed52f284-metrics-server-audit-profiles\") pod \"metrics-server-7544b7b8b8-fk7dc\" (UID: \"e96b6d09-3c97-41ed-860d-02ffed52f284\") " pod="openshift-monitoring/metrics-server-7544b7b8b8-fk7dc"
Sep 16 18:45:58.763234 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:45:58.763197    2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ptts9\" (UniqueName: \"kubernetes.io/projected/e96b6d09-3c97-41ed-860d-02ffed52f284-kube-api-access-ptts9\") pod \"metrics-server-7544b7b8b8-fk7dc\" (UID: \"e96b6d09-3c97-41ed-860d-02ffed52f284\") " pod="openshift-monitoring/metrics-server-7544b7b8b8-fk7dc"
Sep 16 18:45:58.763372 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:45:58.763233    2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/e96b6d09-3c97-41ed-860d-02ffed52f284-client-ca-bundle\") pod \"metrics-server-7544b7b8b8-fk7dc\" (UID: \"e96b6d09-3c97-41ed-860d-02ffed52f284\") " pod="openshift-monitoring/metrics-server-7544b7b8b8-fk7dc"
Sep 16 18:45:58.763372 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:45:58.763301    2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/e96b6d09-3c97-41ed-860d-02ffed52f284-audit-log\") pod \"metrics-server-7544b7b8b8-fk7dc\" (UID: \"e96b6d09-3c97-41ed-860d-02ffed52f284\") " pod="openshift-monitoring/metrics-server-7544b7b8b8-fk7dc"
Sep 16 18:45:58.763759 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:45:58.763735    2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e96b6d09-3c97-41ed-860d-02ffed52f284-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-7544b7b8b8-fk7dc\" (UID: \"e96b6d09-3c97-41ed-860d-02ffed52f284\") " pod="openshift-monitoring/metrics-server-7544b7b8b8-fk7dc"
Sep 16 18:45:58.764504 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:45:58.764482    2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/e96b6d09-3c97-41ed-860d-02ffed52f284-metrics-server-audit-profiles\") pod \"metrics-server-7544b7b8b8-fk7dc\" (UID: \"e96b6d09-3c97-41ed-860d-02ffed52f284\") " pod="openshift-monitoring/metrics-server-7544b7b8b8-fk7dc"
Sep 16 18:45:58.765447 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:45:58.765429    2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/e96b6d09-3c97-41ed-860d-02ffed52f284-secret-metrics-server-tls\") pod \"metrics-server-7544b7b8b8-fk7dc\" (UID: \"e96b6d09-3c97-41ed-860d-02ffed52f284\") " pod="openshift-monitoring/metrics-server-7544b7b8b8-fk7dc"
Sep 16 18:45:58.765568 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:45:58.765439    2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/e96b6d09-3c97-41ed-860d-02ffed52f284-secret-metrics-server-client-certs\") pod \"metrics-server-7544b7b8b8-fk7dc\" (UID: \"e96b6d09-3c97-41ed-860d-02ffed52f284\") " pod="openshift-monitoring/metrics-server-7544b7b8b8-fk7dc"
Sep 16 18:45:58.765779 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:45:58.765763    2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/e96b6d09-3c97-41ed-860d-02ffed52f284-client-ca-bundle\") pod \"metrics-server-7544b7b8b8-fk7dc\" (UID: \"e96b6d09-3c97-41ed-860d-02ffed52f284\") " pod="openshift-monitoring/metrics-server-7544b7b8b8-fk7dc"
Sep 16 18:45:58.771516 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:45:58.771494    2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ptts9\" (UniqueName: \"kubernetes.io/projected/e96b6d09-3c97-41ed-860d-02ffed52f284-kube-api-access-ptts9\") pod \"metrics-server-7544b7b8b8-fk7dc\" (UID: \"e96b6d09-3c97-41ed-860d-02ffed52f284\") " pod="openshift-monitoring/metrics-server-7544b7b8b8-fk7dc"
Sep 16 18:45:58.954123 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:45:58.954039    2544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-7544b7b8b8-fk7dc"
Sep 16 18:45:59.223433 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:45:59.223361    2544 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-pfcb7_9e3f9e34-420e-43ef-96b4-c7cad6b2eaa9/console-operator/2.log"
Sep 16 18:45:59.223951 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:45:59.223438    2544 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-696884fb9c-pfcb7" event={"ID":"9e3f9e34-420e-43ef-96b4-c7cad6b2eaa9","Type":"ContainerStarted","Data":"83b7efd846bc4c1a7ef9289980a8fa4b977344582929a3c184a3068df9b0f5e7"}
Sep 16 18:45:59.223951 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:45:59.223794    2544 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-696884fb9c-pfcb7"
Sep 16 18:45:59.238570 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:45:59.238528    2544 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-7544b7b8b8-fk7dc"]
Sep 16 18:45:59.247718 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:45:59.247661    2544 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console-operator/console-operator-696884fb9c-pfcb7" podStartSLOduration=50.157031009 podStartE2EDuration="54.247644048s" totalImagesPullingTime="4.090613039s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:45:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:45:59.245786404 +0000 UTC m=+122.034788964" watchObservedRunningTime="2026-09-16 18:45:59.247644048 +0000 UTC m=+122.036646583"
Sep 16 18:45:59.552989 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:45:59.552909    2544 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/telemeter-client-598cbd87f8-kmf7z"]
Sep 16 18:45:59.561031 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:45:59.561004    2544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-598cbd87f8-kmf7z"
Sep 16 18:45:59.563469 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:45:59.563441    2544 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-kube-rbac-proxy-config\""
Sep 16 18:45:59.563637 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:45:59.563616    2544 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client\""
Sep 16 18:45:59.563942 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:45:59.563922    2544 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-dockercfg-d6p4m\""
Sep 16 18:45:59.564070 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:45:59.563983    2544 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-client-serving-certs-ca-bundle\""
Sep 16 18:45:59.564135 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:45:59.564116    2544 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"federate-client-certs\""
Sep 16 18:45:59.564203 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:45:59.564114    2544 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-tls\""
Sep 16 18:45:59.570118 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:45:59.568962    2544 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-trusted-ca-bundle-8i12ta5c71j38\""
Sep 16 18:45:59.572099 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:45:59.572077    2544 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-598cbd87f8-kmf7z"]
Sep 16 18:45:59.659415 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:45:59.659382    2544 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console-operator/console-operator-696884fb9c-pfcb7"
Sep 16 18:45:59.671687 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:45:59.671648    2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/ac291abc-ad47-431c-89be-9de4e0f33e88-telemeter-client-tls\") pod \"telemeter-client-598cbd87f8-kmf7z\" (UID: \"ac291abc-ad47-431c-89be-9de4e0f33e88\") " pod="openshift-monitoring/telemeter-client-598cbd87f8-kmf7z"
Sep 16 18:45:59.671860 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:45:59.671733    2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/ac291abc-ad47-431c-89be-9de4e0f33e88-federate-client-tls\") pod \"telemeter-client-598cbd87f8-kmf7z\" (UID: \"ac291abc-ad47-431c-89be-9de4e0f33e88\") " pod="openshift-monitoring/telemeter-client-598cbd87f8-kmf7z"
Sep 16 18:45:59.671860 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:45:59.671777    2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/ac291abc-ad47-431c-89be-9de4e0f33e88-secret-telemeter-client\") pod \"telemeter-client-598cbd87f8-kmf7z\" (UID: \"ac291abc-ad47-431c-89be-9de4e0f33e88\") " pod="openshift-monitoring/telemeter-client-598cbd87f8-kmf7z"
Sep 16 18:45:59.671860 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:45:59.671813    2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/ac291abc-ad47-431c-89be-9de4e0f33e88-metrics-client-ca\") pod \"telemeter-client-598cbd87f8-kmf7z\" (UID: \"ac291abc-ad47-431c-89be-9de4e0f33e88\") " pod="openshift-monitoring/telemeter-client-598cbd87f8-kmf7z"
Sep 16 18:45:59.671989 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:45:59.671916    2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/ac291abc-ad47-431c-89be-9de4e0f33e88-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-598cbd87f8-kmf7z\" (UID: \"ac291abc-ad47-431c-89be-9de4e0f33e88\") " pod="openshift-monitoring/telemeter-client-598cbd87f8-kmf7z"
Sep 16 18:45:59.671989 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:45:59.671965    2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lbkvt\" (UniqueName: \"kubernetes.io/projected/ac291abc-ad47-431c-89be-9de4e0f33e88-kube-api-access-lbkvt\") pod \"telemeter-client-598cbd87f8-kmf7z\" (UID: \"ac291abc-ad47-431c-89be-9de4e0f33e88\") " pod="openshift-monitoring/telemeter-client-598cbd87f8-kmf7z"
Sep 16 18:45:59.672083 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:45:59.672008    2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ac291abc-ad47-431c-89be-9de4e0f33e88-serving-certs-ca-bundle\") pod \"telemeter-client-598cbd87f8-kmf7z\" (UID: \"ac291abc-ad47-431c-89be-9de4e0f33e88\") " pod="openshift-monitoring/telemeter-client-598cbd87f8-kmf7z"
Sep 16 18:45:59.672083 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:45:59.672033    2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ac291abc-ad47-431c-89be-9de4e0f33e88-telemeter-trusted-ca-bundle\") pod \"telemeter-client-598cbd87f8-kmf7z\" (UID: \"ac291abc-ad47-431c-89be-9de4e0f33e88\") " pod="openshift-monitoring/telemeter-client-598cbd87f8-kmf7z"
Sep 16 18:45:59.772920 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:45:59.772884    2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/ac291abc-ad47-431c-89be-9de4e0f33e88-secret-telemeter-client\") pod \"telemeter-client-598cbd87f8-kmf7z\" (UID: \"ac291abc-ad47-431c-89be-9de4e0f33e88\") " pod="openshift-monitoring/telemeter-client-598cbd87f8-kmf7z"
Sep 16 18:45:59.773079 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:45:59.772944    2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/ac291abc-ad47-431c-89be-9de4e0f33e88-metrics-client-ca\") pod \"telemeter-client-598cbd87f8-kmf7z\" (UID: \"ac291abc-ad47-431c-89be-9de4e0f33e88\") " pod="openshift-monitoring/telemeter-client-598cbd87f8-kmf7z"
Sep 16 18:45:59.773079 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:45:59.773005    2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/ac291abc-ad47-431c-89be-9de4e0f33e88-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-598cbd87f8-kmf7z\" (UID: \"ac291abc-ad47-431c-89be-9de4e0f33e88\") " pod="openshift-monitoring/telemeter-client-598cbd87f8-kmf7z"
Sep 16 18:45:59.773079 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:45:59.773046    2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lbkvt\" (UniqueName: \"kubernetes.io/projected/ac291abc-ad47-431c-89be-9de4e0f33e88-kube-api-access-lbkvt\") pod \"telemeter-client-598cbd87f8-kmf7z\" (UID: \"ac291abc-ad47-431c-89be-9de4e0f33e88\") " pod="openshift-monitoring/telemeter-client-598cbd87f8-kmf7z"
Sep 16 18:45:59.773301 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:45:59.773090    2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ac291abc-ad47-431c-89be-9de4e0f33e88-serving-certs-ca-bundle\") pod \"telemeter-client-598cbd87f8-kmf7z\" (UID: \"ac291abc-ad47-431c-89be-9de4e0f33e88\") " pod="openshift-monitoring/telemeter-client-598cbd87f8-kmf7z"
Sep 16 18:45:59.773301 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:45:59.773114    2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ac291abc-ad47-431c-89be-9de4e0f33e88-telemeter-trusted-ca-bundle\") pod \"telemeter-client-598cbd87f8-kmf7z\" (UID: \"ac291abc-ad47-431c-89be-9de4e0f33e88\") " pod="openshift-monitoring/telemeter-client-598cbd87f8-kmf7z"
Sep 16 18:45:59.773301 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:45:59.773187    2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/ac291abc-ad47-431c-89be-9de4e0f33e88-telemeter-client-tls\") pod \"telemeter-client-598cbd87f8-kmf7z\" (UID: \"ac291abc-ad47-431c-89be-9de4e0f33e88\") " pod="openshift-monitoring/telemeter-client-598cbd87f8-kmf7z"
Sep 16 18:45:59.773301 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:45:59.773225    2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/ac291abc-ad47-431c-89be-9de4e0f33e88-federate-client-tls\") pod \"telemeter-client-598cbd87f8-kmf7z\" (UID: \"ac291abc-ad47-431c-89be-9de4e0f33e88\") " pod="openshift-monitoring/telemeter-client-598cbd87f8-kmf7z"
Sep 16 18:45:59.773816 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:45:59.773791    2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/ac291abc-ad47-431c-89be-9de4e0f33e88-metrics-client-ca\") pod \"telemeter-client-598cbd87f8-kmf7z\" (UID: \"ac291abc-ad47-431c-89be-9de4e0f33e88\") " pod="openshift-monitoring/telemeter-client-598cbd87f8-kmf7z"
Sep 16 18:45:59.774123 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:45:59.774094    2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ac291abc-ad47-431c-89be-9de4e0f33e88-serving-certs-ca-bundle\") pod \"telemeter-client-598cbd87f8-kmf7z\" (UID: \"ac291abc-ad47-431c-89be-9de4e0f33e88\") " pod="openshift-monitoring/telemeter-client-598cbd87f8-kmf7z"
Sep 16 18:45:59.774227 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:45:59.774173    2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ac291abc-ad47-431c-89be-9de4e0f33e88-telemeter-trusted-ca-bundle\") pod \"telemeter-client-598cbd87f8-kmf7z\" (UID: \"ac291abc-ad47-431c-89be-9de4e0f33e88\") " pod="openshift-monitoring/telemeter-client-598cbd87f8-kmf7z"
Sep 16 18:45:59.775435 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:45:59.775414    2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/ac291abc-ad47-431c-89be-9de4e0f33e88-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-598cbd87f8-kmf7z\" (UID: \"ac291abc-ad47-431c-89be-9de4e0f33e88\") " pod="openshift-monitoring/telemeter-client-598cbd87f8-kmf7z"
Sep 16 18:45:59.775782 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:45:59.775762    2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/ac291abc-ad47-431c-89be-9de4e0f33e88-federate-client-tls\") pod \"telemeter-client-598cbd87f8-kmf7z\" (UID: \"ac291abc-ad47-431c-89be-9de4e0f33e88\") " pod="openshift-monitoring/telemeter-client-598cbd87f8-kmf7z"
Sep 16 18:45:59.775902 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:45:59.775836    2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/ac291abc-ad47-431c-89be-9de4e0f33e88-telemeter-client-tls\") pod \"telemeter-client-598cbd87f8-kmf7z\" (UID: \"ac291abc-ad47-431c-89be-9de4e0f33e88\") " pod="openshift-monitoring/telemeter-client-598cbd87f8-kmf7z"
Sep 16 18:45:59.776065 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:45:59.776040    2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/ac291abc-ad47-431c-89be-9de4e0f33e88-secret-telemeter-client\") pod \"telemeter-client-598cbd87f8-kmf7z\" (UID: \"ac291abc-ad47-431c-89be-9de4e0f33e88\") " pod="openshift-monitoring/telemeter-client-598cbd87f8-kmf7z"
Sep 16 18:45:59.784192 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:45:59.782029    2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lbkvt\" (UniqueName: \"kubernetes.io/projected/ac291abc-ad47-431c-89be-9de4e0f33e88-kube-api-access-lbkvt\") pod \"telemeter-client-598cbd87f8-kmf7z\" (UID: \"ac291abc-ad47-431c-89be-9de4e0f33e88\") " pod="openshift-monitoring/telemeter-client-598cbd87f8-kmf7z"
Sep 16 18:45:59.853431 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:45:59.853410    2544 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/downloads-76f99584f4-m4dhw"]
Sep 16 18:45:59.857629 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:45:59.857612    2544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-76f99584f4-m4dhw"
Sep 16 18:45:59.860373 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:45:59.860349    2544 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\""
Sep 16 18:45:59.860471 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:45:59.860435    2544 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\""
Sep 16 18:45:59.860471 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:45:59.860459    2544 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"default-dockercfg-n7xct\""
Sep 16 18:45:59.869759 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:45:59.869737    2544 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-76f99584f4-m4dhw"]
Sep 16 18:45:59.874184 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:45:59.874159    2544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-598cbd87f8-kmf7z"
Sep 16 18:45:59.974537 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:45:59.974501    2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d7dgb\" (UniqueName: \"kubernetes.io/projected/e03463ea-f247-46f9-b8a8-57ccef5477d0-kube-api-access-d7dgb\") pod \"downloads-76f99584f4-m4dhw\" (UID: \"e03463ea-f247-46f9-b8a8-57ccef5477d0\") " pod="openshift-console/downloads-76f99584f4-m4dhw"
Sep 16 18:46:00.076281 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:46:00.076252    2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d7dgb\" (UniqueName: \"kubernetes.io/projected/e03463ea-f247-46f9-b8a8-57ccef5477d0-kube-api-access-d7dgb\") pod \"downloads-76f99584f4-m4dhw\" (UID: \"e03463ea-f247-46f9-b8a8-57ccef5477d0\") " pod="openshift-console/downloads-76f99584f4-m4dhw"
Sep 16 18:46:00.100383 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:46:00.100355    2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-d7dgb\" (UniqueName: \"kubernetes.io/projected/e03463ea-f247-46f9-b8a8-57ccef5477d0-kube-api-access-d7dgb\") pod \"downloads-76f99584f4-m4dhw\" (UID: \"e03463ea-f247-46f9-b8a8-57ccef5477d0\") " pod="openshift-console/downloads-76f99584f4-m4dhw"
Sep 16 18:46:00.179392 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:46:00.179342    2544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-76f99584f4-m4dhw"
Sep 16 18:46:00.232802 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:46:00.232713    2544 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-7544b7b8b8-fk7dc" event={"ID":"e96b6d09-3c97-41ed-860d-02ffed52f284","Type":"ContainerStarted","Data":"44eefac3aa39709defde1a9194b43e0a176475e83a5d8972ec19d95e0ee34b39"}
Sep 16 18:46:00.234117 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:46:00.233944    2544 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-598cbd87f8-kmf7z"]
Sep 16 18:46:00.246729 ip-10-0-132-80 kubenswrapper[2544]: W0916 18:46:00.238634    2544 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podac291abc_ad47_431c_89be_9de4e0f33e88.slice/crio-687953ca2f7289ed33557dec17a7e65fcb27dd5b3c23c770315d31e39d438bca WatchSource:0}: Error finding container 687953ca2f7289ed33557dec17a7e65fcb27dd5b3c23c770315d31e39d438bca: Status 404 returned error can't find the container with id 687953ca2f7289ed33557dec17a7e65fcb27dd5b3c23c770315d31e39d438bca
Sep 16 18:46:00.246729 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:46:00.241596    2544 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"7a642ca2-3fee-4a4c-8c92-3f3069a99ac5","Type":"ContainerStarted","Data":"54bdf2fa8350cdfc6440f1011b737bf409d17e83ef4439f97779fdfe602db92a"}
Sep 16 18:46:00.246729 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:46:00.241671    2544 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"7a642ca2-3fee-4a4c-8c92-3f3069a99ac5","Type":"ContainerStarted","Data":"26d9eb49dfb46f1825543ad633aeae54b3bb118af49bc59757034e90de2169e0"}
Sep 16 18:46:00.246729 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:46:00.241710    2544 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"7a642ca2-3fee-4a4c-8c92-3f3069a99ac5","Type":"ContainerStarted","Data":"ee9f9bb76fc17a89b5bc2bbef261d4f3168e436faf413593b3d466b7af49ff27"}
Sep 16 18:46:00.246729 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:46:00.241750    2544 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"7a642ca2-3fee-4a4c-8c92-3f3069a99ac5","Type":"ContainerStarted","Data":"0fea312a41c726c8f76ad6706d748cf7cda93b7bc61ffaf4e203edb5a4133e88"}
Sep 16 18:46:00.246729 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:46:00.241791    2544 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"7a642ca2-3fee-4a4c-8c92-3f3069a99ac5","Type":"ContainerStarted","Data":"00d9804e08c15f830a6e69132dc25ac411f4ca4977e7f7d37dd63e8c38f308c4"}
Sep 16 18:46:00.459665 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:46:00.459583    2544 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-76f99584f4-m4dhw"]
Sep 16 18:46:00.684133 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:46:00.684097    2544 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 16 18:46:00.689283 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:46:00.689249    2544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:46:00.692249 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:46:00.692218    2544 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-4pn6do9rm2eaa\""
Sep 16 18:46:00.692459 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:46:00.692427    2544 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\""
Sep 16 18:46:00.695266 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:46:00.692534    2544 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\""
Sep 16 18:46:00.695933 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:46:00.695910    2544 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\""
Sep 16 18:46:00.696092 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:46:00.696071    2544 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\""
Sep 16 18:46:00.696167 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:46:00.696089    2544 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\""
Sep 16 18:46:00.696167 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:46:00.696156    2544 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\""
Sep 16 18:46:00.696275 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:46:00.696035    2544 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\""
Sep 16 18:46:00.696275 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:46:00.696038    2544 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\""
Sep 16 18:46:00.696379 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:46:00.695914    2544 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\""
Sep 16 18:46:00.696423 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:46:00.696405    2544 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\""
Sep 16 18:46:00.696586 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:46:00.696566    2544 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\""
Sep 16 18:46:00.696586 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:46:00.696582    2544 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\""
Sep 16 18:46:00.696714 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:46:00.696620    2544 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-mvgc6\""
Sep 16 18:46:00.696837 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:46:00.696822    2544 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\""
Sep 16 18:46:00.698229 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:46:00.698209    2544 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\""
Sep 16 18:46:00.710692 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:46:00.710642    2544 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 16 18:46:00.759610 ip-10-0-132-80 kubenswrapper[2544]: W0916 18:46:00.759577    2544 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode03463ea_f247_46f9_b8a8_57ccef5477d0.slice/crio-ad0a7d3c81b7d5a4657c1a03d9c5f74d3c414d318487afcecfd4d9cc8d55ab35 WatchSource:0}: Error finding container ad0a7d3c81b7d5a4657c1a03d9c5f74d3c414d318487afcecfd4d9cc8d55ab35: Status 404 returned error can't find the container with id ad0a7d3c81b7d5a4657c1a03d9c5f74d3c414d318487afcecfd4d9cc8d55ab35
Sep 16 18:46:00.782385 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:46:00.782360    2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/dbb1a91b-a0d3-477a-a98b-82fa11d56733-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"dbb1a91b-a0d3-477a-a98b-82fa11d56733\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:46:00.782494 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:46:00.782391    2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/dbb1a91b-a0d3-477a-a98b-82fa11d56733-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"dbb1a91b-a0d3-477a-a98b-82fa11d56733\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:46:00.782494 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:46:00.782411    2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/dbb1a91b-a0d3-477a-a98b-82fa11d56733-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"dbb1a91b-a0d3-477a-a98b-82fa11d56733\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:46:00.782627 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:46:00.782487    2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/dbb1a91b-a0d3-477a-a98b-82fa11d56733-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"dbb1a91b-a0d3-477a-a98b-82fa11d56733\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:46:00.782627 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:46:00.782522    2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/dbb1a91b-a0d3-477a-a98b-82fa11d56733-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"dbb1a91b-a0d3-477a-a98b-82fa11d56733\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:46:00.782627 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:46:00.782565    2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/dbb1a91b-a0d3-477a-a98b-82fa11d56733-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"dbb1a91b-a0d3-477a-a98b-82fa11d56733\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:46:00.782777 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:46:00.782625    2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xjnkt\" (UniqueName: \"kubernetes.io/projected/dbb1a91b-a0d3-477a-a98b-82fa11d56733-kube-api-access-xjnkt\") pod \"prometheus-k8s-0\" (UID: \"dbb1a91b-a0d3-477a-a98b-82fa11d56733\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:46:00.782777 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:46:00.782661    2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/dbb1a91b-a0d3-477a-a98b-82fa11d56733-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"dbb1a91b-a0d3-477a-a98b-82fa11d56733\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:46:00.782777 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:46:00.782705    2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/dbb1a91b-a0d3-477a-a98b-82fa11d56733-config\") pod \"prometheus-k8s-0\" (UID: \"dbb1a91b-a0d3-477a-a98b-82fa11d56733\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:46:00.782777 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:46:00.782728    2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/dbb1a91b-a0d3-477a-a98b-82fa11d56733-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"dbb1a91b-a0d3-477a-a98b-82fa11d56733\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:46:00.782777 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:46:00.782758    2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/dbb1a91b-a0d3-477a-a98b-82fa11d56733-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"dbb1a91b-a0d3-477a-a98b-82fa11d56733\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:46:00.782960 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:46:00.782780    2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/dbb1a91b-a0d3-477a-a98b-82fa11d56733-web-config\") pod \"prometheus-k8s-0\" (UID: \"dbb1a91b-a0d3-477a-a98b-82fa11d56733\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:46:00.782960 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:46:00.782813    2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/dbb1a91b-a0d3-477a-a98b-82fa11d56733-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"dbb1a91b-a0d3-477a-a98b-82fa11d56733\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:46:00.782960 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:46:00.782864    2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/dbb1a91b-a0d3-477a-a98b-82fa11d56733-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"dbb1a91b-a0d3-477a-a98b-82fa11d56733\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:46:00.782960 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:46:00.782893    2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/dbb1a91b-a0d3-477a-a98b-82fa11d56733-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"dbb1a91b-a0d3-477a-a98b-82fa11d56733\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:46:00.782960 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:46:00.782913    2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/dbb1a91b-a0d3-477a-a98b-82fa11d56733-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"dbb1a91b-a0d3-477a-a98b-82fa11d56733\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:46:00.782960 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:46:00.782933    2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/dbb1a91b-a0d3-477a-a98b-82fa11d56733-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"dbb1a91b-a0d3-477a-a98b-82fa11d56733\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:46:00.783143 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:46:00.783001    2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/dbb1a91b-a0d3-477a-a98b-82fa11d56733-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"dbb1a91b-a0d3-477a-a98b-82fa11d56733\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:46:00.783143 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:46:00.783035    2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/dbb1a91b-a0d3-477a-a98b-82fa11d56733-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"dbb1a91b-a0d3-477a-a98b-82fa11d56733\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:46:00.783143 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:46:00.783057    2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/dbb1a91b-a0d3-477a-a98b-82fa11d56733-config-out\") pod \"prometheus-k8s-0\" (UID: \"dbb1a91b-a0d3-477a-a98b-82fa11d56733\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:46:00.883872 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:46:00.883836    2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/dbb1a91b-a0d3-477a-a98b-82fa11d56733-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"dbb1a91b-a0d3-477a-a98b-82fa11d56733\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:46:00.883872 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:46:00.883876    2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/dbb1a91b-a0d3-477a-a98b-82fa11d56733-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"dbb1a91b-a0d3-477a-a98b-82fa11d56733\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:46:00.884130 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:46:00.883902    2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/dbb1a91b-a0d3-477a-a98b-82fa11d56733-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"dbb1a91b-a0d3-477a-a98b-82fa11d56733\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:46:00.884130 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:46:00.883930    2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/dbb1a91b-a0d3-477a-a98b-82fa11d56733-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"dbb1a91b-a0d3-477a-a98b-82fa11d56733\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:46:00.884130 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:46:00.883970    2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/dbb1a91b-a0d3-477a-a98b-82fa11d56733-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"dbb1a91b-a0d3-477a-a98b-82fa11d56733\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:46:00.884130 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:46:00.883997    2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/dbb1a91b-a0d3-477a-a98b-82fa11d56733-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"dbb1a91b-a0d3-477a-a98b-82fa11d56733\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:46:00.884130 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:46:00.884017    2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/dbb1a91b-a0d3-477a-a98b-82fa11d56733-config-out\") pod \"prometheus-k8s-0\" (UID: \"dbb1a91b-a0d3-477a-a98b-82fa11d56733\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:46:00.884130 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:46:00.884044    2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/dbb1a91b-a0d3-477a-a98b-82fa11d56733-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"dbb1a91b-a0d3-477a-a98b-82fa11d56733\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:46:00.884130 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:46:00.884067    2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/dbb1a91b-a0d3-477a-a98b-82fa11d56733-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"dbb1a91b-a0d3-477a-a98b-82fa11d56733\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:46:00.884130 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:46:00.884091    2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/dbb1a91b-a0d3-477a-a98b-82fa11d56733-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"dbb1a91b-a0d3-477a-a98b-82fa11d56733\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:46:00.884130 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:46:00.884131    2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/dbb1a91b-a0d3-477a-a98b-82fa11d56733-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"dbb1a91b-a0d3-477a-a98b-82fa11d56733\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:46:00.884548 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:46:00.884153    2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/dbb1a91b-a0d3-477a-a98b-82fa11d56733-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"dbb1a91b-a0d3-477a-a98b-82fa11d56733\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:46:00.884548 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:46:00.884179    2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/dbb1a91b-a0d3-477a-a98b-82fa11d56733-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"dbb1a91b-a0d3-477a-a98b-82fa11d56733\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:46:00.884548 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:46:00.884220    2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xjnkt\" (UniqueName: \"kubernetes.io/projected/dbb1a91b-a0d3-477a-a98b-82fa11d56733-kube-api-access-xjnkt\") pod \"prometheus-k8s-0\" (UID: \"dbb1a91b-a0d3-477a-a98b-82fa11d56733\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:46:00.884548 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:46:00.884244    2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/dbb1a91b-a0d3-477a-a98b-82fa11d56733-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"dbb1a91b-a0d3-477a-a98b-82fa11d56733\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:46:00.884548 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:46:00.884289    2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/dbb1a91b-a0d3-477a-a98b-82fa11d56733-config\") pod \"prometheus-k8s-0\" (UID: \"dbb1a91b-a0d3-477a-a98b-82fa11d56733\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:46:00.884548 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:46:00.884310    2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/dbb1a91b-a0d3-477a-a98b-82fa11d56733-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"dbb1a91b-a0d3-477a-a98b-82fa11d56733\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:46:00.884548 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:46:00.884335    2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/dbb1a91b-a0d3-477a-a98b-82fa11d56733-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"dbb1a91b-a0d3-477a-a98b-82fa11d56733\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:46:00.884548 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:46:00.884344    2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/dbb1a91b-a0d3-477a-a98b-82fa11d56733-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"dbb1a91b-a0d3-477a-a98b-82fa11d56733\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:46:00.884548 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:46:00.884405    2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/dbb1a91b-a0d3-477a-a98b-82fa11d56733-web-config\") pod \"prometheus-k8s-0\" (UID: \"dbb1a91b-a0d3-477a-a98b-82fa11d56733\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:46:00.884548 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:46:00.884435    2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/dbb1a91b-a0d3-477a-a98b-82fa11d56733-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"dbb1a91b-a0d3-477a-a98b-82fa11d56733\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:46:00.885087 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:46:00.884658    2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/dbb1a91b-a0d3-477a-a98b-82fa11d56733-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"dbb1a91b-a0d3-477a-a98b-82fa11d56733\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:46:00.885087 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:46:00.884936    2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/dbb1a91b-a0d3-477a-a98b-82fa11d56733-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"dbb1a91b-a0d3-477a-a98b-82fa11d56733\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:46:00.885187 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:46:00.885081    2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/dbb1a91b-a0d3-477a-a98b-82fa11d56733-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"dbb1a91b-a0d3-477a-a98b-82fa11d56733\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:46:00.888612 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:46:00.888063    2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/dbb1a91b-a0d3-477a-a98b-82fa11d56733-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"dbb1a91b-a0d3-477a-a98b-82fa11d56733\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:46:00.888612 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:46:00.888477    2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/dbb1a91b-a0d3-477a-a98b-82fa11d56733-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"dbb1a91b-a0d3-477a-a98b-82fa11d56733\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:46:00.888842 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:46:00.888813    2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/dbb1a91b-a0d3-477a-a98b-82fa11d56733-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"dbb1a91b-a0d3-477a-a98b-82fa11d56733\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:46:00.889010 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:46:00.888952    2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/dbb1a91b-a0d3-477a-a98b-82fa11d56733-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"dbb1a91b-a0d3-477a-a98b-82fa11d56733\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:46:00.889123 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:46:00.889102    2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/dbb1a91b-a0d3-477a-a98b-82fa11d56733-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"dbb1a91b-a0d3-477a-a98b-82fa11d56733\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:46:00.889465 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:46:00.889403    2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/dbb1a91b-a0d3-477a-a98b-82fa11d56733-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"dbb1a91b-a0d3-477a-a98b-82fa11d56733\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:46:00.889803 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:46:00.889752    2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/dbb1a91b-a0d3-477a-a98b-82fa11d56733-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"dbb1a91b-a0d3-477a-a98b-82fa11d56733\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:46:00.889912 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:46:00.889891    2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/dbb1a91b-a0d3-477a-a98b-82fa11d56733-config-out\") pod \"prometheus-k8s-0\" (UID: \"dbb1a91b-a0d3-477a-a98b-82fa11d56733\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:46:00.889981 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:46:00.889910    2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/dbb1a91b-a0d3-477a-a98b-82fa11d56733-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"dbb1a91b-a0d3-477a-a98b-82fa11d56733\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:46:00.890191 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:46:00.890170    2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/dbb1a91b-a0d3-477a-a98b-82fa11d56733-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"dbb1a91b-a0d3-477a-a98b-82fa11d56733\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:46:00.890259 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:46:00.890237    2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/dbb1a91b-a0d3-477a-a98b-82fa11d56733-web-config\") pod \"prometheus-k8s-0\" (UID: \"dbb1a91b-a0d3-477a-a98b-82fa11d56733\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:46:00.890528 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:46:00.890506    2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/dbb1a91b-a0d3-477a-a98b-82fa11d56733-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"dbb1a91b-a0d3-477a-a98b-82fa11d56733\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:46:00.890958 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:46:00.890939    2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/dbb1a91b-a0d3-477a-a98b-82fa11d56733-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"dbb1a91b-a0d3-477a-a98b-82fa11d56733\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:46:00.891406 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:46:00.891387    2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/dbb1a91b-a0d3-477a-a98b-82fa11d56733-config\") pod \"prometheus-k8s-0\" (UID: \"dbb1a91b-a0d3-477a-a98b-82fa11d56733\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:46:00.891406 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:46:00.891395    2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/dbb1a91b-a0d3-477a-a98b-82fa11d56733-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"dbb1a91b-a0d3-477a-a98b-82fa11d56733\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:46:00.905769 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:46:00.905743    2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xjnkt\" (UniqueName: \"kubernetes.io/projected/dbb1a91b-a0d3-477a-a98b-82fa11d56733-kube-api-access-xjnkt\") pod \"prometheus-k8s-0\" (UID: \"dbb1a91b-a0d3-477a-a98b-82fa11d56733\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:46:01.002580 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:46:01.002338    2544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:46:01.246684 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:46:01.246653    2544 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-76f99584f4-m4dhw" event={"ID":"e03463ea-f247-46f9-b8a8-57ccef5477d0","Type":"ContainerStarted","Data":"ad0a7d3c81b7d5a4657c1a03d9c5f74d3c414d318487afcecfd4d9cc8d55ab35"}
Sep 16 18:46:01.248037 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:46:01.248014    2544 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-598cbd87f8-kmf7z" event={"ID":"ac291abc-ad47-431c-89be-9de4e0f33e88","Type":"ContainerStarted","Data":"687953ca2f7289ed33557dec17a7e65fcb27dd5b3c23c770315d31e39d438bca"}
Sep 16 18:46:01.249511 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:46:01.249487    2544 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-7544b7b8b8-fk7dc" event={"ID":"e96b6d09-3c97-41ed-860d-02ffed52f284","Type":"ContainerStarted","Data":"3272c77cf76d4189c8cf9231e774b28833581ff9d0a2d84a4e18a186d0fb7d69"}
Sep 16 18:46:01.269950 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:46:01.269911    2544 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/metrics-server-7544b7b8b8-fk7dc" podStartSLOduration=1.7101730659999999 podStartE2EDuration="3.269896411s" totalImagesPullingTime="1.559723345s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:45:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:46:01.268685134 +0000 UTC m=+124.057687692" watchObservedRunningTime="2026-09-16 18:46:01.269896411 +0000 UTC m=+124.058898945"
Sep 16 18:46:01.305434 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:46:01.305378    2544 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 16 18:46:01.308910 ip-10-0-132-80 kubenswrapper[2544]: W0916 18:46:01.308883    2544 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddbb1a91b_a0d3_477a_a98b_82fa11d56733.slice/crio-7686f099730ae123234922b6be38b9ec81fd3a1d0c67a90af29d130ef404e3c5 WatchSource:0}: Error finding container 7686f099730ae123234922b6be38b9ec81fd3a1d0c67a90af29d130ef404e3c5: Status 404 returned error can't find the container with id 7686f099730ae123234922b6be38b9ec81fd3a1d0c67a90af29d130ef404e3c5
Sep 16 18:46:02.255777 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:46:02.255742    2544 generic.go:356] "Generic (PLEG): container finished" podID="dbb1a91b-a0d3-477a-a98b-82fa11d56733" containerID="d0023e96010bf816b1118e2d024a6cdc27820ffb0e33bcefc75d61f18d94e90f" exitCode=0
Sep 16 18:46:02.256199 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:46:02.255847    2544 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"dbb1a91b-a0d3-477a-a98b-82fa11d56733","Type":"ContainerDied","Data":"d0023e96010bf816b1118e2d024a6cdc27820ffb0e33bcefc75d61f18d94e90f"}
Sep 16 18:46:02.256199 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:46:02.255902    2544 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"dbb1a91b-a0d3-477a-a98b-82fa11d56733","Type":"ContainerStarted","Data":"7686f099730ae123234922b6be38b9ec81fd3a1d0c67a90af29d130ef404e3c5"}
Sep 16 18:46:02.261021 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:46:02.260972    2544 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"7a642ca2-3fee-4a4c-8c92-3f3069a99ac5","Type":"ContainerStarted","Data":"06accc71e639e5500bc2a7e464c9033d53d565ed84ff1acea24ad664215c3256"}
Sep 16 18:46:02.343615 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:46:02.343537    2544 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/alertmanager-main-0" podStartSLOduration=3.4597741109999998 podStartE2EDuration="7.343517666s" totalImagesPullingTime="3.866222512s" totalInitContainerRuntime="17.521043ms" isStatefulPod=false podCreationTimestamp="2026-09-16 18:45:55 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:46:02.34280597 +0000 UTC m=+125.131808534" watchObservedRunningTime="2026-09-16 18:46:02.343517666 +0000 UTC m=+125.132520202"
Sep 16 18:46:03.223941 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:46:03.223907    2544 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-76874c7675-4dhws"]
Sep 16 18:46:03.269802 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:46:03.269512    2544 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-598cbd87f8-kmf7z" event={"ID":"ac291abc-ad47-431c-89be-9de4e0f33e88","Type":"ContainerStarted","Data":"39774d20b4275145b6456971731a7536e3a58a6681aa5bccf1bf95e5ad16e2e9"}
Sep 16 18:46:03.269802 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:46:03.269655    2544 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-598cbd87f8-kmf7z" event={"ID":"ac291abc-ad47-431c-89be-9de4e0f33e88","Type":"ContainerStarted","Data":"4819a125f2800a23db5f04c940ebbb09c2246d56273db7acd501a690f9f5b973"}
Sep 16 18:46:03.269802 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:46:03.269697    2544 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-598cbd87f8-kmf7z" event={"ID":"ac291abc-ad47-431c-89be-9de4e0f33e88","Type":"ContainerStarted","Data":"c4499f08c67887670e307dda0850ef0dd6fa8efb2c96f110f3a7124054259221"}
Sep 16 18:46:03.304320 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:46:03.304151    2544 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/telemeter-client-598cbd87f8-kmf7z" podStartSLOduration=2.164093332 podStartE2EDuration="4.304131365s" totalImagesPullingTime="2.140038033s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:45:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:46:03.29989344 +0000 UTC m=+126.088895998" watchObservedRunningTime="2026-09-16 18:46:03.304131365 +0000 UTC m=+126.093133901"
Sep 16 18:46:07.294256 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:46:07.294213    2544 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"dbb1a91b-a0d3-477a-a98b-82fa11d56733","Type":"ContainerStarted","Data":"728e8310ab4f116ffd9bbc9488059841cc67a45bed2003c19ff4382c6dd3f668"}
Sep 16 18:46:07.294719 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:46:07.294304    2544 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"dbb1a91b-a0d3-477a-a98b-82fa11d56733","Type":"ContainerStarted","Data":"835a005de515fe25533264a18248a7baae64846a00ebea09cebf79567b247378"}
Sep 16 18:46:07.457041 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:46:07.457000    2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/db1e3b32-114c-4bd9-b28c-21a2a9d6efaa-metrics-certs\") pod \"network-metrics-daemon-pdfpq\" (UID: \"db1e3b32-114c-4bd9-b28c-21a2a9d6efaa\") " pod="openshift-multus/network-metrics-daemon-pdfpq"
Sep 16 18:46:07.459884 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:46:07.459851    2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/db1e3b32-114c-4bd9-b28c-21a2a9d6efaa-metrics-certs\") pod \"network-metrics-daemon-pdfpq\" (UID: \"db1e3b32-114c-4bd9-b28c-21a2a9d6efaa\") " pod="openshift-multus/network-metrics-daemon-pdfpq"
Sep 16 18:46:07.737044 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:46:07.737001    2544 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-hvlxk\""
Sep 16 18:46:07.745129 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:46:07.745101    2544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-pdfpq"
Sep 16 18:46:08.633170 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:46:08.633142    2544 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-pdfpq"]
Sep 16 18:46:08.635808 ip-10-0-132-80 kubenswrapper[2544]: W0916 18:46:08.635776    2544 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddb1e3b32_114c_4bd9_b28c_21a2a9d6efaa.slice/crio-03dce9ed0edc7b287eb96090eb2d45480f4415c9367d1978b20a3a7e5d62a3e6 WatchSource:0}: Error finding container 03dce9ed0edc7b287eb96090eb2d45480f4415c9367d1978b20a3a7e5d62a3e6: Status 404 returned error can't find the container with id 03dce9ed0edc7b287eb96090eb2d45480f4415c9367d1978b20a3a7e5d62a3e6
Sep 16 18:46:09.306795 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:46:09.306752    2544 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-pdfpq" event={"ID":"db1e3b32-114c-4bd9-b28c-21a2a9d6efaa","Type":"ContainerStarted","Data":"03dce9ed0edc7b287eb96090eb2d45480f4415c9367d1978b20a3a7e5d62a3e6"}
Sep 16 18:46:09.311141 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:46:09.311090    2544 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"dbb1a91b-a0d3-477a-a98b-82fa11d56733","Type":"ContainerStarted","Data":"4a7850c58d8b3c043861d11cbf77250e4bd6fa9cc2bc2cba630fea990572fc24"}
Sep 16 18:46:09.311296 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:46:09.311161    2544 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"dbb1a91b-a0d3-477a-a98b-82fa11d56733","Type":"ContainerStarted","Data":"ff3a43bc7d1eb2991ea3b2411cb1334586660cefab76175ab4c65913134d761a"}
Sep 16 18:46:09.311296 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:46:09.311201    2544 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"dbb1a91b-a0d3-477a-a98b-82fa11d56733","Type":"ContainerStarted","Data":"360d70a4d0b601301e62bb1da48989c99735804d375503a6a10a3be829ddfe69"}
Sep 16 18:46:09.311296 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:46:09.311241    2544 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"dbb1a91b-a0d3-477a-a98b-82fa11d56733","Type":"ContainerStarted","Data":"51618dd4a3c2b428c5fe23d46774fd4cfa7b60dc0adfedc67bdd1e3ea84e0d89"}
Sep 16 18:46:09.341465 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:46:09.340782    2544 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-k8s-0" podStartSLOduration=3.337379893 podStartE2EDuration="9.340738439s" totalImagesPullingTime="5.978671216s" totalInitContainerRuntime="24.68733ms" isStatefulPod=false podCreationTimestamp="2026-09-16 18:46:00 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:46:09.339764551 +0000 UTC m=+132.128767112" watchObservedRunningTime="2026-09-16 18:46:09.340738439 +0000 UTC m=+132.129740972"
Sep 16 18:46:10.318394 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:46:10.318351    2544 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-pdfpq" event={"ID":"db1e3b32-114c-4bd9-b28c-21a2a9d6efaa","Type":"ContainerStarted","Data":"725635dbdf69a7716d6a71e10dc3c7e499dc62012ec4894fba3c907fcde200b2"}
Sep 16 18:46:10.318841 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:46:10.318428    2544 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-pdfpq" event={"ID":"db1e3b32-114c-4bd9-b28c-21a2a9d6efaa","Type":"ContainerStarted","Data":"eb5a9c5b97de0fa4e30f6bb78f4376fc73a16b106ef0e17a172b42eaf3f847e2"}
Sep 16 18:46:10.336900 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:46:10.336843    2544 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-pdfpq" podStartSLOduration=132.210811463 podStartE2EDuration="2m13.336825926s" totalImagesPullingTime="1.126014463s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:43:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:46:10.33552275 +0000 UTC m=+133.124525307" watchObservedRunningTime="2026-09-16 18:46:10.336825926 +0000 UTC m=+133.125828461"
Sep 16 18:46:11.003256 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:46:11.003201    2544 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:46:18.360626 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:46:18.360581    2544 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-76f99584f4-m4dhw" event={"ID":"e03463ea-f247-46f9-b8a8-57ccef5477d0","Type":"ContainerStarted","Data":"513d102a460fd670e826b7ec2b573924faaa39e4524460923ddc17f3dd007667"}
Sep 16 18:46:18.361068 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:46:18.360836    2544 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/downloads-76f99584f4-m4dhw"
Sep 16 18:46:18.362629 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:46:18.362596    2544 patch_prober.go:28] interesting pod/downloads-76f99584f4-m4dhw container/download-server namespace/openshift-console: Readiness probe status=failure output="Get \"http://10.132.0.20:8080/\": dial tcp 10.132.0.20:8080: connect: connection refused" start-of-body=
Sep 16 18:46:18.362752 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:46:18.362651    2544 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-console/downloads-76f99584f4-m4dhw" podUID="e03463ea-f247-46f9-b8a8-57ccef5477d0" containerName="download-server" probeResult="failure" output="Get \"http://10.132.0.20:8080/\": dial tcp 10.132.0.20:8080: connect: connection refused"
Sep 16 18:46:18.381886 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:46:18.381840    2544 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/downloads-76f99584f4-m4dhw" podStartSLOduration=1.998730788 podStartE2EDuration="19.381825553s" totalImagesPullingTime="17.383094765s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:45:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:46:18.380025752 +0000 UTC m=+141.169028304" watchObservedRunningTime="2026-09-16 18:46:18.381825553 +0000 UTC m=+141.170828087"
Sep 16 18:46:18.955048 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:46:18.955001    2544 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/metrics-server-7544b7b8b8-fk7dc"
Sep 16 18:46:18.955226 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:46:18.955076    2544 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/metrics-server-7544b7b8b8-fk7dc"
Sep 16 18:46:19.376678 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:46:19.376640    2544 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/downloads-76f99584f4-m4dhw"
Sep 16 18:46:19.569638 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:46:19.569602    2544 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_7a642ca2-3fee-4a4c-8c92-3f3069a99ac5/init-config-reloader/0.log"
Sep 16 18:46:19.771467 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:46:19.771373    2544 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_7a642ca2-3fee-4a4c-8c92-3f3069a99ac5/alertmanager/0.log"
Sep 16 18:46:19.974365 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:46:19.974326    2544 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_7a642ca2-3fee-4a4c-8c92-3f3069a99ac5/config-reloader/0.log"
Sep 16 18:46:20.180169 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:46:20.180134    2544 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_7a642ca2-3fee-4a4c-8c92-3f3069a99ac5/kube-rbac-proxy-web/0.log"
Sep 16 18:46:20.370619 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:46:20.370587    2544 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_7a642ca2-3fee-4a4c-8c92-3f3069a99ac5/kube-rbac-proxy/0.log"
Sep 16 18:46:20.577796 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:46:20.577771    2544 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_7a642ca2-3fee-4a4c-8c92-3f3069a99ac5/kube-rbac-proxy-metric/0.log"
Sep 16 18:46:20.796936 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:46:20.796911    2544 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_7a642ca2-3fee-4a4c-8c92-3f3069a99ac5/prom-label-proxy/0.log"
Sep 16 18:46:21.169506 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:46:21.169474    2544 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-59fbdd77d-2fwf6_2b942120-fb59-49f7-a96f-3f7165b19110/kube-state-metrics/0.log"
Sep 16 18:46:21.378974 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:46:21.378933    2544 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-59fbdd77d-2fwf6_2b942120-fb59-49f7-a96f-3f7165b19110/kube-rbac-proxy-main/0.log"
Sep 16 18:46:21.582921 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:46:21.582891    2544 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-59fbdd77d-2fwf6_2b942120-fb59-49f7-a96f-3f7165b19110/kube-rbac-proxy-self/0.log"
Sep 16 18:46:21.770127 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:46:21.770093    2544 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_metrics-server-7544b7b8b8-fk7dc_e96b6d09-3c97-41ed-860d-02ffed52f284/metrics-server/0.log"
Sep 16 18:46:23.376451 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:46:23.376397    2544 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-sljjd_8b245d92-ed58-4240-8b0d-7dab57750690/init-textfile/0.log"
Sep 16 18:46:23.570280 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:46:23.570254    2544 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-sljjd_8b245d92-ed58-4240-8b0d-7dab57750690/node-exporter/0.log"
Sep 16 18:46:23.769381 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:46:23.769307    2544 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-sljjd_8b245d92-ed58-4240-8b0d-7dab57750690/kube-rbac-proxy/0.log"
Sep 16 18:46:23.976242 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:46:23.976209    2544 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-67dcf8c979-694l8_ce1ddaa9-0566-4f7a-bc19-e7a7b958ae22/kube-rbac-proxy-main/0.log"
Sep 16 18:46:24.169724 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:46:24.169690    2544 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-67dcf8c979-694l8_ce1ddaa9-0566-4f7a-bc19-e7a7b958ae22/kube-rbac-proxy-self/0.log"
Sep 16 18:46:24.376770 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:46:24.376737    2544 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-67dcf8c979-694l8_ce1ddaa9-0566-4f7a-bc19-e7a7b958ae22/openshift-state-metrics/0.log"
Sep 16 18:46:24.571191 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:46:24.571164    2544 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_dbb1a91b-a0d3-477a-a98b-82fa11d56733/init-config-reloader/0.log"
Sep 16 18:46:24.774765 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:46:24.774733    2544 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_dbb1a91b-a0d3-477a-a98b-82fa11d56733/prometheus/0.log"
Sep 16 18:46:24.970282 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:46:24.970197    2544 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_dbb1a91b-a0d3-477a-a98b-82fa11d56733/config-reloader/0.log"
Sep 16 18:46:25.175720 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:46:25.175689    2544 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_dbb1a91b-a0d3-477a-a98b-82fa11d56733/thanos-sidecar/0.log"
Sep 16 18:46:25.370992 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:46:25.370953    2544 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_dbb1a91b-a0d3-477a-a98b-82fa11d56733/kube-rbac-proxy-web/0.log"
Sep 16 18:46:25.571932 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:46:25.571893    2544 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_dbb1a91b-a0d3-477a-a98b-82fa11d56733/kube-rbac-proxy/0.log"
Sep 16 18:46:25.770015 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:46:25.769936    2544 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_dbb1a91b-a0d3-477a-a98b-82fa11d56733/kube-rbac-proxy-thanos/0.log"
Sep 16 18:46:26.370901 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:46:26.370869    2544 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-admission-webhook-5864cd99d7-pk89k_837cae96-cf48-488d-8a96-e698f41de5df/prometheus-operator-admission-webhook/0.log"
Sep 16 18:46:26.572876 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:46:26.572825    2544 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-598cbd87f8-kmf7z_ac291abc-ad47-431c-89be-9de4e0f33e88/telemeter-client/0.log"
Sep 16 18:46:26.770024 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:46:26.769948    2544 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-598cbd87f8-kmf7z_ac291abc-ad47-431c-89be-9de4e0f33e88/reload/0.log"
Sep 16 18:46:26.969137 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:46:26.969105    2544 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-598cbd87f8-kmf7z_ac291abc-ad47-431c-89be-9de4e0f33e88/kube-rbac-proxy/0.log"
Sep 16 18:46:28.251221 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:46:28.251172    2544 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-image-registry/image-registry-76874c7675-4dhws" podUID="b03d56d8-b081-41fd-bac9-ba7da38f733a" containerName="registry" containerID="cri-o://4e010777b617ab2a76883143bbebcea44bdb5943d989d9f1a06eb4635b380a0e" gracePeriod=30
Sep 16 18:46:28.369159 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:46:28.369128    2544 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-console_networking-console-plugin-5dcbc7df6b-q4dbv_aa574cca-15b0-43dd-b30c-58d124d977d2/networking-console-plugin/0.log"
Sep 16 18:46:28.407035 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:46:28.406966    2544 generic.go:356] "Generic (PLEG): container finished" podID="b03d56d8-b081-41fd-bac9-ba7da38f733a" containerID="4e010777b617ab2a76883143bbebcea44bdb5943d989d9f1a06eb4635b380a0e" exitCode=0
Sep 16 18:46:28.407204 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:46:28.407045    2544 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-76874c7675-4dhws" event={"ID":"b03d56d8-b081-41fd-bac9-ba7da38f733a","Type":"ContainerDied","Data":"4e010777b617ab2a76883143bbebcea44bdb5943d989d9f1a06eb4635b380a0e"}
Sep 16 18:46:28.522650 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:46:28.522590    2544 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-76874c7675-4dhws"
Sep 16 18:46:28.567926 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:46:28.567905    2544 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-pfcb7_9e3f9e34-420e-43ef-96b4-c7cad6b2eaa9/console-operator/2.log"
Sep 16 18:46:28.651193 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:46:28.651165    2544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/b03d56d8-b081-41fd-bac9-ba7da38f733a-ca-trust-extracted\") pod \"b03d56d8-b081-41fd-bac9-ba7da38f733a\" (UID: \"b03d56d8-b081-41fd-bac9-ba7da38f733a\") "
Sep 16 18:46:28.651360 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:46:28.651218    2544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/b03d56d8-b081-41fd-bac9-ba7da38f733a-image-registry-private-configuration\") pod \"b03d56d8-b081-41fd-bac9-ba7da38f733a\" (UID: \"b03d56d8-b081-41fd-bac9-ba7da38f733a\") "
Sep 16 18:46:28.651360 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:46:28.651249    2544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/b03d56d8-b081-41fd-bac9-ba7da38f733a-trusted-ca\") pod \"b03d56d8-b081-41fd-bac9-ba7da38f733a\" (UID: \"b03d56d8-b081-41fd-bac9-ba7da38f733a\") "
Sep 16 18:46:28.651360 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:46:28.651276    2544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/b03d56d8-b081-41fd-bac9-ba7da38f733a-registry-certificates\") pod \"b03d56d8-b081-41fd-bac9-ba7da38f733a\" (UID: \"b03d56d8-b081-41fd-bac9-ba7da38f733a\") "
Sep 16 18:46:28.651360 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:46:28.651292    2544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/b03d56d8-b081-41fd-bac9-ba7da38f733a-registry-tls\") pod \"b03d56d8-b081-41fd-bac9-ba7da38f733a\" (UID: \"b03d56d8-b081-41fd-bac9-ba7da38f733a\") "
Sep 16 18:46:28.651360 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:46:28.651317    2544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zbbk8\" (UniqueName: \"kubernetes.io/projected/b03d56d8-b081-41fd-bac9-ba7da38f733a-kube-api-access-zbbk8\") pod \"b03d56d8-b081-41fd-bac9-ba7da38f733a\" (UID: \"b03d56d8-b081-41fd-bac9-ba7da38f733a\") "
Sep 16 18:46:28.651360 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:46:28.651343    2544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/b03d56d8-b081-41fd-bac9-ba7da38f733a-bound-sa-token\") pod \"b03d56d8-b081-41fd-bac9-ba7da38f733a\" (UID: \"b03d56d8-b081-41fd-bac9-ba7da38f733a\") "
Sep 16 18:46:28.651727 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:46:28.651377    2544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/b03d56d8-b081-41fd-bac9-ba7da38f733a-installation-pull-secrets\") pod \"b03d56d8-b081-41fd-bac9-ba7da38f733a\" (UID: \"b03d56d8-b081-41fd-bac9-ba7da38f733a\") "
Sep 16 18:46:28.651858 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:46:28.651822    2544 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b03d56d8-b081-41fd-bac9-ba7da38f733a-trusted-ca" (OuterVolumeSpecName: "trusted-ca") pod "b03d56d8-b081-41fd-bac9-ba7da38f733a" (UID: "b03d56d8-b081-41fd-bac9-ba7da38f733a"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 18:46:28.652221 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:46:28.652189    2544 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b03d56d8-b081-41fd-bac9-ba7da38f733a-registry-certificates" (OuterVolumeSpecName: "registry-certificates") pod "b03d56d8-b081-41fd-bac9-ba7da38f733a" (UID: "b03d56d8-b081-41fd-bac9-ba7da38f733a"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 18:46:28.653981 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:46:28.653949    2544 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b03d56d8-b081-41fd-bac9-ba7da38f733a-kube-api-access-zbbk8" (OuterVolumeSpecName: "kube-api-access-zbbk8") pod "b03d56d8-b081-41fd-bac9-ba7da38f733a" (UID: "b03d56d8-b081-41fd-bac9-ba7da38f733a"). InnerVolumeSpecName "kube-api-access-zbbk8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 18:46:28.654198 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:46:28.654148    2544 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b03d56d8-b081-41fd-bac9-ba7da38f733a-bound-sa-token" (OuterVolumeSpecName: "bound-sa-token") pod "b03d56d8-b081-41fd-bac9-ba7da38f733a" (UID: "b03d56d8-b081-41fd-bac9-ba7da38f733a"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 18:46:28.654198 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:46:28.654189    2544 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b03d56d8-b081-41fd-bac9-ba7da38f733a-registry-tls" (OuterVolumeSpecName: "registry-tls") pod "b03d56d8-b081-41fd-bac9-ba7da38f733a" (UID: "b03d56d8-b081-41fd-bac9-ba7da38f733a"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 18:46:28.654333 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:46:28.654205    2544 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b03d56d8-b081-41fd-bac9-ba7da38f733a-image-registry-private-configuration" (OuterVolumeSpecName: "image-registry-private-configuration") pod "b03d56d8-b081-41fd-bac9-ba7da38f733a" (UID: "b03d56d8-b081-41fd-bac9-ba7da38f733a"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 18:46:28.654333 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:46:28.654263    2544 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b03d56d8-b081-41fd-bac9-ba7da38f733a-installation-pull-secrets" (OuterVolumeSpecName: "installation-pull-secrets") pod "b03d56d8-b081-41fd-bac9-ba7da38f733a" (UID: "b03d56d8-b081-41fd-bac9-ba7da38f733a"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 18:46:28.662279 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:46:28.662253    2544 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b03d56d8-b081-41fd-bac9-ba7da38f733a-ca-trust-extracted" (OuterVolumeSpecName: "ca-trust-extracted") pod "b03d56d8-b081-41fd-bac9-ba7da38f733a" (UID: "b03d56d8-b081-41fd-bac9-ba7da38f733a"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 18:46:28.753013 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:46:28.752992    2544 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/b03d56d8-b081-41fd-bac9-ba7da38f733a-trusted-ca\") on node \"ip-10-0-132-80.ec2.internal\" DevicePath \"\""
Sep 16 18:46:28.753013 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:46:28.753014    2544 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/b03d56d8-b081-41fd-bac9-ba7da38f733a-registry-certificates\") on node \"ip-10-0-132-80.ec2.internal\" DevicePath \"\""
Sep 16 18:46:28.753139 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:46:28.753024    2544 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/b03d56d8-b081-41fd-bac9-ba7da38f733a-registry-tls\") on node \"ip-10-0-132-80.ec2.internal\" DevicePath \"\""
Sep 16 18:46:28.753139 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:46:28.753032    2544 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zbbk8\" (UniqueName: \"kubernetes.io/projected/b03d56d8-b081-41fd-bac9-ba7da38f733a-kube-api-access-zbbk8\") on node \"ip-10-0-132-80.ec2.internal\" DevicePath \"\""
Sep 16 18:46:28.753139 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:46:28.753040    2544 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/b03d56d8-b081-41fd-bac9-ba7da38f733a-bound-sa-token\") on node \"ip-10-0-132-80.ec2.internal\" DevicePath \"\""
Sep 16 18:46:28.753139 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:46:28.753048    2544 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/b03d56d8-b081-41fd-bac9-ba7da38f733a-installation-pull-secrets\") on node \"ip-10-0-132-80.ec2.internal\" DevicePath \"\""
Sep 16 18:46:28.753139 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:46:28.753057    2544 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/b03d56d8-b081-41fd-bac9-ba7da38f733a-ca-trust-extracted\") on node \"ip-10-0-132-80.ec2.internal\" DevicePath \"\""
Sep 16 18:46:28.753139 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:46:28.753066    2544 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/b03d56d8-b081-41fd-bac9-ba7da38f733a-image-registry-private-configuration\") on node \"ip-10-0-132-80.ec2.internal\" DevicePath \"\""
Sep 16 18:46:28.771834 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:46:28.771814    2544 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-pfcb7_9e3f9e34-420e-43ef-96b4-c7cad6b2eaa9/console-operator/3.log"
Sep 16 18:46:29.170768 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:46:29.170734    2544 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_downloads-76f99584f4-m4dhw_e03463ea-f247-46f9-b8a8-57ccef5477d0/download-server/0.log"
Sep 16 18:46:29.412633 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:46:29.412597    2544 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-76874c7675-4dhws" event={"ID":"b03d56d8-b081-41fd-bac9-ba7da38f733a","Type":"ContainerDied","Data":"1ea2e77a794610d6355a62d061729ba524f0678192386319965ccdd3fd5698ad"}
Sep 16 18:46:29.413030 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:46:29.412656    2544 scope.go:117] "RemoveContainer" containerID="4e010777b617ab2a76883143bbebcea44bdb5943d989d9f1a06eb4635b380a0e"
Sep 16 18:46:29.413030 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:46:29.412601    2544 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-76874c7675-4dhws"
Sep 16 18:46:29.433300 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:46:29.433275    2544 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-76874c7675-4dhws"]
Sep 16 18:46:29.439370 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:46:29.439347    2544 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-76874c7675-4dhws"]
Sep 16 18:46:29.721648 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:46:29.721577    2544 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b03d56d8-b081-41fd-bac9-ba7da38f733a" path="/var/lib/kubelet/pods/b03d56d8-b081-41fd-bac9-ba7da38f733a/volumes"
Sep 16 18:46:29.775745 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:46:29.775719    2544 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-jmnkv_ee8d40ee-7c8a-496a-a4e4-56cdc3e0aaa2/serve-healthcheck-canary/0.log"
Sep 16 18:46:38.959484 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:46:38.959454    2544 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/metrics-server-7544b7b8b8-fk7dc"
Sep 16 18:46:38.963195 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:46:38.963176    2544 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/metrics-server-7544b7b8b8-fk7dc"
Sep 16 18:47:01.002642 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:47:01.002603    2544 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:47:01.085701 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:47:01.085674    2544 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:47:01.564324 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:47:01.564297    2544 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:47:14.908346 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:47:14.908308    2544 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Sep 16 18:47:14.908853 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:47:14.908621    2544 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="7a642ca2-3fee-4a4c-8c92-3f3069a99ac5" containerName="alertmanager" containerID="cri-o://00d9804e08c15f830a6e69132dc25ac411f4ca4977e7f7d37dd63e8c38f308c4" gracePeriod=120
Sep 16 18:47:14.908853 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:47:14.908699    2544 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="7a642ca2-3fee-4a4c-8c92-3f3069a99ac5" containerName="kube-rbac-proxy-metric" containerID="cri-o://54bdf2fa8350cdfc6440f1011b737bf409d17e83ef4439f97779fdfe602db92a" gracePeriod=120
Sep 16 18:47:14.908853 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:47:14.908712    2544 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="7a642ca2-3fee-4a4c-8c92-3f3069a99ac5" containerName="prom-label-proxy" containerID="cri-o://06accc71e639e5500bc2a7e464c9033d53d565ed84ff1acea24ad664215c3256" gracePeriod=120
Sep 16 18:47:14.908853 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:47:14.908753    2544 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="7a642ca2-3fee-4a4c-8c92-3f3069a99ac5" containerName="kube-rbac-proxy" containerID="cri-o://26d9eb49dfb46f1825543ad633aeae54b3bb118af49bc59757034e90de2169e0" gracePeriod=120
Sep 16 18:47:14.908853 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:47:14.908764    2544 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="7a642ca2-3fee-4a4c-8c92-3f3069a99ac5" containerName="kube-rbac-proxy-web" containerID="cri-o://ee9f9bb76fc17a89b5bc2bbef261d4f3168e436faf413593b3d466b7af49ff27" gracePeriod=120
Sep 16 18:47:14.908853 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:47:14.908778    2544 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="7a642ca2-3fee-4a4c-8c92-3f3069a99ac5" containerName="config-reloader" containerID="cri-o://0fea312a41c726c8f76ad6706d748cf7cda93b7bc61ffaf4e203edb5a4133e88" gracePeriod=120
Sep 16 18:47:15.609187 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:47:15.609150    2544 generic.go:356] "Generic (PLEG): container finished" podID="7a642ca2-3fee-4a4c-8c92-3f3069a99ac5" containerID="06accc71e639e5500bc2a7e464c9033d53d565ed84ff1acea24ad664215c3256" exitCode=0
Sep 16 18:47:15.609187 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:47:15.609178    2544 generic.go:356] "Generic (PLEG): container finished" podID="7a642ca2-3fee-4a4c-8c92-3f3069a99ac5" containerID="26d9eb49dfb46f1825543ad633aeae54b3bb118af49bc59757034e90de2169e0" exitCode=0
Sep 16 18:47:15.609187 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:47:15.609185    2544 generic.go:356] "Generic (PLEG): container finished" podID="7a642ca2-3fee-4a4c-8c92-3f3069a99ac5" containerID="0fea312a41c726c8f76ad6706d748cf7cda93b7bc61ffaf4e203edb5a4133e88" exitCode=0
Sep 16 18:47:15.609187 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:47:15.609191    2544 generic.go:356] "Generic (PLEG): container finished" podID="7a642ca2-3fee-4a4c-8c92-3f3069a99ac5" containerID="00d9804e08c15f830a6e69132dc25ac411f4ca4977e7f7d37dd63e8c38f308c4" exitCode=0
Sep 16 18:47:15.609438 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:47:15.609221    2544 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"7a642ca2-3fee-4a4c-8c92-3f3069a99ac5","Type":"ContainerDied","Data":"06accc71e639e5500bc2a7e464c9033d53d565ed84ff1acea24ad664215c3256"}
Sep 16 18:47:15.609438 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:47:15.609288    2544 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"7a642ca2-3fee-4a4c-8c92-3f3069a99ac5","Type":"ContainerDied","Data":"26d9eb49dfb46f1825543ad633aeae54b3bb118af49bc59757034e90de2169e0"}
Sep 16 18:47:15.609438 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:47:15.609315    2544 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"7a642ca2-3fee-4a4c-8c92-3f3069a99ac5","Type":"ContainerDied","Data":"0fea312a41c726c8f76ad6706d748cf7cda93b7bc61ffaf4e203edb5a4133e88"}
Sep 16 18:47:15.609438 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:47:15.609340    2544 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"7a642ca2-3fee-4a4c-8c92-3f3069a99ac5","Type":"ContainerDied","Data":"00d9804e08c15f830a6e69132dc25ac411f4ca4977e7f7d37dd63e8c38f308c4"}
Sep 16 18:47:16.148212 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:47:16.148189    2544 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Sep 16 18:47:16.243610 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:47:16.243522    2544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/7a642ca2-3fee-4a4c-8c92-3f3069a99ac5-secret-alertmanager-kube-rbac-proxy-metric\") pod \"7a642ca2-3fee-4a4c-8c92-3f3069a99ac5\" (UID: \"7a642ca2-3fee-4a4c-8c92-3f3069a99ac5\") "
Sep 16 18:47:16.243610 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:47:16.243581    2544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/7a642ca2-3fee-4a4c-8c92-3f3069a99ac5-tls-assets\") pod \"7a642ca2-3fee-4a4c-8c92-3f3069a99ac5\" (UID: \"7a642ca2-3fee-4a4c-8c92-3f3069a99ac5\") "
Sep 16 18:47:16.243610 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:47:16.243601    2544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/7a642ca2-3fee-4a4c-8c92-3f3069a99ac5-secret-alertmanager-kube-rbac-proxy-web\") pod \"7a642ca2-3fee-4a4c-8c92-3f3069a99ac5\" (UID: \"7a642ca2-3fee-4a4c-8c92-3f3069a99ac5\") "
Sep 16 18:47:16.243886 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:47:16.243627    2544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7a642ca2-3fee-4a4c-8c92-3f3069a99ac5-alertmanager-trusted-ca-bundle\") pod \"7a642ca2-3fee-4a4c-8c92-3f3069a99ac5\" (UID: \"7a642ca2-3fee-4a4c-8c92-3f3069a99ac5\") "
Sep 16 18:47:16.243886 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:47:16.243677    2544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/7a642ca2-3fee-4a4c-8c92-3f3069a99ac5-config-out\") pod \"7a642ca2-3fee-4a4c-8c92-3f3069a99ac5\" (UID: \"7a642ca2-3fee-4a4c-8c92-3f3069a99ac5\") "
Sep 16 18:47:16.243886 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:47:16.243725    2544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/7a642ca2-3fee-4a4c-8c92-3f3069a99ac5-config-volume\") pod \"7a642ca2-3fee-4a4c-8c92-3f3069a99ac5\" (UID: \"7a642ca2-3fee-4a4c-8c92-3f3069a99ac5\") "
Sep 16 18:47:16.244114 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:47:16.244086    2544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/7a642ca2-3fee-4a4c-8c92-3f3069a99ac5-secret-alertmanager-main-tls\") pod \"7a642ca2-3fee-4a4c-8c92-3f3069a99ac5\" (UID: \"7a642ca2-3fee-4a4c-8c92-3f3069a99ac5\") "
Sep 16 18:47:16.244295 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:47:16.244276    2544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/7a642ca2-3fee-4a4c-8c92-3f3069a99ac5-alertmanager-main-db\") pod \"7a642ca2-3fee-4a4c-8c92-3f3069a99ac5\" (UID: \"7a642ca2-3fee-4a4c-8c92-3f3069a99ac5\") "
Sep 16 18:47:16.244423 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:47:16.244407    2544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/7a642ca2-3fee-4a4c-8c92-3f3069a99ac5-cluster-tls-config\") pod \"7a642ca2-3fee-4a4c-8c92-3f3069a99ac5\" (UID: \"7a642ca2-3fee-4a4c-8c92-3f3069a99ac5\") "
Sep 16 18:47:16.244534 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:47:16.244518    2544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/7a642ca2-3fee-4a4c-8c92-3f3069a99ac5-secret-alertmanager-kube-rbac-proxy\") pod \"7a642ca2-3fee-4a4c-8c92-3f3069a99ac5\" (UID: \"7a642ca2-3fee-4a4c-8c92-3f3069a99ac5\") "
Sep 16 18:47:16.244683 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:47:16.244668    2544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-bgjh4\" (UniqueName: \"kubernetes.io/projected/7a642ca2-3fee-4a4c-8c92-3f3069a99ac5-kube-api-access-bgjh4\") pod \"7a642ca2-3fee-4a4c-8c92-3f3069a99ac5\" (UID: \"7a642ca2-3fee-4a4c-8c92-3f3069a99ac5\") "
Sep 16 18:47:16.244828 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:47:16.244813    2544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/7a642ca2-3fee-4a4c-8c92-3f3069a99ac5-metrics-client-ca\") pod \"7a642ca2-3fee-4a4c-8c92-3f3069a99ac5\" (UID: \"7a642ca2-3fee-4a4c-8c92-3f3069a99ac5\") "
Sep 16 18:47:16.244948 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:47:16.244934    2544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/7a642ca2-3fee-4a4c-8c92-3f3069a99ac5-web-config\") pod \"7a642ca2-3fee-4a4c-8c92-3f3069a99ac5\" (UID: \"7a642ca2-3fee-4a4c-8c92-3f3069a99ac5\") "
Sep 16 18:47:16.246499 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:47:16.244094    2544 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7a642ca2-3fee-4a4c-8c92-3f3069a99ac5-alertmanager-trusted-ca-bundle" (OuterVolumeSpecName: "alertmanager-trusted-ca-bundle") pod "7a642ca2-3fee-4a4c-8c92-3f3069a99ac5" (UID: "7a642ca2-3fee-4a4c-8c92-3f3069a99ac5"). InnerVolumeSpecName "alertmanager-trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 18:47:16.246499 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:47:16.246256    2544 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7a642ca2-3fee-4a4c-8c92-3f3069a99ac5-tls-assets" (OuterVolumeSpecName: "tls-assets") pod "7a642ca2-3fee-4a4c-8c92-3f3069a99ac5" (UID: "7a642ca2-3fee-4a4c-8c92-3f3069a99ac5"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 18:47:16.246499 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:47:16.246317    2544 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7a642ca2-3fee-4a4c-8c92-3f3069a99ac5-secret-alertmanager-kube-rbac-proxy-web" (OuterVolumeSpecName: "secret-alertmanager-kube-rbac-proxy-web") pod "7a642ca2-3fee-4a4c-8c92-3f3069a99ac5" (UID: "7a642ca2-3fee-4a4c-8c92-3f3069a99ac5"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy-web". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 18:47:16.246847 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:47:16.246745    2544 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7a642ca2-3fee-4a4c-8c92-3f3069a99ac5-alertmanager-main-db" (OuterVolumeSpecName: "alertmanager-main-db") pod "7a642ca2-3fee-4a4c-8c92-3f3069a99ac5" (UID: "7a642ca2-3fee-4a4c-8c92-3f3069a99ac5"). InnerVolumeSpecName "alertmanager-main-db". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 18:47:16.246847 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:47:16.246746    2544 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7a642ca2-3fee-4a4c-8c92-3f3069a99ac5-secret-alertmanager-kube-rbac-proxy-metric" (OuterVolumeSpecName: "secret-alertmanager-kube-rbac-proxy-metric") pod "7a642ca2-3fee-4a4c-8c92-3f3069a99ac5" (UID: "7a642ca2-3fee-4a4c-8c92-3f3069a99ac5"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy-metric". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 18:47:16.247037 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:47:16.246982    2544 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7a642ca2-3fee-4a4c-8c92-3f3069a99ac5-secret-alertmanager-main-tls" (OuterVolumeSpecName: "secret-alertmanager-main-tls") pod "7a642ca2-3fee-4a4c-8c92-3f3069a99ac5" (UID: "7a642ca2-3fee-4a4c-8c92-3f3069a99ac5"). InnerVolumeSpecName "secret-alertmanager-main-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 18:47:16.247112 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:47:16.247097    2544 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7a642ca2-3fee-4a4c-8c92-3f3069a99ac5-metrics-client-ca" (OuterVolumeSpecName: "metrics-client-ca") pod "7a642ca2-3fee-4a4c-8c92-3f3069a99ac5" (UID: "7a642ca2-3fee-4a4c-8c92-3f3069a99ac5"). InnerVolumeSpecName "metrics-client-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 18:47:16.247218 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:47:16.247190    2544 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7a642ca2-3fee-4a4c-8c92-3f3069a99ac5-config-out" (OuterVolumeSpecName: "config-out") pod "7a642ca2-3fee-4a4c-8c92-3f3069a99ac5" (UID: "7a642ca2-3fee-4a4c-8c92-3f3069a99ac5"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 18:47:16.247279 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:47:16.247259    2544 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7a642ca2-3fee-4a4c-8c92-3f3069a99ac5-config-volume" (OuterVolumeSpecName: "config-volume") pod "7a642ca2-3fee-4a4c-8c92-3f3069a99ac5" (UID: "7a642ca2-3fee-4a4c-8c92-3f3069a99ac5"). InnerVolumeSpecName "config-volume". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 18:47:16.248496 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:47:16.248475    2544 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7a642ca2-3fee-4a4c-8c92-3f3069a99ac5-kube-api-access-bgjh4" (OuterVolumeSpecName: "kube-api-access-bgjh4") pod "7a642ca2-3fee-4a4c-8c92-3f3069a99ac5" (UID: "7a642ca2-3fee-4a4c-8c92-3f3069a99ac5"). InnerVolumeSpecName "kube-api-access-bgjh4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 18:47:16.248980 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:47:16.248960    2544 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7a642ca2-3fee-4a4c-8c92-3f3069a99ac5-secret-alertmanager-kube-rbac-proxy" (OuterVolumeSpecName: "secret-alertmanager-kube-rbac-proxy") pod "7a642ca2-3fee-4a4c-8c92-3f3069a99ac5" (UID: "7a642ca2-3fee-4a4c-8c92-3f3069a99ac5"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 18:47:16.253150 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:47:16.253125    2544 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7a642ca2-3fee-4a4c-8c92-3f3069a99ac5-cluster-tls-config" (OuterVolumeSpecName: "cluster-tls-config") pod "7a642ca2-3fee-4a4c-8c92-3f3069a99ac5" (UID: "7a642ca2-3fee-4a4c-8c92-3f3069a99ac5"). InnerVolumeSpecName "cluster-tls-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 18:47:16.259367 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:47:16.259346    2544 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7a642ca2-3fee-4a4c-8c92-3f3069a99ac5-web-config" (OuterVolumeSpecName: "web-config") pod "7a642ca2-3fee-4a4c-8c92-3f3069a99ac5" (UID: "7a642ca2-3fee-4a4c-8c92-3f3069a99ac5"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 18:47:16.350676 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:47:16.346195    2544 reconciler_common.go:299] "Volume detached for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/7a642ca2-3fee-4a4c-8c92-3f3069a99ac5-config-volume\") on node \"ip-10-0-132-80.ec2.internal\" DevicePath \"\""
Sep 16 18:47:16.350676 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:47:16.346324    2544 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/7a642ca2-3fee-4a4c-8c92-3f3069a99ac5-secret-alertmanager-main-tls\") on node \"ip-10-0-132-80.ec2.internal\" DevicePath \"\""
Sep 16 18:47:16.350676 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:47:16.346341    2544 reconciler_common.go:299] "Volume detached for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/7a642ca2-3fee-4a4c-8c92-3f3069a99ac5-alertmanager-main-db\") on node \"ip-10-0-132-80.ec2.internal\" DevicePath \"\""
Sep 16 18:47:16.350676 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:47:16.346358    2544 reconciler_common.go:299] "Volume detached for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/7a642ca2-3fee-4a4c-8c92-3f3069a99ac5-cluster-tls-config\") on node \"ip-10-0-132-80.ec2.internal\" DevicePath \"\""
Sep 16 18:47:16.350676 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:47:16.346381    2544 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/7a642ca2-3fee-4a4c-8c92-3f3069a99ac5-secret-alertmanager-kube-rbac-proxy\") on node \"ip-10-0-132-80.ec2.internal\" DevicePath \"\""
Sep 16 18:47:16.350676 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:47:16.346395    2544 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bgjh4\" (UniqueName: \"kubernetes.io/projected/7a642ca2-3fee-4a4c-8c92-3f3069a99ac5-kube-api-access-bgjh4\") on node \"ip-10-0-132-80.ec2.internal\" DevicePath \"\""
Sep 16 18:47:16.350676 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:47:16.346409    2544 reconciler_common.go:299] "Volume detached for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/7a642ca2-3fee-4a4c-8c92-3f3069a99ac5-metrics-client-ca\") on node \"ip-10-0-132-80.ec2.internal\" DevicePath \"\""
Sep 16 18:47:16.350676 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:47:16.346422    2544 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/7a642ca2-3fee-4a4c-8c92-3f3069a99ac5-web-config\") on node \"ip-10-0-132-80.ec2.internal\" DevicePath \"\""
Sep 16 18:47:16.350676 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:47:16.346442    2544 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/7a642ca2-3fee-4a4c-8c92-3f3069a99ac5-secret-alertmanager-kube-rbac-proxy-metric\") on node \"ip-10-0-132-80.ec2.internal\" DevicePath \"\""
Sep 16 18:47:16.350676 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:47:16.346456    2544 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/7a642ca2-3fee-4a4c-8c92-3f3069a99ac5-tls-assets\") on node \"ip-10-0-132-80.ec2.internal\" DevicePath \"\""
Sep 16 18:47:16.350676 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:47:16.346469    2544 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/7a642ca2-3fee-4a4c-8c92-3f3069a99ac5-secret-alertmanager-kube-rbac-proxy-web\") on node \"ip-10-0-132-80.ec2.internal\" DevicePath \"\""
Sep 16 18:47:16.350676 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:47:16.346483    2544 reconciler_common.go:299] "Volume detached for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7a642ca2-3fee-4a4c-8c92-3f3069a99ac5-alertmanager-trusted-ca-bundle\") on node \"ip-10-0-132-80.ec2.internal\" DevicePath \"\""
Sep 16 18:47:16.350676 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:47:16.346501    2544 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/7a642ca2-3fee-4a4c-8c92-3f3069a99ac5-config-out\") on node \"ip-10-0-132-80.ec2.internal\" DevicePath \"\""
Sep 16 18:47:16.615958 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:47:16.615920    2544 generic.go:356] "Generic (PLEG): container finished" podID="7a642ca2-3fee-4a4c-8c92-3f3069a99ac5" containerID="54bdf2fa8350cdfc6440f1011b737bf409d17e83ef4439f97779fdfe602db92a" exitCode=0
Sep 16 18:47:16.615958 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:47:16.615947    2544 generic.go:356] "Generic (PLEG): container finished" podID="7a642ca2-3fee-4a4c-8c92-3f3069a99ac5" containerID="ee9f9bb76fc17a89b5bc2bbef261d4f3168e436faf413593b3d466b7af49ff27" exitCode=0
Sep 16 18:47:16.616143 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:47:16.616001    2544 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"7a642ca2-3fee-4a4c-8c92-3f3069a99ac5","Type":"ContainerDied","Data":"54bdf2fa8350cdfc6440f1011b737bf409d17e83ef4439f97779fdfe602db92a"}
Sep 16 18:47:16.616143 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:47:16.616021    2544 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Sep 16 18:47:16.616143 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:47:16.616117    2544 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"7a642ca2-3fee-4a4c-8c92-3f3069a99ac5","Type":"ContainerDied","Data":"ee9f9bb76fc17a89b5bc2bbef261d4f3168e436faf413593b3d466b7af49ff27"}
Sep 16 18:47:16.616243 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:47:16.616164    2544 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"7a642ca2-3fee-4a4c-8c92-3f3069a99ac5","Type":"ContainerDied","Data":"ed3f704d98d544da2c3c5ee1cb84c96ba83a2c96e97341908f366fa44b5f9020"}
Sep 16 18:47:16.616243 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:47:16.616185    2544 scope.go:117] "RemoveContainer" containerID="06accc71e639e5500bc2a7e464c9033d53d565ed84ff1acea24ad664215c3256"
Sep 16 18:47:16.623712 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:47:16.623580    2544 scope.go:117] "RemoveContainer" containerID="54bdf2fa8350cdfc6440f1011b737bf409d17e83ef4439f97779fdfe602db92a"
Sep 16 18:47:16.630490 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:47:16.630473    2544 scope.go:117] "RemoveContainer" containerID="26d9eb49dfb46f1825543ad633aeae54b3bb118af49bc59757034e90de2169e0"
Sep 16 18:47:16.636830 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:47:16.636814    2544 scope.go:117] "RemoveContainer" containerID="ee9f9bb76fc17a89b5bc2bbef261d4f3168e436faf413593b3d466b7af49ff27"
Sep 16 18:47:16.643264 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:47:16.643245    2544 scope.go:117] "RemoveContainer" containerID="0fea312a41c726c8f76ad6706d748cf7cda93b7bc61ffaf4e203edb5a4133e88"
Sep 16 18:47:16.649949 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:47:16.649932    2544 scope.go:117] "RemoveContainer" containerID="00d9804e08c15f830a6e69132dc25ac411f4ca4977e7f7d37dd63e8c38f308c4"
Sep 16 18:47:16.654313 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:47:16.654157    2544 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Sep 16 18:47:16.655730 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:47:16.655711    2544 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Sep 16 18:47:16.660282 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:47:16.660264    2544 scope.go:117] "RemoveContainer" containerID="fa8b3480b4cd7b3ae46246e3d6cab25c95a961ca441be61b82514a692b4ec3cf"
Sep 16 18:47:16.667007 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:47:16.666992    2544 scope.go:117] "RemoveContainer" containerID="06accc71e639e5500bc2a7e464c9033d53d565ed84ff1acea24ad664215c3256"
Sep 16 18:47:16.667283 ip-10-0-132-80 kubenswrapper[2544]: E0916 18:47:16.667261    2544 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"06accc71e639e5500bc2a7e464c9033d53d565ed84ff1acea24ad664215c3256\": container with ID starting with 06accc71e639e5500bc2a7e464c9033d53d565ed84ff1acea24ad664215c3256 not found: ID does not exist" containerID="06accc71e639e5500bc2a7e464c9033d53d565ed84ff1acea24ad664215c3256"
Sep 16 18:47:16.667342 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:47:16.667290    2544 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"06accc71e639e5500bc2a7e464c9033d53d565ed84ff1acea24ad664215c3256"} err="failed to get container status \"06accc71e639e5500bc2a7e464c9033d53d565ed84ff1acea24ad664215c3256\": rpc error: code = NotFound desc = could not find container \"06accc71e639e5500bc2a7e464c9033d53d565ed84ff1acea24ad664215c3256\": container with ID starting with 06accc71e639e5500bc2a7e464c9033d53d565ed84ff1acea24ad664215c3256 not found: ID does not exist"
Sep 16 18:47:16.667342 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:47:16.667322    2544 scope.go:117] "RemoveContainer" containerID="54bdf2fa8350cdfc6440f1011b737bf409d17e83ef4439f97779fdfe602db92a"
Sep 16 18:47:16.667614 ip-10-0-132-80 kubenswrapper[2544]: E0916 18:47:16.667594    2544 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"54bdf2fa8350cdfc6440f1011b737bf409d17e83ef4439f97779fdfe602db92a\": container with ID starting with 54bdf2fa8350cdfc6440f1011b737bf409d17e83ef4439f97779fdfe602db92a not found: ID does not exist" containerID="54bdf2fa8350cdfc6440f1011b737bf409d17e83ef4439f97779fdfe602db92a"
Sep 16 18:47:16.667676 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:47:16.667620    2544 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"54bdf2fa8350cdfc6440f1011b737bf409d17e83ef4439f97779fdfe602db92a"} err="failed to get container status \"54bdf2fa8350cdfc6440f1011b737bf409d17e83ef4439f97779fdfe602db92a\": rpc error: code = NotFound desc = could not find container \"54bdf2fa8350cdfc6440f1011b737bf409d17e83ef4439f97779fdfe602db92a\": container with ID starting with 54bdf2fa8350cdfc6440f1011b737bf409d17e83ef4439f97779fdfe602db92a not found: ID does not exist"
Sep 16 18:47:16.667676 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:47:16.667638    2544 scope.go:117] "RemoveContainer" containerID="26d9eb49dfb46f1825543ad633aeae54b3bb118af49bc59757034e90de2169e0"
Sep 16 18:47:16.667889 ip-10-0-132-80 kubenswrapper[2544]: E0916 18:47:16.667874    2544 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"26d9eb49dfb46f1825543ad633aeae54b3bb118af49bc59757034e90de2169e0\": container with ID starting with 26d9eb49dfb46f1825543ad633aeae54b3bb118af49bc59757034e90de2169e0 not found: ID does not exist" containerID="26d9eb49dfb46f1825543ad633aeae54b3bb118af49bc59757034e90de2169e0"
Sep 16 18:47:16.667962 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:47:16.667892    2544 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"26d9eb49dfb46f1825543ad633aeae54b3bb118af49bc59757034e90de2169e0"} err="failed to get container status \"26d9eb49dfb46f1825543ad633aeae54b3bb118af49bc59757034e90de2169e0\": rpc error: code = NotFound desc = could not find container \"26d9eb49dfb46f1825543ad633aeae54b3bb118af49bc59757034e90de2169e0\": container with ID starting with 26d9eb49dfb46f1825543ad633aeae54b3bb118af49bc59757034e90de2169e0 not found: ID does not exist"
Sep 16 18:47:16.667962 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:47:16.667912    2544 scope.go:117] "RemoveContainer" containerID="ee9f9bb76fc17a89b5bc2bbef261d4f3168e436faf413593b3d466b7af49ff27"
Sep 16 18:47:16.668145 ip-10-0-132-80 kubenswrapper[2544]: E0916 18:47:16.668131    2544 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ee9f9bb76fc17a89b5bc2bbef261d4f3168e436faf413593b3d466b7af49ff27\": container with ID starting with ee9f9bb76fc17a89b5bc2bbef261d4f3168e436faf413593b3d466b7af49ff27 not found: ID does not exist" containerID="ee9f9bb76fc17a89b5bc2bbef261d4f3168e436faf413593b3d466b7af49ff27"
Sep 16 18:47:16.668182 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:47:16.668149    2544 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ee9f9bb76fc17a89b5bc2bbef261d4f3168e436faf413593b3d466b7af49ff27"} err="failed to get container status \"ee9f9bb76fc17a89b5bc2bbef261d4f3168e436faf413593b3d466b7af49ff27\": rpc error: code = NotFound desc = could not find container \"ee9f9bb76fc17a89b5bc2bbef261d4f3168e436faf413593b3d466b7af49ff27\": container with ID starting with ee9f9bb76fc17a89b5bc2bbef261d4f3168e436faf413593b3d466b7af49ff27 not found: ID does not exist"
Sep 16 18:47:16.668182 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:47:16.668161    2544 scope.go:117] "RemoveContainer" containerID="0fea312a41c726c8f76ad6706d748cf7cda93b7bc61ffaf4e203edb5a4133e88"
Sep 16 18:47:16.668403 ip-10-0-132-80 kubenswrapper[2544]: E0916 18:47:16.668364    2544 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0fea312a41c726c8f76ad6706d748cf7cda93b7bc61ffaf4e203edb5a4133e88\": container with ID starting with 0fea312a41c726c8f76ad6706d748cf7cda93b7bc61ffaf4e203edb5a4133e88 not found: ID does not exist" containerID="0fea312a41c726c8f76ad6706d748cf7cda93b7bc61ffaf4e203edb5a4133e88"
Sep 16 18:47:16.668438 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:47:16.668408    2544 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0fea312a41c726c8f76ad6706d748cf7cda93b7bc61ffaf4e203edb5a4133e88"} err="failed to get container status \"0fea312a41c726c8f76ad6706d748cf7cda93b7bc61ffaf4e203edb5a4133e88\": rpc error: code = NotFound desc = could not find container \"0fea312a41c726c8f76ad6706d748cf7cda93b7bc61ffaf4e203edb5a4133e88\": container with ID starting with 0fea312a41c726c8f76ad6706d748cf7cda93b7bc61ffaf4e203edb5a4133e88 not found: ID does not exist"
Sep 16 18:47:16.668438 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:47:16.668423    2544 scope.go:117] "RemoveContainer" containerID="00d9804e08c15f830a6e69132dc25ac411f4ca4977e7f7d37dd63e8c38f308c4"
Sep 16 18:47:16.668638 ip-10-0-132-80 kubenswrapper[2544]: E0916 18:47:16.668622    2544 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"00d9804e08c15f830a6e69132dc25ac411f4ca4977e7f7d37dd63e8c38f308c4\": container with ID starting with 00d9804e08c15f830a6e69132dc25ac411f4ca4977e7f7d37dd63e8c38f308c4 not found: ID does not exist" containerID="00d9804e08c15f830a6e69132dc25ac411f4ca4977e7f7d37dd63e8c38f308c4"
Sep 16 18:47:16.668684 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:47:16.668641    2544 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"00d9804e08c15f830a6e69132dc25ac411f4ca4977e7f7d37dd63e8c38f308c4"} err="failed to get container status \"00d9804e08c15f830a6e69132dc25ac411f4ca4977e7f7d37dd63e8c38f308c4\": rpc error: code = NotFound desc = could not find container \"00d9804e08c15f830a6e69132dc25ac411f4ca4977e7f7d37dd63e8c38f308c4\": container with ID starting with 00d9804e08c15f830a6e69132dc25ac411f4ca4977e7f7d37dd63e8c38f308c4 not found: ID does not exist"
Sep 16 18:47:16.668684 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:47:16.668653    2544 scope.go:117] "RemoveContainer" containerID="fa8b3480b4cd7b3ae46246e3d6cab25c95a961ca441be61b82514a692b4ec3cf"
Sep 16 18:47:16.668844 ip-10-0-132-80 kubenswrapper[2544]: E0916 18:47:16.668830    2544 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fa8b3480b4cd7b3ae46246e3d6cab25c95a961ca441be61b82514a692b4ec3cf\": container with ID starting with fa8b3480b4cd7b3ae46246e3d6cab25c95a961ca441be61b82514a692b4ec3cf not found: ID does not exist" containerID="fa8b3480b4cd7b3ae46246e3d6cab25c95a961ca441be61b82514a692b4ec3cf"
Sep 16 18:47:16.668880 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:47:16.668847    2544 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fa8b3480b4cd7b3ae46246e3d6cab25c95a961ca441be61b82514a692b4ec3cf"} err="failed to get container status \"fa8b3480b4cd7b3ae46246e3d6cab25c95a961ca441be61b82514a692b4ec3cf\": rpc error: code = NotFound desc = could not find container \"fa8b3480b4cd7b3ae46246e3d6cab25c95a961ca441be61b82514a692b4ec3cf\": container with ID starting with fa8b3480b4cd7b3ae46246e3d6cab25c95a961ca441be61b82514a692b4ec3cf not found: ID does not exist"
Sep 16 18:47:16.668880 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:47:16.668857    2544 scope.go:117] "RemoveContainer" containerID="06accc71e639e5500bc2a7e464c9033d53d565ed84ff1acea24ad664215c3256"
Sep 16 18:47:16.669056 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:47:16.669041    2544 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"06accc71e639e5500bc2a7e464c9033d53d565ed84ff1acea24ad664215c3256"} err="failed to get container status \"06accc71e639e5500bc2a7e464c9033d53d565ed84ff1acea24ad664215c3256\": rpc error: code = NotFound desc = could not find container \"06accc71e639e5500bc2a7e464c9033d53d565ed84ff1acea24ad664215c3256\": container with ID starting with 06accc71e639e5500bc2a7e464c9033d53d565ed84ff1acea24ad664215c3256 not found: ID does not exist"
Sep 16 18:47:16.669094 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:47:16.669056    2544 scope.go:117] "RemoveContainer" containerID="54bdf2fa8350cdfc6440f1011b737bf409d17e83ef4439f97779fdfe602db92a"
Sep 16 18:47:16.669260 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:47:16.669245    2544 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"54bdf2fa8350cdfc6440f1011b737bf409d17e83ef4439f97779fdfe602db92a"} err="failed to get container status \"54bdf2fa8350cdfc6440f1011b737bf409d17e83ef4439f97779fdfe602db92a\": rpc error: code = NotFound desc = could not find container \"54bdf2fa8350cdfc6440f1011b737bf409d17e83ef4439f97779fdfe602db92a\": container with ID starting with 54bdf2fa8350cdfc6440f1011b737bf409d17e83ef4439f97779fdfe602db92a not found: ID does not exist"
Sep 16 18:47:16.669305 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:47:16.669261    2544 scope.go:117] "RemoveContainer" containerID="26d9eb49dfb46f1825543ad633aeae54b3bb118af49bc59757034e90de2169e0"
Sep 16 18:47:16.669443 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:47:16.669428    2544 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"26d9eb49dfb46f1825543ad633aeae54b3bb118af49bc59757034e90de2169e0"} err="failed to get container status \"26d9eb49dfb46f1825543ad633aeae54b3bb118af49bc59757034e90de2169e0\": rpc error: code = NotFound desc = could not find container \"26d9eb49dfb46f1825543ad633aeae54b3bb118af49bc59757034e90de2169e0\": container with ID starting with 26d9eb49dfb46f1825543ad633aeae54b3bb118af49bc59757034e90de2169e0 not found: ID does not exist"
Sep 16 18:47:16.669485 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:47:16.669443    2544 scope.go:117] "RemoveContainer" containerID="ee9f9bb76fc17a89b5bc2bbef261d4f3168e436faf413593b3d466b7af49ff27"
Sep 16 18:47:16.669698 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:47:16.669680    2544 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ee9f9bb76fc17a89b5bc2bbef261d4f3168e436faf413593b3d466b7af49ff27"} err="failed to get container status \"ee9f9bb76fc17a89b5bc2bbef261d4f3168e436faf413593b3d466b7af49ff27\": rpc error: code = NotFound desc = could not find container \"ee9f9bb76fc17a89b5bc2bbef261d4f3168e436faf413593b3d466b7af49ff27\": container with ID starting with ee9f9bb76fc17a89b5bc2bbef261d4f3168e436faf413593b3d466b7af49ff27 not found: ID does not exist"
Sep 16 18:47:16.669747 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:47:16.669698    2544 scope.go:117] "RemoveContainer" containerID="0fea312a41c726c8f76ad6706d748cf7cda93b7bc61ffaf4e203edb5a4133e88"
Sep 16 18:47:16.669928 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:47:16.669911    2544 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0fea312a41c726c8f76ad6706d748cf7cda93b7bc61ffaf4e203edb5a4133e88"} err="failed to get container status \"0fea312a41c726c8f76ad6706d748cf7cda93b7bc61ffaf4e203edb5a4133e88\": rpc error: code = NotFound desc = could not find container \"0fea312a41c726c8f76ad6706d748cf7cda93b7bc61ffaf4e203edb5a4133e88\": container with ID starting with 0fea312a41c726c8f76ad6706d748cf7cda93b7bc61ffaf4e203edb5a4133e88 not found: ID does not exist"
Sep 16 18:47:16.669970 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:47:16.669929    2544 scope.go:117] "RemoveContainer" containerID="00d9804e08c15f830a6e69132dc25ac411f4ca4977e7f7d37dd63e8c38f308c4"
Sep 16 18:47:16.670125 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:47:16.670110    2544 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"00d9804e08c15f830a6e69132dc25ac411f4ca4977e7f7d37dd63e8c38f308c4"} err="failed to get container status \"00d9804e08c15f830a6e69132dc25ac411f4ca4977e7f7d37dd63e8c38f308c4\": rpc error: code = NotFound desc = could not find container \"00d9804e08c15f830a6e69132dc25ac411f4ca4977e7f7d37dd63e8c38f308c4\": container with ID starting with 00d9804e08c15f830a6e69132dc25ac411f4ca4977e7f7d37dd63e8c38f308c4 not found: ID does not exist"
Sep 16 18:47:16.670170 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:47:16.670125    2544 scope.go:117] "RemoveContainer" containerID="fa8b3480b4cd7b3ae46246e3d6cab25c95a961ca441be61b82514a692b4ec3cf"
Sep 16 18:47:16.670307 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:47:16.670292    2544 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fa8b3480b4cd7b3ae46246e3d6cab25c95a961ca441be61b82514a692b4ec3cf"} err="failed to get container status \"fa8b3480b4cd7b3ae46246e3d6cab25c95a961ca441be61b82514a692b4ec3cf\": rpc error: code = NotFound desc = could not find container \"fa8b3480b4cd7b3ae46246e3d6cab25c95a961ca441be61b82514a692b4ec3cf\": container with ID starting with fa8b3480b4cd7b3ae46246e3d6cab25c95a961ca441be61b82514a692b4ec3cf not found: ID does not exist"
Sep 16 18:47:16.699702 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:47:16.699676    2544 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Sep 16 18:47:16.699971 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:47:16.699958    2544 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7a642ca2-3fee-4a4c-8c92-3f3069a99ac5" containerName="kube-rbac-proxy-metric"
Sep 16 18:47:16.700019 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:47:16.699974    2544 state_mem.go:107] "Deleted CPUSet assignment" podUID="7a642ca2-3fee-4a4c-8c92-3f3069a99ac5" containerName="kube-rbac-proxy-metric"
Sep 16 18:47:16.700019 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:47:16.699984    2544 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7a642ca2-3fee-4a4c-8c92-3f3069a99ac5" containerName="config-reloader"
Sep 16 18:47:16.700019 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:47:16.699990    2544 state_mem.go:107] "Deleted CPUSet assignment" podUID="7a642ca2-3fee-4a4c-8c92-3f3069a99ac5" containerName="config-reloader"
Sep 16 18:47:16.700019 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:47:16.700002    2544 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7a642ca2-3fee-4a4c-8c92-3f3069a99ac5" containerName="prom-label-proxy"
Sep 16 18:47:16.700019 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:47:16.700008    2544 state_mem.go:107] "Deleted CPUSet assignment" podUID="7a642ca2-3fee-4a4c-8c92-3f3069a99ac5" containerName="prom-label-proxy"
Sep 16 18:47:16.700019 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:47:16.700017    2544 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7a642ca2-3fee-4a4c-8c92-3f3069a99ac5" containerName="init-config-reloader"
Sep 16 18:47:16.700019 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:47:16.700022    2544 state_mem.go:107] "Deleted CPUSet assignment" podUID="7a642ca2-3fee-4a4c-8c92-3f3069a99ac5" containerName="init-config-reloader"
Sep 16 18:47:16.700237 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:47:16.700032    2544 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7a642ca2-3fee-4a4c-8c92-3f3069a99ac5" containerName="kube-rbac-proxy-web"
Sep 16 18:47:16.700237 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:47:16.700037    2544 state_mem.go:107] "Deleted CPUSet assignment" podUID="7a642ca2-3fee-4a4c-8c92-3f3069a99ac5" containerName="kube-rbac-proxy-web"
Sep 16 18:47:16.700237 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:47:16.700046    2544 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7a642ca2-3fee-4a4c-8c92-3f3069a99ac5" containerName="alertmanager"
Sep 16 18:47:16.700237 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:47:16.700051    2544 state_mem.go:107] "Deleted CPUSet assignment" podUID="7a642ca2-3fee-4a4c-8c92-3f3069a99ac5" containerName="alertmanager"
Sep 16 18:47:16.700237 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:47:16.700061    2544 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7a642ca2-3fee-4a4c-8c92-3f3069a99ac5" containerName="kube-rbac-proxy"
Sep 16 18:47:16.700237 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:47:16.700066    2544 state_mem.go:107] "Deleted CPUSet assignment" podUID="7a642ca2-3fee-4a4c-8c92-3f3069a99ac5" containerName="kube-rbac-proxy"
Sep 16 18:47:16.700237 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:47:16.700072    2544 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b03d56d8-b081-41fd-bac9-ba7da38f733a" containerName="registry"
Sep 16 18:47:16.700237 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:47:16.700078    2544 state_mem.go:107] "Deleted CPUSet assignment" podUID="b03d56d8-b081-41fd-bac9-ba7da38f733a" containerName="registry"
Sep 16 18:47:16.700237 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:47:16.700127    2544 memory_manager.go:368] "RemoveStaleState removing state" podUID="7a642ca2-3fee-4a4c-8c92-3f3069a99ac5" containerName="alertmanager"
Sep 16 18:47:16.700237 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:47:16.700133    2544 memory_manager.go:368] "RemoveStaleState removing state" podUID="7a642ca2-3fee-4a4c-8c92-3f3069a99ac5" containerName="kube-rbac-proxy-metric"
Sep 16 18:47:16.700237 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:47:16.700139    2544 memory_manager.go:368] "RemoveStaleState removing state" podUID="7a642ca2-3fee-4a4c-8c92-3f3069a99ac5" containerName="kube-rbac-proxy"
Sep 16 18:47:16.700237 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:47:16.700146    2544 memory_manager.go:368] "RemoveStaleState removing state" podUID="7a642ca2-3fee-4a4c-8c92-3f3069a99ac5" containerName="prom-label-proxy"
Sep 16 18:47:16.700237 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:47:16.700153    2544 memory_manager.go:368] "RemoveStaleState removing state" podUID="b03d56d8-b081-41fd-bac9-ba7da38f733a" containerName="registry"
Sep 16 18:47:16.700237 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:47:16.700158    2544 memory_manager.go:368] "RemoveStaleState removing state" podUID="7a642ca2-3fee-4a4c-8c92-3f3069a99ac5" containerName="kube-rbac-proxy-web"
Sep 16 18:47:16.700237 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:47:16.700165    2544 memory_manager.go:368] "RemoveStaleState removing state" podUID="7a642ca2-3fee-4a4c-8c92-3f3069a99ac5" containerName="config-reloader"
Sep 16 18:47:16.705150 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:47:16.705132    2544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Sep 16 18:47:16.708412 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:47:16.708394    2544 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\""
Sep 16 18:47:16.708530 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:47:16.708467    2544 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\""
Sep 16 18:47:16.708530 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:47:16.708500    2544 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\""
Sep 16 18:47:16.708693 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:47:16.708500    2544 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\""
Sep 16 18:47:16.708862 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:47:16.708842    2544 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\""
Sep 16 18:47:16.708960 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:47:16.708863    2544 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-nlphx\""
Sep 16 18:47:16.709751 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:47:16.709735    2544 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\""
Sep 16 18:47:16.710159 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:47:16.710136    2544 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\""
Sep 16 18:47:16.714153 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:47:16.714136    2544 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\""
Sep 16 18:47:16.715349 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:47:16.715328    2544 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\""
Sep 16 18:47:16.723885 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:47:16.723861    2544 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Sep 16 18:47:16.850932 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:47:16.850895    2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/c1055a71-f078-4e9d-a1d4-7320f07566c8-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"c1055a71-f078-4e9d-a1d4-7320f07566c8\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 18:47:16.851097 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:47:16.850936    2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/c1055a71-f078-4e9d-a1d4-7320f07566c8-tls-assets\") pod \"alertmanager-main-0\" (UID: \"c1055a71-f078-4e9d-a1d4-7320f07566c8\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 18:47:16.851097 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:47:16.850964    2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/c1055a71-f078-4e9d-a1d4-7320f07566c8-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"c1055a71-f078-4e9d-a1d4-7320f07566c8\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 18:47:16.851097 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:47:16.850986    2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cjzp6\" (UniqueName: \"kubernetes.io/projected/c1055a71-f078-4e9d-a1d4-7320f07566c8-kube-api-access-cjzp6\") pod \"alertmanager-main-0\" (UID: \"c1055a71-f078-4e9d-a1d4-7320f07566c8\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 18:47:16.851097 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:47:16.851003    2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/c1055a71-f078-4e9d-a1d4-7320f07566c8-config-volume\") pod \"alertmanager-main-0\" (UID: \"c1055a71-f078-4e9d-a1d4-7320f07566c8\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 18:47:16.851097 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:47:16.851020    2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/c1055a71-f078-4e9d-a1d4-7320f07566c8-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"c1055a71-f078-4e9d-a1d4-7320f07566c8\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 18:47:16.851097 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:47:16.851052    2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/c1055a71-f078-4e9d-a1d4-7320f07566c8-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"c1055a71-f078-4e9d-a1d4-7320f07566c8\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 18:47:16.851097 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:47:16.851088    2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/c1055a71-f078-4e9d-a1d4-7320f07566c8-config-out\") pod \"alertmanager-main-0\" (UID: \"c1055a71-f078-4e9d-a1d4-7320f07566c8\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 18:47:16.851366 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:47:16.851123    2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/c1055a71-f078-4e9d-a1d4-7320f07566c8-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"c1055a71-f078-4e9d-a1d4-7320f07566c8\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 18:47:16.851366 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:47:16.851147    2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/c1055a71-f078-4e9d-a1d4-7320f07566c8-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"c1055a71-f078-4e9d-a1d4-7320f07566c8\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 18:47:16.851366 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:47:16.851167    2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/c1055a71-f078-4e9d-a1d4-7320f07566c8-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"c1055a71-f078-4e9d-a1d4-7320f07566c8\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 18:47:16.851366 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:47:16.851206    2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c1055a71-f078-4e9d-a1d4-7320f07566c8-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"c1055a71-f078-4e9d-a1d4-7320f07566c8\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 18:47:16.851366 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:47:16.851244    2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/c1055a71-f078-4e9d-a1d4-7320f07566c8-web-config\") pod \"alertmanager-main-0\" (UID: \"c1055a71-f078-4e9d-a1d4-7320f07566c8\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 18:47:16.951824 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:47:16.951743    2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/c1055a71-f078-4e9d-a1d4-7320f07566c8-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"c1055a71-f078-4e9d-a1d4-7320f07566c8\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 18:47:16.951824 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:47:16.951780    2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c1055a71-f078-4e9d-a1d4-7320f07566c8-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"c1055a71-f078-4e9d-a1d4-7320f07566c8\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 18:47:16.951824 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:47:16.951801    2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/c1055a71-f078-4e9d-a1d4-7320f07566c8-web-config\") pod \"alertmanager-main-0\" (UID: \"c1055a71-f078-4e9d-a1d4-7320f07566c8\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 18:47:16.952074 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:47:16.951836    2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/c1055a71-f078-4e9d-a1d4-7320f07566c8-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"c1055a71-f078-4e9d-a1d4-7320f07566c8\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 18:47:16.952074 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:47:16.951861    2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/c1055a71-f078-4e9d-a1d4-7320f07566c8-tls-assets\") pod \"alertmanager-main-0\" (UID: \"c1055a71-f078-4e9d-a1d4-7320f07566c8\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 18:47:16.952074 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:47:16.951876    2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/c1055a71-f078-4e9d-a1d4-7320f07566c8-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"c1055a71-f078-4e9d-a1d4-7320f07566c8\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 18:47:16.952074 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:47:16.951891    2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cjzp6\" (UniqueName: \"kubernetes.io/projected/c1055a71-f078-4e9d-a1d4-7320f07566c8-kube-api-access-cjzp6\") pod \"alertmanager-main-0\" (UID: \"c1055a71-f078-4e9d-a1d4-7320f07566c8\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 18:47:16.952074 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:47:16.951909    2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/c1055a71-f078-4e9d-a1d4-7320f07566c8-config-volume\") pod \"alertmanager-main-0\" (UID: \"c1055a71-f078-4e9d-a1d4-7320f07566c8\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 18:47:16.952074 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:47:16.952050    2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/c1055a71-f078-4e9d-a1d4-7320f07566c8-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"c1055a71-f078-4e9d-a1d4-7320f07566c8\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 18:47:16.952362 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:47:16.952085    2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/c1055a71-f078-4e9d-a1d4-7320f07566c8-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"c1055a71-f078-4e9d-a1d4-7320f07566c8\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 18:47:16.952362 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:47:16.952132    2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/c1055a71-f078-4e9d-a1d4-7320f07566c8-config-out\") pod \"alertmanager-main-0\" (UID: \"c1055a71-f078-4e9d-a1d4-7320f07566c8\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 18:47:16.952362 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:47:16.952169    2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/c1055a71-f078-4e9d-a1d4-7320f07566c8-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"c1055a71-f078-4e9d-a1d4-7320f07566c8\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 18:47:16.952362 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:47:16.952204    2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/c1055a71-f078-4e9d-a1d4-7320f07566c8-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"c1055a71-f078-4e9d-a1d4-7320f07566c8\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 18:47:16.952864 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:47:16.952828    2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/c1055a71-f078-4e9d-a1d4-7320f07566c8-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"c1055a71-f078-4e9d-a1d4-7320f07566c8\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 18:47:16.953045 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:47:16.953011    2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/c1055a71-f078-4e9d-a1d4-7320f07566c8-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"c1055a71-f078-4e9d-a1d4-7320f07566c8\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 18:47:16.953131 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:47:16.953115    2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c1055a71-f078-4e9d-a1d4-7320f07566c8-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"c1055a71-f078-4e9d-a1d4-7320f07566c8\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 18:47:16.955079 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:47:16.954968    2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/c1055a71-f078-4e9d-a1d4-7320f07566c8-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"c1055a71-f078-4e9d-a1d4-7320f07566c8\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 18:47:16.955079 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:47:16.955063    2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/c1055a71-f078-4e9d-a1d4-7320f07566c8-web-config\") pod \"alertmanager-main-0\" (UID: \"c1055a71-f078-4e9d-a1d4-7320f07566c8\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 18:47:16.955245 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:47:16.955091    2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/c1055a71-f078-4e9d-a1d4-7320f07566c8-config-volume\") pod \"alertmanager-main-0\" (UID: \"c1055a71-f078-4e9d-a1d4-7320f07566c8\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 18:47:16.955245 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:47:16.955097    2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/c1055a71-f078-4e9d-a1d4-7320f07566c8-tls-assets\") pod \"alertmanager-main-0\" (UID: \"c1055a71-f078-4e9d-a1d4-7320f07566c8\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 18:47:16.955245 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:47:16.955144    2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/c1055a71-f078-4e9d-a1d4-7320f07566c8-config-out\") pod \"alertmanager-main-0\" (UID: \"c1055a71-f078-4e9d-a1d4-7320f07566c8\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 18:47:16.955620 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:47:16.955601    2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/c1055a71-f078-4e9d-a1d4-7320f07566c8-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"c1055a71-f078-4e9d-a1d4-7320f07566c8\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 18:47:16.955845 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:47:16.955821    2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/c1055a71-f078-4e9d-a1d4-7320f07566c8-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"c1055a71-f078-4e9d-a1d4-7320f07566c8\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 18:47:16.956479 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:47:16.956454    2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/c1055a71-f078-4e9d-a1d4-7320f07566c8-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"c1055a71-f078-4e9d-a1d4-7320f07566c8\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 18:47:16.956573 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:47:16.956455    2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/c1055a71-f078-4e9d-a1d4-7320f07566c8-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"c1055a71-f078-4e9d-a1d4-7320f07566c8\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 18:47:16.960064 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:47:16.960049    2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cjzp6\" (UniqueName: \"kubernetes.io/projected/c1055a71-f078-4e9d-a1d4-7320f07566c8-kube-api-access-cjzp6\") pod \"alertmanager-main-0\" (UID: \"c1055a71-f078-4e9d-a1d4-7320f07566c8\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 18:47:17.015109 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:47:17.015092    2544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Sep 16 18:47:17.296938 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:47:17.296913    2544 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Sep 16 18:47:17.299067 ip-10-0-132-80 kubenswrapper[2544]: W0916 18:47:17.299041    2544 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc1055a71_f078_4e9d_a1d4_7320f07566c8.slice/crio-e2ebc401e6d1a444e2c9cf802ec0e35d4fcbd5f023ea6fd3b76014ffda91d3dc WatchSource:0}: Error finding container e2ebc401e6d1a444e2c9cf802ec0e35d4fcbd5f023ea6fd3b76014ffda91d3dc: Status 404 returned error can't find the container with id e2ebc401e6d1a444e2c9cf802ec0e35d4fcbd5f023ea6fd3b76014ffda91d3dc
Sep 16 18:47:17.622158 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:47:17.622119    2544 generic.go:356] "Generic (PLEG): container finished" podID="c1055a71-f078-4e9d-a1d4-7320f07566c8" containerID="9881aaf8a2907d8870eb07b3f6d85484c73e4bf259f561dd2215190fddaf1f7e" exitCode=0
Sep 16 18:47:17.622350 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:47:17.622203    2544 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"c1055a71-f078-4e9d-a1d4-7320f07566c8","Type":"ContainerDied","Data":"9881aaf8a2907d8870eb07b3f6d85484c73e4bf259f561dd2215190fddaf1f7e"}
Sep 16 18:47:17.622350 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:47:17.622274    2544 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"c1055a71-f078-4e9d-a1d4-7320f07566c8","Type":"ContainerStarted","Data":"e2ebc401e6d1a444e2c9cf802ec0e35d4fcbd5f023ea6fd3b76014ffda91d3dc"}
Sep 16 18:47:17.723668 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:47:17.723633    2544 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7a642ca2-3fee-4a4c-8c92-3f3069a99ac5" path="/var/lib/kubelet/pods/7a642ca2-3fee-4a4c-8c92-3f3069a99ac5/volumes"
Sep 16 18:47:18.630878 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:47:18.630839    2544 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"c1055a71-f078-4e9d-a1d4-7320f07566c8","Type":"ContainerStarted","Data":"0f134914281498237416f0aa098eb78f61b2edf9c742362eac9846010bd02ae6"}
Sep 16 18:47:18.631297 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:47:18.630946    2544 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"c1055a71-f078-4e9d-a1d4-7320f07566c8","Type":"ContainerStarted","Data":"e07fe8bcd10d7ffc7611742a984b5362afba5db846e18f2b89368254a49f9467"}
Sep 16 18:47:18.631297 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:47:18.630974    2544 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"c1055a71-f078-4e9d-a1d4-7320f07566c8","Type":"ContainerStarted","Data":"a9b8d293536e55c72f455989aae83558a3f895ecd46500b536a132d5c2c4101d"}
Sep 16 18:47:18.631297 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:47:18.631006    2544 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"c1055a71-f078-4e9d-a1d4-7320f07566c8","Type":"ContainerStarted","Data":"6268403fe33afc4ee1907684439ce9dd0707a718654a1245b39a1f7747dbaa7b"}
Sep 16 18:47:18.631297 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:47:18.631044    2544 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"c1055a71-f078-4e9d-a1d4-7320f07566c8","Type":"ContainerStarted","Data":"ba8e670fa8784a12026973297d7d88cd6440d693aee4e33ca9b37ebd2d0b0b0f"}
Sep 16 18:47:18.631297 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:47:18.631072    2544 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"c1055a71-f078-4e9d-a1d4-7320f07566c8","Type":"ContainerStarted","Data":"535ea188e8d71a8805102b7d1a0a9a710253be746496d76a645de9460725b76b"}
Sep 16 18:47:18.659058 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:47:18.658978    2544 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/alertmanager-main-0" podStartSLOduration=2.644515636 podStartE2EDuration="2.658960084s" totalImagesPullingTime="0s" totalInitContainerRuntime="14.444448ms" isStatefulPod=false podCreationTimestamp="2026-09-16 18:47:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:47:18.658199399 +0000 UTC m=+201.447201955" watchObservedRunningTime="2026-09-16 18:47:18.658960084 +0000 UTC m=+201.447962619"
Sep 16 18:47:19.205122 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:47:19.205087    2544 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 16 18:47:19.205407 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:47:19.205369    2544 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="dbb1a91b-a0d3-477a-a98b-82fa11d56733" containerName="prometheus" containerID="cri-o://835a005de515fe25533264a18248a7baae64846a00ebea09cebf79567b247378" gracePeriod=600
Sep 16 18:47:19.205564 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:47:19.205405    2544 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="dbb1a91b-a0d3-477a-a98b-82fa11d56733" containerName="kube-rbac-proxy-web" containerID="cri-o://360d70a4d0b601301e62bb1da48989c99735804d375503a6a10a3be829ddfe69" gracePeriod=600
Sep 16 18:47:19.205564 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:47:19.205425    2544 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="dbb1a91b-a0d3-477a-a98b-82fa11d56733" containerName="kube-rbac-proxy-thanos" containerID="cri-o://4a7850c58d8b3c043861d11cbf77250e4bd6fa9cc2bc2cba630fea990572fc24" gracePeriod=600
Sep 16 18:47:19.205564 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:47:19.205424    2544 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="dbb1a91b-a0d3-477a-a98b-82fa11d56733" containerName="config-reloader" containerID="cri-o://728e8310ab4f116ffd9bbc9488059841cc67a45bed2003c19ff4382c6dd3f668" gracePeriod=600
Sep 16 18:47:19.205564 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:47:19.205379    2544 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="dbb1a91b-a0d3-477a-a98b-82fa11d56733" containerName="kube-rbac-proxy" containerID="cri-o://ff3a43bc7d1eb2991ea3b2411cb1334586660cefab76175ab4c65913134d761a" gracePeriod=600
Sep 16 18:47:19.205564 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:47:19.205408    2544 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="dbb1a91b-a0d3-477a-a98b-82fa11d56733" containerName="thanos-sidecar" containerID="cri-o://51618dd4a3c2b428c5fe23d46774fd4cfa7b60dc0adfedc67bdd1e3ea84e0d89" gracePeriod=600
Sep 16 18:47:19.442293 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:47:19.442269    2544 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:47:19.576347 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:47:19.576320    2544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/dbb1a91b-a0d3-477a-a98b-82fa11d56733-prometheus-k8s-rulefiles-1\") pod \"dbb1a91b-a0d3-477a-a98b-82fa11d56733\" (UID: \"dbb1a91b-a0d3-477a-a98b-82fa11d56733\") "
Sep 16 18:47:19.576484 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:47:19.576363    2544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/dbb1a91b-a0d3-477a-a98b-82fa11d56733-config\") pod \"dbb1a91b-a0d3-477a-a98b-82fa11d56733\" (UID: \"dbb1a91b-a0d3-477a-a98b-82fa11d56733\") "
Sep 16 18:47:19.576484 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:47:19.576393    2544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/dbb1a91b-a0d3-477a-a98b-82fa11d56733-configmap-metrics-client-ca\") pod \"dbb1a91b-a0d3-477a-a98b-82fa11d56733\" (UID: \"dbb1a91b-a0d3-477a-a98b-82fa11d56733\") "
Sep 16 18:47:19.576484 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:47:19.576432    2544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/dbb1a91b-a0d3-477a-a98b-82fa11d56733-configmap-kubelet-serving-ca-bundle\") pod \"dbb1a91b-a0d3-477a-a98b-82fa11d56733\" (UID: \"dbb1a91b-a0d3-477a-a98b-82fa11d56733\") "
Sep 16 18:47:19.576484 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:47:19.576461    2544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/dbb1a91b-a0d3-477a-a98b-82fa11d56733-tls-assets\") pod \"dbb1a91b-a0d3-477a-a98b-82fa11d56733\" (UID: \"dbb1a91b-a0d3-477a-a98b-82fa11d56733\") "
Sep 16 18:47:19.576753 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:47:19.576489    2544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/dbb1a91b-a0d3-477a-a98b-82fa11d56733-secret-prometheus-k8s-tls\") pod \"dbb1a91b-a0d3-477a-a98b-82fa11d56733\" (UID: \"dbb1a91b-a0d3-477a-a98b-82fa11d56733\") "
Sep 16 18:47:19.576753 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:47:19.576521    2544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/dbb1a91b-a0d3-477a-a98b-82fa11d56733-secret-grpc-tls\") pod \"dbb1a91b-a0d3-477a-a98b-82fa11d56733\" (UID: \"dbb1a91b-a0d3-477a-a98b-82fa11d56733\") "
Sep 16 18:47:19.576753 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:47:19.576583    2544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xjnkt\" (UniqueName: \"kubernetes.io/projected/dbb1a91b-a0d3-477a-a98b-82fa11d56733-kube-api-access-xjnkt\") pod \"dbb1a91b-a0d3-477a-a98b-82fa11d56733\" (UID: \"dbb1a91b-a0d3-477a-a98b-82fa11d56733\") "
Sep 16 18:47:19.576753 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:47:19.576616    2544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/dbb1a91b-a0d3-477a-a98b-82fa11d56733-secret-kube-rbac-proxy\") pod \"dbb1a91b-a0d3-477a-a98b-82fa11d56733\" (UID: \"dbb1a91b-a0d3-477a-a98b-82fa11d56733\") "
Sep 16 18:47:19.576753 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:47:19.576640    2544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/dbb1a91b-a0d3-477a-a98b-82fa11d56733-configmap-serving-certs-ca-bundle\") pod \"dbb1a91b-a0d3-477a-a98b-82fa11d56733\" (UID: \"dbb1a91b-a0d3-477a-a98b-82fa11d56733\") "
Sep 16 18:47:19.576753 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:47:19.576676    2544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/dbb1a91b-a0d3-477a-a98b-82fa11d56733-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"dbb1a91b-a0d3-477a-a98b-82fa11d56733\" (UID: \"dbb1a91b-a0d3-477a-a98b-82fa11d56733\") "
Sep 16 18:47:19.576753 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:47:19.576703    2544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/dbb1a91b-a0d3-477a-a98b-82fa11d56733-secret-metrics-client-certs\") pod \"dbb1a91b-a0d3-477a-a98b-82fa11d56733\" (UID: \"dbb1a91b-a0d3-477a-a98b-82fa11d56733\") "
Sep 16 18:47:19.576753 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:47:19.576735    2544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/dbb1a91b-a0d3-477a-a98b-82fa11d56733-prometheus-k8s-rulefiles-0\") pod \"dbb1a91b-a0d3-477a-a98b-82fa11d56733\" (UID: \"dbb1a91b-a0d3-477a-a98b-82fa11d56733\") "
Sep 16 18:47:19.576753 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:47:19.576742    2544 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dbb1a91b-a0d3-477a-a98b-82fa11d56733-prometheus-k8s-rulefiles-1" (OuterVolumeSpecName: "prometheus-k8s-rulefiles-1") pod "dbb1a91b-a0d3-477a-a98b-82fa11d56733" (UID: "dbb1a91b-a0d3-477a-a98b-82fa11d56733"). InnerVolumeSpecName "prometheus-k8s-rulefiles-1". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 18:47:19.577180 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:47:19.576767    2544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/dbb1a91b-a0d3-477a-a98b-82fa11d56733-prometheus-k8s-rulefiles-2\") pod \"dbb1a91b-a0d3-477a-a98b-82fa11d56733\" (UID: \"dbb1a91b-a0d3-477a-a98b-82fa11d56733\") "
Sep 16 18:47:19.577180 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:47:19.576791    2544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/dbb1a91b-a0d3-477a-a98b-82fa11d56733-web-config\") pod \"dbb1a91b-a0d3-477a-a98b-82fa11d56733\" (UID: \"dbb1a91b-a0d3-477a-a98b-82fa11d56733\") "
Sep 16 18:47:19.577180 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:47:19.576835    2544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/dbb1a91b-a0d3-477a-a98b-82fa11d56733-prometheus-trusted-ca-bundle\") pod \"dbb1a91b-a0d3-477a-a98b-82fa11d56733\" (UID: \"dbb1a91b-a0d3-477a-a98b-82fa11d56733\") "
Sep 16 18:47:19.577180 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:47:19.576867    2544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/dbb1a91b-a0d3-477a-a98b-82fa11d56733-thanos-prometheus-http-client-file\") pod \"dbb1a91b-a0d3-477a-a98b-82fa11d56733\" (UID: \"dbb1a91b-a0d3-477a-a98b-82fa11d56733\") "
Sep 16 18:47:19.577180 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:47:19.576892    2544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/dbb1a91b-a0d3-477a-a98b-82fa11d56733-config-out\") pod \"dbb1a91b-a0d3-477a-a98b-82fa11d56733\" (UID: \"dbb1a91b-a0d3-477a-a98b-82fa11d56733\") "
Sep 16 18:47:19.577180 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:47:19.576897    2544 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dbb1a91b-a0d3-477a-a98b-82fa11d56733-configmap-metrics-client-ca" (OuterVolumeSpecName: "configmap-metrics-client-ca") pod "dbb1a91b-a0d3-477a-a98b-82fa11d56733" (UID: "dbb1a91b-a0d3-477a-a98b-82fa11d56733"). InnerVolumeSpecName "configmap-metrics-client-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 18:47:19.577180 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:47:19.576928    2544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/dbb1a91b-a0d3-477a-a98b-82fa11d56733-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"dbb1a91b-a0d3-477a-a98b-82fa11d56733\" (UID: \"dbb1a91b-a0d3-477a-a98b-82fa11d56733\") "
Sep 16 18:47:19.577180 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:47:19.576979    2544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/dbb1a91b-a0d3-477a-a98b-82fa11d56733-prometheus-k8s-db\") pod \"dbb1a91b-a0d3-477a-a98b-82fa11d56733\" (UID: \"dbb1a91b-a0d3-477a-a98b-82fa11d56733\") "
Sep 16 18:47:19.577585 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:47:19.577230    2544 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dbb1a91b-a0d3-477a-a98b-82fa11d56733-configmap-kubelet-serving-ca-bundle" (OuterVolumeSpecName: "configmap-kubelet-serving-ca-bundle") pod "dbb1a91b-a0d3-477a-a98b-82fa11d56733" (UID: "dbb1a91b-a0d3-477a-a98b-82fa11d56733"). InnerVolumeSpecName "configmap-kubelet-serving-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 18:47:19.577585 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:47:19.577255    2544 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/dbb1a91b-a0d3-477a-a98b-82fa11d56733-prometheus-k8s-rulefiles-1\") on node \"ip-10-0-132-80.ec2.internal\" DevicePath \"\""
Sep 16 18:47:19.577585 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:47:19.577275    2544 reconciler_common.go:299] "Volume detached for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/dbb1a91b-a0d3-477a-a98b-82fa11d56733-configmap-metrics-client-ca\") on node \"ip-10-0-132-80.ec2.internal\" DevicePath \"\""
Sep 16 18:47:19.577744 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:47:19.577615    2544 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dbb1a91b-a0d3-477a-a98b-82fa11d56733-prometheus-k8s-rulefiles-2" (OuterVolumeSpecName: "prometheus-k8s-rulefiles-2") pod "dbb1a91b-a0d3-477a-a98b-82fa11d56733" (UID: "dbb1a91b-a0d3-477a-a98b-82fa11d56733"). InnerVolumeSpecName "prometheus-k8s-rulefiles-2". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 18:47:19.579526 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:47:19.578056    2544 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dbb1a91b-a0d3-477a-a98b-82fa11d56733-prometheus-trusted-ca-bundle" (OuterVolumeSpecName: "prometheus-trusted-ca-bundle") pod "dbb1a91b-a0d3-477a-a98b-82fa11d56733" (UID: "dbb1a91b-a0d3-477a-a98b-82fa11d56733"). InnerVolumeSpecName "prometheus-trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 18:47:19.579526 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:47:19.579420    2544 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dbb1a91b-a0d3-477a-a98b-82fa11d56733-tls-assets" (OuterVolumeSpecName: "tls-assets") pod "dbb1a91b-a0d3-477a-a98b-82fa11d56733" (UID: "dbb1a91b-a0d3-477a-a98b-82fa11d56733"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 18:47:19.579526 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:47:19.579479    2544 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dbb1a91b-a0d3-477a-a98b-82fa11d56733-configmap-serving-certs-ca-bundle" (OuterVolumeSpecName: "configmap-serving-certs-ca-bundle") pod "dbb1a91b-a0d3-477a-a98b-82fa11d56733" (UID: "dbb1a91b-a0d3-477a-a98b-82fa11d56733"). InnerVolumeSpecName "configmap-serving-certs-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 18:47:19.579779 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:47:19.579632    2544 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/dbb1a91b-a0d3-477a-a98b-82fa11d56733-secret-grpc-tls" (OuterVolumeSpecName: "secret-grpc-tls") pod "dbb1a91b-a0d3-477a-a98b-82fa11d56733" (UID: "dbb1a91b-a0d3-477a-a98b-82fa11d56733"). InnerVolumeSpecName "secret-grpc-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 18:47:19.579779 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:47:19.579677    2544 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dbb1a91b-a0d3-477a-a98b-82fa11d56733-prometheus-k8s-db" (OuterVolumeSpecName: "prometheus-k8s-db") pod "dbb1a91b-a0d3-477a-a98b-82fa11d56733" (UID: "dbb1a91b-a0d3-477a-a98b-82fa11d56733"). InnerVolumeSpecName "prometheus-k8s-db". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 18:47:19.579880 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:47:19.579784    2544 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dbb1a91b-a0d3-477a-a98b-82fa11d56733-prometheus-k8s-rulefiles-0" (OuterVolumeSpecName: "prometheus-k8s-rulefiles-0") pod "dbb1a91b-a0d3-477a-a98b-82fa11d56733" (UID: "dbb1a91b-a0d3-477a-a98b-82fa11d56733"). InnerVolumeSpecName "prometheus-k8s-rulefiles-0". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 18:47:19.580044 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:47:19.580010    2544 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dbb1a91b-a0d3-477a-a98b-82fa11d56733-kube-api-access-xjnkt" (OuterVolumeSpecName: "kube-api-access-xjnkt") pod "dbb1a91b-a0d3-477a-a98b-82fa11d56733" (UID: "dbb1a91b-a0d3-477a-a98b-82fa11d56733"). InnerVolumeSpecName "kube-api-access-xjnkt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 18:47:19.580044 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:47:19.580010    2544 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/dbb1a91b-a0d3-477a-a98b-82fa11d56733-config" (OuterVolumeSpecName: "config") pod "dbb1a91b-a0d3-477a-a98b-82fa11d56733" (UID: "dbb1a91b-a0d3-477a-a98b-82fa11d56733"). InnerVolumeSpecName "config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 18:47:19.580227 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:47:19.580126    2544 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/dbb1a91b-a0d3-477a-a98b-82fa11d56733-secret-prometheus-k8s-tls" (OuterVolumeSpecName: "secret-prometheus-k8s-tls") pod "dbb1a91b-a0d3-477a-a98b-82fa11d56733" (UID: "dbb1a91b-a0d3-477a-a98b-82fa11d56733"). InnerVolumeSpecName "secret-prometheus-k8s-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 18:47:19.580410 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:47:19.580387    2544 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/dbb1a91b-a0d3-477a-a98b-82fa11d56733-secret-kube-rbac-proxy" (OuterVolumeSpecName: "secret-kube-rbac-proxy") pod "dbb1a91b-a0d3-477a-a98b-82fa11d56733" (UID: "dbb1a91b-a0d3-477a-a98b-82fa11d56733"). InnerVolumeSpecName "secret-kube-rbac-proxy". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 18:47:19.581273 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:47:19.581247    2544 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dbb1a91b-a0d3-477a-a98b-82fa11d56733-config-out" (OuterVolumeSpecName: "config-out") pod "dbb1a91b-a0d3-477a-a98b-82fa11d56733" (UID: "dbb1a91b-a0d3-477a-a98b-82fa11d56733"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 18:47:19.581355 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:47:19.581241    2544 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/dbb1a91b-a0d3-477a-a98b-82fa11d56733-secret-prometheus-k8s-thanos-sidecar-tls" (OuterVolumeSpecName: "secret-prometheus-k8s-thanos-sidecar-tls") pod "dbb1a91b-a0d3-477a-a98b-82fa11d56733" (UID: "dbb1a91b-a0d3-477a-a98b-82fa11d56733"). InnerVolumeSpecName "secret-prometheus-k8s-thanos-sidecar-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 18:47:19.581596 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:47:19.581568    2544 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/dbb1a91b-a0d3-477a-a98b-82fa11d56733-secret-metrics-client-certs" (OuterVolumeSpecName: "secret-metrics-client-certs") pod "dbb1a91b-a0d3-477a-a98b-82fa11d56733" (UID: "dbb1a91b-a0d3-477a-a98b-82fa11d56733"). InnerVolumeSpecName "secret-metrics-client-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 18:47:19.581983 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:47:19.581962    2544 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/dbb1a91b-a0d3-477a-a98b-82fa11d56733-secret-prometheus-k8s-kube-rbac-proxy-web" (OuterVolumeSpecName: "secret-prometheus-k8s-kube-rbac-proxy-web") pod "dbb1a91b-a0d3-477a-a98b-82fa11d56733" (UID: "dbb1a91b-a0d3-477a-a98b-82fa11d56733"). InnerVolumeSpecName "secret-prometheus-k8s-kube-rbac-proxy-web". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 18:47:19.582046 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:47:19.582023    2544 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/dbb1a91b-a0d3-477a-a98b-82fa11d56733-thanos-prometheus-http-client-file" (OuterVolumeSpecName: "thanos-prometheus-http-client-file") pod "dbb1a91b-a0d3-477a-a98b-82fa11d56733" (UID: "dbb1a91b-a0d3-477a-a98b-82fa11d56733"). InnerVolumeSpecName "thanos-prometheus-http-client-file". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 18:47:19.591560 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:47:19.591529    2544 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/dbb1a91b-a0d3-477a-a98b-82fa11d56733-web-config" (OuterVolumeSpecName: "web-config") pod "dbb1a91b-a0d3-477a-a98b-82fa11d56733" (UID: "dbb1a91b-a0d3-477a-a98b-82fa11d56733"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 18:47:19.638248 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:47:19.638206    2544 generic.go:356] "Generic (PLEG): container finished" podID="dbb1a91b-a0d3-477a-a98b-82fa11d56733" containerID="4a7850c58d8b3c043861d11cbf77250e4bd6fa9cc2bc2cba630fea990572fc24" exitCode=0
Sep 16 18:47:19.638248 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:47:19.638242    2544 generic.go:356] "Generic (PLEG): container finished" podID="dbb1a91b-a0d3-477a-a98b-82fa11d56733" containerID="ff3a43bc7d1eb2991ea3b2411cb1334586660cefab76175ab4c65913134d761a" exitCode=0
Sep 16 18:47:19.638653 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:47:19.638253    2544 generic.go:356] "Generic (PLEG): container finished" podID="dbb1a91b-a0d3-477a-a98b-82fa11d56733" containerID="360d70a4d0b601301e62bb1da48989c99735804d375503a6a10a3be829ddfe69" exitCode=0
Sep 16 18:47:19.638653 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:47:19.638263    2544 generic.go:356] "Generic (PLEG): container finished" podID="dbb1a91b-a0d3-477a-a98b-82fa11d56733" containerID="51618dd4a3c2b428c5fe23d46774fd4cfa7b60dc0adfedc67bdd1e3ea84e0d89" exitCode=0
Sep 16 18:47:19.638653 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:47:19.638269    2544 generic.go:356] "Generic (PLEG): container finished" podID="dbb1a91b-a0d3-477a-a98b-82fa11d56733" containerID="728e8310ab4f116ffd9bbc9488059841cc67a45bed2003c19ff4382c6dd3f668" exitCode=0
Sep 16 18:47:19.638653 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:47:19.638274    2544 generic.go:356] "Generic (PLEG): container finished" podID="dbb1a91b-a0d3-477a-a98b-82fa11d56733" containerID="835a005de515fe25533264a18248a7baae64846a00ebea09cebf79567b247378" exitCode=0
Sep 16 18:47:19.638653 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:47:19.638289    2544 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:47:19.638653 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:47:19.638289    2544 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"dbb1a91b-a0d3-477a-a98b-82fa11d56733","Type":"ContainerDied","Data":"4a7850c58d8b3c043861d11cbf77250e4bd6fa9cc2bc2cba630fea990572fc24"}
Sep 16 18:47:19.638653 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:47:19.638417    2544 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"dbb1a91b-a0d3-477a-a98b-82fa11d56733","Type":"ContainerDied","Data":"ff3a43bc7d1eb2991ea3b2411cb1334586660cefab76175ab4c65913134d761a"}
Sep 16 18:47:19.638653 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:47:19.638448    2544 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"dbb1a91b-a0d3-477a-a98b-82fa11d56733","Type":"ContainerDied","Data":"360d70a4d0b601301e62bb1da48989c99735804d375503a6a10a3be829ddfe69"}
Sep 16 18:47:19.638653 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:47:19.638474    2544 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"dbb1a91b-a0d3-477a-a98b-82fa11d56733","Type":"ContainerDied","Data":"51618dd4a3c2b428c5fe23d46774fd4cfa7b60dc0adfedc67bdd1e3ea84e0d89"}
Sep 16 18:47:19.638653 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:47:19.638481    2544 scope.go:117] "RemoveContainer" containerID="4a7850c58d8b3c043861d11cbf77250e4bd6fa9cc2bc2cba630fea990572fc24"
Sep 16 18:47:19.638653 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:47:19.638501    2544 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"dbb1a91b-a0d3-477a-a98b-82fa11d56733","Type":"ContainerDied","Data":"728e8310ab4f116ffd9bbc9488059841cc67a45bed2003c19ff4382c6dd3f668"}
Sep 16 18:47:19.638653 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:47:19.638525    2544 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"dbb1a91b-a0d3-477a-a98b-82fa11d56733","Type":"ContainerDied","Data":"835a005de515fe25533264a18248a7baae64846a00ebea09cebf79567b247378"}
Sep 16 18:47:19.638653 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:47:19.638567    2544 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"dbb1a91b-a0d3-477a-a98b-82fa11d56733","Type":"ContainerDied","Data":"7686f099730ae123234922b6be38b9ec81fd3a1d0c67a90af29d130ef404e3c5"}
Sep 16 18:47:19.647477 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:47:19.647383    2544 scope.go:117] "RemoveContainer" containerID="ff3a43bc7d1eb2991ea3b2411cb1334586660cefab76175ab4c65913134d761a"
Sep 16 18:47:19.656232 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:47:19.656209    2544 scope.go:117] "RemoveContainer" containerID="360d70a4d0b601301e62bb1da48989c99735804d375503a6a10a3be829ddfe69"
Sep 16 18:47:19.663120 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:47:19.663100    2544 scope.go:117] "RemoveContainer" containerID="51618dd4a3c2b428c5fe23d46774fd4cfa7b60dc0adfedc67bdd1e3ea84e0d89"
Sep 16 18:47:19.663344 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:47:19.663328    2544 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 16 18:47:19.667696 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:47:19.667673    2544 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 16 18:47:19.670589 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:47:19.670571    2544 scope.go:117] "RemoveContainer" containerID="728e8310ab4f116ffd9bbc9488059841cc67a45bed2003c19ff4382c6dd3f668"
Sep 16 18:47:19.677120 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:47:19.677106    2544 scope.go:117] "RemoveContainer" containerID="835a005de515fe25533264a18248a7baae64846a00ebea09cebf79567b247378"
Sep 16 18:47:19.677770 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:47:19.677750    2544 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/dbb1a91b-a0d3-477a-a98b-82fa11d56733-prometheus-k8s-db\") on node \"ip-10-0-132-80.ec2.internal\" DevicePath \"\""
Sep 16 18:47:19.677770 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:47:19.677771    2544 reconciler_common.go:299] "Volume detached for volume \"config\" (UniqueName: \"kubernetes.io/secret/dbb1a91b-a0d3-477a-a98b-82fa11d56733-config\") on node \"ip-10-0-132-80.ec2.internal\" DevicePath \"\""
Sep 16 18:47:19.677879 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:47:19.677782    2544 reconciler_common.go:299] "Volume detached for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/dbb1a91b-a0d3-477a-a98b-82fa11d56733-configmap-kubelet-serving-ca-bundle\") on node \"ip-10-0-132-80.ec2.internal\" DevicePath \"\""
Sep 16 18:47:19.677879 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:47:19.677790    2544 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/dbb1a91b-a0d3-477a-a98b-82fa11d56733-tls-assets\") on node \"ip-10-0-132-80.ec2.internal\" DevicePath \"\""
Sep 16 18:47:19.677879 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:47:19.677799    2544 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/dbb1a91b-a0d3-477a-a98b-82fa11d56733-secret-prometheus-k8s-tls\") on node \"ip-10-0-132-80.ec2.internal\" DevicePath \"\""
Sep 16 18:47:19.677879 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:47:19.677810    2544 reconciler_common.go:299] "Volume detached for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/dbb1a91b-a0d3-477a-a98b-82fa11d56733-secret-grpc-tls\") on node \"ip-10-0-132-80.ec2.internal\" DevicePath \"\""
Sep 16 18:47:19.677879 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:47:19.677821    2544 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xjnkt\" (UniqueName: \"kubernetes.io/projected/dbb1a91b-a0d3-477a-a98b-82fa11d56733-kube-api-access-xjnkt\") on node \"ip-10-0-132-80.ec2.internal\" DevicePath \"\""
Sep 16 18:47:19.677879 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:47:19.677829    2544 reconciler_common.go:299] "Volume detached for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/dbb1a91b-a0d3-477a-a98b-82fa11d56733-secret-kube-rbac-proxy\") on node \"ip-10-0-132-80.ec2.internal\" DevicePath \"\""
Sep 16 18:47:19.677879 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:47:19.677837    2544 reconciler_common.go:299] "Volume detached for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/dbb1a91b-a0d3-477a-a98b-82fa11d56733-configmap-serving-certs-ca-bundle\") on node \"ip-10-0-132-80.ec2.internal\" DevicePath \"\""
Sep 16 18:47:19.677879 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:47:19.677846    2544 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/dbb1a91b-a0d3-477a-a98b-82fa11d56733-secret-prometheus-k8s-thanos-sidecar-tls\") on node \"ip-10-0-132-80.ec2.internal\" DevicePath \"\""
Sep 16 18:47:19.677879 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:47:19.677854    2544 reconciler_common.go:299] "Volume detached for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/dbb1a91b-a0d3-477a-a98b-82fa11d56733-secret-metrics-client-certs\") on node \"ip-10-0-132-80.ec2.internal\" DevicePath \"\""
Sep 16 18:47:19.677879 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:47:19.677862    2544 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/dbb1a91b-a0d3-477a-a98b-82fa11d56733-prometheus-k8s-rulefiles-0\") on node \"ip-10-0-132-80.ec2.internal\" DevicePath \"\""
Sep 16 18:47:19.677879 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:47:19.677870    2544 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/dbb1a91b-a0d3-477a-a98b-82fa11d56733-prometheus-k8s-rulefiles-2\") on node \"ip-10-0-132-80.ec2.internal\" DevicePath \"\""
Sep 16 18:47:19.677879 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:47:19.677878    2544 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/dbb1a91b-a0d3-477a-a98b-82fa11d56733-web-config\") on node \"ip-10-0-132-80.ec2.internal\" DevicePath \"\""
Sep 16 18:47:19.678222 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:47:19.677890    2544 reconciler_common.go:299] "Volume detached for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/dbb1a91b-a0d3-477a-a98b-82fa11d56733-prometheus-trusted-ca-bundle\") on node \"ip-10-0-132-80.ec2.internal\" DevicePath \"\""
Sep 16 18:47:19.678222 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:47:19.677902    2544 reconciler_common.go:299] "Volume detached for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/dbb1a91b-a0d3-477a-a98b-82fa11d56733-thanos-prometheus-http-client-file\") on node \"ip-10-0-132-80.ec2.internal\" DevicePath \"\""
Sep 16 18:47:19.678222 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:47:19.677914    2544 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/dbb1a91b-a0d3-477a-a98b-82fa11d56733-config-out\") on node \"ip-10-0-132-80.ec2.internal\" DevicePath \"\""
Sep 16 18:47:19.678222 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:47:19.677925    2544 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/dbb1a91b-a0d3-477a-a98b-82fa11d56733-secret-prometheus-k8s-kube-rbac-proxy-web\") on node \"ip-10-0-132-80.ec2.internal\" DevicePath \"\""
Sep 16 18:47:19.684017 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:47:19.684002    2544 scope.go:117] "RemoveContainer" containerID="d0023e96010bf816b1118e2d024a6cdc27820ffb0e33bcefc75d61f18d94e90f"
Sep 16 18:47:19.690507 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:47:19.690488    2544 scope.go:117] "RemoveContainer" containerID="4a7850c58d8b3c043861d11cbf77250e4bd6fa9cc2bc2cba630fea990572fc24"
Sep 16 18:47:19.690780 ip-10-0-132-80 kubenswrapper[2544]: E0916 18:47:19.690763    2544 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4a7850c58d8b3c043861d11cbf77250e4bd6fa9cc2bc2cba630fea990572fc24\": container with ID starting with 4a7850c58d8b3c043861d11cbf77250e4bd6fa9cc2bc2cba630fea990572fc24 not found: ID does not exist" containerID="4a7850c58d8b3c043861d11cbf77250e4bd6fa9cc2bc2cba630fea990572fc24"
Sep 16 18:47:19.690849 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:47:19.690790    2544 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4a7850c58d8b3c043861d11cbf77250e4bd6fa9cc2bc2cba630fea990572fc24"} err="failed to get container status \"4a7850c58d8b3c043861d11cbf77250e4bd6fa9cc2bc2cba630fea990572fc24\": rpc error: code = NotFound desc = could not find container \"4a7850c58d8b3c043861d11cbf77250e4bd6fa9cc2bc2cba630fea990572fc24\": container with ID starting with 4a7850c58d8b3c043861d11cbf77250e4bd6fa9cc2bc2cba630fea990572fc24 not found: ID does not exist"
Sep 16 18:47:19.690849 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:47:19.690813    2544 scope.go:117] "RemoveContainer" containerID="ff3a43bc7d1eb2991ea3b2411cb1334586660cefab76175ab4c65913134d761a"
Sep 16 18:47:19.691065 ip-10-0-132-80 kubenswrapper[2544]: E0916 18:47:19.691048    2544 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ff3a43bc7d1eb2991ea3b2411cb1334586660cefab76175ab4c65913134d761a\": container with ID starting with ff3a43bc7d1eb2991ea3b2411cb1334586660cefab76175ab4c65913134d761a not found: ID does not exist" containerID="ff3a43bc7d1eb2991ea3b2411cb1334586660cefab76175ab4c65913134d761a"
Sep 16 18:47:19.691101 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:47:19.691070    2544 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ff3a43bc7d1eb2991ea3b2411cb1334586660cefab76175ab4c65913134d761a"} err="failed to get container status \"ff3a43bc7d1eb2991ea3b2411cb1334586660cefab76175ab4c65913134d761a\": rpc error: code = NotFound desc = could not find container \"ff3a43bc7d1eb2991ea3b2411cb1334586660cefab76175ab4c65913134d761a\": container with ID starting with ff3a43bc7d1eb2991ea3b2411cb1334586660cefab76175ab4c65913134d761a not found: ID does not exist"
Sep 16 18:47:19.691101 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:47:19.691086    2544 scope.go:117] "RemoveContainer" containerID="360d70a4d0b601301e62bb1da48989c99735804d375503a6a10a3be829ddfe69"
Sep 16 18:47:19.691291 ip-10-0-132-80 kubenswrapper[2544]: E0916 18:47:19.691275    2544 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"360d70a4d0b601301e62bb1da48989c99735804d375503a6a10a3be829ddfe69\": container with ID starting with 360d70a4d0b601301e62bb1da48989c99735804d375503a6a10a3be829ddfe69 not found: ID does not exist" containerID="360d70a4d0b601301e62bb1da48989c99735804d375503a6a10a3be829ddfe69"
Sep 16 18:47:19.691357 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:47:19.691298    2544 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"360d70a4d0b601301e62bb1da48989c99735804d375503a6a10a3be829ddfe69"} err="failed to get container status \"360d70a4d0b601301e62bb1da48989c99735804d375503a6a10a3be829ddfe69\": rpc error: code = NotFound desc = could not find container \"360d70a4d0b601301e62bb1da48989c99735804d375503a6a10a3be829ddfe69\": container with ID starting with 360d70a4d0b601301e62bb1da48989c99735804d375503a6a10a3be829ddfe69 not found: ID does not exist"
Sep 16 18:47:19.691357 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:47:19.691317    2544 scope.go:117] "RemoveContainer" containerID="51618dd4a3c2b428c5fe23d46774fd4cfa7b60dc0adfedc67bdd1e3ea84e0d89"
Sep 16 18:47:19.691595 ip-10-0-132-80 kubenswrapper[2544]: E0916 18:47:19.691571    2544 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"51618dd4a3c2b428c5fe23d46774fd4cfa7b60dc0adfedc67bdd1e3ea84e0d89\": container with ID starting with 51618dd4a3c2b428c5fe23d46774fd4cfa7b60dc0adfedc67bdd1e3ea84e0d89 not found: ID does not exist" containerID="51618dd4a3c2b428c5fe23d46774fd4cfa7b60dc0adfedc67bdd1e3ea84e0d89"
Sep 16 18:47:19.691648 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:47:19.691599    2544 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"51618dd4a3c2b428c5fe23d46774fd4cfa7b60dc0adfedc67bdd1e3ea84e0d89"} err="failed to get container status \"51618dd4a3c2b428c5fe23d46774fd4cfa7b60dc0adfedc67bdd1e3ea84e0d89\": rpc error: code = NotFound desc = could not find container \"51618dd4a3c2b428c5fe23d46774fd4cfa7b60dc0adfedc67bdd1e3ea84e0d89\": container with ID starting with 51618dd4a3c2b428c5fe23d46774fd4cfa7b60dc0adfedc67bdd1e3ea84e0d89 not found: ID does not exist"
Sep 16 18:47:19.691648 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:47:19.691613    2544 scope.go:117] "RemoveContainer" containerID="728e8310ab4f116ffd9bbc9488059841cc67a45bed2003c19ff4382c6dd3f668"
Sep 16 18:47:19.691886 ip-10-0-132-80 kubenswrapper[2544]: E0916 18:47:19.691864    2544 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"728e8310ab4f116ffd9bbc9488059841cc67a45bed2003c19ff4382c6dd3f668\": container with ID starting with 728e8310ab4f116ffd9bbc9488059841cc67a45bed2003c19ff4382c6dd3f668 not found: ID does not exist" containerID="728e8310ab4f116ffd9bbc9488059841cc67a45bed2003c19ff4382c6dd3f668"
Sep 16 18:47:19.691959 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:47:19.691894    2544 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"728e8310ab4f116ffd9bbc9488059841cc67a45bed2003c19ff4382c6dd3f668"} err="failed to get container status \"728e8310ab4f116ffd9bbc9488059841cc67a45bed2003c19ff4382c6dd3f668\": rpc error: code = NotFound desc = could not find container \"728e8310ab4f116ffd9bbc9488059841cc67a45bed2003c19ff4382c6dd3f668\": container with ID starting with 728e8310ab4f116ffd9bbc9488059841cc67a45bed2003c19ff4382c6dd3f668 not found: ID does not exist"
Sep 16 18:47:19.691959 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:47:19.691911    2544 scope.go:117] "RemoveContainer" containerID="835a005de515fe25533264a18248a7baae64846a00ebea09cebf79567b247378"
Sep 16 18:47:19.692291 ip-10-0-132-80 kubenswrapper[2544]: E0916 18:47:19.692254    2544 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"835a005de515fe25533264a18248a7baae64846a00ebea09cebf79567b247378\": container with ID starting with 835a005de515fe25533264a18248a7baae64846a00ebea09cebf79567b247378 not found: ID does not exist" containerID="835a005de515fe25533264a18248a7baae64846a00ebea09cebf79567b247378"
Sep 16 18:47:19.692371 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:47:19.692297    2544 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"835a005de515fe25533264a18248a7baae64846a00ebea09cebf79567b247378"} err="failed to get container status \"835a005de515fe25533264a18248a7baae64846a00ebea09cebf79567b247378\": rpc error: code = NotFound desc = could not find container \"835a005de515fe25533264a18248a7baae64846a00ebea09cebf79567b247378\": container with ID starting with 835a005de515fe25533264a18248a7baae64846a00ebea09cebf79567b247378 not found: ID does not exist"
Sep 16 18:47:19.692371 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:47:19.692316    2544 scope.go:117] "RemoveContainer" containerID="d0023e96010bf816b1118e2d024a6cdc27820ffb0e33bcefc75d61f18d94e90f"
Sep 16 18:47:19.692713 ip-10-0-132-80 kubenswrapper[2544]: E0916 18:47:19.692694    2544 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d0023e96010bf816b1118e2d024a6cdc27820ffb0e33bcefc75d61f18d94e90f\": container with ID starting with d0023e96010bf816b1118e2d024a6cdc27820ffb0e33bcefc75d61f18d94e90f not found: ID does not exist" containerID="d0023e96010bf816b1118e2d024a6cdc27820ffb0e33bcefc75d61f18d94e90f"
Sep 16 18:47:19.692776 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:47:19.692715    2544 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d0023e96010bf816b1118e2d024a6cdc27820ffb0e33bcefc75d61f18d94e90f"} err="failed to get container status \"d0023e96010bf816b1118e2d024a6cdc27820ffb0e33bcefc75d61f18d94e90f\": rpc error: code = NotFound desc = could not find container \"d0023e96010bf816b1118e2d024a6cdc27820ffb0e33bcefc75d61f18d94e90f\": container with ID starting with d0023e96010bf816b1118e2d024a6cdc27820ffb0e33bcefc75d61f18d94e90f not found: ID does not exist"
Sep 16 18:47:19.692776 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:47:19.692731    2544 scope.go:117] "RemoveContainer" containerID="4a7850c58d8b3c043861d11cbf77250e4bd6fa9cc2bc2cba630fea990572fc24"
Sep 16 18:47:19.693002 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:47:19.692978    2544 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4a7850c58d8b3c043861d11cbf77250e4bd6fa9cc2bc2cba630fea990572fc24"} err="failed to get container status \"4a7850c58d8b3c043861d11cbf77250e4bd6fa9cc2bc2cba630fea990572fc24\": rpc error: code = NotFound desc = could not find container \"4a7850c58d8b3c043861d11cbf77250e4bd6fa9cc2bc2cba630fea990572fc24\": container with ID starting with 4a7850c58d8b3c043861d11cbf77250e4bd6fa9cc2bc2cba630fea990572fc24 not found: ID does not exist"
Sep 16 18:47:19.693059 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:47:19.693007    2544 scope.go:117] "RemoveContainer" containerID="ff3a43bc7d1eb2991ea3b2411cb1334586660cefab76175ab4c65913134d761a"
Sep 16 18:47:19.693285 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:47:19.693266    2544 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ff3a43bc7d1eb2991ea3b2411cb1334586660cefab76175ab4c65913134d761a"} err="failed to get container status \"ff3a43bc7d1eb2991ea3b2411cb1334586660cefab76175ab4c65913134d761a\": rpc error: code = NotFound desc = could not find container \"ff3a43bc7d1eb2991ea3b2411cb1334586660cefab76175ab4c65913134d761a\": container with ID starting with ff3a43bc7d1eb2991ea3b2411cb1334586660cefab76175ab4c65913134d761a not found: ID does not exist"
Sep 16 18:47:19.693355 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:47:19.693287    2544 scope.go:117] "RemoveContainer" containerID="360d70a4d0b601301e62bb1da48989c99735804d375503a6a10a3be829ddfe69"
Sep 16 18:47:19.693524 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:47:19.693506    2544 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"360d70a4d0b601301e62bb1da48989c99735804d375503a6a10a3be829ddfe69"} err="failed to get container status \"360d70a4d0b601301e62bb1da48989c99735804d375503a6a10a3be829ddfe69\": rpc error: code = NotFound desc = could not find container \"360d70a4d0b601301e62bb1da48989c99735804d375503a6a10a3be829ddfe69\": container with ID starting with 360d70a4d0b601301e62bb1da48989c99735804d375503a6a10a3be829ddfe69 not found: ID does not exist"
Sep 16 18:47:19.693624 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:47:19.693525    2544 scope.go:117] "RemoveContainer" containerID="51618dd4a3c2b428c5fe23d46774fd4cfa7b60dc0adfedc67bdd1e3ea84e0d89"
Sep 16 18:47:19.693692 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:47:19.693674    2544 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 16 18:47:19.693842 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:47:19.693820    2544 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"51618dd4a3c2b428c5fe23d46774fd4cfa7b60dc0adfedc67bdd1e3ea84e0d89"} err="failed to get container status \"51618dd4a3c2b428c5fe23d46774fd4cfa7b60dc0adfedc67bdd1e3ea84e0d89\": rpc error: code = NotFound desc = could not find container \"51618dd4a3c2b428c5fe23d46774fd4cfa7b60dc0adfedc67bdd1e3ea84e0d89\": container with ID starting with 51618dd4a3c2b428c5fe23d46774fd4cfa7b60dc0adfedc67bdd1e3ea84e0d89 not found: ID does not exist"
Sep 16 18:47:19.693879 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:47:19.693844    2544 scope.go:117] "RemoveContainer" containerID="728e8310ab4f116ffd9bbc9488059841cc67a45bed2003c19ff4382c6dd3f668"
Sep 16 18:47:19.694032 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:47:19.694018    2544 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="dbb1a91b-a0d3-477a-a98b-82fa11d56733" containerName="config-reloader"
Sep 16 18:47:19.694080 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:47:19.694037    2544 state_mem.go:107] "Deleted CPUSet assignment" podUID="dbb1a91b-a0d3-477a-a98b-82fa11d56733" containerName="config-reloader"
Sep 16 18:47:19.694080 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:47:19.694052    2544 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="dbb1a91b-a0d3-477a-a98b-82fa11d56733" containerName="init-config-reloader"
Sep 16 18:47:19.694080 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:47:19.694062    2544 state_mem.go:107] "Deleted CPUSet assignment" podUID="dbb1a91b-a0d3-477a-a98b-82fa11d56733" containerName="init-config-reloader"
Sep 16 18:47:19.694080 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:47:19.694071    2544 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="dbb1a91b-a0d3-477a-a98b-82fa11d56733" containerName="thanos-sidecar"
Sep 16 18:47:19.694080 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:47:19.694070    2544 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"728e8310ab4f116ffd9bbc9488059841cc67a45bed2003c19ff4382c6dd3f668"} err="failed to get container status \"728e8310ab4f116ffd9bbc9488059841cc67a45bed2003c19ff4382c6dd3f668\": rpc error: code = NotFound desc = could not find container \"728e8310ab4f116ffd9bbc9488059841cc67a45bed2003c19ff4382c6dd3f668\": container with ID starting with 728e8310ab4f116ffd9bbc9488059841cc67a45bed2003c19ff4382c6dd3f668 not found: ID does not exist"
Sep 16 18:47:19.694237 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:47:19.694094    2544 scope.go:117] "RemoveContainer" containerID="835a005de515fe25533264a18248a7baae64846a00ebea09cebf79567b247378"
Sep 16 18:47:19.694237 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:47:19.694079    2544 state_mem.go:107] "Deleted CPUSet assignment" podUID="dbb1a91b-a0d3-477a-a98b-82fa11d56733" containerName="thanos-sidecar"
Sep 16 18:47:19.694237 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:47:19.694140    2544 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="dbb1a91b-a0d3-477a-a98b-82fa11d56733" containerName="kube-rbac-proxy-thanos"
Sep 16 18:47:19.694237 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:47:19.694147    2544 state_mem.go:107] "Deleted CPUSet assignment" podUID="dbb1a91b-a0d3-477a-a98b-82fa11d56733" containerName="kube-rbac-proxy-thanos"
Sep 16 18:47:19.694237 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:47:19.694160    2544 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="dbb1a91b-a0d3-477a-a98b-82fa11d56733" containerName="kube-rbac-proxy"
Sep 16 18:47:19.694237 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:47:19.694168    2544 state_mem.go:107] "Deleted CPUSet assignment" podUID="dbb1a91b-a0d3-477a-a98b-82fa11d56733" containerName="kube-rbac-proxy"
Sep 16 18:47:19.694237 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:47:19.694186    2544 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="dbb1a91b-a0d3-477a-a98b-82fa11d56733" containerName="prometheus"
Sep 16 18:47:19.694237 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:47:19.694191    2544 state_mem.go:107] "Deleted CPUSet assignment" podUID="dbb1a91b-a0d3-477a-a98b-82fa11d56733" containerName="prometheus"
Sep 16 18:47:19.694237 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:47:19.694200    2544 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="dbb1a91b-a0d3-477a-a98b-82fa11d56733" containerName="kube-rbac-proxy-web"
Sep 16 18:47:19.694237 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:47:19.694205    2544 state_mem.go:107] "Deleted CPUSet assignment" podUID="dbb1a91b-a0d3-477a-a98b-82fa11d56733" containerName="kube-rbac-proxy-web"
Sep 16 18:47:19.694627 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:47:19.694271    2544 memory_manager.go:368] "RemoveStaleState removing state" podUID="dbb1a91b-a0d3-477a-a98b-82fa11d56733" containerName="prometheus"
Sep 16 18:47:19.694627 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:47:19.694283    2544 memory_manager.go:368] "RemoveStaleState removing state" podUID="dbb1a91b-a0d3-477a-a98b-82fa11d56733" containerName="thanos-sidecar"
Sep 16 18:47:19.694627 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:47:19.694291    2544 memory_manager.go:368] "RemoveStaleState removing state" podUID="dbb1a91b-a0d3-477a-a98b-82fa11d56733" containerName="kube-rbac-proxy-web"
Sep 16 18:47:19.694627 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:47:19.694300    2544 memory_manager.go:368] "RemoveStaleState removing state" podUID="dbb1a91b-a0d3-477a-a98b-82fa11d56733" containerName="kube-rbac-proxy"
Sep 16 18:47:19.694627 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:47:19.694308    2544 memory_manager.go:368] "RemoveStaleState removing state" podUID="dbb1a91b-a0d3-477a-a98b-82fa11d56733" containerName="kube-rbac-proxy-thanos"
Sep 16 18:47:19.694627 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:47:19.694321    2544 memory_manager.go:368] "RemoveStaleState removing state" podUID="dbb1a91b-a0d3-477a-a98b-82fa11d56733" containerName="config-reloader"
Sep 16 18:47:19.694627 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:47:19.694345    2544 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"835a005de515fe25533264a18248a7baae64846a00ebea09cebf79567b247378"} err="failed to get container status \"835a005de515fe25533264a18248a7baae64846a00ebea09cebf79567b247378\": rpc error: code = NotFound desc = could not find container \"835a005de515fe25533264a18248a7baae64846a00ebea09cebf79567b247378\": container with ID starting with 835a005de515fe25533264a18248a7baae64846a00ebea09cebf79567b247378 not found: ID does not exist"
Sep 16 18:47:19.694627 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:47:19.694362    2544 scope.go:117] "RemoveContainer" containerID="d0023e96010bf816b1118e2d024a6cdc27820ffb0e33bcefc75d61f18d94e90f"
Sep 16 18:47:19.694627 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:47:19.694605    2544 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d0023e96010bf816b1118e2d024a6cdc27820ffb0e33bcefc75d61f18d94e90f"} err="failed to get container status \"d0023e96010bf816b1118e2d024a6cdc27820ffb0e33bcefc75d61f18d94e90f\": rpc error: code = NotFound desc = could not find container \"d0023e96010bf816b1118e2d024a6cdc27820ffb0e33bcefc75d61f18d94e90f\": container with ID starting with d0023e96010bf816b1118e2d024a6cdc27820ffb0e33bcefc75d61f18d94e90f not found: ID does not exist"
Sep 16 18:47:19.694627 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:47:19.694619    2544 scope.go:117] "RemoveContainer" containerID="4a7850c58d8b3c043861d11cbf77250e4bd6fa9cc2bc2cba630fea990572fc24"
Sep 16 18:47:19.694994 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:47:19.694806    2544 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4a7850c58d8b3c043861d11cbf77250e4bd6fa9cc2bc2cba630fea990572fc24"} err="failed to get container status \"4a7850c58d8b3c043861d11cbf77250e4bd6fa9cc2bc2cba630fea990572fc24\": rpc error: code = NotFound desc = could not find container \"4a7850c58d8b3c043861d11cbf77250e4bd6fa9cc2bc2cba630fea990572fc24\": container with ID starting with 4a7850c58d8b3c043861d11cbf77250e4bd6fa9cc2bc2cba630fea990572fc24 not found: ID does not exist"
Sep 16 18:47:19.694994 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:47:19.694819    2544 scope.go:117] "RemoveContainer" containerID="ff3a43bc7d1eb2991ea3b2411cb1334586660cefab76175ab4c65913134d761a"
Sep 16 18:47:19.695075 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:47:19.695006    2544 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ff3a43bc7d1eb2991ea3b2411cb1334586660cefab76175ab4c65913134d761a"} err="failed to get container status \"ff3a43bc7d1eb2991ea3b2411cb1334586660cefab76175ab4c65913134d761a\": rpc error: code = NotFound desc = could not find container \"ff3a43bc7d1eb2991ea3b2411cb1334586660cefab76175ab4c65913134d761a\": container with ID starting with ff3a43bc7d1eb2991ea3b2411cb1334586660cefab76175ab4c65913134d761a not found: ID does not exist"
Sep 16 18:47:19.695075 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:47:19.695020    2544 scope.go:117] "RemoveContainer" containerID="360d70a4d0b601301e62bb1da48989c99735804d375503a6a10a3be829ddfe69"
Sep 16 18:47:19.695204 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:47:19.695188    2544 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"360d70a4d0b601301e62bb1da48989c99735804d375503a6a10a3be829ddfe69"} err="failed to get container status \"360d70a4d0b601301e62bb1da48989c99735804d375503a6a10a3be829ddfe69\": rpc error: code = NotFound desc = could not find container \"360d70a4d0b601301e62bb1da48989c99735804d375503a6a10a3be829ddfe69\": container with ID starting with 360d70a4d0b601301e62bb1da48989c99735804d375503a6a10a3be829ddfe69 not found: ID does not exist"
Sep 16 18:47:19.695246 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:47:19.695205    2544 scope.go:117] "RemoveContainer" containerID="51618dd4a3c2b428c5fe23d46774fd4cfa7b60dc0adfedc67bdd1e3ea84e0d89"
Sep 16 18:47:19.695450 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:47:19.695435    2544 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"51618dd4a3c2b428c5fe23d46774fd4cfa7b60dc0adfedc67bdd1e3ea84e0d89"} err="failed to get container status \"51618dd4a3c2b428c5fe23d46774fd4cfa7b60dc0adfedc67bdd1e3ea84e0d89\": rpc error: code = NotFound desc = could not find container \"51618dd4a3c2b428c5fe23d46774fd4cfa7b60dc0adfedc67bdd1e3ea84e0d89\": container with ID starting with 51618dd4a3c2b428c5fe23d46774fd4cfa7b60dc0adfedc67bdd1e3ea84e0d89 not found: ID does not exist"
Sep 16 18:47:19.695450 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:47:19.695449    2544 scope.go:117] "RemoveContainer" containerID="728e8310ab4f116ffd9bbc9488059841cc67a45bed2003c19ff4382c6dd3f668"
Sep 16 18:47:19.695654 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:47:19.695637    2544 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"728e8310ab4f116ffd9bbc9488059841cc67a45bed2003c19ff4382c6dd3f668"} err="failed to get container status \"728e8310ab4f116ffd9bbc9488059841cc67a45bed2003c19ff4382c6dd3f668\": rpc error: code = NotFound desc = could not find container \"728e8310ab4f116ffd9bbc9488059841cc67a45bed2003c19ff4382c6dd3f668\": container with ID starting with 728e8310ab4f116ffd9bbc9488059841cc67a45bed2003c19ff4382c6dd3f668 not found: ID does not exist"
Sep 16 18:47:19.695654 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:47:19.695652    2544 scope.go:117] "RemoveContainer" containerID="835a005de515fe25533264a18248a7baae64846a00ebea09cebf79567b247378"
Sep 16 18:47:19.695840 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:47:19.695820    2544 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"835a005de515fe25533264a18248a7baae64846a00ebea09cebf79567b247378"} err="failed to get container status \"835a005de515fe25533264a18248a7baae64846a00ebea09cebf79567b247378\": rpc error: code = NotFound desc = could not find container \"835a005de515fe25533264a18248a7baae64846a00ebea09cebf79567b247378\": container with ID starting with 835a005de515fe25533264a18248a7baae64846a00ebea09cebf79567b247378 not found: ID does not exist"
Sep 16 18:47:19.695907 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:47:19.695842    2544 scope.go:117] "RemoveContainer" containerID="d0023e96010bf816b1118e2d024a6cdc27820ffb0e33bcefc75d61f18d94e90f"
Sep 16 18:47:19.696082 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:47:19.696066    2544 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d0023e96010bf816b1118e2d024a6cdc27820ffb0e33bcefc75d61f18d94e90f"} err="failed to get container status \"d0023e96010bf816b1118e2d024a6cdc27820ffb0e33bcefc75d61f18d94e90f\": rpc error: code = NotFound desc = could not find container \"d0023e96010bf816b1118e2d024a6cdc27820ffb0e33bcefc75d61f18d94e90f\": container with ID starting with d0023e96010bf816b1118e2d024a6cdc27820ffb0e33bcefc75d61f18d94e90f not found: ID does not exist"
Sep 16 18:47:19.696129 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:47:19.696082    2544 scope.go:117] "RemoveContainer" containerID="4a7850c58d8b3c043861d11cbf77250e4bd6fa9cc2bc2cba630fea990572fc24"
Sep 16 18:47:19.696264 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:47:19.696248    2544 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4a7850c58d8b3c043861d11cbf77250e4bd6fa9cc2bc2cba630fea990572fc24"} err="failed to get container status \"4a7850c58d8b3c043861d11cbf77250e4bd6fa9cc2bc2cba630fea990572fc24\": rpc error: code = NotFound desc = could not find container \"4a7850c58d8b3c043861d11cbf77250e4bd6fa9cc2bc2cba630fea990572fc24\": container with ID starting with 4a7850c58d8b3c043861d11cbf77250e4bd6fa9cc2bc2cba630fea990572fc24 not found: ID does not exist"
Sep 16 18:47:19.696264 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:47:19.696263    2544 scope.go:117] "RemoveContainer" containerID="ff3a43bc7d1eb2991ea3b2411cb1334586660cefab76175ab4c65913134d761a"
Sep 16 18:47:19.696479 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:47:19.696461    2544 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ff3a43bc7d1eb2991ea3b2411cb1334586660cefab76175ab4c65913134d761a"} err="failed to get container status \"ff3a43bc7d1eb2991ea3b2411cb1334586660cefab76175ab4c65913134d761a\": rpc error: code = NotFound desc = could not find container \"ff3a43bc7d1eb2991ea3b2411cb1334586660cefab76175ab4c65913134d761a\": container with ID starting with ff3a43bc7d1eb2991ea3b2411cb1334586660cefab76175ab4c65913134d761a not found: ID does not exist"
Sep 16 18:47:19.696519 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:47:19.696479    2544 scope.go:117] "RemoveContainer" containerID="360d70a4d0b601301e62bb1da48989c99735804d375503a6a10a3be829ddfe69"
Sep 16 18:47:19.696753 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:47:19.696731    2544 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"360d70a4d0b601301e62bb1da48989c99735804d375503a6a10a3be829ddfe69"} err="failed to get container status \"360d70a4d0b601301e62bb1da48989c99735804d375503a6a10a3be829ddfe69\": rpc error: code = NotFound desc = could not find container \"360d70a4d0b601301e62bb1da48989c99735804d375503a6a10a3be829ddfe69\": container with ID starting with 360d70a4d0b601301e62bb1da48989c99735804d375503a6a10a3be829ddfe69 not found: ID does not exist"
Sep 16 18:47:19.696753 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:47:19.696752    2544 scope.go:117] "RemoveContainer" containerID="51618dd4a3c2b428c5fe23d46774fd4cfa7b60dc0adfedc67bdd1e3ea84e0d89"
Sep 16 18:47:19.697026 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:47:19.697007    2544 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"51618dd4a3c2b428c5fe23d46774fd4cfa7b60dc0adfedc67bdd1e3ea84e0d89"} err="failed to get container status \"51618dd4a3c2b428c5fe23d46774fd4cfa7b60dc0adfedc67bdd1e3ea84e0d89\": rpc error: code = NotFound desc = could not find container \"51618dd4a3c2b428c5fe23d46774fd4cfa7b60dc0adfedc67bdd1e3ea84e0d89\": container with ID starting with 51618dd4a3c2b428c5fe23d46774fd4cfa7b60dc0adfedc67bdd1e3ea84e0d89 not found: ID does not exist"
Sep 16 18:47:19.697090 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:47:19.697027    2544 scope.go:117] "RemoveContainer" containerID="728e8310ab4f116ffd9bbc9488059841cc67a45bed2003c19ff4382c6dd3f668"
Sep 16 18:47:19.697272 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:47:19.697254    2544 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"728e8310ab4f116ffd9bbc9488059841cc67a45bed2003c19ff4382c6dd3f668"} err="failed to get container status \"728e8310ab4f116ffd9bbc9488059841cc67a45bed2003c19ff4382c6dd3f668\": rpc error: code = NotFound desc = could not find container \"728e8310ab4f116ffd9bbc9488059841cc67a45bed2003c19ff4382c6dd3f668\": container with ID starting with 728e8310ab4f116ffd9bbc9488059841cc67a45bed2003c19ff4382c6dd3f668 not found: ID does not exist"
Sep 16 18:47:19.697313 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:47:19.697273    2544 scope.go:117] "RemoveContainer" containerID="835a005de515fe25533264a18248a7baae64846a00ebea09cebf79567b247378"
Sep 16 18:47:19.697521 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:47:19.697504    2544 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"835a005de515fe25533264a18248a7baae64846a00ebea09cebf79567b247378"} err="failed to get container status \"835a005de515fe25533264a18248a7baae64846a00ebea09cebf79567b247378\": rpc error: code = NotFound desc = could not find container \"835a005de515fe25533264a18248a7baae64846a00ebea09cebf79567b247378\": container with ID starting with 835a005de515fe25533264a18248a7baae64846a00ebea09cebf79567b247378 not found: ID does not exist"
Sep 16 18:47:19.697592 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:47:19.697522    2544 scope.go:117] "RemoveContainer" containerID="d0023e96010bf816b1118e2d024a6cdc27820ffb0e33bcefc75d61f18d94e90f"
Sep 16 18:47:19.697801 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:47:19.697779    2544 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d0023e96010bf816b1118e2d024a6cdc27820ffb0e33bcefc75d61f18d94e90f"} err="failed to get container status \"d0023e96010bf816b1118e2d024a6cdc27820ffb0e33bcefc75d61f18d94e90f\": rpc error: code = NotFound desc = could not find container \"d0023e96010bf816b1118e2d024a6cdc27820ffb0e33bcefc75d61f18d94e90f\": container with ID starting with d0023e96010bf816b1118e2d024a6cdc27820ffb0e33bcefc75d61f18d94e90f not found: ID does not exist"
Sep 16 18:47:19.697844 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:47:19.697801    2544 scope.go:117] "RemoveContainer" containerID="4a7850c58d8b3c043861d11cbf77250e4bd6fa9cc2bc2cba630fea990572fc24"
Sep 16 18:47:19.698020 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:47:19.697986    2544 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4a7850c58d8b3c043861d11cbf77250e4bd6fa9cc2bc2cba630fea990572fc24"} err="failed to get container status \"4a7850c58d8b3c043861d11cbf77250e4bd6fa9cc2bc2cba630fea990572fc24\": rpc error: code = NotFound desc = could not find container \"4a7850c58d8b3c043861d11cbf77250e4bd6fa9cc2bc2cba630fea990572fc24\": container with ID starting with 4a7850c58d8b3c043861d11cbf77250e4bd6fa9cc2bc2cba630fea990572fc24 not found: ID does not exist"
Sep 16 18:47:19.698020 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:47:19.698005    2544 scope.go:117] "RemoveContainer" containerID="ff3a43bc7d1eb2991ea3b2411cb1334586660cefab76175ab4c65913134d761a"
Sep 16 18:47:19.698246 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:47:19.698227    2544 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ff3a43bc7d1eb2991ea3b2411cb1334586660cefab76175ab4c65913134d761a"} err="failed to get container status \"ff3a43bc7d1eb2991ea3b2411cb1334586660cefab76175ab4c65913134d761a\": rpc error: code = NotFound desc = could not find container \"ff3a43bc7d1eb2991ea3b2411cb1334586660cefab76175ab4c65913134d761a\": container with ID starting with ff3a43bc7d1eb2991ea3b2411cb1334586660cefab76175ab4c65913134d761a not found: ID does not exist"
Sep 16 18:47:19.698289 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:47:19.698246    2544 scope.go:117] "RemoveContainer" containerID="360d70a4d0b601301e62bb1da48989c99735804d375503a6a10a3be829ddfe69"
Sep 16 18:47:19.698473 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:47:19.698454    2544 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"360d70a4d0b601301e62bb1da48989c99735804d375503a6a10a3be829ddfe69"} err="failed to get container status \"360d70a4d0b601301e62bb1da48989c99735804d375503a6a10a3be829ddfe69\": rpc error: code = NotFound desc = could not find container \"360d70a4d0b601301e62bb1da48989c99735804d375503a6a10a3be829ddfe69\": container with ID starting with 360d70a4d0b601301e62bb1da48989c99735804d375503a6a10a3be829ddfe69 not found: ID does not exist"
Sep 16 18:47:19.698512 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:47:19.698474    2544 scope.go:117] "RemoveContainer" containerID="51618dd4a3c2b428c5fe23d46774fd4cfa7b60dc0adfedc67bdd1e3ea84e0d89"
Sep 16 18:47:19.698703 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:47:19.698685    2544 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"51618dd4a3c2b428c5fe23d46774fd4cfa7b60dc0adfedc67bdd1e3ea84e0d89"} err="failed to get container status \"51618dd4a3c2b428c5fe23d46774fd4cfa7b60dc0adfedc67bdd1e3ea84e0d89\": rpc error: code = NotFound desc = could not find container \"51618dd4a3c2b428c5fe23d46774fd4cfa7b60dc0adfedc67bdd1e3ea84e0d89\": container with ID starting with 51618dd4a3c2b428c5fe23d46774fd4cfa7b60dc0adfedc67bdd1e3ea84e0d89 not found: ID does not exist"
Sep 16 18:47:19.698746 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:47:19.698703    2544 scope.go:117] "RemoveContainer" containerID="728e8310ab4f116ffd9bbc9488059841cc67a45bed2003c19ff4382c6dd3f668"
Sep 16 18:47:19.698910 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:47:19.698894    2544 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"728e8310ab4f116ffd9bbc9488059841cc67a45bed2003c19ff4382c6dd3f668"} err="failed to get container status \"728e8310ab4f116ffd9bbc9488059841cc67a45bed2003c19ff4382c6dd3f668\": rpc error: code = NotFound desc = could not find container \"728e8310ab4f116ffd9bbc9488059841cc67a45bed2003c19ff4382c6dd3f668\": container with ID starting with 728e8310ab4f116ffd9bbc9488059841cc67a45bed2003c19ff4382c6dd3f668 not found: ID does not exist"
Sep 16 18:47:19.698949 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:47:19.698912    2544 scope.go:117] "RemoveContainer" containerID="835a005de515fe25533264a18248a7baae64846a00ebea09cebf79567b247378"
Sep 16 18:47:19.699234 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:47:19.699110    2544 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"835a005de515fe25533264a18248a7baae64846a00ebea09cebf79567b247378"} err="failed to get container status \"835a005de515fe25533264a18248a7baae64846a00ebea09cebf79567b247378\": rpc error: code = NotFound desc = could not find container \"835a005de515fe25533264a18248a7baae64846a00ebea09cebf79567b247378\": container with ID starting with 835a005de515fe25533264a18248a7baae64846a00ebea09cebf79567b247378 not found: ID does not exist"
Sep 16 18:47:19.699234 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:47:19.699126    2544 scope.go:117] "RemoveContainer" containerID="d0023e96010bf816b1118e2d024a6cdc27820ffb0e33bcefc75d61f18d94e90f"
Sep 16 18:47:19.699339 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:47:19.699311    2544 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d0023e96010bf816b1118e2d024a6cdc27820ffb0e33bcefc75d61f18d94e90f"} err="failed to get container status \"d0023e96010bf816b1118e2d024a6cdc27820ffb0e33bcefc75d61f18d94e90f\": rpc error: code = NotFound desc = could not find container \"d0023e96010bf816b1118e2d024a6cdc27820ffb0e33bcefc75d61f18d94e90f\": container with ID starting with d0023e96010bf816b1118e2d024a6cdc27820ffb0e33bcefc75d61f18d94e90f not found: ID does not exist"
Sep 16 18:47:19.699339 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:47:19.699324    2544 scope.go:117] "RemoveContainer" containerID="4a7850c58d8b3c043861d11cbf77250e4bd6fa9cc2bc2cba630fea990572fc24"
Sep 16 18:47:19.699579 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:47:19.699563    2544 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4a7850c58d8b3c043861d11cbf77250e4bd6fa9cc2bc2cba630fea990572fc24"} err="failed to get container status \"4a7850c58d8b3c043861d11cbf77250e4bd6fa9cc2bc2cba630fea990572fc24\": rpc error: code = NotFound desc = could not find container \"4a7850c58d8b3c043861d11cbf77250e4bd6fa9cc2bc2cba630fea990572fc24\": container with ID starting with 4a7850c58d8b3c043861d11cbf77250e4bd6fa9cc2bc2cba630fea990572fc24 not found: ID does not exist"
Sep 16 18:47:19.699620 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:47:19.699579    2544 scope.go:117] "RemoveContainer" containerID="ff3a43bc7d1eb2991ea3b2411cb1334586660cefab76175ab4c65913134d761a"
Sep 16 18:47:19.699755 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:47:19.699740    2544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:47:19.699799 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:47:19.699769    2544 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ff3a43bc7d1eb2991ea3b2411cb1334586660cefab76175ab4c65913134d761a"} err="failed to get container status \"ff3a43bc7d1eb2991ea3b2411cb1334586660cefab76175ab4c65913134d761a\": rpc error: code = NotFound desc = could not find container \"ff3a43bc7d1eb2991ea3b2411cb1334586660cefab76175ab4c65913134d761a\": container with ID starting with ff3a43bc7d1eb2991ea3b2411cb1334586660cefab76175ab4c65913134d761a not found: ID does not exist"
Sep 16 18:47:19.699799 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:47:19.699783    2544 scope.go:117] "RemoveContainer" containerID="360d70a4d0b601301e62bb1da48989c99735804d375503a6a10a3be829ddfe69"
Sep 16 18:47:19.700030 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:47:19.700003    2544 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"360d70a4d0b601301e62bb1da48989c99735804d375503a6a10a3be829ddfe69"} err="failed to get container status \"360d70a4d0b601301e62bb1da48989c99735804d375503a6a10a3be829ddfe69\": rpc error: code = NotFound desc = could not find container \"360d70a4d0b601301e62bb1da48989c99735804d375503a6a10a3be829ddfe69\": container with ID starting with 360d70a4d0b601301e62bb1da48989c99735804d375503a6a10a3be829ddfe69 not found: ID does not exist"
Sep 16 18:47:19.700030 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:47:19.700029    2544 scope.go:117] "RemoveContainer" containerID="51618dd4a3c2b428c5fe23d46774fd4cfa7b60dc0adfedc67bdd1e3ea84e0d89"
Sep 16 18:47:19.700361 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:47:19.700338    2544 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"51618dd4a3c2b428c5fe23d46774fd4cfa7b60dc0adfedc67bdd1e3ea84e0d89"} err="failed to get container status \"51618dd4a3c2b428c5fe23d46774fd4cfa7b60dc0adfedc67bdd1e3ea84e0d89\": rpc error: code = NotFound desc = could not find container \"51618dd4a3c2b428c5fe23d46774fd4cfa7b60dc0adfedc67bdd1e3ea84e0d89\": container with ID starting with 51618dd4a3c2b428c5fe23d46774fd4cfa7b60dc0adfedc67bdd1e3ea84e0d89 not found: ID does not exist"
Sep 16 18:47:19.700424 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:47:19.700364    2544 scope.go:117] "RemoveContainer" containerID="728e8310ab4f116ffd9bbc9488059841cc67a45bed2003c19ff4382c6dd3f668"
Sep 16 18:47:19.700645 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:47:19.700629    2544 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"728e8310ab4f116ffd9bbc9488059841cc67a45bed2003c19ff4382c6dd3f668"} err="failed to get container status \"728e8310ab4f116ffd9bbc9488059841cc67a45bed2003c19ff4382c6dd3f668\": rpc error: code = NotFound desc = could not find container \"728e8310ab4f116ffd9bbc9488059841cc67a45bed2003c19ff4382c6dd3f668\": container with ID starting with 728e8310ab4f116ffd9bbc9488059841cc67a45bed2003c19ff4382c6dd3f668 not found: ID does not exist"
Sep 16 18:47:19.700645 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:47:19.700645    2544 scope.go:117] "RemoveContainer" containerID="835a005de515fe25533264a18248a7baae64846a00ebea09cebf79567b247378"
Sep 16 18:47:19.700931 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:47:19.700911    2544 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"835a005de515fe25533264a18248a7baae64846a00ebea09cebf79567b247378"} err="failed to get container status \"835a005de515fe25533264a18248a7baae64846a00ebea09cebf79567b247378\": rpc error: code = NotFound desc = could not find container \"835a005de515fe25533264a18248a7baae64846a00ebea09cebf79567b247378\": container with ID starting with 835a005de515fe25533264a18248a7baae64846a00ebea09cebf79567b247378 not found: ID does not exist"
Sep 16 18:47:19.700982 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:47:19.700933    2544 scope.go:117] "RemoveContainer" containerID="d0023e96010bf816b1118e2d024a6cdc27820ffb0e33bcefc75d61f18d94e90f"
Sep 16 18:47:19.701242 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:47:19.701218    2544 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d0023e96010bf816b1118e2d024a6cdc27820ffb0e33bcefc75d61f18d94e90f"} err="failed to get container status \"d0023e96010bf816b1118e2d024a6cdc27820ffb0e33bcefc75d61f18d94e90f\": rpc error: code = NotFound desc = could not find container \"d0023e96010bf816b1118e2d024a6cdc27820ffb0e33bcefc75d61f18d94e90f\": container with ID starting with d0023e96010bf816b1118e2d024a6cdc27820ffb0e33bcefc75d61f18d94e90f not found: ID does not exist"
Sep 16 18:47:19.701860 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:47:19.701826    2544 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\""
Sep 16 18:47:19.701986 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:47:19.701867    2544 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\""
Sep 16 18:47:19.701986 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:47:19.701908    2544 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-mvgc6\""
Sep 16 18:47:19.701986 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:47:19.701870    2544 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\""
Sep 16 18:47:19.702147 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:47:19.701867    2544 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\""
Sep 16 18:47:19.702147 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:47:19.702001    2544 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\""
Sep 16 18:47:19.702147 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:47:19.702109    2544 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\""
Sep 16 18:47:19.702276 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:47:19.702112    2544 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-4pn6do9rm2eaa\""
Sep 16 18:47:19.702338 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:47:19.702324    2544 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\""
Sep 16 18:47:19.702501 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:47:19.702485    2544 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\""
Sep 16 18:47:19.702757 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:47:19.702738    2544 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\""
Sep 16 18:47:19.702856 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:47:19.702760    2544 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\""
Sep 16 18:47:19.702856 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:47:19.702819    2544 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\""
Sep 16 18:47:19.702974 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:47:19.702856    2544 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\""
Sep 16 18:47:19.705804 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:47:19.705773    2544 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\""
Sep 16 18:47:19.712233 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:47:19.709737    2544 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\""
Sep 16 18:47:19.715173 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:47:19.715145    2544 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 16 18:47:19.722013 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:47:19.721988    2544 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="dbb1a91b-a0d3-477a-a98b-82fa11d56733" path="/var/lib/kubelet/pods/dbb1a91b-a0d3-477a-a98b-82fa11d56733/volumes"
Sep 16 18:47:19.778865 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:47:19.778832    2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/c650a6db-a02b-40ca-a614-b2d37d2b2173-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"c650a6db-a02b-40ca-a614-b2d37d2b2173\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:47:19.778865 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:47:19.778864    2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/c650a6db-a02b-40ca-a614-b2d37d2b2173-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"c650a6db-a02b-40ca-a614-b2d37d2b2173\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:47:19.779087 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:47:19.779069    2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/c650a6db-a02b-40ca-a614-b2d37d2b2173-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"c650a6db-a02b-40ca-a614-b2d37d2b2173\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:47:19.779149 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:47:19.779107    2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/c650a6db-a02b-40ca-a614-b2d37d2b2173-web-config\") pod \"prometheus-k8s-0\" (UID: \"c650a6db-a02b-40ca-a614-b2d37d2b2173\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:47:19.779205 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:47:19.779157    2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/c650a6db-a02b-40ca-a614-b2d37d2b2173-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"c650a6db-a02b-40ca-a614-b2d37d2b2173\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:47:19.779205 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:47:19.779174    2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-42kg2\" (UniqueName: \"kubernetes.io/projected/c650a6db-a02b-40ca-a614-b2d37d2b2173-kube-api-access-42kg2\") pod \"prometheus-k8s-0\" (UID: \"c650a6db-a02b-40ca-a614-b2d37d2b2173\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:47:19.779344 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:47:19.779199    2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/c650a6db-a02b-40ca-a614-b2d37d2b2173-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"c650a6db-a02b-40ca-a614-b2d37d2b2173\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:47:19.779344 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:47:19.779244    2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/c650a6db-a02b-40ca-a614-b2d37d2b2173-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"c650a6db-a02b-40ca-a614-b2d37d2b2173\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:47:19.779344 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:47:19.779270    2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/c650a6db-a02b-40ca-a614-b2d37d2b2173-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"c650a6db-a02b-40ca-a614-b2d37d2b2173\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:47:19.779344 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:47:19.779301    2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/c650a6db-a02b-40ca-a614-b2d37d2b2173-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"c650a6db-a02b-40ca-a614-b2d37d2b2173\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:47:19.779546 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:47:19.779517    2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c650a6db-a02b-40ca-a614-b2d37d2b2173-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"c650a6db-a02b-40ca-a614-b2d37d2b2173\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:47:19.779709 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:47:19.779690    2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/c650a6db-a02b-40ca-a614-b2d37d2b2173-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"c650a6db-a02b-40ca-a614-b2d37d2b2173\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:47:19.779770 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:47:19.779731    2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/c650a6db-a02b-40ca-a614-b2d37d2b2173-config\") pod \"prometheus-k8s-0\" (UID: \"c650a6db-a02b-40ca-a614-b2d37d2b2173\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:47:19.779770 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:47:19.779758    2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/c650a6db-a02b-40ca-a614-b2d37d2b2173-config-out\") pod \"prometheus-k8s-0\" (UID: \"c650a6db-a02b-40ca-a614-b2d37d2b2173\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:47:19.779871 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:47:19.779788    2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/c650a6db-a02b-40ca-a614-b2d37d2b2173-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"c650a6db-a02b-40ca-a614-b2d37d2b2173\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:47:19.779871 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:47:19.779817    2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c650a6db-a02b-40ca-a614-b2d37d2b2173-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"c650a6db-a02b-40ca-a614-b2d37d2b2173\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:47:19.779871 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:47:19.779840    2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c650a6db-a02b-40ca-a614-b2d37d2b2173-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"c650a6db-a02b-40ca-a614-b2d37d2b2173\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:47:19.779985 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:47:19.779895    2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/c650a6db-a02b-40ca-a614-b2d37d2b2173-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"c650a6db-a02b-40ca-a614-b2d37d2b2173\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:47:19.779985 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:47:19.779922    2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/c650a6db-a02b-40ca-a614-b2d37d2b2173-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"c650a6db-a02b-40ca-a614-b2d37d2b2173\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:47:19.779985 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:47:19.779952    2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/c650a6db-a02b-40ca-a614-b2d37d2b2173-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"c650a6db-a02b-40ca-a614-b2d37d2b2173\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:47:19.880365 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:47:19.880282    2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/c650a6db-a02b-40ca-a614-b2d37d2b2173-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"c650a6db-a02b-40ca-a614-b2d37d2b2173\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:47:19.880365 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:47:19.880323    2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c650a6db-a02b-40ca-a614-b2d37d2b2173-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"c650a6db-a02b-40ca-a614-b2d37d2b2173\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:47:19.880365 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:47:19.880357    2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/c650a6db-a02b-40ca-a614-b2d37d2b2173-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"c650a6db-a02b-40ca-a614-b2d37d2b2173\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:47:19.880609 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:47:19.880385    2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/c650a6db-a02b-40ca-a614-b2d37d2b2173-config\") pod \"prometheus-k8s-0\" (UID: \"c650a6db-a02b-40ca-a614-b2d37d2b2173\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:47:19.880609 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:47:19.880407    2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/c650a6db-a02b-40ca-a614-b2d37d2b2173-config-out\") pod \"prometheus-k8s-0\" (UID: \"c650a6db-a02b-40ca-a614-b2d37d2b2173\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:47:19.880609 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:47:19.880430    2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/c650a6db-a02b-40ca-a614-b2d37d2b2173-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"c650a6db-a02b-40ca-a614-b2d37d2b2173\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:47:19.880609 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:47:19.880456    2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c650a6db-a02b-40ca-a614-b2d37d2b2173-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"c650a6db-a02b-40ca-a614-b2d37d2b2173\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:47:19.880609 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:47:19.880501    2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c650a6db-a02b-40ca-a614-b2d37d2b2173-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"c650a6db-a02b-40ca-a614-b2d37d2b2173\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:47:19.880609 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:47:19.880546    2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/c650a6db-a02b-40ca-a614-b2d37d2b2173-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"c650a6db-a02b-40ca-a614-b2d37d2b2173\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:47:19.880609 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:47:19.880594    2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/c650a6db-a02b-40ca-a614-b2d37d2b2173-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"c650a6db-a02b-40ca-a614-b2d37d2b2173\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:47:19.880944 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:47:19.880618    2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/c650a6db-a02b-40ca-a614-b2d37d2b2173-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"c650a6db-a02b-40ca-a614-b2d37d2b2173\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:47:19.880944 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:47:19.880652    2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/c650a6db-a02b-40ca-a614-b2d37d2b2173-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"c650a6db-a02b-40ca-a614-b2d37d2b2173\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:47:19.880944 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:47:19.880682    2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/c650a6db-a02b-40ca-a614-b2d37d2b2173-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"c650a6db-a02b-40ca-a614-b2d37d2b2173\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:47:19.880944 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:47:19.880711    2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/c650a6db-a02b-40ca-a614-b2d37d2b2173-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"c650a6db-a02b-40ca-a614-b2d37d2b2173\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:47:19.880944 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:47:19.880736    2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/c650a6db-a02b-40ca-a614-b2d37d2b2173-web-config\") pod \"prometheus-k8s-0\" (UID: \"c650a6db-a02b-40ca-a614-b2d37d2b2173\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:47:19.880944 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:47:19.880774    2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/c650a6db-a02b-40ca-a614-b2d37d2b2173-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"c650a6db-a02b-40ca-a614-b2d37d2b2173\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:47:19.880944 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:47:19.880799    2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-42kg2\" (UniqueName: \"kubernetes.io/projected/c650a6db-a02b-40ca-a614-b2d37d2b2173-kube-api-access-42kg2\") pod \"prometheus-k8s-0\" (UID: \"c650a6db-a02b-40ca-a614-b2d37d2b2173\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:47:19.880944 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:47:19.880837    2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/c650a6db-a02b-40ca-a614-b2d37d2b2173-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"c650a6db-a02b-40ca-a614-b2d37d2b2173\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:47:19.880944 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:47:19.880873    2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/c650a6db-a02b-40ca-a614-b2d37d2b2173-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"c650a6db-a02b-40ca-a614-b2d37d2b2173\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:47:19.880944 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:47:19.880901    2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/c650a6db-a02b-40ca-a614-b2d37d2b2173-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"c650a6db-a02b-40ca-a614-b2d37d2b2173\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:47:19.881430 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:47:19.881138    2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c650a6db-a02b-40ca-a614-b2d37d2b2173-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"c650a6db-a02b-40ca-a614-b2d37d2b2173\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:47:19.881430 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:47:19.881138    2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/c650a6db-a02b-40ca-a614-b2d37d2b2173-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"c650a6db-a02b-40ca-a614-b2d37d2b2173\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:47:19.881430 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:47:19.881219    2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/c650a6db-a02b-40ca-a614-b2d37d2b2173-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"c650a6db-a02b-40ca-a614-b2d37d2b2173\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:47:19.881430 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:47:19.881399    2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c650a6db-a02b-40ca-a614-b2d37d2b2173-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"c650a6db-a02b-40ca-a614-b2d37d2b2173\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:47:19.881688 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:47:19.881471    2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c650a6db-a02b-40ca-a614-b2d37d2b2173-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"c650a6db-a02b-40ca-a614-b2d37d2b2173\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:47:19.881688 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:47:19.881667    2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/c650a6db-a02b-40ca-a614-b2d37d2b2173-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"c650a6db-a02b-40ca-a614-b2d37d2b2173\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:47:19.882213 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:47:19.882181    2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/c650a6db-a02b-40ca-a614-b2d37d2b2173-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"c650a6db-a02b-40ca-a614-b2d37d2b2173\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:47:19.883943 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:47:19.883619    2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/c650a6db-a02b-40ca-a614-b2d37d2b2173-config\") pod \"prometheus-k8s-0\" (UID: \"c650a6db-a02b-40ca-a614-b2d37d2b2173\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:47:19.883943 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:47:19.883848    2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/c650a6db-a02b-40ca-a614-b2d37d2b2173-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"c650a6db-a02b-40ca-a614-b2d37d2b2173\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:47:19.883943 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:47:19.883895    2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/c650a6db-a02b-40ca-a614-b2d37d2b2173-web-config\") pod \"prometheus-k8s-0\" (UID: \"c650a6db-a02b-40ca-a614-b2d37d2b2173\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:47:19.884152 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:47:19.884112    2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/c650a6db-a02b-40ca-a614-b2d37d2b2173-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"c650a6db-a02b-40ca-a614-b2d37d2b2173\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:47:19.884314 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:47:19.884285    2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/c650a6db-a02b-40ca-a614-b2d37d2b2173-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"c650a6db-a02b-40ca-a614-b2d37d2b2173\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:47:19.884545 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:47:19.884520    2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/c650a6db-a02b-40ca-a614-b2d37d2b2173-config-out\") pod \"prometheus-k8s-0\" (UID: \"c650a6db-a02b-40ca-a614-b2d37d2b2173\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:47:19.885036 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:47:19.885010    2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/c650a6db-a02b-40ca-a614-b2d37d2b2173-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"c650a6db-a02b-40ca-a614-b2d37d2b2173\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:47:19.885289 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:47:19.885268    2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/c650a6db-a02b-40ca-a614-b2d37d2b2173-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"c650a6db-a02b-40ca-a614-b2d37d2b2173\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:47:19.885935 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:47:19.885916    2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/c650a6db-a02b-40ca-a614-b2d37d2b2173-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"c650a6db-a02b-40ca-a614-b2d37d2b2173\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:47:19.886048 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:47:19.886032    2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/c650a6db-a02b-40ca-a614-b2d37d2b2173-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"c650a6db-a02b-40ca-a614-b2d37d2b2173\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:47:19.886480 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:47:19.886456    2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/c650a6db-a02b-40ca-a614-b2d37d2b2173-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"c650a6db-a02b-40ca-a614-b2d37d2b2173\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:47:19.886608 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:47:19.886592    2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/c650a6db-a02b-40ca-a614-b2d37d2b2173-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"c650a6db-a02b-40ca-a614-b2d37d2b2173\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:47:19.889984 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:47:19.889960    2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-42kg2\" (UniqueName: \"kubernetes.io/projected/c650a6db-a02b-40ca-a614-b2d37d2b2173-kube-api-access-42kg2\") pod \"prometheus-k8s-0\" (UID: \"c650a6db-a02b-40ca-a614-b2d37d2b2173\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:47:20.011952 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:47:20.011901    2544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:47:20.297760 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:47:20.297736    2544 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 16 18:47:20.299104 ip-10-0-132-80 kubenswrapper[2544]: W0916 18:47:20.299078    2544 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc650a6db_a02b_40ca_a614_b2d37d2b2173.slice/crio-2f931d69affc3a4117fe69d385c5c321b3bbf841095fac600fad7c3066ad12da WatchSource:0}: Error finding container 2f931d69affc3a4117fe69d385c5c321b3bbf841095fac600fad7c3066ad12da: Status 404 returned error can't find the container with id 2f931d69affc3a4117fe69d385c5c321b3bbf841095fac600fad7c3066ad12da
Sep 16 18:47:20.645018 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:47:20.644988    2544 generic.go:356] "Generic (PLEG): container finished" podID="c650a6db-a02b-40ca-a614-b2d37d2b2173" containerID="e7fdac8cb21053f181d86ebfbd9ddc90174b37a0627c415cf43225f16c76f87b" exitCode=0
Sep 16 18:47:20.645355 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:47:20.645035    2544 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"c650a6db-a02b-40ca-a614-b2d37d2b2173","Type":"ContainerDied","Data":"e7fdac8cb21053f181d86ebfbd9ddc90174b37a0627c415cf43225f16c76f87b"}
Sep 16 18:47:20.645355 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:47:20.645084    2544 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"c650a6db-a02b-40ca-a614-b2d37d2b2173","Type":"ContainerStarted","Data":"2f931d69affc3a4117fe69d385c5c321b3bbf841095fac600fad7c3066ad12da"}
Sep 16 18:47:21.652738 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:47:21.652691    2544 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"c650a6db-a02b-40ca-a614-b2d37d2b2173","Type":"ContainerStarted","Data":"72e7f23e8a20ef69c48447223ef20b2d6ad5f2173bdfd63acf5e2b2c682c0186"}
Sep 16 18:47:21.653141 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:47:21.652759    2544 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"c650a6db-a02b-40ca-a614-b2d37d2b2173","Type":"ContainerStarted","Data":"2291a94806245b10b8017a20f693543c4133522c63864727db17e760d3fb1071"}
Sep 16 18:47:21.653141 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:47:21.652786    2544 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"c650a6db-a02b-40ca-a614-b2d37d2b2173","Type":"ContainerStarted","Data":"271089482b1af4d69f7e05f2c65d0dff1b98e96456ff9b4a04b6cce37921af4b"}
Sep 16 18:47:21.653141 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:47:21.652832    2544 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"c650a6db-a02b-40ca-a614-b2d37d2b2173","Type":"ContainerStarted","Data":"931fd8ecef45020e2b01d8967f6be4e70023dcf232ed3f3125c75cb9e7785fc1"}
Sep 16 18:47:21.653141 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:47:21.652860    2544 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"c650a6db-a02b-40ca-a614-b2d37d2b2173","Type":"ContainerStarted","Data":"3fae4c061750a030c42dc687993e7edab8daebabbe4d4675ec8954c2f75f1cae"}
Sep 16 18:47:21.653141 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:47:21.652884    2544 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"c650a6db-a02b-40ca-a614-b2d37d2b2173","Type":"ContainerStarted","Data":"f4f8a1bde86b017175f23cb8bf3fb40c262d0f165af3733849495d3a3c1bb592"}
Sep 16 18:47:21.714647 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:47:21.714587    2544 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-k8s-0" podStartSLOduration=2.696738105 podStartE2EDuration="2.714546702s" totalImagesPullingTime="0s" totalInitContainerRuntime="17.808597ms" isStatefulPod=false podCreationTimestamp="2026-09-16 18:47:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:47:21.709868106 +0000 UTC m=+204.498870649" watchObservedRunningTime="2026-09-16 18:47:21.714546702 +0000 UTC m=+204.503549237"
Sep 16 18:47:25.012033 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:47:25.011989    2544 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:48:20.012998 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:48:20.012906    2544 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:48:20.028599 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:48:20.028571    2544 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:48:20.906876 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:48:20.906849    2544 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:48:57.641028 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:48:57.641003    2544 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-pfcb7_9e3f9e34-420e-43ef-96b4-c7cad6b2eaa9/console-operator/2.log"
Sep 16 18:48:57.642651 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:48:57.642624    2544 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-pfcb7_9e3f9e34-420e-43ef-96b4-c7cad6b2eaa9/console-operator/2.log"
Sep 16 18:48:57.652319 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:48:57.652295    2544 kubelet.go:1648] "Image garbage collection succeeded"
Sep 16 18:51:37.148932 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:51:37.148875    2544 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/s3-init-ljlnm"]
Sep 16 18:51:37.152719 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:51:37.152696    2544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/s3-init-ljlnm"
Sep 16 18:51:37.160632 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:51:37.160606    2544 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"hf-token\""
Sep 16 18:51:37.160738 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:51:37.160657    2544 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"mlpipeline-s3-artifact\""
Sep 16 18:51:37.160797 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:51:37.160777    2544 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve\"/\"kube-root-ca.crt\""
Sep 16 18:51:37.160885 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:51:37.160869    2544 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"default-dockercfg-d9pmv\""
Sep 16 18:51:37.161449 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:51:37.161436    2544 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve\"/\"openshift-service-ca.crt\""
Sep 16 18:51:37.161684 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:51:37.161665    2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x98hb\" (UniqueName: \"kubernetes.io/projected/d2391c30-9a0d-44b8-9f6e-b9d433288c96-kube-api-access-x98hb\") pod \"s3-init-ljlnm\" (UID: \"d2391c30-9a0d-44b8-9f6e-b9d433288c96\") " pod="kserve/s3-init-ljlnm"
Sep 16 18:51:37.161763 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:51:37.161725    2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d2391c30-9a0d-44b8-9f6e-b9d433288c96-model-cache\") pod \"s3-init-ljlnm\" (UID: \"d2391c30-9a0d-44b8-9f6e-b9d433288c96\") " pod="kserve/s3-init-ljlnm"
Sep 16 18:51:37.168243 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:51:37.168222    2544 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/s3-init-ljlnm"]
Sep 16 18:51:37.262717 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:51:37.262679    2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x98hb\" (UniqueName: \"kubernetes.io/projected/d2391c30-9a0d-44b8-9f6e-b9d433288c96-kube-api-access-x98hb\") pod \"s3-init-ljlnm\" (UID: \"d2391c30-9a0d-44b8-9f6e-b9d433288c96\") " pod="kserve/s3-init-ljlnm"
Sep 16 18:51:37.262884 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:51:37.262729    2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d2391c30-9a0d-44b8-9f6e-b9d433288c96-model-cache\") pod \"s3-init-ljlnm\" (UID: \"d2391c30-9a0d-44b8-9f6e-b9d433288c96\") " pod="kserve/s3-init-ljlnm"
Sep 16 18:51:37.263053 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:51:37.263039    2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d2391c30-9a0d-44b8-9f6e-b9d433288c96-model-cache\") pod \"s3-init-ljlnm\" (UID: \"d2391c30-9a0d-44b8-9f6e-b9d433288c96\") " pod="kserve/s3-init-ljlnm"
Sep 16 18:51:37.277664 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:51:37.277641    2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-x98hb\" (UniqueName: \"kubernetes.io/projected/d2391c30-9a0d-44b8-9f6e-b9d433288c96-kube-api-access-x98hb\") pod \"s3-init-ljlnm\" (UID: \"d2391c30-9a0d-44b8-9f6e-b9d433288c96\") " pod="kserve/s3-init-ljlnm"
Sep 16 18:51:37.461023 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:51:37.460937    2544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/s3-init-ljlnm"
Sep 16 18:51:37.776682 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:51:37.776655    2544 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/s3-init-ljlnm"]
Sep 16 18:51:37.777862 ip-10-0-132-80 kubenswrapper[2544]: W0916 18:51:37.777838    2544 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podd2391c30_9a0d_44b8_9f6e_b9d433288c96.slice/crio-0e60f99c59cb58481895236d5f5b9ed41488fed924e909e0f25fad755ed5c12f WatchSource:0}: Error finding container 0e60f99c59cb58481895236d5f5b9ed41488fed924e909e0f25fad755ed5c12f: Status 404 returned error can't find the container with id 0e60f99c59cb58481895236d5f5b9ed41488fed924e909e0f25fad755ed5c12f
Sep 16 18:51:37.780079 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:51:37.780064    2544 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 16 18:51:38.679153 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:51:38.679119    2544 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-ljlnm" event={"ID":"d2391c30-9a0d-44b8-9f6e-b9d433288c96","Type":"ContainerStarted","Data":"0e60f99c59cb58481895236d5f5b9ed41488fed924e909e0f25fad755ed5c12f"}
Sep 16 18:51:41.699189 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:51:41.699153    2544 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-ljlnm" event={"ID":"d2391c30-9a0d-44b8-9f6e-b9d433288c96","Type":"ContainerStarted","Data":"0b06ba13ba3651fd0937de55e8a6372d46bfea92c64a7dabf9b1f11d3e69ee11"}
Sep 16 18:51:43.709652 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:51:43.709620    2544 generic.go:356] "Generic (PLEG): container finished" podID="d2391c30-9a0d-44b8-9f6e-b9d433288c96" containerID="0b06ba13ba3651fd0937de55e8a6372d46bfea92c64a7dabf9b1f11d3e69ee11" exitCode=0
Sep 16 18:51:43.710035 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:51:43.709703    2544 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-ljlnm" event={"ID":"d2391c30-9a0d-44b8-9f6e-b9d433288c96","Type":"ContainerDied","Data":"0b06ba13ba3651fd0937de55e8a6372d46bfea92c64a7dabf9b1f11d3e69ee11"}
Sep 16 18:51:48.734538 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:51:48.734506    2544 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-ljlnm" event={"ID":"d2391c30-9a0d-44b8-9f6e-b9d433288c96","Type":"ContainerStarted","Data":"600b788c9b2f83146d9616b3b95d4edd6c06096491c3d6119e75af6d7a0f690a"}
Sep 16 18:52:52.986777 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:52:52.986744    2544 generic.go:356] "Generic (PLEG): container finished" podID="d2391c30-9a0d-44b8-9f6e-b9d433288c96" containerID="600b788c9b2f83146d9616b3b95d4edd6c06096491c3d6119e75af6d7a0f690a" exitCode=0
Sep 16 18:52:52.987198 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:52:52.986786    2544 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-ljlnm" event={"ID":"d2391c30-9a0d-44b8-9f6e-b9d433288c96","Type":"ContainerDied","Data":"600b788c9b2f83146d9616b3b95d4edd6c06096491c3d6119e75af6d7a0f690a"}
Sep 16 18:52:54.116229 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:52:54.116205    2544 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/s3-init-ljlnm"
Sep 16 18:52:54.189412 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:52:54.189387    2544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d2391c30-9a0d-44b8-9f6e-b9d433288c96-model-cache\") pod \"d2391c30-9a0d-44b8-9f6e-b9d433288c96\" (UID: \"d2391c30-9a0d-44b8-9f6e-b9d433288c96\") "
Sep 16 18:52:54.189571 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:52:54.189480    2544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-x98hb\" (UniqueName: \"kubernetes.io/projected/d2391c30-9a0d-44b8-9f6e-b9d433288c96-kube-api-access-x98hb\") pod \"d2391c30-9a0d-44b8-9f6e-b9d433288c96\" (UID: \"d2391c30-9a0d-44b8-9f6e-b9d433288c96\") "
Sep 16 18:52:54.191541 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:52:54.191518    2544 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d2391c30-9a0d-44b8-9f6e-b9d433288c96-kube-api-access-x98hb" (OuterVolumeSpecName: "kube-api-access-x98hb") pod "d2391c30-9a0d-44b8-9f6e-b9d433288c96" (UID: "d2391c30-9a0d-44b8-9f6e-b9d433288c96"). InnerVolumeSpecName "kube-api-access-x98hb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 18:52:54.207721 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:52:54.207691    2544 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d2391c30-9a0d-44b8-9f6e-b9d433288c96-model-cache" (OuterVolumeSpecName: "model-cache") pod "d2391c30-9a0d-44b8-9f6e-b9d433288c96" (UID: "d2391c30-9a0d-44b8-9f6e-b9d433288c96"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 18:52:54.290892 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:52:54.290841    2544 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d2391c30-9a0d-44b8-9f6e-b9d433288c96-model-cache\") on node \"ip-10-0-132-80.ec2.internal\" DevicePath \"\""
Sep 16 18:52:54.290892 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:52:54.290863    2544 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x98hb\" (UniqueName: \"kubernetes.io/projected/d2391c30-9a0d-44b8-9f6e-b9d433288c96-kube-api-access-x98hb\") on node \"ip-10-0-132-80.ec2.internal\" DevicePath \"\""
Sep 16 18:52:54.997477 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:52:54.997448    2544 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/s3-init-ljlnm"
Sep 16 18:52:54.997648 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:52:54.997452    2544 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-ljlnm" event={"ID":"d2391c30-9a0d-44b8-9f6e-b9d433288c96","Type":"ContainerDied","Data":"0e60f99c59cb58481895236d5f5b9ed41488fed924e909e0f25fad755ed5c12f"}
Sep 16 18:52:54.997648 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:52:54.997585    2544 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0e60f99c59cb58481895236d5f5b9ed41488fed924e909e0f25fad755ed5c12f"
Sep 16 18:53:06.969736 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:53:06.969699    2544 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-7dc597df9b-f5gr6"]
Sep 16 18:53:06.970111 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:53:06.970089    2544 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d2391c30-9a0d-44b8-9f6e-b9d433288c96" containerName="download-hf-model"
Sep 16 18:53:06.970111 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:53:06.970102    2544 state_mem.go:107] "Deleted CPUSet assignment" podUID="d2391c30-9a0d-44b8-9f6e-b9d433288c96" containerName="download-hf-model"
Sep 16 18:53:06.970186 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:53:06.970117    2544 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d2391c30-9a0d-44b8-9f6e-b9d433288c96" containerName="s3-init"
Sep 16 18:53:06.970186 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:53:06.970124    2544 state_mem.go:107] "Deleted CPUSet assignment" podUID="d2391c30-9a0d-44b8-9f6e-b9d433288c96" containerName="s3-init"
Sep 16 18:53:06.970186 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:53:06.970180    2544 memory_manager.go:368] "RemoveStaleState removing state" podUID="d2391c30-9a0d-44b8-9f6e-b9d433288c96" containerName="s3-init"
Sep 16 18:53:06.973845 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:53:06.973825    2544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-7dc597df9b-f5gr6"
Sep 16 18:53:06.976125 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:53:06.976099    2544 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-rollback-v2-predictor-serving-cert\""
Sep 16 18:53:06.976264 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:53:06.976124    2544 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-rollback-kube-rbac-proxy-sar-config\""
Sep 16 18:53:06.976264 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:53:06.976212    2544 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Sep 16 18:53:06.976264 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:53:06.976232    2544 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-zmsvt\""
Sep 16 18:53:06.976264 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:53:06.976234    2544 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Sep 16 18:53:06.976479 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:53:06.976217    2544 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Sep 16 18:53:06.976607 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:53:06.976592    2544 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Sep 16 18:53:06.983981 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:53:06.983956    2544 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-7dc597df9b-f5gr6"]
Sep 16 18:53:07.084045 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:53:07.084012    2544 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-predictor-665c868b97-4lpk8"]
Sep 16 18:53:07.087604 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:53:07.087584    2544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-665c868b97-4lpk8"
Sep 16 18:53:07.089455 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:53:07.089438    2544 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-rollback-predictor-serving-cert\""
Sep 16 18:53:07.096239 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:53:07.096216    2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/7a5f744f-a31c-4006-b6a5-b8d2b4561c53-proxy-tls\") pod \"isvc-canary-rollback-v2-predictor-7dc597df9b-f5gr6\" (UID: \"7a5f744f-a31c-4006-b6a5-b8d2b4561c53\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-7dc597df9b-f5gr6"
Sep 16 18:53:07.096348 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:53:07.096249    2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f94kq\" (UniqueName: \"kubernetes.io/projected/7a5f744f-a31c-4006-b6a5-b8d2b4561c53-kube-api-access-f94kq\") pod \"isvc-canary-rollback-v2-predictor-7dc597df9b-f5gr6\" (UID: \"7a5f744f-a31c-4006-b6a5-b8d2b4561c53\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-7dc597df9b-f5gr6"
Sep 16 18:53:07.096348 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:53:07.096279    2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/7a5f744f-a31c-4006-b6a5-b8d2b4561c53-kserve-provision-location\") pod \"isvc-canary-rollback-v2-predictor-7dc597df9b-f5gr6\" (UID: \"7a5f744f-a31c-4006-b6a5-b8d2b4561c53\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-7dc597df9b-f5gr6"
Sep 16 18:53:07.096446 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:53:07.096346    2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/7a5f744f-a31c-4006-b6a5-b8d2b4561c53-cabundle-cert\") pod \"isvc-canary-rollback-v2-predictor-7dc597df9b-f5gr6\" (UID: \"7a5f744f-a31c-4006-b6a5-b8d2b4561c53\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-7dc597df9b-f5gr6"
Sep 16 18:53:07.096446 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:53:07.096389    2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/7a5f744f-a31c-4006-b6a5-b8d2b4561c53-kube-rbac-proxy-sar-config\") pod \"isvc-canary-rollback-v2-predictor-7dc597df9b-f5gr6\" (UID: \"7a5f744f-a31c-4006-b6a5-b8d2b4561c53\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-7dc597df9b-f5gr6"
Sep 16 18:53:07.100082 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:53:07.100061    2544 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-predictor-665c868b97-4lpk8"]
Sep 16 18:53:07.191578 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:53:07.191524    2544 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-raw-sklearn-batcher-a31c2-predictor-5857f79896-m4b9r"]
Sep 16 18:53:07.195268 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:53:07.195251    2544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-a31c2-predictor-5857f79896-m4b9r"
Sep 16 18:53:07.197509 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:53:07.197491    2544 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-raw-sklearn-batcher-a31c2-predictor-serving-cert\""
Sep 16 18:53:07.197618 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:53:07.197499    2544 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-raw-sklearn-batcher-a31c2-kube-rbac-proxy-sar-config\""
Sep 16 18:53:07.198042 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:53:07.198005    2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/7a5f744f-a31c-4006-b6a5-b8d2b4561c53-proxy-tls\") pod \"isvc-canary-rollback-v2-predictor-7dc597df9b-f5gr6\" (UID: \"7a5f744f-a31c-4006-b6a5-b8d2b4561c53\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-7dc597df9b-f5gr6"
Sep 16 18:53:07.198157 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:53:07.198055    2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/75b40cfe-b463-4afe-a48a-6c803b06b74d-proxy-tls\") pod \"isvc-canary-rollback-predictor-665c868b97-4lpk8\" (UID: \"75b40cfe-b463-4afe-a48a-6c803b06b74d\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-665c868b97-4lpk8"
Sep 16 18:53:07.198157 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:53:07.198088    2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/75b40cfe-b463-4afe-a48a-6c803b06b74d-cabundle-cert\") pod \"isvc-canary-rollback-predictor-665c868b97-4lpk8\" (UID: \"75b40cfe-b463-4afe-a48a-6c803b06b74d\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-665c868b97-4lpk8"
Sep 16 18:53:07.198157 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:53:07.198126    2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f94kq\" (UniqueName: \"kubernetes.io/projected/7a5f744f-a31c-4006-b6a5-b8d2b4561c53-kube-api-access-f94kq\") pod \"isvc-canary-rollback-v2-predictor-7dc597df9b-f5gr6\" (UID: \"7a5f744f-a31c-4006-b6a5-b8d2b4561c53\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-7dc597df9b-f5gr6"
Sep 16 18:53:07.198314 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:53:07.198169    2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/7a5f744f-a31c-4006-b6a5-b8d2b4561c53-kserve-provision-location\") pod \"isvc-canary-rollback-v2-predictor-7dc597df9b-f5gr6\" (UID: \"7a5f744f-a31c-4006-b6a5-b8d2b4561c53\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-7dc597df9b-f5gr6"
Sep 16 18:53:07.198314 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:53:07.198270    2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/75b40cfe-b463-4afe-a48a-6c803b06b74d-kserve-provision-location\") pod \"isvc-canary-rollback-predictor-665c868b97-4lpk8\" (UID: \"75b40cfe-b463-4afe-a48a-6c803b06b74d\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-665c868b97-4lpk8"
Sep 16 18:53:07.198417 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:53:07.198367    2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/75b40cfe-b463-4afe-a48a-6c803b06b74d-kube-rbac-proxy-sar-config\") pod \"isvc-canary-rollback-predictor-665c868b97-4lpk8\" (UID: \"75b40cfe-b463-4afe-a48a-6c803b06b74d\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-665c868b97-4lpk8"
Sep 16 18:53:07.199854 ip-10-0-132-80 kubenswrapper[2544]: E0916 18:53:07.198460    2544 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-serving-cert: secret "isvc-canary-rollback-v2-predictor-serving-cert" not found
Sep 16 18:53:07.199854 ip-10-0-132-80 kubenswrapper[2544]: E0916 18:53:07.198543    2544 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7a5f744f-a31c-4006-b6a5-b8d2b4561c53-proxy-tls podName:7a5f744f-a31c-4006-b6a5-b8d2b4561c53 nodeName:}" failed. No retries permitted until 2026-09-16 18:53:07.698523402 +0000 UTC m=+550.487525931 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/7a5f744f-a31c-4006-b6a5-b8d2b4561c53-proxy-tls") pod "isvc-canary-rollback-v2-predictor-7dc597df9b-f5gr6" (UID: "7a5f744f-a31c-4006-b6a5-b8d2b4561c53") : secret "isvc-canary-rollback-v2-predictor-serving-cert" not found
Sep 16 18:53:07.199854 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:53:07.198612    2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/7a5f744f-a31c-4006-b6a5-b8d2b4561c53-kserve-provision-location\") pod \"isvc-canary-rollback-v2-predictor-7dc597df9b-f5gr6\" (UID: \"7a5f744f-a31c-4006-b6a5-b8d2b4561c53\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-7dc597df9b-f5gr6"
Sep 16 18:53:07.199854 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:53:07.198786    2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/7a5f744f-a31c-4006-b6a5-b8d2b4561c53-cabundle-cert\") pod \"isvc-canary-rollback-v2-predictor-7dc597df9b-f5gr6\" (UID: \"7a5f744f-a31c-4006-b6a5-b8d2b4561c53\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-7dc597df9b-f5gr6"
Sep 16 18:53:07.199854 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:53:07.198848    2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g4k5r\" (UniqueName: \"kubernetes.io/projected/75b40cfe-b463-4afe-a48a-6c803b06b74d-kube-api-access-g4k5r\") pod \"isvc-canary-rollback-predictor-665c868b97-4lpk8\" (UID: \"75b40cfe-b463-4afe-a48a-6c803b06b74d\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-665c868b97-4lpk8"
Sep 16 18:53:07.199854 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:53:07.198906    2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/7a5f744f-a31c-4006-b6a5-b8d2b4561c53-kube-rbac-proxy-sar-config\") pod \"isvc-canary-rollback-v2-predictor-7dc597df9b-f5gr6\" (UID: \"7a5f744f-a31c-4006-b6a5-b8d2b4561c53\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-7dc597df9b-f5gr6"
Sep 16 18:53:07.199854 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:53:07.199506    2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/7a5f744f-a31c-4006-b6a5-b8d2b4561c53-cabundle-cert\") pod \"isvc-canary-rollback-v2-predictor-7dc597df9b-f5gr6\" (UID: \"7a5f744f-a31c-4006-b6a5-b8d2b4561c53\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-7dc597df9b-f5gr6"
Sep 16 18:53:07.203622 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:53:07.199892    2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/7a5f744f-a31c-4006-b6a5-b8d2b4561c53-kube-rbac-proxy-sar-config\") pod \"isvc-canary-rollback-v2-predictor-7dc597df9b-f5gr6\" (UID: \"7a5f744f-a31c-4006-b6a5-b8d2b4561c53\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-7dc597df9b-f5gr6"
Sep 16 18:53:07.208976 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:53:07.208955    2544 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-raw-sklearn-batcher-a31c2-predictor-5857f79896-m4b9r"]
Sep 16 18:53:07.215192 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:53:07.215167    2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-f94kq\" (UniqueName: \"kubernetes.io/projected/7a5f744f-a31c-4006-b6a5-b8d2b4561c53-kube-api-access-f94kq\") pod \"isvc-canary-rollback-v2-predictor-7dc597df9b-f5gr6\" (UID: \"7a5f744f-a31c-4006-b6a5-b8d2b4561c53\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-7dc597df9b-f5gr6"
Sep 16 18:53:07.300661 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:53:07.300575    2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/4df56c4f-7740-4f22-be89-ebfee0e47da5-kube-rbac-proxy-sar-config\") pod \"isvc-raw-sklearn-batcher-a31c2-predictor-5857f79896-m4b9r\" (UID: \"4df56c4f-7740-4f22-be89-ebfee0e47da5\") " pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-a31c2-predictor-5857f79896-m4b9r"
Sep 16 18:53:07.300661 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:53:07.300620    2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/75b40cfe-b463-4afe-a48a-6c803b06b74d-kube-rbac-proxy-sar-config\") pod \"isvc-canary-rollback-predictor-665c868b97-4lpk8\" (UID: \"75b40cfe-b463-4afe-a48a-6c803b06b74d\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-665c868b97-4lpk8"
Sep 16 18:53:07.300843 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:53:07.300670    2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g4k5r\" (UniqueName: \"kubernetes.io/projected/75b40cfe-b463-4afe-a48a-6c803b06b74d-kube-api-access-g4k5r\") pod \"isvc-canary-rollback-predictor-665c868b97-4lpk8\" (UID: \"75b40cfe-b463-4afe-a48a-6c803b06b74d\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-665c868b97-4lpk8"
Sep 16 18:53:07.300843 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:53:07.300742    2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7cc57\" (UniqueName: \"kubernetes.io/projected/4df56c4f-7740-4f22-be89-ebfee0e47da5-kube-api-access-7cc57\") pod \"isvc-raw-sklearn-batcher-a31c2-predictor-5857f79896-m4b9r\" (UID: \"4df56c4f-7740-4f22-be89-ebfee0e47da5\") " pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-a31c2-predictor-5857f79896-m4b9r"
Sep 16 18:53:07.300843 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:53:07.300791    2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/4df56c4f-7740-4f22-be89-ebfee0e47da5-kserve-provision-location\") pod \"isvc-raw-sklearn-batcher-a31c2-predictor-5857f79896-m4b9r\" (UID: \"4df56c4f-7740-4f22-be89-ebfee0e47da5\") " pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-a31c2-predictor-5857f79896-m4b9r"
Sep 16 18:53:07.300955 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:53:07.300888    2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/75b40cfe-b463-4afe-a48a-6c803b06b74d-proxy-tls\") pod \"isvc-canary-rollback-predictor-665c868b97-4lpk8\" (UID: \"75b40cfe-b463-4afe-a48a-6c803b06b74d\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-665c868b97-4lpk8"
Sep 16 18:53:07.300955 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:53:07.300924    2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/75b40cfe-b463-4afe-a48a-6c803b06b74d-cabundle-cert\") pod \"isvc-canary-rollback-predictor-665c868b97-4lpk8\" (UID: \"75b40cfe-b463-4afe-a48a-6c803b06b74d\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-665c868b97-4lpk8"
Sep 16 18:53:07.301034 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:53:07.300953    2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/4df56c4f-7740-4f22-be89-ebfee0e47da5-cabundle-cert\") pod \"isvc-raw-sklearn-batcher-a31c2-predictor-5857f79896-m4b9r\" (UID: \"4df56c4f-7740-4f22-be89-ebfee0e47da5\") " pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-a31c2-predictor-5857f79896-m4b9r"
Sep 16 18:53:07.301034 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:53:07.300983    2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/4df56c4f-7740-4f22-be89-ebfee0e47da5-proxy-tls\") pod \"isvc-raw-sklearn-batcher-a31c2-predictor-5857f79896-m4b9r\" (UID: \"4df56c4f-7740-4f22-be89-ebfee0e47da5\") " pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-a31c2-predictor-5857f79896-m4b9r"
Sep 16 18:53:07.301034 ip-10-0-132-80 kubenswrapper[2544]: E0916 18:53:07.301003    2544 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-canary-rollback-predictor-serving-cert: secret "isvc-canary-rollback-predictor-serving-cert" not found
Sep 16 18:53:07.301228 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:53:07.301040    2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/75b40cfe-b463-4afe-a48a-6c803b06b74d-kserve-provision-location\") pod \"isvc-canary-rollback-predictor-665c868b97-4lpk8\" (UID: \"75b40cfe-b463-4afe-a48a-6c803b06b74d\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-665c868b97-4lpk8"
Sep 16 18:53:07.301228 ip-10-0-132-80 kubenswrapper[2544]: E0916 18:53:07.301068    2544 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/75b40cfe-b463-4afe-a48a-6c803b06b74d-proxy-tls podName:75b40cfe-b463-4afe-a48a-6c803b06b74d nodeName:}" failed. No retries permitted until 2026-09-16 18:53:07.801052763 +0000 UTC m=+550.590055275 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/75b40cfe-b463-4afe-a48a-6c803b06b74d-proxy-tls") pod "isvc-canary-rollback-predictor-665c868b97-4lpk8" (UID: "75b40cfe-b463-4afe-a48a-6c803b06b74d") : secret "isvc-canary-rollback-predictor-serving-cert" not found
Sep 16 18:53:07.301499 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:53:07.301452    2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/75b40cfe-b463-4afe-a48a-6c803b06b74d-kserve-provision-location\") pod \"isvc-canary-rollback-predictor-665c868b97-4lpk8\" (UID: \"75b40cfe-b463-4afe-a48a-6c803b06b74d\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-665c868b97-4lpk8"
Sep 16 18:53:07.301833 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:53:07.301810    2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/75b40cfe-b463-4afe-a48a-6c803b06b74d-kube-rbac-proxy-sar-config\") pod \"isvc-canary-rollback-predictor-665c868b97-4lpk8\" (UID: \"75b40cfe-b463-4afe-a48a-6c803b06b74d\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-665c868b97-4lpk8"
Sep 16 18:53:07.301908 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:53:07.301810    2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/75b40cfe-b463-4afe-a48a-6c803b06b74d-cabundle-cert\") pod \"isvc-canary-rollback-predictor-665c868b97-4lpk8\" (UID: \"75b40cfe-b463-4afe-a48a-6c803b06b74d\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-665c868b97-4lpk8"
Sep 16 18:53:07.310578 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:53:07.310536    2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-g4k5r\" (UniqueName: \"kubernetes.io/projected/75b40cfe-b463-4afe-a48a-6c803b06b74d-kube-api-access-g4k5r\") pod \"isvc-canary-rollback-predictor-665c868b97-4lpk8\" (UID: \"75b40cfe-b463-4afe-a48a-6c803b06b74d\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-665c868b97-4lpk8"
Sep 16 18:53:07.402472 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:53:07.402421    2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/4df56c4f-7740-4f22-be89-ebfee0e47da5-kube-rbac-proxy-sar-config\") pod \"isvc-raw-sklearn-batcher-a31c2-predictor-5857f79896-m4b9r\" (UID: \"4df56c4f-7740-4f22-be89-ebfee0e47da5\") " pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-a31c2-predictor-5857f79896-m4b9r"
Sep 16 18:53:07.402693 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:53:07.402512    2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7cc57\" (UniqueName: \"kubernetes.io/projected/4df56c4f-7740-4f22-be89-ebfee0e47da5-kube-api-access-7cc57\") pod \"isvc-raw-sklearn-batcher-a31c2-predictor-5857f79896-m4b9r\" (UID: \"4df56c4f-7740-4f22-be89-ebfee0e47da5\") " pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-a31c2-predictor-5857f79896-m4b9r"
Sep 16 18:53:07.402693 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:53:07.402540    2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/4df56c4f-7740-4f22-be89-ebfee0e47da5-kserve-provision-location\") pod \"isvc-raw-sklearn-batcher-a31c2-predictor-5857f79896-m4b9r\" (UID: \"4df56c4f-7740-4f22-be89-ebfee0e47da5\") " pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-a31c2-predictor-5857f79896-m4b9r"
Sep 16 18:53:07.402693 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:53:07.402613    2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/4df56c4f-7740-4f22-be89-ebfee0e47da5-cabundle-cert\") pod \"isvc-raw-sklearn-batcher-a31c2-predictor-5857f79896-m4b9r\" (UID: \"4df56c4f-7740-4f22-be89-ebfee0e47da5\") " pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-a31c2-predictor-5857f79896-m4b9r"
Sep 16 18:53:07.402693 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:53:07.402641    2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/4df56c4f-7740-4f22-be89-ebfee0e47da5-proxy-tls\") pod \"isvc-raw-sklearn-batcher-a31c2-predictor-5857f79896-m4b9r\" (UID: \"4df56c4f-7740-4f22-be89-ebfee0e47da5\") " pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-a31c2-predictor-5857f79896-m4b9r"
Sep 16 18:53:07.403040 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:53:07.403009    2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/4df56c4f-7740-4f22-be89-ebfee0e47da5-kserve-provision-location\") pod \"isvc-raw-sklearn-batcher-a31c2-predictor-5857f79896-m4b9r\" (UID: \"4df56c4f-7740-4f22-be89-ebfee0e47da5\") " pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-a31c2-predictor-5857f79896-m4b9r"
Sep 16 18:53:07.403290 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:53:07.403269    2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/4df56c4f-7740-4f22-be89-ebfee0e47da5-cabundle-cert\") pod \"isvc-raw-sklearn-batcher-a31c2-predictor-5857f79896-m4b9r\" (UID: \"4df56c4f-7740-4f22-be89-ebfee0e47da5\") " pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-a31c2-predictor-5857f79896-m4b9r"
Sep 16 18:53:07.403354 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:53:07.403286    2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/4df56c4f-7740-4f22-be89-ebfee0e47da5-kube-rbac-proxy-sar-config\") pod \"isvc-raw-sklearn-batcher-a31c2-predictor-5857f79896-m4b9r\" (UID: \"4df56c4f-7740-4f22-be89-ebfee0e47da5\") " pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-a31c2-predictor-5857f79896-m4b9r"
Sep 16 18:53:07.405154 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:53:07.405134    2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/4df56c4f-7740-4f22-be89-ebfee0e47da5-proxy-tls\") pod \"isvc-raw-sklearn-batcher-a31c2-predictor-5857f79896-m4b9r\" (UID: \"4df56c4f-7740-4f22-be89-ebfee0e47da5\") " pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-a31c2-predictor-5857f79896-m4b9r"
Sep 16 18:53:07.412820 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:53:07.412801    2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7cc57\" (UniqueName: \"kubernetes.io/projected/4df56c4f-7740-4f22-be89-ebfee0e47da5-kube-api-access-7cc57\") pod \"isvc-raw-sklearn-batcher-a31c2-predictor-5857f79896-m4b9r\" (UID: \"4df56c4f-7740-4f22-be89-ebfee0e47da5\") " pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-a31c2-predictor-5857f79896-m4b9r"
Sep 16 18:53:07.508614 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:53:07.508544    2544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-a31c2-predictor-5857f79896-m4b9r"
Sep 16 18:53:07.706063 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:53:07.706024    2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/7a5f744f-a31c-4006-b6a5-b8d2b4561c53-proxy-tls\") pod \"isvc-canary-rollback-v2-predictor-7dc597df9b-f5gr6\" (UID: \"7a5f744f-a31c-4006-b6a5-b8d2b4561c53\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-7dc597df9b-f5gr6"
Sep 16 18:53:07.708475 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:53:07.708443    2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/7a5f744f-a31c-4006-b6a5-b8d2b4561c53-proxy-tls\") pod \"isvc-canary-rollback-v2-predictor-7dc597df9b-f5gr6\" (UID: \"7a5f744f-a31c-4006-b6a5-b8d2b4561c53\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-7dc597df9b-f5gr6"
Sep 16 18:53:07.784799 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:53:07.784770    2544 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-raw-sklearn-batcher-a31c2-predictor-5857f79896-m4b9r"]
Sep 16 18:53:07.787142 ip-10-0-132-80 kubenswrapper[2544]: W0916 18:53:07.787116    2544 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4df56c4f_7740_4f22_be89_ebfee0e47da5.slice/crio-1bb9066c0d571a414a842760aa9ca7841ccf5e4e442a20fd35ca58ef2808c0a8 WatchSource:0}: Error finding container 1bb9066c0d571a414a842760aa9ca7841ccf5e4e442a20fd35ca58ef2808c0a8: Status 404 returned error can't find the container with id 1bb9066c0d571a414a842760aa9ca7841ccf5e4e442a20fd35ca58ef2808c0a8
Sep 16 18:53:07.806577 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:53:07.806532    2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/75b40cfe-b463-4afe-a48a-6c803b06b74d-proxy-tls\") pod \"isvc-canary-rollback-predictor-665c868b97-4lpk8\" (UID: \"75b40cfe-b463-4afe-a48a-6c803b06b74d\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-665c868b97-4lpk8"
Sep 16 18:53:07.808764 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:53:07.808713    2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/75b40cfe-b463-4afe-a48a-6c803b06b74d-proxy-tls\") pod \"isvc-canary-rollback-predictor-665c868b97-4lpk8\" (UID: \"75b40cfe-b463-4afe-a48a-6c803b06b74d\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-665c868b97-4lpk8"
Sep 16 18:53:07.883985 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:53:07.883954    2544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-7dc597df9b-f5gr6"
Sep 16 18:53:07.997663 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:53:07.997624    2544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-665c868b97-4lpk8"
Sep 16 18:53:08.053015 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:53:08.052968    2544 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-a31c2-predictor-5857f79896-m4b9r" event={"ID":"4df56c4f-7740-4f22-be89-ebfee0e47da5","Type":"ContainerStarted","Data":"fa50551d58591fb01f9a73afe19ab9c6cace7def59a0f9488959d35e0f2a9e1d"}
Sep 16 18:53:08.053150 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:53:08.053069    2544 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-a31c2-predictor-5857f79896-m4b9r" event={"ID":"4df56c4f-7740-4f22-be89-ebfee0e47da5","Type":"ContainerStarted","Data":"1bb9066c0d571a414a842760aa9ca7841ccf5e4e442a20fd35ca58ef2808c0a8"}
Sep 16 18:53:08.155858 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:53:08.155822    2544 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-7dc597df9b-f5gr6"]
Sep 16 18:53:08.158314 ip-10-0-132-80 kubenswrapper[2544]: W0916 18:53:08.158287    2544 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7a5f744f_a31c_4006_b6a5_b8d2b4561c53.slice/crio-21d829cca545d1d73e9807c2f24eeede7d325d8c7d0852d0f4871cec5a1fedca WatchSource:0}: Error finding container 21d829cca545d1d73e9807c2f24eeede7d325d8c7d0852d0f4871cec5a1fedca: Status 404 returned error can't find the container with id 21d829cca545d1d73e9807c2f24eeede7d325d8c7d0852d0f4871cec5a1fedca
Sep 16 18:53:08.273260 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:53:08.273230    2544 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-predictor-665c868b97-4lpk8"]
Sep 16 18:53:08.276305 ip-10-0-132-80 kubenswrapper[2544]: W0916 18:53:08.276277    2544 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod75b40cfe_b463_4afe_a48a_6c803b06b74d.slice/crio-9908bacd26da9c2c9e148a6def4056cdf71d6899eded80e1b5da431a43aa3707 WatchSource:0}: Error finding container 9908bacd26da9c2c9e148a6def4056cdf71d6899eded80e1b5da431a43aa3707: Status 404 returned error can't find the container with id 9908bacd26da9c2c9e148a6def4056cdf71d6899eded80e1b5da431a43aa3707
Sep 16 18:53:09.058182 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:53:09.058141    2544 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-7dc597df9b-f5gr6" event={"ID":"7a5f744f-a31c-4006-b6a5-b8d2b4561c53","Type":"ContainerStarted","Data":"a04f3c48c8106948f06f6ca5c305bb7fabb787473e16e1d59601040a2a5946e3"}
Sep 16 18:53:09.058666 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:53:09.058209    2544 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-7dc597df9b-f5gr6" event={"ID":"7a5f744f-a31c-4006-b6a5-b8d2b4561c53","Type":"ContainerStarted","Data":"21d829cca545d1d73e9807c2f24eeede7d325d8c7d0852d0f4871cec5a1fedca"}
Sep 16 18:53:09.059679 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:53:09.059652    2544 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-665c868b97-4lpk8" event={"ID":"75b40cfe-b463-4afe-a48a-6c803b06b74d","Type":"ContainerStarted","Data":"b95ed9eae3042fc79d84e50d2dad53fd422c027dd2adc087f11968900823fa4a"}
Sep 16 18:53:09.059770 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:53:09.059701    2544 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-665c868b97-4lpk8" event={"ID":"75b40cfe-b463-4afe-a48a-6c803b06b74d","Type":"ContainerStarted","Data":"9908bacd26da9c2c9e148a6def4056cdf71d6899eded80e1b5da431a43aa3707"}
Sep 16 18:53:12.074670 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:53:12.074633    2544 generic.go:356] "Generic (PLEG): container finished" podID="4df56c4f-7740-4f22-be89-ebfee0e47da5" containerID="fa50551d58591fb01f9a73afe19ab9c6cace7def59a0f9488959d35e0f2a9e1d" exitCode=0
Sep 16 18:53:12.074670 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:53:12.074673    2544 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-a31c2-predictor-5857f79896-m4b9r" event={"ID":"4df56c4f-7740-4f22-be89-ebfee0e47da5","Type":"ContainerDied","Data":"fa50551d58591fb01f9a73afe19ab9c6cace7def59a0f9488959d35e0f2a9e1d"}
Sep 16 18:53:13.087499 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:53:13.086502    2544 generic.go:356] "Generic (PLEG): container finished" podID="75b40cfe-b463-4afe-a48a-6c803b06b74d" containerID="b95ed9eae3042fc79d84e50d2dad53fd422c027dd2adc087f11968900823fa4a" exitCode=0
Sep 16 18:53:13.087499 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:53:13.086687    2544 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-665c868b97-4lpk8" event={"ID":"75b40cfe-b463-4afe-a48a-6c803b06b74d","Type":"ContainerDied","Data":"b95ed9eae3042fc79d84e50d2dad53fd422c027dd2adc087f11968900823fa4a"}
Sep 16 18:53:13.094990 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:53:13.094173    2544 generic.go:356] "Generic (PLEG): container finished" podID="7a5f744f-a31c-4006-b6a5-b8d2b4561c53" containerID="a04f3c48c8106948f06f6ca5c305bb7fabb787473e16e1d59601040a2a5946e3" exitCode=0
Sep 16 18:53:13.094990 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:53:13.094235    2544 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-7dc597df9b-f5gr6" event={"ID":"7a5f744f-a31c-4006-b6a5-b8d2b4561c53","Type":"ContainerDied","Data":"a04f3c48c8106948f06f6ca5c305bb7fabb787473e16e1d59601040a2a5946e3"}
Sep 16 18:53:32.286319 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:53:32.286275    2544 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-a31c2-predictor-5857f79896-m4b9r" event={"ID":"4df56c4f-7740-4f22-be89-ebfee0e47da5","Type":"ContainerStarted","Data":"45b5f198c276027b9c7a8cb07c48ff104c1a2d43c5153f7663eae21bccf46743"}
Sep 16 18:53:32.290329 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:53:32.290297    2544 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-7dc597df9b-f5gr6" event={"ID":"7a5f744f-a31c-4006-b6a5-b8d2b4561c53","Type":"ContainerStarted","Data":"f78a9ff879832ed81559f36976e1c5b4f64d160d390bb33e73735290360f66c8"}
Sep 16 18:53:32.295420 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:53:32.295337    2544 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-665c868b97-4lpk8" event={"ID":"75b40cfe-b463-4afe-a48a-6c803b06b74d","Type":"ContainerStarted","Data":"bbf7eb254136cbad6e6204318fd31a499abca68c07fc3e4fd43e7bd9f9ba8e37"}
Sep 16 18:53:34.309789 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:53:34.309731    2544 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-665c868b97-4lpk8" event={"ID":"75b40cfe-b463-4afe-a48a-6c803b06b74d","Type":"ContainerStarted","Data":"b8b414a51b397fd78c7db5101c3b8b5b7bc4f722328be4a51c334cc450101db8"}
Sep 16 18:53:34.310250 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:53:34.309819    2544 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-665c868b97-4lpk8"
Sep 16 18:53:34.310250 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:53:34.309993    2544 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-665c868b97-4lpk8"
Sep 16 18:53:34.311405 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:53:34.311371    2544 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-665c868b97-4lpk8" podUID="75b40cfe-b463-4afe-a48a-6c803b06b74d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.26:8080: connect: connection refused"
Sep 16 18:53:34.312611 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:53:34.312593    2544 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-a31c2-predictor-5857f79896-m4b9r" event={"ID":"4df56c4f-7740-4f22-be89-ebfee0e47da5","Type":"ContainerStarted","Data":"1fb67b0d035cf5459ddc5cb8b1a927c3d87ee28a0a9946c6f38c1e9809a8df19"}
Sep 16 18:53:34.314487 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:53:34.314466    2544 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-7dc597df9b-f5gr6" event={"ID":"7a5f744f-a31c-4006-b6a5-b8d2b4561c53","Type":"ContainerStarted","Data":"942edcb2d4c852444220f19a464a876700f1fdab69de5905d7ba8a682f34ffb9"}
Sep 16 18:53:34.314671 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:53:34.314653    2544 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-7dc597df9b-f5gr6"
Sep 16 18:53:34.314751 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:53:34.314706    2544 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-7dc597df9b-f5gr6"
Sep 16 18:53:34.315449 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:53:34.315424    2544 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-7dc597df9b-f5gr6" podUID="7a5f744f-a31c-4006-b6a5-b8d2b4561c53" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.25:8080: connect: connection refused"
Sep 16 18:53:34.334841 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:53:34.334801    2544 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-665c868b97-4lpk8" podStartSLOduration=2.448572042 podStartE2EDuration="27.334787402s" totalImagesPullingTime="21.022046284s" totalInitContainerRuntime="3.864169076s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:53:07 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:53:34.332894544 +0000 UTC m=+577.121897089" watchObservedRunningTime="2026-09-16 18:53:34.334787402 +0000 UTC m=+577.123789935"
Sep 16 18:53:34.356452 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:53:34.356416    2544 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-7dc597df9b-f5gr6" podStartSLOduration=3.280978377 podStartE2EDuration="28.356403952s" totalImagesPullingTime="21.056963594s" totalInitContainerRuntime="4.018461981s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:53:06 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:53:34.355730781 +0000 UTC m=+577.144733333" watchObservedRunningTime="2026-09-16 18:53:34.356403952 +0000 UTC m=+577.145406486"
Sep 16 18:53:35.318767 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:53:35.318720    2544 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-665c868b97-4lpk8" podUID="75b40cfe-b463-4afe-a48a-6c803b06b74d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.26:8080: connect: connection refused"
Sep 16 18:53:35.319136 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:53:35.318821    2544 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-7dc597df9b-f5gr6" podUID="7a5f744f-a31c-4006-b6a5-b8d2b4561c53" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.25:8080: connect: connection refused"
Sep 16 18:53:37.330731 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:53:37.330700    2544 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-a31c2-predictor-5857f79896-m4b9r" event={"ID":"4df56c4f-7740-4f22-be89-ebfee0e47da5","Type":"ContainerStarted","Data":"20409c96989a0ffc8d2039b56cf1ead4cd317c9e2b5343e712b37db4f0dc906f"}
Sep 16 18:53:37.331104 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:53:37.330869    2544 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-a31c2-predictor-5857f79896-m4b9r"
Sep 16 18:53:37.354203 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:53:37.354160    2544 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-a31c2-predictor-5857f79896-m4b9r" podStartSLOduration=2.298247081 podStartE2EDuration="30.354144794s" totalImagesPullingTime="24.425388845s" totalInitContainerRuntime="3.630508868s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:53:07 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:53:37.353720556 +0000 UTC m=+580.142723138" watchObservedRunningTime="2026-09-16 18:53:37.354144794 +0000 UTC m=+580.143147330"
Sep 16 18:53:38.335149 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:53:38.335116    2544 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-a31c2-predictor-5857f79896-m4b9r"
Sep 16 18:53:38.335512 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:53:38.335181    2544 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-a31c2-predictor-5857f79896-m4b9r"
Sep 16 18:53:38.336317 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:53:38.336290    2544 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-a31c2-predictor-5857f79896-m4b9r" podUID="4df56c4f-7740-4f22-be89-ebfee0e47da5" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.27:8080: connect: connection refused"
Sep 16 18:53:38.336950 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:53:38.336921    2544 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-a31c2-predictor-5857f79896-m4b9r" podUID="4df56c4f-7740-4f22-be89-ebfee0e47da5" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 16 18:53:39.339192 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:53:39.339155    2544 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-a31c2-predictor-5857f79896-m4b9r" podUID="4df56c4f-7740-4f22-be89-ebfee0e47da5" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.27:8080: connect: connection refused"
Sep 16 18:53:39.339665 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:53:39.339542    2544 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-a31c2-predictor-5857f79896-m4b9r" podUID="4df56c4f-7740-4f22-be89-ebfee0e47da5" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 16 18:53:39.343781 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:53:39.343759    2544 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-a31c2-predictor-5857f79896-m4b9r"
Sep 16 18:53:40.323289 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:53:40.323251    2544 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-665c868b97-4lpk8"
Sep 16 18:53:40.323575 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:53:40.323530    2544 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-7dc597df9b-f5gr6"
Sep 16 18:53:40.323813 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:53:40.323780    2544 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-665c868b97-4lpk8" podUID="75b40cfe-b463-4afe-a48a-6c803b06b74d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.26:8080: connect: connection refused"
Sep 16 18:53:40.324161 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:53:40.324137    2544 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-7dc597df9b-f5gr6" podUID="7a5f744f-a31c-4006-b6a5-b8d2b4561c53" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.25:8080: connect: connection refused"
Sep 16 18:53:40.347181 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:53:40.347138    2544 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-a31c2-predictor-5857f79896-m4b9r" podUID="4df56c4f-7740-4f22-be89-ebfee0e47da5" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.27:8080: connect: connection refused"
Sep 16 18:53:40.347521 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:53:40.347379    2544 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-a31c2-predictor-5857f79896-m4b9r" podUID="4df56c4f-7740-4f22-be89-ebfee0e47da5" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 16 18:53:50.324059 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:53:50.324015    2544 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-665c868b97-4lpk8" podUID="75b40cfe-b463-4afe-a48a-6c803b06b74d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.26:8080: connect: connection refused"
Sep 16 18:53:50.324519 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:53:50.324153    2544 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-7dc597df9b-f5gr6" podUID="7a5f744f-a31c-4006-b6a5-b8d2b4561c53" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.25:8080: connect: connection refused"
Sep 16 18:53:50.347506 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:53:50.347456    2544 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-a31c2-predictor-5857f79896-m4b9r" podUID="4df56c4f-7740-4f22-be89-ebfee0e47da5" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.27:8080: connect: connection refused"
Sep 16 18:53:50.347854 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:53:50.347827    2544 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-a31c2-predictor-5857f79896-m4b9r" podUID="4df56c4f-7740-4f22-be89-ebfee0e47da5" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 16 18:53:57.677352 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:53:57.677326    2544 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-pfcb7_9e3f9e34-420e-43ef-96b4-c7cad6b2eaa9/console-operator/2.log"
Sep 16 18:53:57.678144 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:53:57.678122    2544 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-pfcb7_9e3f9e34-420e-43ef-96b4-c7cad6b2eaa9/console-operator/2.log"
Sep 16 18:54:00.323895 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:54:00.323830    2544 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-665c868b97-4lpk8" podUID="75b40cfe-b463-4afe-a48a-6c803b06b74d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.26:8080: connect: connection refused"
Sep 16 18:54:00.324628 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:54:00.324592    2544 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-7dc597df9b-f5gr6" podUID="7a5f744f-a31c-4006-b6a5-b8d2b4561c53" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.25:8080: connect: connection refused"
Sep 16 18:54:00.347176 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:54:00.347140    2544 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-a31c2-predictor-5857f79896-m4b9r" podUID="4df56c4f-7740-4f22-be89-ebfee0e47da5" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.27:8080: connect: connection refused"
Sep 16 18:54:00.347612 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:54:00.347594    2544 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-a31c2-predictor-5857f79896-m4b9r" podUID="4df56c4f-7740-4f22-be89-ebfee0e47da5" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 16 18:54:10.324542 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:54:10.324497    2544 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-665c868b97-4lpk8" podUID="75b40cfe-b463-4afe-a48a-6c803b06b74d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.26:8080: connect: connection refused"
Sep 16 18:54:10.324542 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:54:10.324496    2544 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-7dc597df9b-f5gr6" podUID="7a5f744f-a31c-4006-b6a5-b8d2b4561c53" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.25:8080: connect: connection refused"
Sep 16 18:54:10.347214 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:54:10.347178    2544 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-a31c2-predictor-5857f79896-m4b9r" podUID="4df56c4f-7740-4f22-be89-ebfee0e47da5" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.27:8080: connect: connection refused"
Sep 16 18:54:10.347629 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:54:10.347602    2544 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-a31c2-predictor-5857f79896-m4b9r" podUID="4df56c4f-7740-4f22-be89-ebfee0e47da5" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 16 18:54:20.324039 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:54:20.323951    2544 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-665c868b97-4lpk8" podUID="75b40cfe-b463-4afe-a48a-6c803b06b74d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.26:8080: connect: connection refused"
Sep 16 18:54:20.324467 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:54:20.324223    2544 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-7dc597df9b-f5gr6" podUID="7a5f744f-a31c-4006-b6a5-b8d2b4561c53" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.25:8080: connect: connection refused"
Sep 16 18:54:20.347170 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:54:20.347129    2544 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-a31c2-predictor-5857f79896-m4b9r" podUID="4df56c4f-7740-4f22-be89-ebfee0e47da5" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.27:8080: connect: connection refused"
Sep 16 18:54:20.347685 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:54:20.347659    2544 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-a31c2-predictor-5857f79896-m4b9r" podUID="4df56c4f-7740-4f22-be89-ebfee0e47da5" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 16 18:54:30.323930 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:54:30.323881    2544 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-665c868b97-4lpk8" podUID="75b40cfe-b463-4afe-a48a-6c803b06b74d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.26:8080: connect: connection refused"
Sep 16 18:54:30.324465 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:54:30.324375    2544 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-7dc597df9b-f5gr6" podUID="7a5f744f-a31c-4006-b6a5-b8d2b4561c53" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.25:8080: connect: connection refused"
Sep 16 18:54:30.347088 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:54:30.347054    2544 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-a31c2-predictor-5857f79896-m4b9r" podUID="4df56c4f-7740-4f22-be89-ebfee0e47da5" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.27:8080: connect: connection refused"
Sep 16 18:54:30.347471 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:54:30.347450    2544 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-a31c2-predictor-5857f79896-m4b9r" podUID="4df56c4f-7740-4f22-be89-ebfee0e47da5" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 16 18:54:40.324319 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:54:40.324282    2544 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-665c868b97-4lpk8" podUID="75b40cfe-b463-4afe-a48a-6c803b06b74d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.26:8080: connect: connection refused"
Sep 16 18:54:40.324319 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:54:40.324309    2544 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-7dc597df9b-f5gr6" podUID="7a5f744f-a31c-4006-b6a5-b8d2b4561c53" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.25:8080: connect: connection refused"
Sep 16 18:54:40.347173 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:54:40.347140    2544 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-a31c2-predictor-5857f79896-m4b9r" podUID="4df56c4f-7740-4f22-be89-ebfee0e47da5" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.27:8080: connect: connection refused"
Sep 16 18:54:40.347573 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:54:40.347529    2544 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-a31c2-predictor-5857f79896-m4b9r" podUID="4df56c4f-7740-4f22-be89-ebfee0e47da5" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 16 18:54:50.325127 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:54:50.325098    2544 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-665c868b97-4lpk8"
Sep 16 18:54:50.325493 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:54:50.325181    2544 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-7dc597df9b-f5gr6"
Sep 16 18:54:50.348297 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:54:50.348269    2544 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-a31c2-predictor-5857f79896-m4b9r"
Sep 16 18:54:50.348443 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:54:50.348389    2544 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-a31c2-predictor-5857f79896-m4b9r"
Sep 16 18:54:50.475853 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:54:50.475824    2544 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-predictor-665c868b97-4lpk8"]
Sep 16 18:54:50.664716 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:54:50.664607    2544 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-665c868b97-4lpk8" podUID="75b40cfe-b463-4afe-a48a-6c803b06b74d" containerName="kserve-container" containerID="cri-o://bbf7eb254136cbad6e6204318fd31a499abca68c07fc3e4fd43e7bd9f9ba8e37" gracePeriod=30
Sep 16 18:54:50.664716 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:54:50.664690    2544 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-665c868b97-4lpk8" podUID="75b40cfe-b463-4afe-a48a-6c803b06b74d" containerName="kube-rbac-proxy" containerID="cri-o://b8b414a51b397fd78c7db5101c3b8b5b7bc4f722328be4a51c334cc450101db8" gracePeriod=30
Sep 16 18:54:51.670396 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:54:51.670361    2544 generic.go:356] "Generic (PLEG): container finished" podID="75b40cfe-b463-4afe-a48a-6c803b06b74d" containerID="b8b414a51b397fd78c7db5101c3b8b5b7bc4f722328be4a51c334cc450101db8" exitCode=2
Sep 16 18:54:51.670773 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:54:51.670408    2544 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-665c868b97-4lpk8" event={"ID":"75b40cfe-b463-4afe-a48a-6c803b06b74d","Type":"ContainerDied","Data":"b8b414a51b397fd78c7db5101c3b8b5b7bc4f722328be4a51c334cc450101db8"}
Sep 16 18:54:55.204279 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:54:55.204257    2544 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-665c868b97-4lpk8"
Sep 16 18:54:55.305143 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:54:55.305066    2544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/75b40cfe-b463-4afe-a48a-6c803b06b74d-proxy-tls\") pod \"75b40cfe-b463-4afe-a48a-6c803b06b74d\" (UID: \"75b40cfe-b463-4afe-a48a-6c803b06b74d\") "
Sep 16 18:54:55.305143 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:54:55.305105    2544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/75b40cfe-b463-4afe-a48a-6c803b06b74d-kube-rbac-proxy-sar-config\") pod \"75b40cfe-b463-4afe-a48a-6c803b06b74d\" (UID: \"75b40cfe-b463-4afe-a48a-6c803b06b74d\") "
Sep 16 18:54:55.305343 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:54:55.305152    2544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/75b40cfe-b463-4afe-a48a-6c803b06b74d-cabundle-cert\") pod \"75b40cfe-b463-4afe-a48a-6c803b06b74d\" (UID: \"75b40cfe-b463-4afe-a48a-6c803b06b74d\") "
Sep 16 18:54:55.305343 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:54:55.305171    2544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/75b40cfe-b463-4afe-a48a-6c803b06b74d-kserve-provision-location\") pod \"75b40cfe-b463-4afe-a48a-6c803b06b74d\" (UID: \"75b40cfe-b463-4afe-a48a-6c803b06b74d\") "
Sep 16 18:54:55.305343 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:54:55.305226    2544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-g4k5r\" (UniqueName: \"kubernetes.io/projected/75b40cfe-b463-4afe-a48a-6c803b06b74d-kube-api-access-g4k5r\") pod \"75b40cfe-b463-4afe-a48a-6c803b06b74d\" (UID: \"75b40cfe-b463-4afe-a48a-6c803b06b74d\") "
Sep 16 18:54:55.305511 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:54:55.305464    2544 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/75b40cfe-b463-4afe-a48a-6c803b06b74d-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "75b40cfe-b463-4afe-a48a-6c803b06b74d" (UID: "75b40cfe-b463-4afe-a48a-6c803b06b74d"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 18:54:55.305596 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:54:55.305543    2544 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/75b40cfe-b463-4afe-a48a-6c803b06b74d-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "75b40cfe-b463-4afe-a48a-6c803b06b74d" (UID: "75b40cfe-b463-4afe-a48a-6c803b06b74d"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 18:54:55.305596 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:54:55.305576    2544 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/75b40cfe-b463-4afe-a48a-6c803b06b74d-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "75b40cfe-b463-4afe-a48a-6c803b06b74d" (UID: "75b40cfe-b463-4afe-a48a-6c803b06b74d"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 18:54:55.307347 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:54:55.307318    2544 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/75b40cfe-b463-4afe-a48a-6c803b06b74d-kube-api-access-g4k5r" (OuterVolumeSpecName: "kube-api-access-g4k5r") pod "75b40cfe-b463-4afe-a48a-6c803b06b74d" (UID: "75b40cfe-b463-4afe-a48a-6c803b06b74d"). InnerVolumeSpecName "kube-api-access-g4k5r". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 18:54:55.307347 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:54:55.307343    2544 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/75b40cfe-b463-4afe-a48a-6c803b06b74d-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "75b40cfe-b463-4afe-a48a-6c803b06b74d" (UID: "75b40cfe-b463-4afe-a48a-6c803b06b74d"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 18:54:55.406400 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:54:55.406367    2544 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/75b40cfe-b463-4afe-a48a-6c803b06b74d-proxy-tls\") on node \"ip-10-0-132-80.ec2.internal\" DevicePath \"\""
Sep 16 18:54:55.406400 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:54:55.406395    2544 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/75b40cfe-b463-4afe-a48a-6c803b06b74d-kube-rbac-proxy-sar-config\") on node \"ip-10-0-132-80.ec2.internal\" DevicePath \"\""
Sep 16 18:54:55.406605 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:54:55.406408    2544 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/75b40cfe-b463-4afe-a48a-6c803b06b74d-cabundle-cert\") on node \"ip-10-0-132-80.ec2.internal\" DevicePath \"\""
Sep 16 18:54:55.406605 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:54:55.406422    2544 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/75b40cfe-b463-4afe-a48a-6c803b06b74d-kserve-provision-location\") on node \"ip-10-0-132-80.ec2.internal\" DevicePath \"\""
Sep 16 18:54:55.406605 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:54:55.406433    2544 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g4k5r\" (UniqueName: \"kubernetes.io/projected/75b40cfe-b463-4afe-a48a-6c803b06b74d-kube-api-access-g4k5r\") on node \"ip-10-0-132-80.ec2.internal\" DevicePath \"\""
Sep 16 18:54:55.689093 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:54:55.689061    2544 generic.go:356] "Generic (PLEG): container finished" podID="75b40cfe-b463-4afe-a48a-6c803b06b74d" containerID="bbf7eb254136cbad6e6204318fd31a499abca68c07fc3e4fd43e7bd9f9ba8e37" exitCode=0
Sep 16 18:54:55.689093 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:54:55.689097    2544 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-665c868b97-4lpk8" event={"ID":"75b40cfe-b463-4afe-a48a-6c803b06b74d","Type":"ContainerDied","Data":"bbf7eb254136cbad6e6204318fd31a499abca68c07fc3e4fd43e7bd9f9ba8e37"}
Sep 16 18:54:55.689328 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:54:55.689128    2544 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-665c868b97-4lpk8"
Sep 16 18:54:55.689328 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:54:55.689143    2544 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-665c868b97-4lpk8" event={"ID":"75b40cfe-b463-4afe-a48a-6c803b06b74d","Type":"ContainerDied","Data":"9908bacd26da9c2c9e148a6def4056cdf71d6899eded80e1b5da431a43aa3707"}
Sep 16 18:54:55.689328 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:54:55.689168    2544 scope.go:117] "RemoveContainer" containerID="b8b414a51b397fd78c7db5101c3b8b5b7bc4f722328be4a51c334cc450101db8"
Sep 16 18:54:55.701268 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:54:55.699337    2544 scope.go:117] "RemoveContainer" containerID="bbf7eb254136cbad6e6204318fd31a499abca68c07fc3e4fd43e7bd9f9ba8e37"
Sep 16 18:54:55.708029 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:54:55.708010    2544 scope.go:117] "RemoveContainer" containerID="b95ed9eae3042fc79d84e50d2dad53fd422c027dd2adc087f11968900823fa4a"
Sep 16 18:54:55.715329 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:54:55.715307    2544 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-predictor-665c868b97-4lpk8"]
Sep 16 18:54:55.716090 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:54:55.716074    2544 scope.go:117] "RemoveContainer" containerID="b8b414a51b397fd78c7db5101c3b8b5b7bc4f722328be4a51c334cc450101db8"
Sep 16 18:54:55.716372 ip-10-0-132-80 kubenswrapper[2544]: E0916 18:54:55.716348    2544 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b8b414a51b397fd78c7db5101c3b8b5b7bc4f722328be4a51c334cc450101db8\": container with ID starting with b8b414a51b397fd78c7db5101c3b8b5b7bc4f722328be4a51c334cc450101db8 not found: ID does not exist" containerID="b8b414a51b397fd78c7db5101c3b8b5b7bc4f722328be4a51c334cc450101db8"
Sep 16 18:54:55.716428 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:54:55.716384    2544 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b8b414a51b397fd78c7db5101c3b8b5b7bc4f722328be4a51c334cc450101db8"} err="failed to get container status \"b8b414a51b397fd78c7db5101c3b8b5b7bc4f722328be4a51c334cc450101db8\": rpc error: code = NotFound desc = could not find container \"b8b414a51b397fd78c7db5101c3b8b5b7bc4f722328be4a51c334cc450101db8\": container with ID starting with b8b414a51b397fd78c7db5101c3b8b5b7bc4f722328be4a51c334cc450101db8 not found: ID does not exist"
Sep 16 18:54:55.716428 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:54:55.716408    2544 scope.go:117] "RemoveContainer" containerID="bbf7eb254136cbad6e6204318fd31a499abca68c07fc3e4fd43e7bd9f9ba8e37"
Sep 16 18:54:55.716787 ip-10-0-132-80 kubenswrapper[2544]: E0916 18:54:55.716763    2544 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bbf7eb254136cbad6e6204318fd31a499abca68c07fc3e4fd43e7bd9f9ba8e37\": container with ID starting with bbf7eb254136cbad6e6204318fd31a499abca68c07fc3e4fd43e7bd9f9ba8e37 not found: ID does not exist" containerID="bbf7eb254136cbad6e6204318fd31a499abca68c07fc3e4fd43e7bd9f9ba8e37"
Sep 16 18:54:55.716900 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:54:55.716796    2544 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bbf7eb254136cbad6e6204318fd31a499abca68c07fc3e4fd43e7bd9f9ba8e37"} err="failed to get container status \"bbf7eb254136cbad6e6204318fd31a499abca68c07fc3e4fd43e7bd9f9ba8e37\": rpc error: code = NotFound desc = could not find container \"bbf7eb254136cbad6e6204318fd31a499abca68c07fc3e4fd43e7bd9f9ba8e37\": container with ID starting with bbf7eb254136cbad6e6204318fd31a499abca68c07fc3e4fd43e7bd9f9ba8e37 not found: ID does not exist"
Sep 16 18:54:55.716900 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:54:55.716817    2544 scope.go:117] "RemoveContainer" containerID="b95ed9eae3042fc79d84e50d2dad53fd422c027dd2adc087f11968900823fa4a"
Sep 16 18:54:55.717517 ip-10-0-132-80 kubenswrapper[2544]: E0916 18:54:55.717494    2544 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b95ed9eae3042fc79d84e50d2dad53fd422c027dd2adc087f11968900823fa4a\": container with ID starting with b95ed9eae3042fc79d84e50d2dad53fd422c027dd2adc087f11968900823fa4a not found: ID does not exist" containerID="b95ed9eae3042fc79d84e50d2dad53fd422c027dd2adc087f11968900823fa4a"
Sep 16 18:54:55.717618 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:54:55.717523    2544 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b95ed9eae3042fc79d84e50d2dad53fd422c027dd2adc087f11968900823fa4a"} err="failed to get container status \"b95ed9eae3042fc79d84e50d2dad53fd422c027dd2adc087f11968900823fa4a\": rpc error: code = NotFound desc = could not find container \"b95ed9eae3042fc79d84e50d2dad53fd422c027dd2adc087f11968900823fa4a\": container with ID starting with b95ed9eae3042fc79d84e50d2dad53fd422c027dd2adc087f11968900823fa4a not found: ID does not exist"
Sep 16 18:54:55.722059 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:54:55.722039    2544 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-predictor-665c868b97-4lpk8"]
Sep 16 18:54:56.886128 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:54:56.886090    2544 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-7dc597df9b-f5gr6"]
Sep 16 18:54:56.886590 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:54:56.886416    2544 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-7dc597df9b-f5gr6" podUID="7a5f744f-a31c-4006-b6a5-b8d2b4561c53" containerName="kserve-container" containerID="cri-o://f78a9ff879832ed81559f36976e1c5b4f64d160d390bb33e73735290360f66c8" gracePeriod=30
Sep 16 18:54:56.886590 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:54:56.886457    2544 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-7dc597df9b-f5gr6" podUID="7a5f744f-a31c-4006-b6a5-b8d2b4561c53" containerName="kube-rbac-proxy" containerID="cri-o://942edcb2d4c852444220f19a464a876700f1fdab69de5905d7ba8a682f34ffb9" gracePeriod=30
Sep 16 18:54:56.959123 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:54:56.959094    2544 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-predictor-665c868b97-mb7km"]
Sep 16 18:54:56.959447 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:54:56.959435    2544 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="75b40cfe-b463-4afe-a48a-6c803b06b74d" containerName="kserve-container"
Sep 16 18:54:56.959497 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:54:56.959449    2544 state_mem.go:107] "Deleted CPUSet assignment" podUID="75b40cfe-b463-4afe-a48a-6c803b06b74d" containerName="kserve-container"
Sep 16 18:54:56.959497 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:54:56.959468    2544 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="75b40cfe-b463-4afe-a48a-6c803b06b74d" containerName="storage-initializer"
Sep 16 18:54:56.959497 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:54:56.959473    2544 state_mem.go:107] "Deleted CPUSet assignment" podUID="75b40cfe-b463-4afe-a48a-6c803b06b74d" containerName="storage-initializer"
Sep 16 18:54:56.959497 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:54:56.959483    2544 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="75b40cfe-b463-4afe-a48a-6c803b06b74d" containerName="kube-rbac-proxy"
Sep 16 18:54:56.959497 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:54:56.959488    2544 state_mem.go:107] "Deleted CPUSet assignment" podUID="75b40cfe-b463-4afe-a48a-6c803b06b74d" containerName="kube-rbac-proxy"
Sep 16 18:54:56.959706 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:54:56.959568    2544 memory_manager.go:368] "RemoveStaleState removing state" podUID="75b40cfe-b463-4afe-a48a-6c803b06b74d" containerName="kube-rbac-proxy"
Sep 16 18:54:56.959706 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:54:56.959584    2544 memory_manager.go:368] "RemoveStaleState removing state" podUID="75b40cfe-b463-4afe-a48a-6c803b06b74d" containerName="kserve-container"
Sep 16 18:54:56.964137 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:54:56.964120    2544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-665c868b97-mb7km"
Sep 16 18:54:56.966125 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:54:56.966104    2544 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-rollback-predictor-serving-cert\""
Sep 16 18:54:56.972171 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:54:56.972150    2544 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-predictor-665c868b97-mb7km"]
Sep 16 18:54:57.020520 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:54:57.020499    2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/328264d8-24da-4c7e-8920-9b9f5b9a03dc-kube-rbac-proxy-sar-config\") pod \"isvc-canary-rollback-predictor-665c868b97-mb7km\" (UID: \"328264d8-24da-4c7e-8920-9b9f5b9a03dc\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-665c868b97-mb7km"
Sep 16 18:54:57.020617 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:54:57.020531    2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/328264d8-24da-4c7e-8920-9b9f5b9a03dc-cabundle-cert\") pod \"isvc-canary-rollback-predictor-665c868b97-mb7km\" (UID: \"328264d8-24da-4c7e-8920-9b9f5b9a03dc\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-665c868b97-mb7km"
Sep 16 18:54:57.020617 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:54:57.020581    2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/328264d8-24da-4c7e-8920-9b9f5b9a03dc-proxy-tls\") pod \"isvc-canary-rollback-predictor-665c868b97-mb7km\" (UID: \"328264d8-24da-4c7e-8920-9b9f5b9a03dc\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-665c868b97-mb7km"
Sep 16 18:54:57.020706 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:54:57.020624    2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7bfnp\" (UniqueName: \"kubernetes.io/projected/328264d8-24da-4c7e-8920-9b9f5b9a03dc-kube-api-access-7bfnp\") pod \"isvc-canary-rollback-predictor-665c868b97-mb7km\" (UID: \"328264d8-24da-4c7e-8920-9b9f5b9a03dc\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-665c868b97-mb7km"
Sep 16 18:54:57.020706 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:54:57.020688    2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/328264d8-24da-4c7e-8920-9b9f5b9a03dc-kserve-provision-location\") pod \"isvc-canary-rollback-predictor-665c868b97-mb7km\" (UID: \"328264d8-24da-4c7e-8920-9b9f5b9a03dc\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-665c868b97-mb7km"
Sep 16 18:54:57.121114 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:54:57.121074    2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/328264d8-24da-4c7e-8920-9b9f5b9a03dc-kube-rbac-proxy-sar-config\") pod \"isvc-canary-rollback-predictor-665c868b97-mb7km\" (UID: \"328264d8-24da-4c7e-8920-9b9f5b9a03dc\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-665c868b97-mb7km"
Sep 16 18:54:57.121238 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:54:57.121126    2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/328264d8-24da-4c7e-8920-9b9f5b9a03dc-cabundle-cert\") pod \"isvc-canary-rollback-predictor-665c868b97-mb7km\" (UID: \"328264d8-24da-4c7e-8920-9b9f5b9a03dc\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-665c868b97-mb7km"
Sep 16 18:54:57.121238 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:54:57.121151    2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/328264d8-24da-4c7e-8920-9b9f5b9a03dc-proxy-tls\") pod \"isvc-canary-rollback-predictor-665c868b97-mb7km\" (UID: \"328264d8-24da-4c7e-8920-9b9f5b9a03dc\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-665c868b97-mb7km"
Sep 16 18:54:57.121238 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:54:57.121208    2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7bfnp\" (UniqueName: \"kubernetes.io/projected/328264d8-24da-4c7e-8920-9b9f5b9a03dc-kube-api-access-7bfnp\") pod \"isvc-canary-rollback-predictor-665c868b97-mb7km\" (UID: \"328264d8-24da-4c7e-8920-9b9f5b9a03dc\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-665c868b97-mb7km"
Sep 16 18:54:57.121367 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:54:57.121238    2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/328264d8-24da-4c7e-8920-9b9f5b9a03dc-kserve-provision-location\") pod \"isvc-canary-rollback-predictor-665c868b97-mb7km\" (UID: \"328264d8-24da-4c7e-8920-9b9f5b9a03dc\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-665c868b97-mb7km"
Sep 16 18:54:57.121677 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:54:57.121645    2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/328264d8-24da-4c7e-8920-9b9f5b9a03dc-kserve-provision-location\") pod \"isvc-canary-rollback-predictor-665c868b97-mb7km\" (UID: \"328264d8-24da-4c7e-8920-9b9f5b9a03dc\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-665c868b97-mb7km"
Sep 16 18:54:57.121898 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:54:57.121881    2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/328264d8-24da-4c7e-8920-9b9f5b9a03dc-kube-rbac-proxy-sar-config\") pod \"isvc-canary-rollback-predictor-665c868b97-mb7km\" (UID: \"328264d8-24da-4c7e-8920-9b9f5b9a03dc\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-665c868b97-mb7km"
Sep 16 18:54:57.121941 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:54:57.121884    2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/328264d8-24da-4c7e-8920-9b9f5b9a03dc-cabundle-cert\") pod \"isvc-canary-rollback-predictor-665c868b97-mb7km\" (UID: \"328264d8-24da-4c7e-8920-9b9f5b9a03dc\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-665c868b97-mb7km"
Sep 16 18:54:57.123634 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:54:57.123616    2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/328264d8-24da-4c7e-8920-9b9f5b9a03dc-proxy-tls\") pod \"isvc-canary-rollback-predictor-665c868b97-mb7km\" (UID: \"328264d8-24da-4c7e-8920-9b9f5b9a03dc\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-665c868b97-mb7km"
Sep 16 18:54:57.128463 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:54:57.128443    2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7bfnp\" (UniqueName: \"kubernetes.io/projected/328264d8-24da-4c7e-8920-9b9f5b9a03dc-kube-api-access-7bfnp\") pod \"isvc-canary-rollback-predictor-665c868b97-mb7km\" (UID: \"328264d8-24da-4c7e-8920-9b9f5b9a03dc\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-665c868b97-mb7km"
Sep 16 18:54:57.275310 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:54:57.275230    2544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-665c868b97-mb7km"
Sep 16 18:54:57.549620 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:54:57.549591    2544 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-predictor-665c868b97-mb7km"]
Sep 16 18:54:57.552092 ip-10-0-132-80 kubenswrapper[2544]: W0916 18:54:57.552059    2544 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod328264d8_24da_4c7e_8920_9b9f5b9a03dc.slice/crio-4f31adc1cfc9936f233048dd4edaf56edf46f0db79b898aff8cfc829f58f0ff8 WatchSource:0}: Error finding container 4f31adc1cfc9936f233048dd4edaf56edf46f0db79b898aff8cfc829f58f0ff8: Status 404 returned error can't find the container with id 4f31adc1cfc9936f233048dd4edaf56edf46f0db79b898aff8cfc829f58f0ff8
Sep 16 18:54:57.700365 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:54:57.700322    2544 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-665c868b97-mb7km" event={"ID":"328264d8-24da-4c7e-8920-9b9f5b9a03dc","Type":"ContainerStarted","Data":"8b9bbf937b2e4ef435ecf5fb6232e10b1c76a7dca64aa223963d56d81dfc88af"}
Sep 16 18:54:57.700501 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:54:57.700399    2544 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-665c868b97-mb7km" event={"ID":"328264d8-24da-4c7e-8920-9b9f5b9a03dc","Type":"ContainerStarted","Data":"4f31adc1cfc9936f233048dd4edaf56edf46f0db79b898aff8cfc829f58f0ff8"}
Sep 16 18:54:57.702758 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:54:57.702731    2544 generic.go:356] "Generic (PLEG): container finished" podID="7a5f744f-a31c-4006-b6a5-b8d2b4561c53" containerID="942edcb2d4c852444220f19a464a876700f1fdab69de5905d7ba8a682f34ffb9" exitCode=2
Sep 16 18:54:57.702866 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:54:57.702800    2544 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-7dc597df9b-f5gr6" event={"ID":"7a5f744f-a31c-4006-b6a5-b8d2b4561c53","Type":"ContainerDied","Data":"942edcb2d4c852444220f19a464a876700f1fdab69de5905d7ba8a682f34ffb9"}
Sep 16 18:54:57.725403 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:54:57.725371    2544 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="75b40cfe-b463-4afe-a48a-6c803b06b74d" path="/var/lib/kubelet/pods/75b40cfe-b463-4afe-a48a-6c803b06b74d/volumes"
Sep 16 18:55:00.319072 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:55:00.319025    2544 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-7dc597df9b-f5gr6" podUID="7a5f744f-a31c-4006-b6a5-b8d2b4561c53" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.25:8643/healthz\": dial tcp 10.132.0.25:8643: connect: connection refused"
Sep 16 18:55:00.324429 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:55:00.324397    2544 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-7dc597df9b-f5gr6" podUID="7a5f744f-a31c-4006-b6a5-b8d2b4561c53" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.25:8080: connect: connection refused"
Sep 16 18:55:01.723185 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:55:01.723157    2544 generic.go:356] "Generic (PLEG): container finished" podID="328264d8-24da-4c7e-8920-9b9f5b9a03dc" containerID="8b9bbf937b2e4ef435ecf5fb6232e10b1c76a7dca64aa223963d56d81dfc88af" exitCode=0
Sep 16 18:55:01.723548 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:55:01.723222    2544 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-665c868b97-mb7km" event={"ID":"328264d8-24da-4c7e-8920-9b9f5b9a03dc","Type":"ContainerDied","Data":"8b9bbf937b2e4ef435ecf5fb6232e10b1c76a7dca64aa223963d56d81dfc88af"}
Sep 16 18:55:01.725408 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:55:01.725384    2544 generic.go:356] "Generic (PLEG): container finished" podID="7a5f744f-a31c-4006-b6a5-b8d2b4561c53" containerID="f78a9ff879832ed81559f36976e1c5b4f64d160d390bb33e73735290360f66c8" exitCode=0
Sep 16 18:55:01.725491 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:55:01.725448    2544 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-7dc597df9b-f5gr6" event={"ID":"7a5f744f-a31c-4006-b6a5-b8d2b4561c53","Type":"ContainerDied","Data":"f78a9ff879832ed81559f36976e1c5b4f64d160d390bb33e73735290360f66c8"}
Sep 16 18:55:01.725532 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:55:01.725497    2544 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-7dc597df9b-f5gr6" event={"ID":"7a5f744f-a31c-4006-b6a5-b8d2b4561c53","Type":"ContainerDied","Data":"21d829cca545d1d73e9807c2f24eeede7d325d8c7d0852d0f4871cec5a1fedca"}
Sep 16 18:55:01.725532 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:55:01.725517    2544 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="21d829cca545d1d73e9807c2f24eeede7d325d8c7d0852d0f4871cec5a1fedca"
Sep 16 18:55:01.736304 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:55:01.736278    2544 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-7dc597df9b-f5gr6"
Sep 16 18:55:01.858879 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:55:01.858852    2544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/7a5f744f-a31c-4006-b6a5-b8d2b4561c53-proxy-tls\") pod \"7a5f744f-a31c-4006-b6a5-b8d2b4561c53\" (UID: \"7a5f744f-a31c-4006-b6a5-b8d2b4561c53\") "
Sep 16 18:55:01.858987 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:55:01.858915    2544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-f94kq\" (UniqueName: \"kubernetes.io/projected/7a5f744f-a31c-4006-b6a5-b8d2b4561c53-kube-api-access-f94kq\") pod \"7a5f744f-a31c-4006-b6a5-b8d2b4561c53\" (UID: \"7a5f744f-a31c-4006-b6a5-b8d2b4561c53\") "
Sep 16 18:55:01.859032 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:55:01.859003    2544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/7a5f744f-a31c-4006-b6a5-b8d2b4561c53-kserve-provision-location\") pod \"7a5f744f-a31c-4006-b6a5-b8d2b4561c53\" (UID: \"7a5f744f-a31c-4006-b6a5-b8d2b4561c53\") "
Sep 16 18:55:01.859078 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:55:01.859030    2544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/7a5f744f-a31c-4006-b6a5-b8d2b4561c53-cabundle-cert\") pod \"7a5f744f-a31c-4006-b6a5-b8d2b4561c53\" (UID: \"7a5f744f-a31c-4006-b6a5-b8d2b4561c53\") "
Sep 16 18:55:01.859078 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:55:01.859059    2544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/7a5f744f-a31c-4006-b6a5-b8d2b4561c53-kube-rbac-proxy-sar-config\") pod \"7a5f744f-a31c-4006-b6a5-b8d2b4561c53\" (UID: \"7a5f744f-a31c-4006-b6a5-b8d2b4561c53\") "
Sep 16 18:55:01.859351 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:55:01.859326    2544 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7a5f744f-a31c-4006-b6a5-b8d2b4561c53-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "7a5f744f-a31c-4006-b6a5-b8d2b4561c53" (UID: "7a5f744f-a31c-4006-b6a5-b8d2b4561c53"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 18:55:01.859411 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:55:01.859372    2544 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7a5f744f-a31c-4006-b6a5-b8d2b4561c53-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "7a5f744f-a31c-4006-b6a5-b8d2b4561c53" (UID: "7a5f744f-a31c-4006-b6a5-b8d2b4561c53"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 18:55:01.859487 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:55:01.859461    2544 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7a5f744f-a31c-4006-b6a5-b8d2b4561c53-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "7a5f744f-a31c-4006-b6a5-b8d2b4561c53" (UID: "7a5f744f-a31c-4006-b6a5-b8d2b4561c53"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 18:55:01.861007 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:55:01.860984    2544 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7a5f744f-a31c-4006-b6a5-b8d2b4561c53-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "7a5f744f-a31c-4006-b6a5-b8d2b4561c53" (UID: "7a5f744f-a31c-4006-b6a5-b8d2b4561c53"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 18:55:01.861095 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:55:01.861070    2544 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7a5f744f-a31c-4006-b6a5-b8d2b4561c53-kube-api-access-f94kq" (OuterVolumeSpecName: "kube-api-access-f94kq") pod "7a5f744f-a31c-4006-b6a5-b8d2b4561c53" (UID: "7a5f744f-a31c-4006-b6a5-b8d2b4561c53"). InnerVolumeSpecName "kube-api-access-f94kq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 18:55:01.925602 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:55:01.925506    2544 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-raw-sklearn-batcher-a31c2-predictor-5857f79896-m4b9r"]
Sep 16 18:55:01.925845 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:55:01.925791    2544 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-a31c2-predictor-5857f79896-m4b9r" podUID="4df56c4f-7740-4f22-be89-ebfee0e47da5" containerName="kserve-container" containerID="cri-o://45b5f198c276027b9c7a8cb07c48ff104c1a2d43c5153f7663eae21bccf46743" gracePeriod=30
Sep 16 18:55:01.925968 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:55:01.925836    2544 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-a31c2-predictor-5857f79896-m4b9r" podUID="4df56c4f-7740-4f22-be89-ebfee0e47da5" containerName="kube-rbac-proxy" containerID="cri-o://1fb67b0d035cf5459ddc5cb8b1a927c3d87ee28a0a9946c6f38c1e9809a8df19" gracePeriod=30
Sep 16 18:55:01.925968 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:55:01.925805    2544 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-a31c2-predictor-5857f79896-m4b9r" podUID="4df56c4f-7740-4f22-be89-ebfee0e47da5" containerName="agent" containerID="cri-o://20409c96989a0ffc8d2039b56cf1ead4cd317c9e2b5343e712b37db4f0dc906f" gracePeriod=30
Sep 16 18:55:01.960199 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:55:01.960164    2544 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f94kq\" (UniqueName: \"kubernetes.io/projected/7a5f744f-a31c-4006-b6a5-b8d2b4561c53-kube-api-access-f94kq\") on node \"ip-10-0-132-80.ec2.internal\" DevicePath \"\""
Sep 16 18:55:01.960199 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:55:01.960192    2544 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/7a5f744f-a31c-4006-b6a5-b8d2b4561c53-kserve-provision-location\") on node \"ip-10-0-132-80.ec2.internal\" DevicePath \"\""
Sep 16 18:55:01.960199 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:55:01.960203    2544 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/7a5f744f-a31c-4006-b6a5-b8d2b4561c53-cabundle-cert\") on node \"ip-10-0-132-80.ec2.internal\" DevicePath \"\""
Sep 16 18:55:01.960437 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:55:01.960211    2544 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/7a5f744f-a31c-4006-b6a5-b8d2b4561c53-kube-rbac-proxy-sar-config\") on node \"ip-10-0-132-80.ec2.internal\" DevicePath \"\""
Sep 16 18:55:01.960437 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:55:01.960221    2544 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/7a5f744f-a31c-4006-b6a5-b8d2b4561c53-proxy-tls\") on node \"ip-10-0-132-80.ec2.internal\" DevicePath \"\""
Sep 16 18:55:02.732115 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:55:02.732025    2544 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-665c868b97-mb7km" event={"ID":"328264d8-24da-4c7e-8920-9b9f5b9a03dc","Type":"ContainerStarted","Data":"1353582a247a4f35f1480b1127fe0e4a6b211e91c5e3be72560bbefc780fa849"}
Sep 16 18:55:02.732115 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:55:02.732081    2544 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-665c868b97-mb7km" event={"ID":"328264d8-24da-4c7e-8920-9b9f5b9a03dc","Type":"ContainerStarted","Data":"cf897faa339790fd4b9aa4963bea36367f7300c0b742aee61a4327d1c743da61"}
Sep 16 18:55:02.732588 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:55:02.732331    2544 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-665c868b97-mb7km"
Sep 16 18:55:02.732588 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:55:02.732495    2544 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-665c868b97-mb7km"
Sep 16 18:55:02.733621 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:55:02.733594    2544 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-665c868b97-mb7km" podUID="328264d8-24da-4c7e-8920-9b9f5b9a03dc" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.28:8080: connect: connection refused"
Sep 16 18:55:02.734733 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:55:02.734708    2544 generic.go:356] "Generic (PLEG): container finished" podID="4df56c4f-7740-4f22-be89-ebfee0e47da5" containerID="1fb67b0d035cf5459ddc5cb8b1a927c3d87ee28a0a9946c6f38c1e9809a8df19" exitCode=2
Sep 16 18:55:02.734840 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:55:02.734776    2544 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-a31c2-predictor-5857f79896-m4b9r" event={"ID":"4df56c4f-7740-4f22-be89-ebfee0e47da5","Type":"ContainerDied","Data":"1fb67b0d035cf5459ddc5cb8b1a927c3d87ee28a0a9946c6f38c1e9809a8df19"}
Sep 16 18:55:02.734840 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:55:02.734806    2544 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-7dc597df9b-f5gr6"
Sep 16 18:55:02.750576 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:55:02.750497    2544 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-665c868b97-mb7km" podStartSLOduration=2.959422908 podStartE2EDuration="6.750477821s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.791054913s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:54:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:55:02.750088392 +0000 UTC m=+665.539090951" watchObservedRunningTime="2026-09-16 18:55:02.750477821 +0000 UTC m=+665.539480358"
Sep 16 18:55:02.762487 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:55:02.762462    2544 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-7dc597df9b-f5gr6"]
Sep 16 18:55:02.766496 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:55:02.766465    2544 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-7dc597df9b-f5gr6"]
Sep 16 18:55:03.721993 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:55:03.721957    2544 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7a5f744f-a31c-4006-b6a5-b8d2b4561c53" path="/var/lib/kubelet/pods/7a5f744f-a31c-4006-b6a5-b8d2b4561c53/volumes"
Sep 16 18:55:03.738768 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:55:03.738732    2544 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-665c868b97-mb7km" podUID="328264d8-24da-4c7e-8920-9b9f5b9a03dc" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.28:8080: connect: connection refused"
Sep 16 18:55:04.339974 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:55:04.339925    2544 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-a31c2-predictor-5857f79896-m4b9r" podUID="4df56c4f-7740-4f22-be89-ebfee0e47da5" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.27:8643/healthz\": dial tcp 10.132.0.27:8643: connect: connection refused"
Sep 16 18:55:06.752783 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:55:06.752750    2544 generic.go:356] "Generic (PLEG): container finished" podID="4df56c4f-7740-4f22-be89-ebfee0e47da5" containerID="45b5f198c276027b9c7a8cb07c48ff104c1a2d43c5153f7663eae21bccf46743" exitCode=0
Sep 16 18:55:06.753154 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:55:06.752822    2544 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-a31c2-predictor-5857f79896-m4b9r" event={"ID":"4df56c4f-7740-4f22-be89-ebfee0e47da5","Type":"ContainerDied","Data":"45b5f198c276027b9c7a8cb07c48ff104c1a2d43c5153f7663eae21bccf46743"}
Sep 16 18:55:08.743047 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:55:08.743018    2544 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-665c868b97-mb7km"
Sep 16 18:55:08.743621 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:55:08.743592    2544 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-665c868b97-mb7km" podUID="328264d8-24da-4c7e-8920-9b9f5b9a03dc" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.28:8080: connect: connection refused"
Sep 16 18:55:09.339768 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:55:09.339725    2544 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-a31c2-predictor-5857f79896-m4b9r" podUID="4df56c4f-7740-4f22-be89-ebfee0e47da5" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.27:8643/healthz\": dial tcp 10.132.0.27:8643: connect: connection refused"
Sep 16 18:55:10.347912 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:55:10.347856    2544 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-a31c2-predictor-5857f79896-m4b9r" podUID="4df56c4f-7740-4f22-be89-ebfee0e47da5" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.27:8080: connect: connection refused"
Sep 16 18:55:10.348310 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:55:10.348178    2544 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-a31c2-predictor-5857f79896-m4b9r" podUID="4df56c4f-7740-4f22-be89-ebfee0e47da5" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 16 18:55:14.339512 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:55:14.339468    2544 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-a31c2-predictor-5857f79896-m4b9r" podUID="4df56c4f-7740-4f22-be89-ebfee0e47da5" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.27:8643/healthz\": dial tcp 10.132.0.27:8643: connect: connection refused"
Sep 16 18:55:14.339907 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:55:14.339651    2544 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-a31c2-predictor-5857f79896-m4b9r"
Sep 16 18:55:18.744076 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:55:18.744034    2544 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-665c868b97-mb7km" podUID="328264d8-24da-4c7e-8920-9b9f5b9a03dc" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.28:8080: connect: connection refused"
Sep 16 18:55:19.340148 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:55:19.340099    2544 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-a31c2-predictor-5857f79896-m4b9r" podUID="4df56c4f-7740-4f22-be89-ebfee0e47da5" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.27:8643/healthz\": dial tcp 10.132.0.27:8643: connect: connection refused"
Sep 16 18:55:20.347866 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:55:20.347821    2544 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-a31c2-predictor-5857f79896-m4b9r" podUID="4df56c4f-7740-4f22-be89-ebfee0e47da5" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.27:8080: connect: connection refused"
Sep 16 18:55:20.348275 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:55:20.348137    2544 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-a31c2-predictor-5857f79896-m4b9r" podUID="4df56c4f-7740-4f22-be89-ebfee0e47da5" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 16 18:55:24.339857 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:55:24.339813    2544 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-a31c2-predictor-5857f79896-m4b9r" podUID="4df56c4f-7740-4f22-be89-ebfee0e47da5" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.27:8643/healthz\": dial tcp 10.132.0.27:8643: connect: connection refused"
Sep 16 18:55:28.743833 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:55:28.743789    2544 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-665c868b97-mb7km" podUID="328264d8-24da-4c7e-8920-9b9f5b9a03dc" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.28:8080: connect: connection refused"
Sep 16 18:55:29.339206 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:55:29.339172    2544 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-a31c2-predictor-5857f79896-m4b9r" podUID="4df56c4f-7740-4f22-be89-ebfee0e47da5" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.27:8643/healthz\": dial tcp 10.132.0.27:8643: connect: connection refused"
Sep 16 18:55:30.347114 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:55:30.347071    2544 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-a31c2-predictor-5857f79896-m4b9r" podUID="4df56c4f-7740-4f22-be89-ebfee0e47da5" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.27:8080: connect: connection refused"
Sep 16 18:55:30.347609 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:55:30.347262    2544 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-a31c2-predictor-5857f79896-m4b9r"
Sep 16 18:55:30.347609 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:55:30.347349    2544 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-a31c2-predictor-5857f79896-m4b9r" podUID="4df56c4f-7740-4f22-be89-ebfee0e47da5" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 16 18:55:30.347609 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:55:30.347440    2544 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-a31c2-predictor-5857f79896-m4b9r"
Sep 16 18:55:32.066985 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:55:32.066961    2544 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-a31c2-predictor-5857f79896-m4b9r"
Sep 16 18:55:32.120169 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:55:32.120134    2544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7cc57\" (UniqueName: \"kubernetes.io/projected/4df56c4f-7740-4f22-be89-ebfee0e47da5-kube-api-access-7cc57\") pod \"4df56c4f-7740-4f22-be89-ebfee0e47da5\" (UID: \"4df56c4f-7740-4f22-be89-ebfee0e47da5\") "
Sep 16 18:55:32.120169 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:55:32.120175    2544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/4df56c4f-7740-4f22-be89-ebfee0e47da5-proxy-tls\") pod \"4df56c4f-7740-4f22-be89-ebfee0e47da5\" (UID: \"4df56c4f-7740-4f22-be89-ebfee0e47da5\") "
Sep 16 18:55:32.120407 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:55:32.120220    2544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/4df56c4f-7740-4f22-be89-ebfee0e47da5-kube-rbac-proxy-sar-config\") pod \"4df56c4f-7740-4f22-be89-ebfee0e47da5\" (UID: \"4df56c4f-7740-4f22-be89-ebfee0e47da5\") "
Sep 16 18:55:32.120407 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:55:32.120252    2544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/4df56c4f-7740-4f22-be89-ebfee0e47da5-cabundle-cert\") pod \"4df56c4f-7740-4f22-be89-ebfee0e47da5\" (UID: \"4df56c4f-7740-4f22-be89-ebfee0e47da5\") "
Sep 16 18:55:32.120407 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:55:32.120321    2544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/4df56c4f-7740-4f22-be89-ebfee0e47da5-kserve-provision-location\") pod \"4df56c4f-7740-4f22-be89-ebfee0e47da5\" (UID: \"4df56c4f-7740-4f22-be89-ebfee0e47da5\") "
Sep 16 18:55:32.120699 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:55:32.120670    2544 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4df56c4f-7740-4f22-be89-ebfee0e47da5-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "4df56c4f-7740-4f22-be89-ebfee0e47da5" (UID: "4df56c4f-7740-4f22-be89-ebfee0e47da5"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 18:55:32.120804 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:55:32.120699    2544 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4df56c4f-7740-4f22-be89-ebfee0e47da5-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "4df56c4f-7740-4f22-be89-ebfee0e47da5" (UID: "4df56c4f-7740-4f22-be89-ebfee0e47da5"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 18:55:32.120804 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:55:32.120727    2544 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4df56c4f-7740-4f22-be89-ebfee0e47da5-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "4df56c4f-7740-4f22-be89-ebfee0e47da5" (UID: "4df56c4f-7740-4f22-be89-ebfee0e47da5"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 18:55:32.122358 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:55:32.122338    2544 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4df56c4f-7740-4f22-be89-ebfee0e47da5-kube-api-access-7cc57" (OuterVolumeSpecName: "kube-api-access-7cc57") pod "4df56c4f-7740-4f22-be89-ebfee0e47da5" (UID: "4df56c4f-7740-4f22-be89-ebfee0e47da5"). InnerVolumeSpecName "kube-api-access-7cc57". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 18:55:32.122457 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:55:32.122389    2544 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4df56c4f-7740-4f22-be89-ebfee0e47da5-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "4df56c4f-7740-4f22-be89-ebfee0e47da5" (UID: "4df56c4f-7740-4f22-be89-ebfee0e47da5"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 18:55:32.221013 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:55:32.220938    2544 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/4df56c4f-7740-4f22-be89-ebfee0e47da5-proxy-tls\") on node \"ip-10-0-132-80.ec2.internal\" DevicePath \"\""
Sep 16 18:55:32.221013 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:55:32.220968    2544 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/4df56c4f-7740-4f22-be89-ebfee0e47da5-kube-rbac-proxy-sar-config\") on node \"ip-10-0-132-80.ec2.internal\" DevicePath \"\""
Sep 16 18:55:32.221013 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:55:32.220978    2544 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/4df56c4f-7740-4f22-be89-ebfee0e47da5-cabundle-cert\") on node \"ip-10-0-132-80.ec2.internal\" DevicePath \"\""
Sep 16 18:55:32.221013 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:55:32.220987    2544 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/4df56c4f-7740-4f22-be89-ebfee0e47da5-kserve-provision-location\") on node \"ip-10-0-132-80.ec2.internal\" DevicePath \"\""
Sep 16 18:55:32.221013 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:55:32.220996    2544 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7cc57\" (UniqueName: \"kubernetes.io/projected/4df56c4f-7740-4f22-be89-ebfee0e47da5-kube-api-access-7cc57\") on node \"ip-10-0-132-80.ec2.internal\" DevicePath \"\""
Sep 16 18:55:32.865334 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:55:32.865303    2544 generic.go:356] "Generic (PLEG): container finished" podID="4df56c4f-7740-4f22-be89-ebfee0e47da5" containerID="20409c96989a0ffc8d2039b56cf1ead4cd317c9e2b5343e712b37db4f0dc906f" exitCode=0
Sep 16 18:55:32.865503 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:55:32.865361    2544 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-a31c2-predictor-5857f79896-m4b9r"
Sep 16 18:55:32.865503 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:55:32.865365    2544 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-a31c2-predictor-5857f79896-m4b9r" event={"ID":"4df56c4f-7740-4f22-be89-ebfee0e47da5","Type":"ContainerDied","Data":"20409c96989a0ffc8d2039b56cf1ead4cd317c9e2b5343e712b37db4f0dc906f"}
Sep 16 18:55:32.865644 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:55:32.865512    2544 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-a31c2-predictor-5857f79896-m4b9r" event={"ID":"4df56c4f-7740-4f22-be89-ebfee0e47da5","Type":"ContainerDied","Data":"1bb9066c0d571a414a842760aa9ca7841ccf5e4e442a20fd35ca58ef2808c0a8"}
Sep 16 18:55:32.865644 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:55:32.865576    2544 scope.go:117] "RemoveContainer" containerID="20409c96989a0ffc8d2039b56cf1ead4cd317c9e2b5343e712b37db4f0dc906f"
Sep 16 18:55:32.874314 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:55:32.874298    2544 scope.go:117] "RemoveContainer" containerID="1fb67b0d035cf5459ddc5cb8b1a927c3d87ee28a0a9946c6f38c1e9809a8df19"
Sep 16 18:55:32.881509 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:55:32.881490    2544 scope.go:117] "RemoveContainer" containerID="45b5f198c276027b9c7a8cb07c48ff104c1a2d43c5153f7663eae21bccf46743"
Sep 16 18:55:32.887107 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:55:32.887080    2544 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-raw-sklearn-batcher-a31c2-predictor-5857f79896-m4b9r"]
Sep 16 18:55:32.888941 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:55:32.888926    2544 scope.go:117] "RemoveContainer" containerID="fa50551d58591fb01f9a73afe19ab9c6cace7def59a0f9488959d35e0f2a9e1d"
Sep 16 18:55:32.891709 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:55:32.891691    2544 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-raw-sklearn-batcher-a31c2-predictor-5857f79896-m4b9r"]
Sep 16 18:55:32.897242 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:55:32.897223    2544 scope.go:117] "RemoveContainer" containerID="20409c96989a0ffc8d2039b56cf1ead4cd317c9e2b5343e712b37db4f0dc906f"
Sep 16 18:55:32.897502 ip-10-0-132-80 kubenswrapper[2544]: E0916 18:55:32.897485    2544 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"20409c96989a0ffc8d2039b56cf1ead4cd317c9e2b5343e712b37db4f0dc906f\": container with ID starting with 20409c96989a0ffc8d2039b56cf1ead4cd317c9e2b5343e712b37db4f0dc906f not found: ID does not exist" containerID="20409c96989a0ffc8d2039b56cf1ead4cd317c9e2b5343e712b37db4f0dc906f"
Sep 16 18:55:32.897565 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:55:32.897509    2544 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"20409c96989a0ffc8d2039b56cf1ead4cd317c9e2b5343e712b37db4f0dc906f"} err="failed to get container status \"20409c96989a0ffc8d2039b56cf1ead4cd317c9e2b5343e712b37db4f0dc906f\": rpc error: code = NotFound desc = could not find container \"20409c96989a0ffc8d2039b56cf1ead4cd317c9e2b5343e712b37db4f0dc906f\": container with ID starting with 20409c96989a0ffc8d2039b56cf1ead4cd317c9e2b5343e712b37db4f0dc906f not found: ID does not exist"
Sep 16 18:55:32.897565 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:55:32.897526    2544 scope.go:117] "RemoveContainer" containerID="1fb67b0d035cf5459ddc5cb8b1a927c3d87ee28a0a9946c6f38c1e9809a8df19"
Sep 16 18:55:32.897800 ip-10-0-132-80 kubenswrapper[2544]: E0916 18:55:32.897783    2544 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1fb67b0d035cf5459ddc5cb8b1a927c3d87ee28a0a9946c6f38c1e9809a8df19\": container with ID starting with 1fb67b0d035cf5459ddc5cb8b1a927c3d87ee28a0a9946c6f38c1e9809a8df19 not found: ID does not exist" containerID="1fb67b0d035cf5459ddc5cb8b1a927c3d87ee28a0a9946c6f38c1e9809a8df19"
Sep 16 18:55:32.897856 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:55:32.897803    2544 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1fb67b0d035cf5459ddc5cb8b1a927c3d87ee28a0a9946c6f38c1e9809a8df19"} err="failed to get container status \"1fb67b0d035cf5459ddc5cb8b1a927c3d87ee28a0a9946c6f38c1e9809a8df19\": rpc error: code = NotFound desc = could not find container \"1fb67b0d035cf5459ddc5cb8b1a927c3d87ee28a0a9946c6f38c1e9809a8df19\": container with ID starting with 1fb67b0d035cf5459ddc5cb8b1a927c3d87ee28a0a9946c6f38c1e9809a8df19 not found: ID does not exist"
Sep 16 18:55:32.897856 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:55:32.897817    2544 scope.go:117] "RemoveContainer" containerID="45b5f198c276027b9c7a8cb07c48ff104c1a2d43c5153f7663eae21bccf46743"
Sep 16 18:55:32.898066 ip-10-0-132-80 kubenswrapper[2544]: E0916 18:55:32.898047    2544 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"45b5f198c276027b9c7a8cb07c48ff104c1a2d43c5153f7663eae21bccf46743\": container with ID starting with 45b5f198c276027b9c7a8cb07c48ff104c1a2d43c5153f7663eae21bccf46743 not found: ID does not exist" containerID="45b5f198c276027b9c7a8cb07c48ff104c1a2d43c5153f7663eae21bccf46743"
Sep 16 18:55:32.898107 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:55:32.898072    2544 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"45b5f198c276027b9c7a8cb07c48ff104c1a2d43c5153f7663eae21bccf46743"} err="failed to get container status \"45b5f198c276027b9c7a8cb07c48ff104c1a2d43c5153f7663eae21bccf46743\": rpc error: code = NotFound desc = could not find container \"45b5f198c276027b9c7a8cb07c48ff104c1a2d43c5153f7663eae21bccf46743\": container with ID starting with 45b5f198c276027b9c7a8cb07c48ff104c1a2d43c5153f7663eae21bccf46743 not found: ID does not exist"
Sep 16 18:55:32.898107 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:55:32.898091    2544 scope.go:117] "RemoveContainer" containerID="fa50551d58591fb01f9a73afe19ab9c6cace7def59a0f9488959d35e0f2a9e1d"
Sep 16 18:55:32.898355 ip-10-0-132-80 kubenswrapper[2544]: E0916 18:55:32.898333    2544 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fa50551d58591fb01f9a73afe19ab9c6cace7def59a0f9488959d35e0f2a9e1d\": container with ID starting with fa50551d58591fb01f9a73afe19ab9c6cace7def59a0f9488959d35e0f2a9e1d not found: ID does not exist" containerID="fa50551d58591fb01f9a73afe19ab9c6cace7def59a0f9488959d35e0f2a9e1d"
Sep 16 18:55:32.898417 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:55:32.898359    2544 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fa50551d58591fb01f9a73afe19ab9c6cace7def59a0f9488959d35e0f2a9e1d"} err="failed to get container status \"fa50551d58591fb01f9a73afe19ab9c6cace7def59a0f9488959d35e0f2a9e1d\": rpc error: code = NotFound desc = could not find container \"fa50551d58591fb01f9a73afe19ab9c6cace7def59a0f9488959d35e0f2a9e1d\": container with ID starting with fa50551d58591fb01f9a73afe19ab9c6cace7def59a0f9488959d35e0f2a9e1d not found: ID does not exist"
Sep 16 18:55:33.722531 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:55:33.722500    2544 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4df56c4f-7740-4f22-be89-ebfee0e47da5" path="/var/lib/kubelet/pods/4df56c4f-7740-4f22-be89-ebfee0e47da5/volumes"
Sep 16 18:55:38.744045 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:55:38.744003    2544 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-665c868b97-mb7km" podUID="328264d8-24da-4c7e-8920-9b9f5b9a03dc" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.28:8080: connect: connection refused"
Sep 16 18:55:48.744014 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:55:48.743965    2544 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-665c868b97-mb7km" podUID="328264d8-24da-4c7e-8920-9b9f5b9a03dc" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.28:8080: connect: connection refused"
Sep 16 18:55:58.743672 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:55:58.743632    2544 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-665c868b97-mb7km" podUID="328264d8-24da-4c7e-8920-9b9f5b9a03dc" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.28:8080: connect: connection refused"
Sep 16 18:56:08.743503 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:56:08.743460    2544 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-665c868b97-mb7km" podUID="328264d8-24da-4c7e-8920-9b9f5b9a03dc" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.28:8080: connect: connection refused"
Sep 16 18:56:18.744722 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:56:18.744689    2544 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-665c868b97-mb7km"
Sep 16 18:56:27.002797 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:56:27.002758    2544 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-predictor-665c868b97-mb7km"]
Sep 16 18:56:27.003304 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:56:27.003041    2544 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-665c868b97-mb7km" podUID="328264d8-24da-4c7e-8920-9b9f5b9a03dc" containerName="kserve-container" containerID="cri-o://cf897faa339790fd4b9aa4963bea36367f7300c0b742aee61a4327d1c743da61" gracePeriod=30
Sep 16 18:56:27.003304 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:56:27.003051    2544 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-665c868b97-mb7km" podUID="328264d8-24da-4c7e-8920-9b9f5b9a03dc" containerName="kube-rbac-proxy" containerID="cri-o://1353582a247a4f35f1480b1127fe0e4a6b211e91c5e3be72560bbefc780fa849" gracePeriod=30
Sep 16 18:56:28.109617 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:56:28.109584    2544 generic.go:356] "Generic (PLEG): container finished" podID="328264d8-24da-4c7e-8920-9b9f5b9a03dc" containerID="1353582a247a4f35f1480b1127fe0e4a6b211e91c5e3be72560bbefc780fa849" exitCode=2
Sep 16 18:56:28.110004 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:56:28.109659    2544 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-665c868b97-mb7km" event={"ID":"328264d8-24da-4c7e-8920-9b9f5b9a03dc","Type":"ContainerDied","Data":"1353582a247a4f35f1480b1127fe0e4a6b211e91c5e3be72560bbefc780fa849"}
Sep 16 18:56:28.739254 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:56:28.739209    2544 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-665c868b97-mb7km" podUID="328264d8-24da-4c7e-8920-9b9f5b9a03dc" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.28:8643/healthz\": dial tcp 10.132.0.28:8643: connect: connection refused"
Sep 16 18:56:28.744044 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:56:28.744015    2544 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-665c868b97-mb7km" podUID="328264d8-24da-4c7e-8920-9b9f5b9a03dc" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.28:8080: connect: connection refused"
Sep 16 18:56:31.443008 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:56:31.442986    2544 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-665c868b97-mb7km"
Sep 16 18:56:31.513183 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:56:31.513149    2544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/328264d8-24da-4c7e-8920-9b9f5b9a03dc-cabundle-cert\") pod \"328264d8-24da-4c7e-8920-9b9f5b9a03dc\" (UID: \"328264d8-24da-4c7e-8920-9b9f5b9a03dc\") "
Sep 16 18:56:31.513331 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:56:31.513223    2544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/328264d8-24da-4c7e-8920-9b9f5b9a03dc-kube-rbac-proxy-sar-config\") pod \"328264d8-24da-4c7e-8920-9b9f5b9a03dc\" (UID: \"328264d8-24da-4c7e-8920-9b9f5b9a03dc\") "
Sep 16 18:56:31.513331 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:56:31.513241    2544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/328264d8-24da-4c7e-8920-9b9f5b9a03dc-kserve-provision-location\") pod \"328264d8-24da-4c7e-8920-9b9f5b9a03dc\" (UID: \"328264d8-24da-4c7e-8920-9b9f5b9a03dc\") "
Sep 16 18:56:31.513331 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:56:31.513321    2544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/328264d8-24da-4c7e-8920-9b9f5b9a03dc-proxy-tls\") pod \"328264d8-24da-4c7e-8920-9b9f5b9a03dc\" (UID: \"328264d8-24da-4c7e-8920-9b9f5b9a03dc\") "
Sep 16 18:56:31.513511 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:56:31.513360    2544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7bfnp\" (UniqueName: \"kubernetes.io/projected/328264d8-24da-4c7e-8920-9b9f5b9a03dc-kube-api-access-7bfnp\") pod \"328264d8-24da-4c7e-8920-9b9f5b9a03dc\" (UID: \"328264d8-24da-4c7e-8920-9b9f5b9a03dc\") "
Sep 16 18:56:31.513586 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:56:31.513533    2544 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/328264d8-24da-4c7e-8920-9b9f5b9a03dc-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "328264d8-24da-4c7e-8920-9b9f5b9a03dc" (UID: "328264d8-24da-4c7e-8920-9b9f5b9a03dc"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 18:56:31.513643 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:56:31.513584    2544 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/328264d8-24da-4c7e-8920-9b9f5b9a03dc-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "328264d8-24da-4c7e-8920-9b9f5b9a03dc" (UID: "328264d8-24da-4c7e-8920-9b9f5b9a03dc"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 18:56:31.513643 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:56:31.513633    2544 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/328264d8-24da-4c7e-8920-9b9f5b9a03dc-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "328264d8-24da-4c7e-8920-9b9f5b9a03dc" (UID: "328264d8-24da-4c7e-8920-9b9f5b9a03dc"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 18:56:31.513735 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:56:31.513659    2544 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/328264d8-24da-4c7e-8920-9b9f5b9a03dc-kserve-provision-location\") on node \"ip-10-0-132-80.ec2.internal\" DevicePath \"\""
Sep 16 18:56:31.513735 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:56:31.513679    2544 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/328264d8-24da-4c7e-8920-9b9f5b9a03dc-cabundle-cert\") on node \"ip-10-0-132-80.ec2.internal\" DevicePath \"\""
Sep 16 18:56:31.515333 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:56:31.515311    2544 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/328264d8-24da-4c7e-8920-9b9f5b9a03dc-kube-api-access-7bfnp" (OuterVolumeSpecName: "kube-api-access-7bfnp") pod "328264d8-24da-4c7e-8920-9b9f5b9a03dc" (UID: "328264d8-24da-4c7e-8920-9b9f5b9a03dc"). InnerVolumeSpecName "kube-api-access-7bfnp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 18:56:31.515333 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:56:31.515323    2544 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/328264d8-24da-4c7e-8920-9b9f5b9a03dc-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "328264d8-24da-4c7e-8920-9b9f5b9a03dc" (UID: "328264d8-24da-4c7e-8920-9b9f5b9a03dc"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 18:56:31.614009 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:56:31.613987    2544 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/328264d8-24da-4c7e-8920-9b9f5b9a03dc-kube-rbac-proxy-sar-config\") on node \"ip-10-0-132-80.ec2.internal\" DevicePath \"\""
Sep 16 18:56:31.614009 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:56:31.614007    2544 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/328264d8-24da-4c7e-8920-9b9f5b9a03dc-proxy-tls\") on node \"ip-10-0-132-80.ec2.internal\" DevicePath \"\""
Sep 16 18:56:31.614140 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:56:31.614018    2544 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7bfnp\" (UniqueName: \"kubernetes.io/projected/328264d8-24da-4c7e-8920-9b9f5b9a03dc-kube-api-access-7bfnp\") on node \"ip-10-0-132-80.ec2.internal\" DevicePath \"\""
Sep 16 18:56:32.129161 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:56:32.129127    2544 generic.go:356] "Generic (PLEG): container finished" podID="328264d8-24da-4c7e-8920-9b9f5b9a03dc" containerID="cf897faa339790fd4b9aa4963bea36367f7300c0b742aee61a4327d1c743da61" exitCode=0
Sep 16 18:56:32.129345 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:56:32.129166    2544 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-665c868b97-mb7km" event={"ID":"328264d8-24da-4c7e-8920-9b9f5b9a03dc","Type":"ContainerDied","Data":"cf897faa339790fd4b9aa4963bea36367f7300c0b742aee61a4327d1c743da61"}
Sep 16 18:56:32.129345 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:56:32.129204    2544 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-665c868b97-mb7km"
Sep 16 18:56:32.129345 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:56:32.129222    2544 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-665c868b97-mb7km" event={"ID":"328264d8-24da-4c7e-8920-9b9f5b9a03dc","Type":"ContainerDied","Data":"4f31adc1cfc9936f233048dd4edaf56edf46f0db79b898aff8cfc829f58f0ff8"}
Sep 16 18:56:32.129345 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:56:32.129251    2544 scope.go:117] "RemoveContainer" containerID="1353582a247a4f35f1480b1127fe0e4a6b211e91c5e3be72560bbefc780fa849"
Sep 16 18:56:32.138740 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:56:32.138720    2544 scope.go:117] "RemoveContainer" containerID="cf897faa339790fd4b9aa4963bea36367f7300c0b742aee61a4327d1c743da61"
Sep 16 18:56:32.146062 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:56:32.146043    2544 scope.go:117] "RemoveContainer" containerID="8b9bbf937b2e4ef435ecf5fb6232e10b1c76a7dca64aa223963d56d81dfc88af"
Sep 16 18:56:32.152319 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:56:32.152296    2544 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-predictor-665c868b97-mb7km"]
Sep 16 18:56:32.155362 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:56:32.155345    2544 scope.go:117] "RemoveContainer" containerID="1353582a247a4f35f1480b1127fe0e4a6b211e91c5e3be72560bbefc780fa849"
Sep 16 18:56:32.155669 ip-10-0-132-80 kubenswrapper[2544]: E0916 18:56:32.155649    2544 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1353582a247a4f35f1480b1127fe0e4a6b211e91c5e3be72560bbefc780fa849\": container with ID starting with 1353582a247a4f35f1480b1127fe0e4a6b211e91c5e3be72560bbefc780fa849 not found: ID does not exist" containerID="1353582a247a4f35f1480b1127fe0e4a6b211e91c5e3be72560bbefc780fa849"
Sep 16 18:56:32.155772 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:56:32.155675    2544 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1353582a247a4f35f1480b1127fe0e4a6b211e91c5e3be72560bbefc780fa849"} err="failed to get container status \"1353582a247a4f35f1480b1127fe0e4a6b211e91c5e3be72560bbefc780fa849\": rpc error: code = NotFound desc = could not find container \"1353582a247a4f35f1480b1127fe0e4a6b211e91c5e3be72560bbefc780fa849\": container with ID starting with 1353582a247a4f35f1480b1127fe0e4a6b211e91c5e3be72560bbefc780fa849 not found: ID does not exist"
Sep 16 18:56:32.155772 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:56:32.155692    2544 scope.go:117] "RemoveContainer" containerID="cf897faa339790fd4b9aa4963bea36367f7300c0b742aee61a4327d1c743da61"
Sep 16 18:56:32.155990 ip-10-0-132-80 kubenswrapper[2544]: E0916 18:56:32.155972    2544 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cf897faa339790fd4b9aa4963bea36367f7300c0b742aee61a4327d1c743da61\": container with ID starting with cf897faa339790fd4b9aa4963bea36367f7300c0b742aee61a4327d1c743da61 not found: ID does not exist" containerID="cf897faa339790fd4b9aa4963bea36367f7300c0b742aee61a4327d1c743da61"
Sep 16 18:56:32.156047 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:56:32.155995    2544 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cf897faa339790fd4b9aa4963bea36367f7300c0b742aee61a4327d1c743da61"} err="failed to get container status \"cf897faa339790fd4b9aa4963bea36367f7300c0b742aee61a4327d1c743da61\": rpc error: code = NotFound desc = could not find container \"cf897faa339790fd4b9aa4963bea36367f7300c0b742aee61a4327d1c743da61\": container with ID starting with cf897faa339790fd4b9aa4963bea36367f7300c0b742aee61a4327d1c743da61 not found: ID does not exist"
Sep 16 18:56:32.156047 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:56:32.156017    2544 scope.go:117] "RemoveContainer" containerID="8b9bbf937b2e4ef435ecf5fb6232e10b1c76a7dca64aa223963d56d81dfc88af"
Sep 16 18:56:32.156245 ip-10-0-132-80 kubenswrapper[2544]: E0916 18:56:32.156230    2544 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8b9bbf937b2e4ef435ecf5fb6232e10b1c76a7dca64aa223963d56d81dfc88af\": container with ID starting with 8b9bbf937b2e4ef435ecf5fb6232e10b1c76a7dca64aa223963d56d81dfc88af not found: ID does not exist" containerID="8b9bbf937b2e4ef435ecf5fb6232e10b1c76a7dca64aa223963d56d81dfc88af"
Sep 16 18:56:32.156292 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:56:32.156250    2544 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8b9bbf937b2e4ef435ecf5fb6232e10b1c76a7dca64aa223963d56d81dfc88af"} err="failed to get container status \"8b9bbf937b2e4ef435ecf5fb6232e10b1c76a7dca64aa223963d56d81dfc88af\": rpc error: code = NotFound desc = could not find container \"8b9bbf937b2e4ef435ecf5fb6232e10b1c76a7dca64aa223963d56d81dfc88af\": container with ID starting with 8b9bbf937b2e4ef435ecf5fb6232e10b1c76a7dca64aa223963d56d81dfc88af not found: ID does not exist"
Sep 16 18:56:32.156487 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:56:32.156461    2544 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-predictor-665c868b97-mb7km"]
Sep 16 18:56:32.216413 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:56:32.216386    2544 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-b7c9ff648-nvcth"]
Sep 16 18:56:32.216725 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:56:32.216713    2544 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="328264d8-24da-4c7e-8920-9b9f5b9a03dc" containerName="storage-initializer"
Sep 16 18:56:32.216782 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:56:32.216727    2544 state_mem.go:107] "Deleted CPUSet assignment" podUID="328264d8-24da-4c7e-8920-9b9f5b9a03dc" containerName="storage-initializer"
Sep 16 18:56:32.216782 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:56:32.216736    2544 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4df56c4f-7740-4f22-be89-ebfee0e47da5" containerName="kserve-container"
Sep 16 18:56:32.216782 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:56:32.216741    2544 state_mem.go:107] "Deleted CPUSet assignment" podUID="4df56c4f-7740-4f22-be89-ebfee0e47da5" containerName="kserve-container"
Sep 16 18:56:32.216782 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:56:32.216753    2544 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7a5f744f-a31c-4006-b6a5-b8d2b4561c53" containerName="kserve-container"
Sep 16 18:56:32.216782 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:56:32.216759    2544 state_mem.go:107] "Deleted CPUSet assignment" podUID="7a5f744f-a31c-4006-b6a5-b8d2b4561c53" containerName="kserve-container"
Sep 16 18:56:32.216782 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:56:32.216767    2544 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4df56c4f-7740-4f22-be89-ebfee0e47da5" containerName="kube-rbac-proxy"
Sep 16 18:56:32.216782 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:56:32.216772    2544 state_mem.go:107] "Deleted CPUSet assignment" podUID="4df56c4f-7740-4f22-be89-ebfee0e47da5" containerName="kube-rbac-proxy"
Sep 16 18:56:32.216782 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:56:32.216780    2544 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4df56c4f-7740-4f22-be89-ebfee0e47da5" containerName="agent"
Sep 16 18:56:32.216782 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:56:32.216785    2544 state_mem.go:107] "Deleted CPUSet assignment" podUID="4df56c4f-7740-4f22-be89-ebfee0e47da5" containerName="agent"
Sep 16 18:56:32.217146 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:56:32.216792    2544 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7a5f744f-a31c-4006-b6a5-b8d2b4561c53" containerName="storage-initializer"
Sep 16 18:56:32.217146 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:56:32.216797    2544 state_mem.go:107] "Deleted CPUSet assignment" podUID="7a5f744f-a31c-4006-b6a5-b8d2b4561c53" containerName="storage-initializer"
Sep 16 18:56:32.217146 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:56:32.216812    2544 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="328264d8-24da-4c7e-8920-9b9f5b9a03dc" containerName="kube-rbac-proxy"
Sep 16 18:56:32.217146 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:56:32.216818    2544 state_mem.go:107] "Deleted CPUSet assignment" podUID="328264d8-24da-4c7e-8920-9b9f5b9a03dc" containerName="kube-rbac-proxy"
Sep 16 18:56:32.217146 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:56:32.216825    2544 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4df56c4f-7740-4f22-be89-ebfee0e47da5" containerName="storage-initializer"
Sep 16 18:56:32.217146 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:56:32.216830    2544 state_mem.go:107] "Deleted CPUSet assignment" podUID="4df56c4f-7740-4f22-be89-ebfee0e47da5" containerName="storage-initializer"
Sep 16 18:56:32.217146 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:56:32.216836    2544 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="328264d8-24da-4c7e-8920-9b9f5b9a03dc" containerName="kserve-container"
Sep 16 18:56:32.217146 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:56:32.216841    2544 state_mem.go:107] "Deleted CPUSet assignment" podUID="328264d8-24da-4c7e-8920-9b9f5b9a03dc" containerName="kserve-container"
Sep 16 18:56:32.217146 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:56:32.216846    2544 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7a5f744f-a31c-4006-b6a5-b8d2b4561c53" containerName="kube-rbac-proxy"
Sep 16 18:56:32.217146 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:56:32.216851    2544 state_mem.go:107] "Deleted CPUSet assignment" podUID="7a5f744f-a31c-4006-b6a5-b8d2b4561c53" containerName="kube-rbac-proxy"
Sep 16 18:56:32.217146 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:56:32.216901    2544 memory_manager.go:368] "RemoveStaleState removing state" podUID="4df56c4f-7740-4f22-be89-ebfee0e47da5" containerName="kube-rbac-proxy"
Sep 16 18:56:32.217146 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:56:32.216909    2544 memory_manager.go:368] "RemoveStaleState removing state" podUID="4df56c4f-7740-4f22-be89-ebfee0e47da5" containerName="kserve-container"
Sep 16 18:56:32.217146 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:56:32.216916    2544 memory_manager.go:368] "RemoveStaleState removing state" podUID="328264d8-24da-4c7e-8920-9b9f5b9a03dc" containerName="kserve-container"
Sep 16 18:56:32.217146 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:56:32.216921    2544 memory_manager.go:368] "RemoveStaleState removing state" podUID="7a5f744f-a31c-4006-b6a5-b8d2b4561c53" containerName="kserve-container"
Sep 16 18:56:32.217146 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:56:32.216926    2544 memory_manager.go:368] "RemoveStaleState removing state" podUID="328264d8-24da-4c7e-8920-9b9f5b9a03dc" containerName="kube-rbac-proxy"
Sep 16 18:56:32.217146 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:56:32.216932    2544 memory_manager.go:368] "RemoveStaleState removing state" podUID="4df56c4f-7740-4f22-be89-ebfee0e47da5" containerName="agent"
Sep 16 18:56:32.217146 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:56:32.216938    2544 memory_manager.go:368] "RemoveStaleState removing state" podUID="7a5f744f-a31c-4006-b6a5-b8d2b4561c53" containerName="kube-rbac-proxy"
Sep 16 18:56:32.221526 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:56:32.221509    2544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-b7c9ff648-nvcth"
Sep 16 18:56:32.225180 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:56:32.225159    2544 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-zmsvt\""
Sep 16 18:56:32.225180 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:56:32.225173    2544 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-stop-kube-rbac-proxy-sar-config\""
Sep 16 18:56:32.225343 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:56:32.225181    2544 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-stop-v2-predictor-serving-cert\""
Sep 16 18:56:32.225343 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:56:32.225167    2544 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Sep 16 18:56:32.225467 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:56:32.225453    2544 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Sep 16 18:56:32.225532 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:56:32.225516    2544 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Sep 16 18:56:32.225639 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:56:32.225610    2544 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Sep 16 18:56:32.231512 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:56:32.231493    2544 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-b7c9ff648-nvcth"]
Sep 16 18:56:32.312321 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:56:32.312286    2544 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-canary-stop-predictor-7bb9c548b5-ln4lk"]
Sep 16 18:56:32.315871 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:56:32.315852    2544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-7bb9c548b5-ln4lk"
Sep 16 18:56:32.317873 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:56:32.317854    2544 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-stop-predictor-serving-cert\""
Sep 16 18:56:32.319422 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:56:32.319406    2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/7ae7a71d-424c-4d16-8916-642cfa6e34b1-proxy-tls\") pod \"isvc-canary-stop-v2-predictor-b7c9ff648-nvcth\" (UID: \"7ae7a71d-424c-4d16-8916-642cfa6e34b1\") " pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-b7c9ff648-nvcth"
Sep 16 18:56:32.319473 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:56:32.319442    2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/7ae7a71d-424c-4d16-8916-642cfa6e34b1-cabundle-cert\") pod \"isvc-canary-stop-v2-predictor-b7c9ff648-nvcth\" (UID: \"7ae7a71d-424c-4d16-8916-642cfa6e34b1\") " pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-b7c9ff648-nvcth"
Sep 16 18:56:32.319597 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:56:32.319572    2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5cljp\" (UniqueName: \"kubernetes.io/projected/7ae7a71d-424c-4d16-8916-642cfa6e34b1-kube-api-access-5cljp\") pod \"isvc-canary-stop-v2-predictor-b7c9ff648-nvcth\" (UID: \"7ae7a71d-424c-4d16-8916-642cfa6e34b1\") " pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-b7c9ff648-nvcth"
Sep 16 18:56:32.319727 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:56:32.319633    2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/7ae7a71d-424c-4d16-8916-642cfa6e34b1-kube-rbac-proxy-sar-config\") pod \"isvc-canary-stop-v2-predictor-b7c9ff648-nvcth\" (UID: \"7ae7a71d-424c-4d16-8916-642cfa6e34b1\") " pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-b7c9ff648-nvcth"
Sep 16 18:56:32.319727 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:56:32.319654    2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/7ae7a71d-424c-4d16-8916-642cfa6e34b1-kserve-provision-location\") pod \"isvc-canary-stop-v2-predictor-b7c9ff648-nvcth\" (UID: \"7ae7a71d-424c-4d16-8916-642cfa6e34b1\") " pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-b7c9ff648-nvcth"
Sep 16 18:56:32.330225 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:56:32.330198    2544 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-stop-predictor-7bb9c548b5-ln4lk"]
Sep 16 18:56:32.420542 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:56:32.420460    2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/09fd6613-c8cb-4b5d-b5b5-75a2e70d9adb-proxy-tls\") pod \"isvc-canary-stop-predictor-7bb9c548b5-ln4lk\" (UID: \"09fd6613-c8cb-4b5d-b5b5-75a2e70d9adb\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-7bb9c548b5-ln4lk"
Sep 16 18:56:32.420542 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:56:32.420493    2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/09fd6613-c8cb-4b5d-b5b5-75a2e70d9adb-kserve-provision-location\") pod \"isvc-canary-stop-predictor-7bb9c548b5-ln4lk\" (UID: \"09fd6613-c8cb-4b5d-b5b5-75a2e70d9adb\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-7bb9c548b5-ln4lk"
Sep 16 18:56:32.420542 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:56:32.420511    2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/09fd6613-c8cb-4b5d-b5b5-75a2e70d9adb-cabundle-cert\") pod \"isvc-canary-stop-predictor-7bb9c548b5-ln4lk\" (UID: \"09fd6613-c8cb-4b5d-b5b5-75a2e70d9adb\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-7bb9c548b5-ln4lk"
Sep 16 18:56:32.420542 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:56:32.420543    2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/7ae7a71d-424c-4d16-8916-642cfa6e34b1-cabundle-cert\") pod \"isvc-canary-stop-v2-predictor-b7c9ff648-nvcth\" (UID: \"7ae7a71d-424c-4d16-8916-642cfa6e34b1\") " pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-b7c9ff648-nvcth"
Sep 16 18:56:32.420944 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:56:32.420609    2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/09fd6613-c8cb-4b5d-b5b5-75a2e70d9adb-kube-rbac-proxy-sar-config\") pod \"isvc-canary-stop-predictor-7bb9c548b5-ln4lk\" (UID: \"09fd6613-c8cb-4b5d-b5b5-75a2e70d9adb\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-7bb9c548b5-ln4lk"
Sep 16 18:56:32.420944 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:56:32.420632    2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5cljp\" (UniqueName: \"kubernetes.io/projected/7ae7a71d-424c-4d16-8916-642cfa6e34b1-kube-api-access-5cljp\") pod \"isvc-canary-stop-v2-predictor-b7c9ff648-nvcth\" (UID: \"7ae7a71d-424c-4d16-8916-642cfa6e34b1\") " pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-b7c9ff648-nvcth"
Sep 16 18:56:32.420944 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:56:32.420657    2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-knljz\" (UniqueName: \"kubernetes.io/projected/09fd6613-c8cb-4b5d-b5b5-75a2e70d9adb-kube-api-access-knljz\") pod \"isvc-canary-stop-predictor-7bb9c548b5-ln4lk\" (UID: \"09fd6613-c8cb-4b5d-b5b5-75a2e70d9adb\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-7bb9c548b5-ln4lk"
Sep 16 18:56:32.420944 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:56:32.420784    2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/7ae7a71d-424c-4d16-8916-642cfa6e34b1-kube-rbac-proxy-sar-config\") pod \"isvc-canary-stop-v2-predictor-b7c9ff648-nvcth\" (UID: \"7ae7a71d-424c-4d16-8916-642cfa6e34b1\") " pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-b7c9ff648-nvcth"
Sep 16 18:56:32.420944 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:56:32.420819    2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/7ae7a71d-424c-4d16-8916-642cfa6e34b1-kserve-provision-location\") pod \"isvc-canary-stop-v2-predictor-b7c9ff648-nvcth\" (UID: \"7ae7a71d-424c-4d16-8916-642cfa6e34b1\") " pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-b7c9ff648-nvcth"
Sep 16 18:56:32.420944 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:56:32.420853    2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/7ae7a71d-424c-4d16-8916-642cfa6e34b1-proxy-tls\") pod \"isvc-canary-stop-v2-predictor-b7c9ff648-nvcth\" (UID: \"7ae7a71d-424c-4d16-8916-642cfa6e34b1\") " pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-b7c9ff648-nvcth"
Sep 16 18:56:32.421263 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:56:32.421149    2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/7ae7a71d-424c-4d16-8916-642cfa6e34b1-kserve-provision-location\") pod \"isvc-canary-stop-v2-predictor-b7c9ff648-nvcth\" (UID: \"7ae7a71d-424c-4d16-8916-642cfa6e34b1\") " pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-b7c9ff648-nvcth"
Sep 16 18:56:32.421322 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:56:32.421292    2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/7ae7a71d-424c-4d16-8916-642cfa6e34b1-cabundle-cert\") pod \"isvc-canary-stop-v2-predictor-b7c9ff648-nvcth\" (UID: \"7ae7a71d-424c-4d16-8916-642cfa6e34b1\") " pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-b7c9ff648-nvcth"
Sep 16 18:56:32.421465 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:56:32.421448    2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/7ae7a71d-424c-4d16-8916-642cfa6e34b1-kube-rbac-proxy-sar-config\") pod \"isvc-canary-stop-v2-predictor-b7c9ff648-nvcth\" (UID: \"7ae7a71d-424c-4d16-8916-642cfa6e34b1\") " pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-b7c9ff648-nvcth"
Sep 16 18:56:32.423123 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:56:32.423107    2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/7ae7a71d-424c-4d16-8916-642cfa6e34b1-proxy-tls\") pod \"isvc-canary-stop-v2-predictor-b7c9ff648-nvcth\" (UID: \"7ae7a71d-424c-4d16-8916-642cfa6e34b1\") " pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-b7c9ff648-nvcth"
Sep 16 18:56:32.428382 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:56:32.428350    2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5cljp\" (UniqueName: \"kubernetes.io/projected/7ae7a71d-424c-4d16-8916-642cfa6e34b1-kube-api-access-5cljp\") pod \"isvc-canary-stop-v2-predictor-b7c9ff648-nvcth\" (UID: \"7ae7a71d-424c-4d16-8916-642cfa6e34b1\") " pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-b7c9ff648-nvcth"
Sep 16 18:56:32.521229 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:56:32.521203    2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/09fd6613-c8cb-4b5d-b5b5-75a2e70d9adb-proxy-tls\") pod \"isvc-canary-stop-predictor-7bb9c548b5-ln4lk\" (UID: \"09fd6613-c8cb-4b5d-b5b5-75a2e70d9adb\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-7bb9c548b5-ln4lk"
Sep 16 18:56:32.521608 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:56:32.521235    2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/09fd6613-c8cb-4b5d-b5b5-75a2e70d9adb-kserve-provision-location\") pod \"isvc-canary-stop-predictor-7bb9c548b5-ln4lk\" (UID: \"09fd6613-c8cb-4b5d-b5b5-75a2e70d9adb\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-7bb9c548b5-ln4lk"
Sep 16 18:56:32.521608 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:56:32.521252    2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/09fd6613-c8cb-4b5d-b5b5-75a2e70d9adb-cabundle-cert\") pod \"isvc-canary-stop-predictor-7bb9c548b5-ln4lk\" (UID: \"09fd6613-c8cb-4b5d-b5b5-75a2e70d9adb\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-7bb9c548b5-ln4lk"
Sep 16 18:56:32.521608 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:56:32.521304    2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/09fd6613-c8cb-4b5d-b5b5-75a2e70d9adb-kube-rbac-proxy-sar-config\") pod \"isvc-canary-stop-predictor-7bb9c548b5-ln4lk\" (UID: \"09fd6613-c8cb-4b5d-b5b5-75a2e70d9adb\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-7bb9c548b5-ln4lk"
Sep 16 18:56:32.521608 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:56:32.521347    2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-knljz\" (UniqueName: \"kubernetes.io/projected/09fd6613-c8cb-4b5d-b5b5-75a2e70d9adb-kube-api-access-knljz\") pod \"isvc-canary-stop-predictor-7bb9c548b5-ln4lk\" (UID: \"09fd6613-c8cb-4b5d-b5b5-75a2e70d9adb\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-7bb9c548b5-ln4lk"
Sep 16 18:56:32.521771 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:56:32.521664    2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/09fd6613-c8cb-4b5d-b5b5-75a2e70d9adb-kserve-provision-location\") pod \"isvc-canary-stop-predictor-7bb9c548b5-ln4lk\" (UID: \"09fd6613-c8cb-4b5d-b5b5-75a2e70d9adb\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-7bb9c548b5-ln4lk"
Sep 16 18:56:32.521958 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:56:32.521940    2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/09fd6613-c8cb-4b5d-b5b5-75a2e70d9adb-cabundle-cert\") pod \"isvc-canary-stop-predictor-7bb9c548b5-ln4lk\" (UID: \"09fd6613-c8cb-4b5d-b5b5-75a2e70d9adb\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-7bb9c548b5-ln4lk"
Sep 16 18:56:32.521993 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:56:32.521972    2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/09fd6613-c8cb-4b5d-b5b5-75a2e70d9adb-kube-rbac-proxy-sar-config\") pod \"isvc-canary-stop-predictor-7bb9c548b5-ln4lk\" (UID: \"09fd6613-c8cb-4b5d-b5b5-75a2e70d9adb\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-7bb9c548b5-ln4lk"
Sep 16 18:56:32.523622 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:56:32.523601    2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/09fd6613-c8cb-4b5d-b5b5-75a2e70d9adb-proxy-tls\") pod \"isvc-canary-stop-predictor-7bb9c548b5-ln4lk\" (UID: \"09fd6613-c8cb-4b5d-b5b5-75a2e70d9adb\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-7bb9c548b5-ln4lk"
Sep 16 18:56:32.529864 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:56:32.529842    2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-knljz\" (UniqueName: \"kubernetes.io/projected/09fd6613-c8cb-4b5d-b5b5-75a2e70d9adb-kube-api-access-knljz\") pod \"isvc-canary-stop-predictor-7bb9c548b5-ln4lk\" (UID: \"09fd6613-c8cb-4b5d-b5b5-75a2e70d9adb\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-7bb9c548b5-ln4lk"
Sep 16 18:56:32.532630 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:56:32.532610    2544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-b7c9ff648-nvcth"
Sep 16 18:56:32.625814 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:56:32.625780    2544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-7bb9c548b5-ln4lk"
Sep 16 18:56:32.805148 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:56:32.805124    2544 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-b7c9ff648-nvcth"]
Sep 16 18:56:32.807522 ip-10-0-132-80 kubenswrapper[2544]: W0916 18:56:32.807494    2544 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7ae7a71d_424c_4d16_8916_642cfa6e34b1.slice/crio-5d5a29c840c662bac5fe0f0a22fc1f72fad644ea60fd3c6eeb49705e6e34ccae WatchSource:0}: Error finding container 5d5a29c840c662bac5fe0f0a22fc1f72fad644ea60fd3c6eeb49705e6e34ccae: Status 404 returned error can't find the container with id 5d5a29c840c662bac5fe0f0a22fc1f72fad644ea60fd3c6eeb49705e6e34ccae
Sep 16 18:56:32.901931 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:56:32.901901    2544 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-stop-predictor-7bb9c548b5-ln4lk"]
Sep 16 18:56:32.903098 ip-10-0-132-80 kubenswrapper[2544]: W0916 18:56:32.903071    2544 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod09fd6613_c8cb_4b5d_b5b5_75a2e70d9adb.slice/crio-00c03ac85d7dbfb0f42eb318ec9bc5b4b9b5ba6aa37f9b0e3474292d65c6eb4a WatchSource:0}: Error finding container 00c03ac85d7dbfb0f42eb318ec9bc5b4b9b5ba6aa37f9b0e3474292d65c6eb4a: Status 404 returned error can't find the container with id 00c03ac85d7dbfb0f42eb318ec9bc5b4b9b5ba6aa37f9b0e3474292d65c6eb4a
Sep 16 18:56:33.136231 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:56:33.136193    2544 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-7bb9c548b5-ln4lk" event={"ID":"09fd6613-c8cb-4b5d-b5b5-75a2e70d9adb","Type":"ContainerStarted","Data":"c78626128efa0bb53db1684ce57a2a42df71a696f7dc3eed8e144bd71f79014d"}
Sep 16 18:56:33.136408 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:56:33.136268    2544 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-7bb9c548b5-ln4lk" event={"ID":"09fd6613-c8cb-4b5d-b5b5-75a2e70d9adb","Type":"ContainerStarted","Data":"00c03ac85d7dbfb0f42eb318ec9bc5b4b9b5ba6aa37f9b0e3474292d65c6eb4a"}
Sep 16 18:56:33.138899 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:56:33.138863    2544 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-b7c9ff648-nvcth" event={"ID":"7ae7a71d-424c-4d16-8916-642cfa6e34b1","Type":"ContainerStarted","Data":"23d0b7d6e8f486fc8564b72c2418041d61a0ff182b815810aa75904bf6334225"}
Sep 16 18:56:33.139044 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:56:33.138929    2544 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-b7c9ff648-nvcth" event={"ID":"7ae7a71d-424c-4d16-8916-642cfa6e34b1","Type":"ContainerStarted","Data":"5d5a29c840c662bac5fe0f0a22fc1f72fad644ea60fd3c6eeb49705e6e34ccae"}
Sep 16 18:56:33.721374 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:56:33.721343    2544 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="328264d8-24da-4c7e-8920-9b9f5b9a03dc" path="/var/lib/kubelet/pods/328264d8-24da-4c7e-8920-9b9f5b9a03dc/volumes"
Sep 16 18:56:37.156654 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:56:37.156616    2544 generic.go:356] "Generic (PLEG): container finished" podID="7ae7a71d-424c-4d16-8916-642cfa6e34b1" containerID="23d0b7d6e8f486fc8564b72c2418041d61a0ff182b815810aa75904bf6334225" exitCode=0
Sep 16 18:56:37.157133 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:56:37.156690    2544 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-b7c9ff648-nvcth" event={"ID":"7ae7a71d-424c-4d16-8916-642cfa6e34b1","Type":"ContainerDied","Data":"23d0b7d6e8f486fc8564b72c2418041d61a0ff182b815810aa75904bf6334225"}
Sep 16 18:56:37.158262 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:56:37.158243    2544 generic.go:356] "Generic (PLEG): container finished" podID="09fd6613-c8cb-4b5d-b5b5-75a2e70d9adb" containerID="c78626128efa0bb53db1684ce57a2a42df71a696f7dc3eed8e144bd71f79014d" exitCode=0
Sep 16 18:56:37.158330 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:56:37.158313    2544 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-7bb9c548b5-ln4lk" event={"ID":"09fd6613-c8cb-4b5d-b5b5-75a2e70d9adb","Type":"ContainerDied","Data":"c78626128efa0bb53db1684ce57a2a42df71a696f7dc3eed8e144bd71f79014d"}
Sep 16 18:56:38.165203 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:56:38.165165    2544 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-b7c9ff648-nvcth" event={"ID":"7ae7a71d-424c-4d16-8916-642cfa6e34b1","Type":"ContainerStarted","Data":"8b918eee02e5ebf4ce91f7f4908d7a8c1a34a432981d135039f85ec8b8da29c0"}
Sep 16 18:56:38.165694 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:56:38.165240    2544 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-b7c9ff648-nvcth" event={"ID":"7ae7a71d-424c-4d16-8916-642cfa6e34b1","Type":"ContainerStarted","Data":"c006c91df31e32a45d918a30163000fd4ce9da92f092a18c41dd134f8b2d48d2"}
Sep 16 18:56:38.165694 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:56:38.165518    2544 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-b7c9ff648-nvcth"
Sep 16 18:56:38.165694 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:56:38.165592    2544 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-b7c9ff648-nvcth"
Sep 16 18:56:38.167203 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:56:38.167165    2544 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-b7c9ff648-nvcth" podUID="7ae7a71d-424c-4d16-8916-642cfa6e34b1" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.29:8080: connect: connection refused"
Sep 16 18:56:38.167595 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:56:38.167565    2544 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-7bb9c548b5-ln4lk" event={"ID":"09fd6613-c8cb-4b5d-b5b5-75a2e70d9adb","Type":"ContainerStarted","Data":"11acfbb3890d82fb84418d2b709912383df14723d33d69d8e23d2e9ea4f55cd2"}
Sep 16 18:56:38.167679 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:56:38.167621    2544 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-7bb9c548b5-ln4lk" event={"ID":"09fd6613-c8cb-4b5d-b5b5-75a2e70d9adb","Type":"ContainerStarted","Data":"fdb9c6f4a725af0dbfd5dbac21545d9e50c697111b1b18b2ba497323dc24a2be"}
Sep 16 18:56:38.167809 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:56:38.167794    2544 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-7bb9c548b5-ln4lk"
Sep 16 18:56:38.183803 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:56:38.183757    2544 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-b7c9ff648-nvcth" podStartSLOduration=2.282307796 podStartE2EDuration="6.183742779s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.901434983s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:56:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:56:38.182529519 +0000 UTC m=+760.971532074" watchObservedRunningTime="2026-09-16 18:56:38.183742779 +0000 UTC m=+760.972745315"
Sep 16 18:56:38.203310 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:56:38.203255    2544 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-7bb9c548b5-ln4lk" podStartSLOduration=2.309209539 podStartE2EDuration="6.203241679s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.89403214s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:56:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:56:38.200215365 +0000 UTC m=+760.989217919" watchObservedRunningTime="2026-09-16 18:56:38.203241679 +0000 UTC m=+760.992244212"
Sep 16 18:56:39.171995 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:56:39.171964    2544 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-7bb9c548b5-ln4lk"
Sep 16 18:56:39.172446 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:56:39.171998    2544 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-b7c9ff648-nvcth" podUID="7ae7a71d-424c-4d16-8916-642cfa6e34b1" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.29:8080: connect: connection refused"
Sep 16 18:56:39.173303 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:56:39.173276    2544 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-7bb9c548b5-ln4lk" podUID="09fd6613-c8cb-4b5d-b5b5-75a2e70d9adb" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.30:8080: connect: connection refused"
Sep 16 18:56:40.176850 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:56:40.176807    2544 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-7bb9c548b5-ln4lk" podUID="09fd6613-c8cb-4b5d-b5b5-75a2e70d9adb" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.30:8080: connect: connection refused"
Sep 16 18:56:44.176992 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:56:44.176960    2544 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-b7c9ff648-nvcth"
Sep 16 18:56:44.177486 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:56:44.177460    2544 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-b7c9ff648-nvcth" podUID="7ae7a71d-424c-4d16-8916-642cfa6e34b1" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.29:8080: connect: connection refused"
Sep 16 18:56:45.181187 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:56:45.181157    2544 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-7bb9c548b5-ln4lk"
Sep 16 18:56:45.181669 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:56:45.181642    2544 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-7bb9c548b5-ln4lk" podUID="09fd6613-c8cb-4b5d-b5b5-75a2e70d9adb" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.30:8080: connect: connection refused"
Sep 16 18:56:52.434199 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:56:52.434162    2544 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-22075-predictor-754b4f7b64-9stpg"]
Sep 16 18:56:52.437992 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:56:52.437975    2544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-22075-predictor-754b4f7b64-9stpg"
Sep 16 18:56:52.440307 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:56:52.440284    2544 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-graph-raw-hpa-22075-kube-rbac-proxy-sar-config\""
Sep 16 18:56:52.440581 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:56:52.440546    2544 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-graph-raw-hpa-22075-predictor-serving-cert\""
Sep 16 18:56:52.453395 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:56:52.453373    2544 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-22075-predictor-754b4f7b64-9stpg"]
Sep 16 18:56:52.489986 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:56:52.489957    2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/bb199f20-09bc-4c3f-aa96-d9a976f62292-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-graph-raw-hpa-22075-predictor-754b4f7b64-9stpg\" (UID: \"bb199f20-09bc-4c3f-aa96-d9a976f62292\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-22075-predictor-754b4f7b64-9stpg"
Sep 16 18:56:52.490130 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:56:52.489990    2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f5rqf\" (UniqueName: \"kubernetes.io/projected/bb199f20-09bc-4c3f-aa96-d9a976f62292-kube-api-access-f5rqf\") pod \"isvc-sklearn-graph-raw-hpa-22075-predictor-754b4f7b64-9stpg\" (UID: \"bb199f20-09bc-4c3f-aa96-d9a976f62292\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-22075-predictor-754b4f7b64-9stpg"
Sep 16 18:56:52.490130 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:56:52.490066    2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/bb199f20-09bc-4c3f-aa96-d9a976f62292-cabundle-cert\") pod \"isvc-sklearn-graph-raw-hpa-22075-predictor-754b4f7b64-9stpg\" (UID: \"bb199f20-09bc-4c3f-aa96-d9a976f62292\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-22075-predictor-754b4f7b64-9stpg"
Sep 16 18:56:52.490240 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:56:52.490134    2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/bb199f20-09bc-4c3f-aa96-d9a976f62292-kserve-provision-location\") pod \"isvc-sklearn-graph-raw-hpa-22075-predictor-754b4f7b64-9stpg\" (UID: \"bb199f20-09bc-4c3f-aa96-d9a976f62292\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-22075-predictor-754b4f7b64-9stpg"
Sep 16 18:56:52.490240 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:56:52.490171    2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/bb199f20-09bc-4c3f-aa96-d9a976f62292-proxy-tls\") pod \"isvc-sklearn-graph-raw-hpa-22075-predictor-754b4f7b64-9stpg\" (UID: \"bb199f20-09bc-4c3f-aa96-d9a976f62292\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-22075-predictor-754b4f7b64-9stpg"
Sep 16 18:56:52.574985 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:56:52.574952    2544 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-22075-predictor-5799df6789-ns959"]
Sep 16 18:56:52.578630 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:56:52.578613    2544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-22075-predictor-5799df6789-ns959"
Sep 16 18:56:52.580790 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:56:52.580767    2544 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-xgboost-graph-raw-hpa-22075-predictor-serving-cert\""
Sep 16 18:56:52.581105 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:56:52.581089    2544 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-xgboost-graph-raw-hpa-22075-kube-rbac-proxy-sar-config\""
Sep 16 18:56:52.590679 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:56:52.590655    2544 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-22075-predictor-5799df6789-ns959"]
Sep 16 18:56:52.590934 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:56:52.590916    2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/bb199f20-09bc-4c3f-aa96-d9a976f62292-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-graph-raw-hpa-22075-predictor-754b4f7b64-9stpg\" (UID: \"bb199f20-09bc-4c3f-aa96-d9a976f62292\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-22075-predictor-754b4f7b64-9stpg"
Sep 16 18:56:52.590982 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:56:52.590943    2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f5rqf\" (UniqueName: \"kubernetes.io/projected/bb199f20-09bc-4c3f-aa96-d9a976f62292-kube-api-access-f5rqf\") pod \"isvc-sklearn-graph-raw-hpa-22075-predictor-754b4f7b64-9stpg\" (UID: \"bb199f20-09bc-4c3f-aa96-d9a976f62292\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-22075-predictor-754b4f7b64-9stpg"
Sep 16 18:56:52.590982 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:56:52.590967    2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/bb199f20-09bc-4c3f-aa96-d9a976f62292-cabundle-cert\") pod \"isvc-sklearn-graph-raw-hpa-22075-predictor-754b4f7b64-9stpg\" (UID: \"bb199f20-09bc-4c3f-aa96-d9a976f62292\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-22075-predictor-754b4f7b64-9stpg"
Sep 16 18:56:52.591048 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:56:52.591009    2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/bb199f20-09bc-4c3f-aa96-d9a976f62292-kserve-provision-location\") pod \"isvc-sklearn-graph-raw-hpa-22075-predictor-754b4f7b64-9stpg\" (UID: \"bb199f20-09bc-4c3f-aa96-d9a976f62292\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-22075-predictor-754b4f7b64-9stpg"
Sep 16 18:56:52.591048 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:56:52.591029    2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/bb199f20-09bc-4c3f-aa96-d9a976f62292-proxy-tls\") pod \"isvc-sklearn-graph-raw-hpa-22075-predictor-754b4f7b64-9stpg\" (UID: \"bb199f20-09bc-4c3f-aa96-d9a976f62292\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-22075-predictor-754b4f7b64-9stpg"
Sep 16 18:56:52.591378 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:56:52.591361    2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/bb199f20-09bc-4c3f-aa96-d9a976f62292-kserve-provision-location\") pod \"isvc-sklearn-graph-raw-hpa-22075-predictor-754b4f7b64-9stpg\" (UID: \"bb199f20-09bc-4c3f-aa96-d9a976f62292\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-22075-predictor-754b4f7b64-9stpg"
Sep 16 18:56:52.591712 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:56:52.591687    2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/bb199f20-09bc-4c3f-aa96-d9a976f62292-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-graph-raw-hpa-22075-predictor-754b4f7b64-9stpg\" (UID: \"bb199f20-09bc-4c3f-aa96-d9a976f62292\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-22075-predictor-754b4f7b64-9stpg"
Sep 16 18:56:52.591798 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:56:52.591726    2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/bb199f20-09bc-4c3f-aa96-d9a976f62292-cabundle-cert\") pod \"isvc-sklearn-graph-raw-hpa-22075-predictor-754b4f7b64-9stpg\" (UID: \"bb199f20-09bc-4c3f-aa96-d9a976f62292\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-22075-predictor-754b4f7b64-9stpg"
Sep 16 18:56:52.593577 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:56:52.593526    2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/bb199f20-09bc-4c3f-aa96-d9a976f62292-proxy-tls\") pod \"isvc-sklearn-graph-raw-hpa-22075-predictor-754b4f7b64-9stpg\" (UID: \"bb199f20-09bc-4c3f-aa96-d9a976f62292\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-22075-predictor-754b4f7b64-9stpg"
Sep 16 18:56:52.605125 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:56:52.605108    2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-f5rqf\" (UniqueName: \"kubernetes.io/projected/bb199f20-09bc-4c3f-aa96-d9a976f62292-kube-api-access-f5rqf\") pod \"isvc-sklearn-graph-raw-hpa-22075-predictor-754b4f7b64-9stpg\" (UID: \"bb199f20-09bc-4c3f-aa96-d9a976f62292\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-22075-predictor-754b4f7b64-9stpg"
Sep 16 18:56:52.691903 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:56:52.691836    2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/2985e5ca-4da7-4cd5-911e-39decc92e56d-kserve-provision-location\") pod \"isvc-xgboost-graph-raw-hpa-22075-predictor-5799df6789-ns959\" (UID: \"2985e5ca-4da7-4cd5-911e-39decc92e56d\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-22075-predictor-5799df6789-ns959"
Sep 16 18:56:52.691903 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:56:52.691870    2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/2985e5ca-4da7-4cd5-911e-39decc92e56d-cabundle-cert\") pod \"isvc-xgboost-graph-raw-hpa-22075-predictor-5799df6789-ns959\" (UID: \"2985e5ca-4da7-4cd5-911e-39decc92e56d\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-22075-predictor-5799df6789-ns959"
Sep 16 18:56:52.692058 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:56:52.691963    2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n5mp5\" (UniqueName: \"kubernetes.io/projected/2985e5ca-4da7-4cd5-911e-39decc92e56d-kube-api-access-n5mp5\") pod \"isvc-xgboost-graph-raw-hpa-22075-predictor-5799df6789-ns959\" (UID: \"2985e5ca-4da7-4cd5-911e-39decc92e56d\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-22075-predictor-5799df6789-ns959"
Sep 16 18:56:52.692058 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:56:52.692006    2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/2985e5ca-4da7-4cd5-911e-39decc92e56d-proxy-tls\") pod \"isvc-xgboost-graph-raw-hpa-22075-predictor-5799df6789-ns959\" (UID: \"2985e5ca-4da7-4cd5-911e-39decc92e56d\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-22075-predictor-5799df6789-ns959"
Sep 16 18:56:52.692058 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:56:52.692029    2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/2985e5ca-4da7-4cd5-911e-39decc92e56d-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-graph-raw-hpa-22075-predictor-5799df6789-ns959\" (UID: \"2985e5ca-4da7-4cd5-911e-39decc92e56d\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-22075-predictor-5799df6789-ns959"
Sep 16 18:56:52.748672 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:56:52.748646    2544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-22075-predictor-754b4f7b64-9stpg"
Sep 16 18:56:52.793232 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:56:52.793191    2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/2985e5ca-4da7-4cd5-911e-39decc92e56d-kserve-provision-location\") pod \"isvc-xgboost-graph-raw-hpa-22075-predictor-5799df6789-ns959\" (UID: \"2985e5ca-4da7-4cd5-911e-39decc92e56d\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-22075-predictor-5799df6789-ns959"
Sep 16 18:56:52.793379 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:56:52.793236    2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/2985e5ca-4da7-4cd5-911e-39decc92e56d-cabundle-cert\") pod \"isvc-xgboost-graph-raw-hpa-22075-predictor-5799df6789-ns959\" (UID: \"2985e5ca-4da7-4cd5-911e-39decc92e56d\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-22075-predictor-5799df6789-ns959"
Sep 16 18:56:52.793379 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:56:52.793306    2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n5mp5\" (UniqueName: \"kubernetes.io/projected/2985e5ca-4da7-4cd5-911e-39decc92e56d-kube-api-access-n5mp5\") pod \"isvc-xgboost-graph-raw-hpa-22075-predictor-5799df6789-ns959\" (UID: \"2985e5ca-4da7-4cd5-911e-39decc92e56d\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-22075-predictor-5799df6789-ns959"
Sep 16 18:56:52.793379 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:56:52.793335    2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/2985e5ca-4da7-4cd5-911e-39decc92e56d-proxy-tls\") pod \"isvc-xgboost-graph-raw-hpa-22075-predictor-5799df6789-ns959\" (UID: \"2985e5ca-4da7-4cd5-911e-39decc92e56d\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-22075-predictor-5799df6789-ns959"
Sep 16 18:56:52.793379 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:56:52.793372    2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/2985e5ca-4da7-4cd5-911e-39decc92e56d-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-graph-raw-hpa-22075-predictor-5799df6789-ns959\" (UID: \"2985e5ca-4da7-4cd5-911e-39decc92e56d\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-22075-predictor-5799df6789-ns959"
Sep 16 18:56:52.793666 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:56:52.793641    2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/2985e5ca-4da7-4cd5-911e-39decc92e56d-kserve-provision-location\") pod \"isvc-xgboost-graph-raw-hpa-22075-predictor-5799df6789-ns959\" (UID: \"2985e5ca-4da7-4cd5-911e-39decc92e56d\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-22075-predictor-5799df6789-ns959"
Sep 16 18:56:52.793995 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:56:52.793973    2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/2985e5ca-4da7-4cd5-911e-39decc92e56d-cabundle-cert\") pod \"isvc-xgboost-graph-raw-hpa-22075-predictor-5799df6789-ns959\" (UID: \"2985e5ca-4da7-4cd5-911e-39decc92e56d\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-22075-predictor-5799df6789-ns959"
Sep 16 18:56:52.794086 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:56:52.794032    2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/2985e5ca-4da7-4cd5-911e-39decc92e56d-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-graph-raw-hpa-22075-predictor-5799df6789-ns959\" (UID: \"2985e5ca-4da7-4cd5-911e-39decc92e56d\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-22075-predictor-5799df6789-ns959"
Sep 16 18:56:52.796246 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:56:52.796217    2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/2985e5ca-4da7-4cd5-911e-39decc92e56d-proxy-tls\") pod \"isvc-xgboost-graph-raw-hpa-22075-predictor-5799df6789-ns959\" (UID: \"2985e5ca-4da7-4cd5-911e-39decc92e56d\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-22075-predictor-5799df6789-ns959"
Sep 16 18:56:52.805590 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:56:52.804985    2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-n5mp5\" (UniqueName: \"kubernetes.io/projected/2985e5ca-4da7-4cd5-911e-39decc92e56d-kube-api-access-n5mp5\") pod \"isvc-xgboost-graph-raw-hpa-22075-predictor-5799df6789-ns959\" (UID: \"2985e5ca-4da7-4cd5-911e-39decc92e56d\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-22075-predictor-5799df6789-ns959"
Sep 16 18:56:52.888433 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:56:52.888402    2544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-22075-predictor-5799df6789-ns959"
Sep 16 18:56:53.025860 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:56:53.025829    2544 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-22075-predictor-754b4f7b64-9stpg"]
Sep 16 18:56:53.028237 ip-10-0-132-80 kubenswrapper[2544]: W0916 18:56:53.028208    2544 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbb199f20_09bc_4c3f_aa96_d9a976f62292.slice/crio-4628b08b0734616ff954ec75f263b403d85db69a7b76c5166ac39bbc43eb4b46 WatchSource:0}: Error finding container 4628b08b0734616ff954ec75f263b403d85db69a7b76c5166ac39bbc43eb4b46: Status 404 returned error can't find the container with id 4628b08b0734616ff954ec75f263b403d85db69a7b76c5166ac39bbc43eb4b46
Sep 16 18:56:53.168419 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:56:53.168390    2544 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-22075-predictor-5799df6789-ns959"]
Sep 16 18:56:53.170288 ip-10-0-132-80 kubenswrapper[2544]: W0916 18:56:53.170264    2544 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2985e5ca_4da7_4cd5_911e_39decc92e56d.slice/crio-2b8603cfe409efd2a1a807f10f8837f40bdeeaca4cae615d3cfd5a65feb454cb WatchSource:0}: Error finding container 2b8603cfe409efd2a1a807f10f8837f40bdeeaca4cae615d3cfd5a65feb454cb: Status 404 returned error can't find the container with id 2b8603cfe409efd2a1a807f10f8837f40bdeeaca4cae615d3cfd5a65feb454cb
Sep 16 18:56:53.237473 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:56:53.237445    2544 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-22075-predictor-754b4f7b64-9stpg" event={"ID":"bb199f20-09bc-4c3f-aa96-d9a976f62292","Type":"ContainerStarted","Data":"2ab9805704910dcd0fb0584bff3babd0b9ae059d01b0f2411bd502507c64741f"}
Sep 16 18:56:53.237619 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:56:53.237507    2544 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-22075-predictor-754b4f7b64-9stpg" event={"ID":"bb199f20-09bc-4c3f-aa96-d9a976f62292","Type":"ContainerStarted","Data":"4628b08b0734616ff954ec75f263b403d85db69a7b76c5166ac39bbc43eb4b46"}
Sep 16 18:56:53.239275 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:56:53.239252    2544 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-22075-predictor-5799df6789-ns959" event={"ID":"2985e5ca-4da7-4cd5-911e-39decc92e56d","Type":"ContainerStarted","Data":"e2b8a30c9f17cd15ae23e43fbeca2f289afa0f9e3abf8be8fd0e7213649df077"}
Sep 16 18:56:53.239355 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:56:53.239297    2544 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-22075-predictor-5799df6789-ns959" event={"ID":"2985e5ca-4da7-4cd5-911e-39decc92e56d","Type":"ContainerStarted","Data":"2b8603cfe409efd2a1a807f10f8837f40bdeeaca4cae615d3cfd5a65feb454cb"}
Sep 16 18:56:54.177732 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:56:54.177694    2544 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-b7c9ff648-nvcth" podUID="7ae7a71d-424c-4d16-8916-642cfa6e34b1" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.29:8080: connect: connection refused"
Sep 16 18:56:55.182028 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:56:55.181992    2544 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-7bb9c548b5-ln4lk" podUID="09fd6613-c8cb-4b5d-b5b5-75a2e70d9adb" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.30:8080: connect: connection refused"
Sep 16 18:56:57.261439 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:56:57.261351    2544 generic.go:356] "Generic (PLEG): container finished" podID="bb199f20-09bc-4c3f-aa96-d9a976f62292" containerID="2ab9805704910dcd0fb0584bff3babd0b9ae059d01b0f2411bd502507c64741f" exitCode=0
Sep 16 18:56:57.261935 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:56:57.261434    2544 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-22075-predictor-754b4f7b64-9stpg" event={"ID":"bb199f20-09bc-4c3f-aa96-d9a976f62292","Type":"ContainerDied","Data":"2ab9805704910dcd0fb0584bff3babd0b9ae059d01b0f2411bd502507c64741f"}
Sep 16 18:56:58.268073 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:56:58.268033    2544 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-22075-predictor-754b4f7b64-9stpg" event={"ID":"bb199f20-09bc-4c3f-aa96-d9a976f62292","Type":"ContainerStarted","Data":"226f7c4b4845bc5ae74b981c2d9c2f8de12ed573a24f667d6bc99a8a0bf7ce80"}
Sep 16 18:56:58.268609 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:56:58.268145    2544 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-22075-predictor-754b4f7b64-9stpg" event={"ID":"bb199f20-09bc-4c3f-aa96-d9a976f62292","Type":"ContainerStarted","Data":"b73ca7b84c824cf9ccf800d19bf6879351d2971f29f34ee8944ea449b0f6a655"}
Sep 16 18:56:58.268609 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:56:58.268448    2544 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-22075-predictor-754b4f7b64-9stpg"
Sep 16 18:56:58.268740 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:56:58.268619    2544 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-22075-predictor-754b4f7b64-9stpg"
Sep 16 18:56:58.269794 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:56:58.269768    2544 generic.go:356] "Generic (PLEG): container finished" podID="2985e5ca-4da7-4cd5-911e-39decc92e56d" containerID="e2b8a30c9f17cd15ae23e43fbeca2f289afa0f9e3abf8be8fd0e7213649df077" exitCode=0
Sep 16 18:56:58.269909 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:56:58.269808    2544 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-22075-predictor-5799df6789-ns959" event={"ID":"2985e5ca-4da7-4cd5-911e-39decc92e56d","Type":"ContainerDied","Data":"e2b8a30c9f17cd15ae23e43fbeca2f289afa0f9e3abf8be8fd0e7213649df077"}
Sep 16 18:56:58.270065 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:56:58.270037    2544 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-22075-predictor-754b4f7b64-9stpg" podUID="bb199f20-09bc-4c3f-aa96-d9a976f62292" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.31:8080: connect: connection refused"
Sep 16 18:56:58.271110 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:56:58.271093    2544 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 16 18:56:58.286223 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:56:58.286168    2544 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-22075-predictor-754b4f7b64-9stpg" podStartSLOduration=2.530072781 podStartE2EDuration="6.286151306s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.756078525s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:56:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:56:58.285593302 +0000 UTC m=+781.074595857" watchObservedRunningTime="2026-09-16 18:56:58.286151306 +0000 UTC m=+781.075153841"
Sep 16 18:56:59.276289 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:56:59.276247    2544 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-22075-predictor-754b4f7b64-9stpg" podUID="bb199f20-09bc-4c3f-aa96-d9a976f62292" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.31:8080: connect: connection refused"
Sep 16 18:57:04.178302 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:57:04.178263    2544 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-b7c9ff648-nvcth" podUID="7ae7a71d-424c-4d16-8916-642cfa6e34b1" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.29:8080: connect: connection refused"
Sep 16 18:57:04.283643 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:57:04.283613    2544 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-22075-predictor-754b4f7b64-9stpg"
Sep 16 18:57:04.284372 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:57:04.284220    2544 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-22075-predictor-754b4f7b64-9stpg" podUID="bb199f20-09bc-4c3f-aa96-d9a976f62292" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.31:8080: connect: connection refused"
Sep 16 18:57:05.182127 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:57:05.182085    2544 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-7bb9c548b5-ln4lk" podUID="09fd6613-c8cb-4b5d-b5b5-75a2e70d9adb" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.30:8080: connect: connection refused"
Sep 16 18:57:14.178404 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:57:14.178365    2544 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-b7c9ff648-nvcth" podUID="7ae7a71d-424c-4d16-8916-642cfa6e34b1" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.29:8080: connect: connection refused"
Sep 16 18:57:14.284629 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:57:14.284587    2544 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-22075-predictor-754b4f7b64-9stpg" podUID="bb199f20-09bc-4c3f-aa96-d9a976f62292" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.31:8080: connect: connection refused"
Sep 16 18:57:15.181892 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:57:15.181846    2544 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-7bb9c548b5-ln4lk" podUID="09fd6613-c8cb-4b5d-b5b5-75a2e70d9adb" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.30:8080: connect: connection refused"
Sep 16 18:57:16.371662 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:57:16.371626    2544 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-22075-predictor-5799df6789-ns959" event={"ID":"2985e5ca-4da7-4cd5-911e-39decc92e56d","Type":"ContainerStarted","Data":"23e62e625222e4c7b46bb1ce0de8ab0411cb562b705cb75c4f7fcfbf93a4adb2"}
Sep 16 18:57:16.372025 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:57:16.371687    2544 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-22075-predictor-5799df6789-ns959" event={"ID":"2985e5ca-4da7-4cd5-911e-39decc92e56d","Type":"ContainerStarted","Data":"c13d0d59a9b5f662e46fe2c176f415862eaf6baa831dd5eb0bf6da6fe83bbbf9"}
Sep 16 18:57:16.372025 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:57:16.371871    2544 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-22075-predictor-5799df6789-ns959"
Sep 16 18:57:16.394468 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:57:16.394425    2544 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-22075-predictor-5799df6789-ns959" podStartSLOduration=3.242226827 podStartE2EDuration="24.394411617s" totalImagesPullingTime="17.068065719s" totalInitContainerRuntime="4.084119071s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:56:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:57:16.39282324 +0000 UTC m=+799.181825774" watchObservedRunningTime="2026-09-16 18:57:16.394411617 +0000 UTC m=+799.183414150"
Sep 16 18:57:17.377004 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:57:17.376972    2544 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-22075-predictor-5799df6789-ns959"
Sep 16 18:57:17.378031 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:57:17.377996    2544 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-22075-predictor-5799df6789-ns959" podUID="2985e5ca-4da7-4cd5-911e-39decc92e56d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.32:8080: connect: connection refused"
Sep 16 18:57:18.381700 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:57:18.381660    2544 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-22075-predictor-5799df6789-ns959" podUID="2985e5ca-4da7-4cd5-911e-39decc92e56d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.32:8080: connect: connection refused"
Sep 16 18:57:23.386310 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:57:23.386276    2544 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-22075-predictor-5799df6789-ns959"
Sep 16 18:57:23.386798 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:57:23.386729    2544 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-22075-predictor-5799df6789-ns959" podUID="2985e5ca-4da7-4cd5-911e-39decc92e56d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.32:8080: connect: connection refused"
Sep 16 18:57:24.178195 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:57:24.178152    2544 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-b7c9ff648-nvcth" podUID="7ae7a71d-424c-4d16-8916-642cfa6e34b1" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.29:8080: connect: connection refused"
Sep 16 18:57:24.284288 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:57:24.284247    2544 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-22075-predictor-754b4f7b64-9stpg" podUID="bb199f20-09bc-4c3f-aa96-d9a976f62292" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.31:8080: connect: connection refused"
Sep 16 18:57:25.181783 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:57:25.181742    2544 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-7bb9c548b5-ln4lk" podUID="09fd6613-c8cb-4b5d-b5b5-75a2e70d9adb" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.30:8080: connect: connection refused"
Sep 16 18:57:33.387260 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:57:33.387223    2544 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-22075-predictor-5799df6789-ns959" podUID="2985e5ca-4da7-4cd5-911e-39decc92e56d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.32:8080: connect: connection refused"
Sep 16 18:57:34.178357 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:57:34.178317    2544 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-b7c9ff648-nvcth" podUID="7ae7a71d-424c-4d16-8916-642cfa6e34b1" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.29:8080: connect: connection refused"
Sep 16 18:57:34.284251 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:57:34.284213    2544 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-22075-predictor-754b4f7b64-9stpg" podUID="bb199f20-09bc-4c3f-aa96-d9a976f62292" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.31:8080: connect: connection refused"
Sep 16 18:57:35.181984 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:57:35.181947    2544 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-7bb9c548b5-ln4lk" podUID="09fd6613-c8cb-4b5d-b5b5-75a2e70d9adb" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.30:8080: connect: connection refused"
Sep 16 18:57:43.387139 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:57:43.387098    2544 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-22075-predictor-5799df6789-ns959" podUID="2985e5ca-4da7-4cd5-911e-39decc92e56d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.32:8080: connect: connection refused"
Sep 16 18:57:44.177509 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:57:44.177466    2544 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-b7c9ff648-nvcth" podUID="7ae7a71d-424c-4d16-8916-642cfa6e34b1" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.29:8080: connect: connection refused"
Sep 16 18:57:44.284673 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:57:44.284629    2544 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-22075-predictor-754b4f7b64-9stpg" podUID="bb199f20-09bc-4c3f-aa96-d9a976f62292" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.31:8080: connect: connection refused"
Sep 16 18:57:45.182046 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:57:45.182007    2544 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-7bb9c548b5-ln4lk" podUID="09fd6613-c8cb-4b5d-b5b5-75a2e70d9adb" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.30:8080: connect: connection refused"
Sep 16 18:57:53.386721 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:57:53.386676    2544 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-22075-predictor-5799df6789-ns959" podUID="2985e5ca-4da7-4cd5-911e-39decc92e56d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.32:8080: connect: connection refused"
Sep 16 18:57:54.178004 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:57:54.177969    2544 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-b7c9ff648-nvcth" podUID="7ae7a71d-424c-4d16-8916-642cfa6e34b1" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.29:8080: connect: connection refused"
Sep 16 18:57:54.284781 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:57:54.284739    2544 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-22075-predictor-754b4f7b64-9stpg" podUID="bb199f20-09bc-4c3f-aa96-d9a976f62292" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.31:8080: connect: connection refused"
Sep 16 18:57:55.182526 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:57:55.182489    2544 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-7bb9c548b5-ln4lk" podUID="09fd6613-c8cb-4b5d-b5b5-75a2e70d9adb" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.30:8080: connect: connection refused"
Sep 16 18:57:58.717026 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:57:58.716980    2544 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-b7c9ff648-nvcth" podUID="7ae7a71d-424c-4d16-8916-642cfa6e34b1" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.29:8080: connect: connection refused"
Sep 16 18:58:02.717594 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:58:02.717536    2544 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-7bb9c548b5-ln4lk"
Sep 16 18:58:03.387483 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:58:03.387442    2544 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-22075-predictor-5799df6789-ns959" podUID="2985e5ca-4da7-4cd5-911e-39decc92e56d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.32:8080: connect: connection refused"
Sep 16 18:58:04.284562 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:58:04.284509    2544 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-22075-predictor-754b4f7b64-9stpg" podUID="bb199f20-09bc-4c3f-aa96-d9a976f62292" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.31:8080: connect: connection refused"
Sep 16 18:58:08.717699 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:58:08.717665    2544 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-b7c9ff648-nvcth"
Sep 16 18:58:08.841568 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:58:08.841514    2544 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-stop-predictor-7bb9c548b5-ln4lk"]
Sep 16 18:58:08.841775 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:58:08.841750    2544 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-7bb9c548b5-ln4lk" podUID="09fd6613-c8cb-4b5d-b5b5-75a2e70d9adb" containerName="kserve-container" containerID="cri-o://fdb9c6f4a725af0dbfd5dbac21545d9e50c697111b1b18b2ba497323dc24a2be" gracePeriod=30
Sep 16 18:58:08.841842 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:58:08.841792    2544 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-7bb9c548b5-ln4lk" podUID="09fd6613-c8cb-4b5d-b5b5-75a2e70d9adb" containerName="kube-rbac-proxy" containerID="cri-o://11acfbb3890d82fb84418d2b709912383df14723d33d69d8e23d2e9ea4f55cd2" gracePeriod=30
Sep 16 18:58:09.630548 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:58:09.630512    2544 generic.go:356] "Generic (PLEG): container finished" podID="09fd6613-c8cb-4b5d-b5b5-75a2e70d9adb" containerID="11acfbb3890d82fb84418d2b709912383df14723d33d69d8e23d2e9ea4f55cd2" exitCode=2
Sep 16 18:58:09.630731 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:58:09.630589    2544 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-7bb9c548b5-ln4lk" event={"ID":"09fd6613-c8cb-4b5d-b5b5-75a2e70d9adb","Type":"ContainerDied","Data":"11acfbb3890d82fb84418d2b709912383df14723d33d69d8e23d2e9ea4f55cd2"}
Sep 16 18:58:10.177438 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:58:10.177389    2544 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-7bb9c548b5-ln4lk" podUID="09fd6613-c8cb-4b5d-b5b5-75a2e70d9adb" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.30:8643/healthz\": dial tcp 10.132.0.30:8643: connect: connection refused"
Sep 16 18:58:12.228110 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:58:12.228074    2544 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-b7c9ff648-nvcth"]
Sep 16 18:58:12.228567 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:58:12.228377    2544 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-b7c9ff648-nvcth" podUID="7ae7a71d-424c-4d16-8916-642cfa6e34b1" containerName="kserve-container" containerID="cri-o://c006c91df31e32a45d918a30163000fd4ce9da92f092a18c41dd134f8b2d48d2" gracePeriod=30
Sep 16 18:58:12.228712 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:58:12.228682    2544 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-b7c9ff648-nvcth" podUID="7ae7a71d-424c-4d16-8916-642cfa6e34b1" containerName="kube-rbac-proxy" containerID="cri-o://8b918eee02e5ebf4ce91f7f4908d7a8c1a34a432981d135039f85ec8b8da29c0" gracePeriod=30
Sep 16 18:58:12.648278 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:58:12.648242    2544 generic.go:356] "Generic (PLEG): container finished" podID="7ae7a71d-424c-4d16-8916-642cfa6e34b1" containerID="8b918eee02e5ebf4ce91f7f4908d7a8c1a34a432981d135039f85ec8b8da29c0" exitCode=2
Sep 16 18:58:12.648446 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:58:12.648314    2544 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-b7c9ff648-nvcth" event={"ID":"7ae7a71d-424c-4d16-8916-642cfa6e34b1","Type":"ContainerDied","Data":"8b918eee02e5ebf4ce91f7f4908d7a8c1a34a432981d135039f85ec8b8da29c0"}
Sep 16 18:58:12.718021 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:58:12.717978    2544 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-7bb9c548b5-ln4lk" podUID="09fd6613-c8cb-4b5d-b5b5-75a2e70d9adb" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.30:8080: connect: connection refused"
Sep 16 18:58:13.387334 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:58:13.387290    2544 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-22075-predictor-5799df6789-ns959" podUID="2985e5ca-4da7-4cd5-911e-39decc92e56d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.32:8080: connect: connection refused"
Sep 16 18:58:14.172208 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:58:14.172175    2544 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-b7c9ff648-nvcth" podUID="7ae7a71d-424c-4d16-8916-642cfa6e34b1" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.29:8643/healthz\": dial tcp 10.132.0.29:8643: connect: connection refused"
Sep 16 18:58:14.182526 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:58:14.182495    2544 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-7bb9c548b5-ln4lk"
Sep 16 18:58:14.238930 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:58:14.238901    2544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/09fd6613-c8cb-4b5d-b5b5-75a2e70d9adb-cabundle-cert\") pod \"09fd6613-c8cb-4b5d-b5b5-75a2e70d9adb\" (UID: \"09fd6613-c8cb-4b5d-b5b5-75a2e70d9adb\") "
Sep 16 18:58:14.238930 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:58:14.238936    2544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/09fd6613-c8cb-4b5d-b5b5-75a2e70d9adb-proxy-tls\") pod \"09fd6613-c8cb-4b5d-b5b5-75a2e70d9adb\" (UID: \"09fd6613-c8cb-4b5d-b5b5-75a2e70d9adb\") "
Sep 16 18:58:14.239202 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:58:14.238955    2544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-knljz\" (UniqueName: \"kubernetes.io/projected/09fd6613-c8cb-4b5d-b5b5-75a2e70d9adb-kube-api-access-knljz\") pod \"09fd6613-c8cb-4b5d-b5b5-75a2e70d9adb\" (UID: \"09fd6613-c8cb-4b5d-b5b5-75a2e70d9adb\") "
Sep 16 18:58:14.239202 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:58:14.239000    2544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/09fd6613-c8cb-4b5d-b5b5-75a2e70d9adb-kube-rbac-proxy-sar-config\") pod \"09fd6613-c8cb-4b5d-b5b5-75a2e70d9adb\" (UID: \"09fd6613-c8cb-4b5d-b5b5-75a2e70d9adb\") "
Sep 16 18:58:14.239202 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:58:14.239042    2544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/09fd6613-c8cb-4b5d-b5b5-75a2e70d9adb-kserve-provision-location\") pod \"09fd6613-c8cb-4b5d-b5b5-75a2e70d9adb\" (UID: \"09fd6613-c8cb-4b5d-b5b5-75a2e70d9adb\") "
Sep 16 18:58:14.239398 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:58:14.239371    2544 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/09fd6613-c8cb-4b5d-b5b5-75a2e70d9adb-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "09fd6613-c8cb-4b5d-b5b5-75a2e70d9adb" (UID: "09fd6613-c8cb-4b5d-b5b5-75a2e70d9adb"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 18:58:14.239437 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:58:14.239403    2544 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/09fd6613-c8cb-4b5d-b5b5-75a2e70d9adb-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "09fd6613-c8cb-4b5d-b5b5-75a2e70d9adb" (UID: "09fd6613-c8cb-4b5d-b5b5-75a2e70d9adb"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 18:58:14.239437 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:58:14.239384    2544 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/09fd6613-c8cb-4b5d-b5b5-75a2e70d9adb-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "09fd6613-c8cb-4b5d-b5b5-75a2e70d9adb" (UID: "09fd6613-c8cb-4b5d-b5b5-75a2e70d9adb"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 18:58:14.241530 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:58:14.241509    2544 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/09fd6613-c8cb-4b5d-b5b5-75a2e70d9adb-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "09fd6613-c8cb-4b5d-b5b5-75a2e70d9adb" (UID: "09fd6613-c8cb-4b5d-b5b5-75a2e70d9adb"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 18:58:14.241530 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:58:14.241518    2544 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/09fd6613-c8cb-4b5d-b5b5-75a2e70d9adb-kube-api-access-knljz" (OuterVolumeSpecName: "kube-api-access-knljz") pod "09fd6613-c8cb-4b5d-b5b5-75a2e70d9adb" (UID: "09fd6613-c8cb-4b5d-b5b5-75a2e70d9adb"). InnerVolumeSpecName "kube-api-access-knljz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 18:58:14.284866 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:58:14.284826    2544 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-22075-predictor-754b4f7b64-9stpg" podUID="bb199f20-09bc-4c3f-aa96-d9a976f62292" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.31:8080: connect: connection refused"
Sep 16 18:58:14.340273 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:58:14.340238    2544 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/09fd6613-c8cb-4b5d-b5b5-75a2e70d9adb-cabundle-cert\") on node \"ip-10-0-132-80.ec2.internal\" DevicePath \"\""
Sep 16 18:58:14.340273 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:58:14.340272    2544 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/09fd6613-c8cb-4b5d-b5b5-75a2e70d9adb-proxy-tls\") on node \"ip-10-0-132-80.ec2.internal\" DevicePath \"\""
Sep 16 18:58:14.340442 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:58:14.340286    2544 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-knljz\" (UniqueName: \"kubernetes.io/projected/09fd6613-c8cb-4b5d-b5b5-75a2e70d9adb-kube-api-access-knljz\") on node \"ip-10-0-132-80.ec2.internal\" DevicePath \"\""
Sep 16 18:58:14.340442 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:58:14.340296    2544 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/09fd6613-c8cb-4b5d-b5b5-75a2e70d9adb-kube-rbac-proxy-sar-config\") on node \"ip-10-0-132-80.ec2.internal\" DevicePath \"\""
Sep 16 18:58:14.340442 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:58:14.340307    2544 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/09fd6613-c8cb-4b5d-b5b5-75a2e70d9adb-kserve-provision-location\") on node \"ip-10-0-132-80.ec2.internal\" DevicePath \"\""
Sep 16 18:58:14.659567 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:58:14.659515    2544 generic.go:356] "Generic (PLEG): container finished" podID="09fd6613-c8cb-4b5d-b5b5-75a2e70d9adb" containerID="fdb9c6f4a725af0dbfd5dbac21545d9e50c697111b1b18b2ba497323dc24a2be" exitCode=0
Sep 16 18:58:14.659956 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:58:14.659594    2544 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-7bb9c548b5-ln4lk"
Sep 16 18:58:14.659956 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:58:14.659594    2544 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-7bb9c548b5-ln4lk" event={"ID":"09fd6613-c8cb-4b5d-b5b5-75a2e70d9adb","Type":"ContainerDied","Data":"fdb9c6f4a725af0dbfd5dbac21545d9e50c697111b1b18b2ba497323dc24a2be"}
Sep 16 18:58:14.659956 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:58:14.659667    2544 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-7bb9c548b5-ln4lk" event={"ID":"09fd6613-c8cb-4b5d-b5b5-75a2e70d9adb","Type":"ContainerDied","Data":"00c03ac85d7dbfb0f42eb318ec9bc5b4b9b5ba6aa37f9b0e3474292d65c6eb4a"}
Sep 16 18:58:14.659956 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:58:14.659705    2544 scope.go:117] "RemoveContainer" containerID="11acfbb3890d82fb84418d2b709912383df14723d33d69d8e23d2e9ea4f55cd2"
Sep 16 18:58:14.668725 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:58:14.668709    2544 scope.go:117] "RemoveContainer" containerID="fdb9c6f4a725af0dbfd5dbac21545d9e50c697111b1b18b2ba497323dc24a2be"
Sep 16 18:58:14.675856 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:58:14.675838    2544 scope.go:117] "RemoveContainer" containerID="c78626128efa0bb53db1684ce57a2a42df71a696f7dc3eed8e144bd71f79014d"
Sep 16 18:58:14.680950 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:58:14.680930    2544 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-stop-predictor-7bb9c548b5-ln4lk"]
Sep 16 18:58:14.685356 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:58:14.685329    2544 scope.go:117] "RemoveContainer" containerID="11acfbb3890d82fb84418d2b709912383df14723d33d69d8e23d2e9ea4f55cd2"
Sep 16 18:58:14.686045 ip-10-0-132-80 kubenswrapper[2544]: E0916 18:58:14.686019    2544 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"11acfbb3890d82fb84418d2b709912383df14723d33d69d8e23d2e9ea4f55cd2\": container with ID starting with 11acfbb3890d82fb84418d2b709912383df14723d33d69d8e23d2e9ea4f55cd2 not found: ID does not exist" containerID="11acfbb3890d82fb84418d2b709912383df14723d33d69d8e23d2e9ea4f55cd2"
Sep 16 18:58:14.686045 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:58:14.686054    2544 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"11acfbb3890d82fb84418d2b709912383df14723d33d69d8e23d2e9ea4f55cd2"} err="failed to get container status \"11acfbb3890d82fb84418d2b709912383df14723d33d69d8e23d2e9ea4f55cd2\": rpc error: code = NotFound desc = could not find container \"11acfbb3890d82fb84418d2b709912383df14723d33d69d8e23d2e9ea4f55cd2\": container with ID starting with 11acfbb3890d82fb84418d2b709912383df14723d33d69d8e23d2e9ea4f55cd2 not found: ID does not exist"
Sep 16 18:58:14.686045 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:58:14.686076    2544 scope.go:117] "RemoveContainer" containerID="fdb9c6f4a725af0dbfd5dbac21545d9e50c697111b1b18b2ba497323dc24a2be"
Sep 16 18:58:14.686415 ip-10-0-132-80 kubenswrapper[2544]: E0916 18:58:14.686388    2544 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fdb9c6f4a725af0dbfd5dbac21545d9e50c697111b1b18b2ba497323dc24a2be\": container with ID starting with fdb9c6f4a725af0dbfd5dbac21545d9e50c697111b1b18b2ba497323dc24a2be not found: ID does not exist" containerID="fdb9c6f4a725af0dbfd5dbac21545d9e50c697111b1b18b2ba497323dc24a2be"
Sep 16 18:58:14.686460 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:58:14.686420    2544 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fdb9c6f4a725af0dbfd5dbac21545d9e50c697111b1b18b2ba497323dc24a2be"} err="failed to get container status \"fdb9c6f4a725af0dbfd5dbac21545d9e50c697111b1b18b2ba497323dc24a2be\": rpc error: code = NotFound desc = could not find container \"fdb9c6f4a725af0dbfd5dbac21545d9e50c697111b1b18b2ba497323dc24a2be\": container with ID starting with fdb9c6f4a725af0dbfd5dbac21545d9e50c697111b1b18b2ba497323dc24a2be not found: ID does not exist"
Sep 16 18:58:14.686460 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:58:14.686436    2544 scope.go:117] "RemoveContainer" containerID="c78626128efa0bb53db1684ce57a2a42df71a696f7dc3eed8e144bd71f79014d"
Sep 16 18:58:14.686739 ip-10-0-132-80 kubenswrapper[2544]: E0916 18:58:14.686719    2544 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c78626128efa0bb53db1684ce57a2a42df71a696f7dc3eed8e144bd71f79014d\": container with ID starting with c78626128efa0bb53db1684ce57a2a42df71a696f7dc3eed8e144bd71f79014d not found: ID does not exist" containerID="c78626128efa0bb53db1684ce57a2a42df71a696f7dc3eed8e144bd71f79014d"
Sep 16 18:58:14.686804 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:58:14.686744    2544 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c78626128efa0bb53db1684ce57a2a42df71a696f7dc3eed8e144bd71f79014d"} err="failed to get container status \"c78626128efa0bb53db1684ce57a2a42df71a696f7dc3eed8e144bd71f79014d\": rpc error: code = NotFound desc = could not find container \"c78626128efa0bb53db1684ce57a2a42df71a696f7dc3eed8e144bd71f79014d\": container with ID starting with c78626128efa0bb53db1684ce57a2a42df71a696f7dc3eed8e144bd71f79014d not found: ID does not exist"
Sep 16 18:58:14.688078 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:58:14.688059    2544 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-stop-predictor-7bb9c548b5-ln4lk"]
Sep 16 18:58:15.721603 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:58:15.721547    2544 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="09fd6613-c8cb-4b5d-b5b5-75a2e70d9adb" path="/var/lib/kubelet/pods/09fd6613-c8cb-4b5d-b5b5-75a2e70d9adb/volumes"
Sep 16 18:58:16.975486 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:58:16.975462    2544 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-b7c9ff648-nvcth"
Sep 16 18:58:17.068743 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:58:17.068717    2544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/7ae7a71d-424c-4d16-8916-642cfa6e34b1-proxy-tls\") pod \"7ae7a71d-424c-4d16-8916-642cfa6e34b1\" (UID: \"7ae7a71d-424c-4d16-8916-642cfa6e34b1\") "
Sep 16 18:58:17.068916 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:58:17.068761    2544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5cljp\" (UniqueName: \"kubernetes.io/projected/7ae7a71d-424c-4d16-8916-642cfa6e34b1-kube-api-access-5cljp\") pod \"7ae7a71d-424c-4d16-8916-642cfa6e34b1\" (UID: \"7ae7a71d-424c-4d16-8916-642cfa6e34b1\") "
Sep 16 18:58:17.068916 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:58:17.068795    2544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/7ae7a71d-424c-4d16-8916-642cfa6e34b1-kube-rbac-proxy-sar-config\") pod \"7ae7a71d-424c-4d16-8916-642cfa6e34b1\" (UID: \"7ae7a71d-424c-4d16-8916-642cfa6e34b1\") "
Sep 16 18:58:17.068916 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:58:17.068814    2544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/7ae7a71d-424c-4d16-8916-642cfa6e34b1-cabundle-cert\") pod \"7ae7a71d-424c-4d16-8916-642cfa6e34b1\" (UID: \"7ae7a71d-424c-4d16-8916-642cfa6e34b1\") "
Sep 16 18:58:17.068916 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:58:17.068868    2544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/7ae7a71d-424c-4d16-8916-642cfa6e34b1-kserve-provision-location\") pod \"7ae7a71d-424c-4d16-8916-642cfa6e34b1\" (UID: \"7ae7a71d-424c-4d16-8916-642cfa6e34b1\") "
Sep 16 18:58:17.069241 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:58:17.069214    2544 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7ae7a71d-424c-4d16-8916-642cfa6e34b1-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "7ae7a71d-424c-4d16-8916-642cfa6e34b1" (UID: "7ae7a71d-424c-4d16-8916-642cfa6e34b1"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 18:58:17.069309 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:58:17.069212    2544 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7ae7a71d-424c-4d16-8916-642cfa6e34b1-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "7ae7a71d-424c-4d16-8916-642cfa6e34b1" (UID: "7ae7a71d-424c-4d16-8916-642cfa6e34b1"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 18:58:17.069309 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:58:17.069220    2544 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7ae7a71d-424c-4d16-8916-642cfa6e34b1-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "7ae7a71d-424c-4d16-8916-642cfa6e34b1" (UID: "7ae7a71d-424c-4d16-8916-642cfa6e34b1"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 18:58:17.070920 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:58:17.070896    2544 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7ae7a71d-424c-4d16-8916-642cfa6e34b1-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "7ae7a71d-424c-4d16-8916-642cfa6e34b1" (UID: "7ae7a71d-424c-4d16-8916-642cfa6e34b1"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 18:58:17.070920 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:58:17.070912    2544 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7ae7a71d-424c-4d16-8916-642cfa6e34b1-kube-api-access-5cljp" (OuterVolumeSpecName: "kube-api-access-5cljp") pod "7ae7a71d-424c-4d16-8916-642cfa6e34b1" (UID: "7ae7a71d-424c-4d16-8916-642cfa6e34b1"). InnerVolumeSpecName "kube-api-access-5cljp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 18:58:17.169950 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:58:17.169913    2544 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5cljp\" (UniqueName: \"kubernetes.io/projected/7ae7a71d-424c-4d16-8916-642cfa6e34b1-kube-api-access-5cljp\") on node \"ip-10-0-132-80.ec2.internal\" DevicePath \"\""
Sep 16 18:58:17.169950 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:58:17.169943    2544 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/7ae7a71d-424c-4d16-8916-642cfa6e34b1-kube-rbac-proxy-sar-config\") on node \"ip-10-0-132-80.ec2.internal\" DevicePath \"\""
Sep 16 18:58:17.169950 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:58:17.169953    2544 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/7ae7a71d-424c-4d16-8916-642cfa6e34b1-cabundle-cert\") on node \"ip-10-0-132-80.ec2.internal\" DevicePath \"\""
Sep 16 18:58:17.170174 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:58:17.169962    2544 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/7ae7a71d-424c-4d16-8916-642cfa6e34b1-kserve-provision-location\") on node \"ip-10-0-132-80.ec2.internal\" DevicePath \"\""
Sep 16 18:58:17.170174 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:58:17.169971    2544 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/7ae7a71d-424c-4d16-8916-642cfa6e34b1-proxy-tls\") on node \"ip-10-0-132-80.ec2.internal\" DevicePath \"\""
Sep 16 18:58:17.676541 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:58:17.676502    2544 generic.go:356] "Generic (PLEG): container finished" podID="7ae7a71d-424c-4d16-8916-642cfa6e34b1" containerID="c006c91df31e32a45d918a30163000fd4ce9da92f092a18c41dd134f8b2d48d2" exitCode=0
Sep 16 18:58:17.676749 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:58:17.676580    2544 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-b7c9ff648-nvcth"
Sep 16 18:58:17.676749 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:58:17.676582    2544 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-b7c9ff648-nvcth" event={"ID":"7ae7a71d-424c-4d16-8916-642cfa6e34b1","Type":"ContainerDied","Data":"c006c91df31e32a45d918a30163000fd4ce9da92f092a18c41dd134f8b2d48d2"}
Sep 16 18:58:17.676749 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:58:17.676645    2544 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-b7c9ff648-nvcth" event={"ID":"7ae7a71d-424c-4d16-8916-642cfa6e34b1","Type":"ContainerDied","Data":"5d5a29c840c662bac5fe0f0a22fc1f72fad644ea60fd3c6eeb49705e6e34ccae"}
Sep 16 18:58:17.676749 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:58:17.676673    2544 scope.go:117] "RemoveContainer" containerID="8b918eee02e5ebf4ce91f7f4908d7a8c1a34a432981d135039f85ec8b8da29c0"
Sep 16 18:58:17.685694 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:58:17.685677    2544 scope.go:117] "RemoveContainer" containerID="c006c91df31e32a45d918a30163000fd4ce9da92f092a18c41dd134f8b2d48d2"
Sep 16 18:58:17.692849 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:58:17.692833    2544 scope.go:117] "RemoveContainer" containerID="23d0b7d6e8f486fc8564b72c2418041d61a0ff182b815810aa75904bf6334225"
Sep 16 18:58:17.698978 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:58:17.698952    2544 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-b7c9ff648-nvcth"]
Sep 16 18:58:17.701958 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:58:17.701934    2544 scope.go:117] "RemoveContainer" containerID="8b918eee02e5ebf4ce91f7f4908d7a8c1a34a432981d135039f85ec8b8da29c0"
Sep 16 18:58:17.702245 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:58:17.702222    2544 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-b7c9ff648-nvcth"]
Sep 16 18:58:17.702325 ip-10-0-132-80 kubenswrapper[2544]: E0916 18:58:17.702255    2544 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8b918eee02e5ebf4ce91f7f4908d7a8c1a34a432981d135039f85ec8b8da29c0\": container with ID starting with 8b918eee02e5ebf4ce91f7f4908d7a8c1a34a432981d135039f85ec8b8da29c0 not found: ID does not exist" containerID="8b918eee02e5ebf4ce91f7f4908d7a8c1a34a432981d135039f85ec8b8da29c0"
Sep 16 18:58:17.702325 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:58:17.702292    2544 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8b918eee02e5ebf4ce91f7f4908d7a8c1a34a432981d135039f85ec8b8da29c0"} err="failed to get container status \"8b918eee02e5ebf4ce91f7f4908d7a8c1a34a432981d135039f85ec8b8da29c0\": rpc error: code = NotFound desc = could not find container \"8b918eee02e5ebf4ce91f7f4908d7a8c1a34a432981d135039f85ec8b8da29c0\": container with ID starting with 8b918eee02e5ebf4ce91f7f4908d7a8c1a34a432981d135039f85ec8b8da29c0 not found: ID does not exist"
Sep 16 18:58:17.702325 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:58:17.702317    2544 scope.go:117] "RemoveContainer" containerID="c006c91df31e32a45d918a30163000fd4ce9da92f092a18c41dd134f8b2d48d2"
Sep 16 18:58:17.702635 ip-10-0-132-80 kubenswrapper[2544]: E0916 18:58:17.702618    2544 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c006c91df31e32a45d918a30163000fd4ce9da92f092a18c41dd134f8b2d48d2\": container with ID starting with c006c91df31e32a45d918a30163000fd4ce9da92f092a18c41dd134f8b2d48d2 not found: ID does not exist" containerID="c006c91df31e32a45d918a30163000fd4ce9da92f092a18c41dd134f8b2d48d2"
Sep 16 18:58:17.702704 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:58:17.702655    2544 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c006c91df31e32a45d918a30163000fd4ce9da92f092a18c41dd134f8b2d48d2"} err="failed to get container status \"c006c91df31e32a45d918a30163000fd4ce9da92f092a18c41dd134f8b2d48d2\": rpc error: code = NotFound desc = could not find container \"c006c91df31e32a45d918a30163000fd4ce9da92f092a18c41dd134f8b2d48d2\": container with ID starting with c006c91df31e32a45d918a30163000fd4ce9da92f092a18c41dd134f8b2d48d2 not found: ID does not exist"
Sep 16 18:58:17.702704 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:58:17.702678    2544 scope.go:117] "RemoveContainer" containerID="23d0b7d6e8f486fc8564b72c2418041d61a0ff182b815810aa75904bf6334225"
Sep 16 18:58:17.703015 ip-10-0-132-80 kubenswrapper[2544]: E0916 18:58:17.702979    2544 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"23d0b7d6e8f486fc8564b72c2418041d61a0ff182b815810aa75904bf6334225\": container with ID starting with 23d0b7d6e8f486fc8564b72c2418041d61a0ff182b815810aa75904bf6334225 not found: ID does not exist" containerID="23d0b7d6e8f486fc8564b72c2418041d61a0ff182b815810aa75904bf6334225"
Sep 16 18:58:17.703063 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:58:17.703024    2544 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"23d0b7d6e8f486fc8564b72c2418041d61a0ff182b815810aa75904bf6334225"} err="failed to get container status \"23d0b7d6e8f486fc8564b72c2418041d61a0ff182b815810aa75904bf6334225\": rpc error: code = NotFound desc = could not find container \"23d0b7d6e8f486fc8564b72c2418041d61a0ff182b815810aa75904bf6334225\": container with ID starting with 23d0b7d6e8f486fc8564b72c2418041d61a0ff182b815810aa75904bf6334225 not found: ID does not exist"
Sep 16 18:58:17.721561 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:58:17.721531    2544 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7ae7a71d-424c-4d16-8916-642cfa6e34b1" path="/var/lib/kubelet/pods/7ae7a71d-424c-4d16-8916-642cfa6e34b1/volumes"
Sep 16 18:58:20.717678 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:58:20.717643    2544 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-22075-predictor-754b4f7b64-9stpg"
Sep 16 18:58:23.387267 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:58:23.387238    2544 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-22075-predictor-5799df6789-ns959"
Sep 16 18:58:42.610687 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:58:42.610642    2544 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-22075-predictor-754b4f7b64-9stpg"]
Sep 16 18:58:42.611132 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:58:42.610932    2544 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-22075-predictor-754b4f7b64-9stpg" podUID="bb199f20-09bc-4c3f-aa96-d9a976f62292" containerName="kserve-container" containerID="cri-o://b73ca7b84c824cf9ccf800d19bf6879351d2971f29f34ee8944ea449b0f6a655" gracePeriod=30
Sep 16 18:58:42.611132 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:58:42.611011    2544 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-22075-predictor-754b4f7b64-9stpg" podUID="bb199f20-09bc-4c3f-aa96-d9a976f62292" containerName="kube-rbac-proxy" containerID="cri-o://226f7c4b4845bc5ae74b981c2d9c2f8de12ed573a24f667d6bc99a8a0bf7ce80" gracePeriod=30
Sep 16 18:58:42.628992 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:58:42.628964    2544 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-22075-predictor-5799df6789-ns959"]
Sep 16 18:58:42.629243 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:58:42.629206    2544 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-22075-predictor-5799df6789-ns959" podUID="2985e5ca-4da7-4cd5-911e-39decc92e56d" containerName="kserve-container" containerID="cri-o://c13d0d59a9b5f662e46fe2c176f415862eaf6baa831dd5eb0bf6da6fe83bbbf9" gracePeriod=30
Sep 16 18:58:42.629338 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:58:42.629297    2544 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-22075-predictor-5799df6789-ns959" podUID="2985e5ca-4da7-4cd5-911e-39decc92e56d" containerName="kube-rbac-proxy" containerID="cri-o://23e62e625222e4c7b46bb1ce0de8ab0411cb562b705cb75c4f7fcfbf93a4adb2" gracePeriod=30
Sep 16 18:58:42.794843 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:58:42.794809    2544 generic.go:356] "Generic (PLEG): container finished" podID="bb199f20-09bc-4c3f-aa96-d9a976f62292" containerID="226f7c4b4845bc5ae74b981c2d9c2f8de12ed573a24f667d6bc99a8a0bf7ce80" exitCode=2
Sep 16 18:58:42.795006 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:58:42.794882    2544 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-22075-predictor-754b4f7b64-9stpg" event={"ID":"bb199f20-09bc-4c3f-aa96-d9a976f62292","Type":"ContainerDied","Data":"226f7c4b4845bc5ae74b981c2d9c2f8de12ed573a24f667d6bc99a8a0bf7ce80"}
Sep 16 18:58:42.797439 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:58:42.797417    2544 generic.go:356] "Generic (PLEG): container finished" podID="2985e5ca-4da7-4cd5-911e-39decc92e56d" containerID="23e62e625222e4c7b46bb1ce0de8ab0411cb562b705cb75c4f7fcfbf93a4adb2" exitCode=2
Sep 16 18:58:42.797565 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:58:42.797477    2544 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-22075-predictor-5799df6789-ns959" event={"ID":"2985e5ca-4da7-4cd5-911e-39decc92e56d","Type":"ContainerDied","Data":"23e62e625222e4c7b46bb1ce0de8ab0411cb562b705cb75c4f7fcfbf93a4adb2"}
Sep 16 18:58:43.382662 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:58:43.382615    2544 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-22075-predictor-5799df6789-ns959" podUID="2985e5ca-4da7-4cd5-911e-39decc92e56d" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.32:8643/healthz\": dial tcp 10.132.0.32:8643: connect: connection refused"
Sep 16 18:58:43.386945 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:58:43.386923    2544 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-22075-predictor-5799df6789-ns959" podUID="2985e5ca-4da7-4cd5-911e-39decc92e56d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.32:8080: connect: connection refused"
Sep 16 18:58:44.277111 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:58:44.277065    2544 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-22075-predictor-754b4f7b64-9stpg" podUID="bb199f20-09bc-4c3f-aa96-d9a976f62292" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.31:8643/healthz\": dial tcp 10.132.0.31:8643: connect: connection refused"
Sep 16 18:58:46.174639 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:58:46.174614    2544 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-22075-predictor-5799df6789-ns959"
Sep 16 18:58:46.219627 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:58:46.219598    2544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-n5mp5\" (UniqueName: \"kubernetes.io/projected/2985e5ca-4da7-4cd5-911e-39decc92e56d-kube-api-access-n5mp5\") pod \"2985e5ca-4da7-4cd5-911e-39decc92e56d\" (UID: \"2985e5ca-4da7-4cd5-911e-39decc92e56d\") "
Sep 16 18:58:46.219778 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:58:46.219649    2544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/2985e5ca-4da7-4cd5-911e-39decc92e56d-proxy-tls\") pod \"2985e5ca-4da7-4cd5-911e-39decc92e56d\" (UID: \"2985e5ca-4da7-4cd5-911e-39decc92e56d\") "
Sep 16 18:58:46.219778 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:58:46.219667    2544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/2985e5ca-4da7-4cd5-911e-39decc92e56d-kube-rbac-proxy-sar-config\") pod \"2985e5ca-4da7-4cd5-911e-39decc92e56d\" (UID: \"2985e5ca-4da7-4cd5-911e-39decc92e56d\") "
Sep 16 18:58:46.219778 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:58:46.219696    2544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/2985e5ca-4da7-4cd5-911e-39decc92e56d-kserve-provision-location\") pod \"2985e5ca-4da7-4cd5-911e-39decc92e56d\" (UID: \"2985e5ca-4da7-4cd5-911e-39decc92e56d\") "
Sep 16 18:58:46.219778 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:58:46.219741    2544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/2985e5ca-4da7-4cd5-911e-39decc92e56d-cabundle-cert\") pod \"2985e5ca-4da7-4cd5-911e-39decc92e56d\" (UID: \"2985e5ca-4da7-4cd5-911e-39decc92e56d\") "
Sep 16 18:58:46.220085 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:58:46.220057    2544 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2985e5ca-4da7-4cd5-911e-39decc92e56d-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "2985e5ca-4da7-4cd5-911e-39decc92e56d" (UID: "2985e5ca-4da7-4cd5-911e-39decc92e56d"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 18:58:46.220210 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:58:46.220101    2544 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2985e5ca-4da7-4cd5-911e-39decc92e56d-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "2985e5ca-4da7-4cd5-911e-39decc92e56d" (UID: "2985e5ca-4da7-4cd5-911e-39decc92e56d"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 18:58:46.220210 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:58:46.220103    2544 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2985e5ca-4da7-4cd5-911e-39decc92e56d-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "2985e5ca-4da7-4cd5-911e-39decc92e56d" (UID: "2985e5ca-4da7-4cd5-911e-39decc92e56d"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 18:58:46.221741 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:58:46.221722    2544 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2985e5ca-4da7-4cd5-911e-39decc92e56d-kube-api-access-n5mp5" (OuterVolumeSpecName: "kube-api-access-n5mp5") pod "2985e5ca-4da7-4cd5-911e-39decc92e56d" (UID: "2985e5ca-4da7-4cd5-911e-39decc92e56d"). InnerVolumeSpecName "kube-api-access-n5mp5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 18:58:46.221967 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:58:46.221946    2544 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2985e5ca-4da7-4cd5-911e-39decc92e56d-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "2985e5ca-4da7-4cd5-911e-39decc92e56d" (UID: "2985e5ca-4da7-4cd5-911e-39decc92e56d"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 18:58:46.321010 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:58:46.320983    2544 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/2985e5ca-4da7-4cd5-911e-39decc92e56d-cabundle-cert\") on node \"ip-10-0-132-80.ec2.internal\" DevicePath \"\""
Sep 16 18:58:46.321010 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:58:46.321009    2544 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n5mp5\" (UniqueName: \"kubernetes.io/projected/2985e5ca-4da7-4cd5-911e-39decc92e56d-kube-api-access-n5mp5\") on node \"ip-10-0-132-80.ec2.internal\" DevicePath \"\""
Sep 16 18:58:46.321192 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:58:46.321021    2544 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/2985e5ca-4da7-4cd5-911e-39decc92e56d-proxy-tls\") on node \"ip-10-0-132-80.ec2.internal\" DevicePath \"\""
Sep 16 18:58:46.321192 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:58:46.321031    2544 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/2985e5ca-4da7-4cd5-911e-39decc92e56d-kube-rbac-proxy-sar-config\") on node \"ip-10-0-132-80.ec2.internal\" DevicePath \"\""
Sep 16 18:58:46.321192 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:58:46.321040    2544 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/2985e5ca-4da7-4cd5-911e-39decc92e56d-kserve-provision-location\") on node \"ip-10-0-132-80.ec2.internal\" DevicePath \"\""
Sep 16 18:58:46.818562 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:58:46.818517    2544 generic.go:356] "Generic (PLEG): container finished" podID="2985e5ca-4da7-4cd5-911e-39decc92e56d" containerID="c13d0d59a9b5f662e46fe2c176f415862eaf6baa831dd5eb0bf6da6fe83bbbf9" exitCode=0
Sep 16 18:58:46.818753 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:58:46.818612    2544 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-22075-predictor-5799df6789-ns959"
Sep 16 18:58:46.818753 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:58:46.818604    2544 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-22075-predictor-5799df6789-ns959" event={"ID":"2985e5ca-4da7-4cd5-911e-39decc92e56d","Type":"ContainerDied","Data":"c13d0d59a9b5f662e46fe2c176f415862eaf6baa831dd5eb0bf6da6fe83bbbf9"}
Sep 16 18:58:46.818753 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:58:46.818743    2544 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-22075-predictor-5799df6789-ns959" event={"ID":"2985e5ca-4da7-4cd5-911e-39decc92e56d","Type":"ContainerDied","Data":"2b8603cfe409efd2a1a807f10f8837f40bdeeaca4cae615d3cfd5a65feb454cb"}
Sep 16 18:58:46.818905 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:58:46.818770    2544 scope.go:117] "RemoveContainer" containerID="23e62e625222e4c7b46bb1ce0de8ab0411cb562b705cb75c4f7fcfbf93a4adb2"
Sep 16 18:58:46.827406 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:58:46.827390    2544 scope.go:117] "RemoveContainer" containerID="c13d0d59a9b5f662e46fe2c176f415862eaf6baa831dd5eb0bf6da6fe83bbbf9"
Sep 16 18:58:46.834671 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:58:46.834649    2544 scope.go:117] "RemoveContainer" containerID="e2b8a30c9f17cd15ae23e43fbeca2f289afa0f9e3abf8be8fd0e7213649df077"
Sep 16 18:58:46.840766 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:58:46.840739    2544 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-22075-predictor-5799df6789-ns959"]
Sep 16 18:58:46.842723 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:58:46.842703    2544 scope.go:117] "RemoveContainer" containerID="23e62e625222e4c7b46bb1ce0de8ab0411cb562b705cb75c4f7fcfbf93a4adb2"
Sep 16 18:58:46.843025 ip-10-0-132-80 kubenswrapper[2544]: E0916 18:58:46.843005    2544 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"23e62e625222e4c7b46bb1ce0de8ab0411cb562b705cb75c4f7fcfbf93a4adb2\": container with ID starting with 23e62e625222e4c7b46bb1ce0de8ab0411cb562b705cb75c4f7fcfbf93a4adb2 not found: ID does not exist" containerID="23e62e625222e4c7b46bb1ce0de8ab0411cb562b705cb75c4f7fcfbf93a4adb2"
Sep 16 18:58:46.843083 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:58:46.843035    2544 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"23e62e625222e4c7b46bb1ce0de8ab0411cb562b705cb75c4f7fcfbf93a4adb2"} err="failed to get container status \"23e62e625222e4c7b46bb1ce0de8ab0411cb562b705cb75c4f7fcfbf93a4adb2\": rpc error: code = NotFound desc = could not find container \"23e62e625222e4c7b46bb1ce0de8ab0411cb562b705cb75c4f7fcfbf93a4adb2\": container with ID starting with 23e62e625222e4c7b46bb1ce0de8ab0411cb562b705cb75c4f7fcfbf93a4adb2 not found: ID does not exist"
Sep 16 18:58:46.843083 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:58:46.843055    2544 scope.go:117] "RemoveContainer" containerID="c13d0d59a9b5f662e46fe2c176f415862eaf6baa831dd5eb0bf6da6fe83bbbf9"
Sep 16 18:58:46.843169 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:58:46.843117    2544 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-22075-predictor-5799df6789-ns959"]
Sep 16 18:58:46.843361 ip-10-0-132-80 kubenswrapper[2544]: E0916 18:58:46.843332    2544 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c13d0d59a9b5f662e46fe2c176f415862eaf6baa831dd5eb0bf6da6fe83bbbf9\": container with ID starting with c13d0d59a9b5f662e46fe2c176f415862eaf6baa831dd5eb0bf6da6fe83bbbf9 not found: ID does not exist" containerID="c13d0d59a9b5f662e46fe2c176f415862eaf6baa831dd5eb0bf6da6fe83bbbf9"
Sep 16 18:58:46.843432 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:58:46.843372    2544 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c13d0d59a9b5f662e46fe2c176f415862eaf6baa831dd5eb0bf6da6fe83bbbf9"} err="failed to get container status \"c13d0d59a9b5f662e46fe2c176f415862eaf6baa831dd5eb0bf6da6fe83bbbf9\": rpc error: code = NotFound desc = could not find container \"c13d0d59a9b5f662e46fe2c176f415862eaf6baa831dd5eb0bf6da6fe83bbbf9\": container with ID starting with c13d0d59a9b5f662e46fe2c176f415862eaf6baa831dd5eb0bf6da6fe83bbbf9 not found: ID does not exist"
Sep 16 18:58:46.843432 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:58:46.843395    2544 scope.go:117] "RemoveContainer" containerID="e2b8a30c9f17cd15ae23e43fbeca2f289afa0f9e3abf8be8fd0e7213649df077"
Sep 16 18:58:46.843700 ip-10-0-132-80 kubenswrapper[2544]: E0916 18:58:46.843681    2544 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e2b8a30c9f17cd15ae23e43fbeca2f289afa0f9e3abf8be8fd0e7213649df077\": container with ID starting with e2b8a30c9f17cd15ae23e43fbeca2f289afa0f9e3abf8be8fd0e7213649df077 not found: ID does not exist" containerID="e2b8a30c9f17cd15ae23e43fbeca2f289afa0f9e3abf8be8fd0e7213649df077"
Sep 16 18:58:46.843748 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:58:46.843704    2544 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e2b8a30c9f17cd15ae23e43fbeca2f289afa0f9e3abf8be8fd0e7213649df077"} err="failed to get container status \"e2b8a30c9f17cd15ae23e43fbeca2f289afa0f9e3abf8be8fd0e7213649df077\": rpc error: code = NotFound desc = could not find container \"e2b8a30c9f17cd15ae23e43fbeca2f289afa0f9e3abf8be8fd0e7213649df077\": container with ID starting with e2b8a30c9f17cd15ae23e43fbeca2f289afa0f9e3abf8be8fd0e7213649df077 not found: ID does not exist"
Sep 16 18:58:47.163851 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:58:47.163823    2544 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-22075-predictor-754b4f7b64-9stpg"
Sep 16 18:58:47.227280 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:58:47.227252    2544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-f5rqf\" (UniqueName: \"kubernetes.io/projected/bb199f20-09bc-4c3f-aa96-d9a976f62292-kube-api-access-f5rqf\") pod \"bb199f20-09bc-4c3f-aa96-d9a976f62292\" (UID: \"bb199f20-09bc-4c3f-aa96-d9a976f62292\") "
Sep 16 18:58:47.227280 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:58:47.227284    2544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/bb199f20-09bc-4c3f-aa96-d9a976f62292-proxy-tls\") pod \"bb199f20-09bc-4c3f-aa96-d9a976f62292\" (UID: \"bb199f20-09bc-4c3f-aa96-d9a976f62292\") "
Sep 16 18:58:47.227823 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:58:47.227312    2544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/bb199f20-09bc-4c3f-aa96-d9a976f62292-kube-rbac-proxy-sar-config\") pod \"bb199f20-09bc-4c3f-aa96-d9a976f62292\" (UID: \"bb199f20-09bc-4c3f-aa96-d9a976f62292\") "
Sep 16 18:58:47.227823 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:58:47.227362    2544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/bb199f20-09bc-4c3f-aa96-d9a976f62292-cabundle-cert\") pod \"bb199f20-09bc-4c3f-aa96-d9a976f62292\" (UID: \"bb199f20-09bc-4c3f-aa96-d9a976f62292\") "
Sep 16 18:58:47.227823 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:58:47.227386    2544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/bb199f20-09bc-4c3f-aa96-d9a976f62292-kserve-provision-location\") pod \"bb199f20-09bc-4c3f-aa96-d9a976f62292\" (UID: \"bb199f20-09bc-4c3f-aa96-d9a976f62292\") "
Sep 16 18:58:47.227823 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:58:47.227787    2544 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bb199f20-09bc-4c3f-aa96-d9a976f62292-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "bb199f20-09bc-4c3f-aa96-d9a976f62292" (UID: "bb199f20-09bc-4c3f-aa96-d9a976f62292"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 18:58:47.227823 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:58:47.227794    2544 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bb199f20-09bc-4c3f-aa96-d9a976f62292-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "bb199f20-09bc-4c3f-aa96-d9a976f62292" (UID: "bb199f20-09bc-4c3f-aa96-d9a976f62292"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 18:58:47.227823 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:58:47.227806    2544 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bb199f20-09bc-4c3f-aa96-d9a976f62292-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "bb199f20-09bc-4c3f-aa96-d9a976f62292" (UID: "bb199f20-09bc-4c3f-aa96-d9a976f62292"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 18:58:47.229422 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:58:47.229403    2544 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bb199f20-09bc-4c3f-aa96-d9a976f62292-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "bb199f20-09bc-4c3f-aa96-d9a976f62292" (UID: "bb199f20-09bc-4c3f-aa96-d9a976f62292"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 18:58:47.229521 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:58:47.229440    2544 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bb199f20-09bc-4c3f-aa96-d9a976f62292-kube-api-access-f5rqf" (OuterVolumeSpecName: "kube-api-access-f5rqf") pod "bb199f20-09bc-4c3f-aa96-d9a976f62292" (UID: "bb199f20-09bc-4c3f-aa96-d9a976f62292"). InnerVolumeSpecName "kube-api-access-f5rqf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 18:58:47.328704 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:58:47.328660    2544 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/bb199f20-09bc-4c3f-aa96-d9a976f62292-cabundle-cert\") on node \"ip-10-0-132-80.ec2.internal\" DevicePath \"\""
Sep 16 18:58:47.328704 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:58:47.328692    2544 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/bb199f20-09bc-4c3f-aa96-d9a976f62292-kserve-provision-location\") on node \"ip-10-0-132-80.ec2.internal\" DevicePath \"\""
Sep 16 18:58:47.328704 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:58:47.328702    2544 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f5rqf\" (UniqueName: \"kubernetes.io/projected/bb199f20-09bc-4c3f-aa96-d9a976f62292-kube-api-access-f5rqf\") on node \"ip-10-0-132-80.ec2.internal\" DevicePath \"\""
Sep 16 18:58:47.328704 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:58:47.328711    2544 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/bb199f20-09bc-4c3f-aa96-d9a976f62292-proxy-tls\") on node \"ip-10-0-132-80.ec2.internal\" DevicePath \"\""
Sep 16 18:58:47.328978 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:58:47.328721    2544 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/bb199f20-09bc-4c3f-aa96-d9a976f62292-kube-rbac-proxy-sar-config\") on node \"ip-10-0-132-80.ec2.internal\" DevicePath \"\""
Sep 16 18:58:47.721667 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:58:47.721634    2544 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2985e5ca-4da7-4cd5-911e-39decc92e56d" path="/var/lib/kubelet/pods/2985e5ca-4da7-4cd5-911e-39decc92e56d/volumes"
Sep 16 18:58:47.825277 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:58:47.825244    2544 generic.go:356] "Generic (PLEG): container finished" podID="bb199f20-09bc-4c3f-aa96-d9a976f62292" containerID="b73ca7b84c824cf9ccf800d19bf6879351d2971f29f34ee8944ea449b0f6a655" exitCode=0
Sep 16 18:58:47.825464 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:58:47.825298    2544 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-22075-predictor-754b4f7b64-9stpg"
Sep 16 18:58:47.825464 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:58:47.825329    2544 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-22075-predictor-754b4f7b64-9stpg" event={"ID":"bb199f20-09bc-4c3f-aa96-d9a976f62292","Type":"ContainerDied","Data":"b73ca7b84c824cf9ccf800d19bf6879351d2971f29f34ee8944ea449b0f6a655"}
Sep 16 18:58:47.825464 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:58:47.825401    2544 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-22075-predictor-754b4f7b64-9stpg" event={"ID":"bb199f20-09bc-4c3f-aa96-d9a976f62292","Type":"ContainerDied","Data":"4628b08b0734616ff954ec75f263b403d85db69a7b76c5166ac39bbc43eb4b46"}
Sep 16 18:58:47.825464 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:58:47.825439    2544 scope.go:117] "RemoveContainer" containerID="226f7c4b4845bc5ae74b981c2d9c2f8de12ed573a24f667d6bc99a8a0bf7ce80"
Sep 16 18:58:47.833854 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:58:47.833829    2544 scope.go:117] "RemoveContainer" containerID="b73ca7b84c824cf9ccf800d19bf6879351d2971f29f34ee8944ea449b0f6a655"
Sep 16 18:58:47.841167 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:58:47.841148    2544 scope.go:117] "RemoveContainer" containerID="2ab9805704910dcd0fb0584bff3babd0b9ae059d01b0f2411bd502507c64741f"
Sep 16 18:58:47.845131 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:58:47.845107    2544 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-22075-predictor-754b4f7b64-9stpg"]
Sep 16 18:58:47.849317 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:58:47.849298    2544 scope.go:117] "RemoveContainer" containerID="226f7c4b4845bc5ae74b981c2d9c2f8de12ed573a24f667d6bc99a8a0bf7ce80"
Sep 16 18:58:47.849625 ip-10-0-132-80 kubenswrapper[2544]: E0916 18:58:47.849605    2544 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"226f7c4b4845bc5ae74b981c2d9c2f8de12ed573a24f667d6bc99a8a0bf7ce80\": container with ID starting with 226f7c4b4845bc5ae74b981c2d9c2f8de12ed573a24f667d6bc99a8a0bf7ce80 not found: ID does not exist" containerID="226f7c4b4845bc5ae74b981c2d9c2f8de12ed573a24f667d6bc99a8a0bf7ce80"
Sep 16 18:58:47.849707 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:58:47.849637    2544 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"226f7c4b4845bc5ae74b981c2d9c2f8de12ed573a24f667d6bc99a8a0bf7ce80"} err="failed to get container status \"226f7c4b4845bc5ae74b981c2d9c2f8de12ed573a24f667d6bc99a8a0bf7ce80\": rpc error: code = NotFound desc = could not find container \"226f7c4b4845bc5ae74b981c2d9c2f8de12ed573a24f667d6bc99a8a0bf7ce80\": container with ID starting with 226f7c4b4845bc5ae74b981c2d9c2f8de12ed573a24f667d6bc99a8a0bf7ce80 not found: ID does not exist"
Sep 16 18:58:47.849707 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:58:47.849662    2544 scope.go:117] "RemoveContainer" containerID="b73ca7b84c824cf9ccf800d19bf6879351d2971f29f34ee8944ea449b0f6a655"
Sep 16 18:58:47.849797 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:58:47.849756    2544 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-22075-predictor-754b4f7b64-9stpg"]
Sep 16 18:58:47.849958 ip-10-0-132-80 kubenswrapper[2544]: E0916 18:58:47.849943    2544 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b73ca7b84c824cf9ccf800d19bf6879351d2971f29f34ee8944ea449b0f6a655\": container with ID starting with b73ca7b84c824cf9ccf800d19bf6879351d2971f29f34ee8944ea449b0f6a655 not found: ID does not exist" containerID="b73ca7b84c824cf9ccf800d19bf6879351d2971f29f34ee8944ea449b0f6a655"
Sep 16 18:58:47.850024 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:58:47.849964    2544 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b73ca7b84c824cf9ccf800d19bf6879351d2971f29f34ee8944ea449b0f6a655"} err="failed to get container status \"b73ca7b84c824cf9ccf800d19bf6879351d2971f29f34ee8944ea449b0f6a655\": rpc error: code = NotFound desc = could not find container \"b73ca7b84c824cf9ccf800d19bf6879351d2971f29f34ee8944ea449b0f6a655\": container with ID starting with b73ca7b84c824cf9ccf800d19bf6879351d2971f29f34ee8944ea449b0f6a655 not found: ID does not exist"
Sep 16 18:58:47.850024 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:58:47.849983    2544 scope.go:117] "RemoveContainer" containerID="2ab9805704910dcd0fb0584bff3babd0b9ae059d01b0f2411bd502507c64741f"
Sep 16 18:58:47.850243 ip-10-0-132-80 kubenswrapper[2544]: E0916 18:58:47.850226    2544 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2ab9805704910dcd0fb0584bff3babd0b9ae059d01b0f2411bd502507c64741f\": container with ID starting with 2ab9805704910dcd0fb0584bff3babd0b9ae059d01b0f2411bd502507c64741f not found: ID does not exist" containerID="2ab9805704910dcd0fb0584bff3babd0b9ae059d01b0f2411bd502507c64741f"
Sep 16 18:58:47.850281 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:58:47.850249    2544 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2ab9805704910dcd0fb0584bff3babd0b9ae059d01b0f2411bd502507c64741f"} err="failed to get container status \"2ab9805704910dcd0fb0584bff3babd0b9ae059d01b0f2411bd502507c64741f\": rpc error: code = NotFound desc = could not find container \"2ab9805704910dcd0fb0584bff3babd0b9ae059d01b0f2411bd502507c64741f\": container with ID starting with 2ab9805704910dcd0fb0584bff3babd0b9ae059d01b0f2411bd502507c64741f not found: ID does not exist"
Sep 16 18:58:49.721538 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:58:49.721506    2544 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="bb199f20-09bc-4c3f-aa96-d9a976f62292" path="/var/lib/kubelet/pods/bb199f20-09bc-4c3f-aa96-d9a976f62292/volumes"
Sep 16 18:58:57.708293 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:58:57.708264    2544 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-pfcb7_9e3f9e34-420e-43ef-96b4-c7cad6b2eaa9/console-operator/2.log"
Sep 16 18:58:57.709815 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:58:57.709789    2544 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-pfcb7_9e3f9e34-420e-43ef-96b4-c7cad6b2eaa9/console-operator/2.log"
Sep 16 18:59:02.718053 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:59:02.718023    2544 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-logger-raw-16e43-predictor-84c9b8899d-2ms7b"]
Sep 16 18:59:02.718396 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:59:02.718320    2544 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2985e5ca-4da7-4cd5-911e-39decc92e56d" containerName="storage-initializer"
Sep 16 18:59:02.718396 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:59:02.718331    2544 state_mem.go:107] "Deleted CPUSet assignment" podUID="2985e5ca-4da7-4cd5-911e-39decc92e56d" containerName="storage-initializer"
Sep 16 18:59:02.718396 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:59:02.718338    2544 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2985e5ca-4da7-4cd5-911e-39decc92e56d" containerName="kube-rbac-proxy"
Sep 16 18:59:02.718396 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:59:02.718344    2544 state_mem.go:107] "Deleted CPUSet assignment" podUID="2985e5ca-4da7-4cd5-911e-39decc92e56d" containerName="kube-rbac-proxy"
Sep 16 18:59:02.718396 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:59:02.718355    2544 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="09fd6613-c8cb-4b5d-b5b5-75a2e70d9adb" containerName="storage-initializer"
Sep 16 18:59:02.718396 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:59:02.718361    2544 state_mem.go:107] "Deleted CPUSet assignment" podUID="09fd6613-c8cb-4b5d-b5b5-75a2e70d9adb" containerName="storage-initializer"
Sep 16 18:59:02.718396 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:59:02.718374    2544 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7ae7a71d-424c-4d16-8916-642cfa6e34b1" containerName="storage-initializer"
Sep 16 18:59:02.718396 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:59:02.718379    2544 state_mem.go:107] "Deleted CPUSet assignment" podUID="7ae7a71d-424c-4d16-8916-642cfa6e34b1" containerName="storage-initializer"
Sep 16 18:59:02.718396 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:59:02.718388    2544 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="09fd6613-c8cb-4b5d-b5b5-75a2e70d9adb" containerName="kserve-container"
Sep 16 18:59:02.718396 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:59:02.718394    2544 state_mem.go:107] "Deleted CPUSet assignment" podUID="09fd6613-c8cb-4b5d-b5b5-75a2e70d9adb" containerName="kserve-container"
Sep 16 18:59:02.718396 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:59:02.718400    2544 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="09fd6613-c8cb-4b5d-b5b5-75a2e70d9adb" containerName="kube-rbac-proxy"
Sep 16 18:59:02.718754 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:59:02.718405    2544 state_mem.go:107] "Deleted CPUSet assignment" podUID="09fd6613-c8cb-4b5d-b5b5-75a2e70d9adb" containerName="kube-rbac-proxy"
Sep 16 18:59:02.718754 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:59:02.718415    2544 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7ae7a71d-424c-4d16-8916-642cfa6e34b1" containerName="kserve-container"
Sep 16 18:59:02.718754 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:59:02.718421    2544 state_mem.go:107] "Deleted CPUSet assignment" podUID="7ae7a71d-424c-4d16-8916-642cfa6e34b1" containerName="kserve-container"
Sep 16 18:59:02.718754 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:59:02.718428    2544 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7ae7a71d-424c-4d16-8916-642cfa6e34b1" containerName="kube-rbac-proxy"
Sep 16 18:59:02.718754 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:59:02.718433    2544 state_mem.go:107] "Deleted CPUSet assignment" podUID="7ae7a71d-424c-4d16-8916-642cfa6e34b1" containerName="kube-rbac-proxy"
Sep 16 18:59:02.718754 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:59:02.718439    2544 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2985e5ca-4da7-4cd5-911e-39decc92e56d" containerName="kserve-container"
Sep 16 18:59:02.718754 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:59:02.718444    2544 state_mem.go:107] "Deleted CPUSet assignment" podUID="2985e5ca-4da7-4cd5-911e-39decc92e56d" containerName="kserve-container"
Sep 16 18:59:02.718754 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:59:02.718451    2544 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bb199f20-09bc-4c3f-aa96-d9a976f62292" containerName="storage-initializer"
Sep 16 18:59:02.718754 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:59:02.718456    2544 state_mem.go:107] "Deleted CPUSet assignment" podUID="bb199f20-09bc-4c3f-aa96-d9a976f62292" containerName="storage-initializer"
Sep 16 18:59:02.718754 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:59:02.718461    2544 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bb199f20-09bc-4c3f-aa96-d9a976f62292" containerName="kserve-container"
Sep 16 18:59:02.718754 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:59:02.718465    2544 state_mem.go:107] "Deleted CPUSet assignment" podUID="bb199f20-09bc-4c3f-aa96-d9a976f62292" containerName="kserve-container"
Sep 16 18:59:02.718754 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:59:02.718472    2544 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bb199f20-09bc-4c3f-aa96-d9a976f62292" containerName="kube-rbac-proxy"
Sep 16 18:59:02.718754 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:59:02.718477    2544 state_mem.go:107] "Deleted CPUSet assignment" podUID="bb199f20-09bc-4c3f-aa96-d9a976f62292" containerName="kube-rbac-proxy"
Sep 16 18:59:02.718754 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:59:02.718523    2544 memory_manager.go:368] "RemoveStaleState removing state" podUID="2985e5ca-4da7-4cd5-911e-39decc92e56d" containerName="kube-rbac-proxy"
Sep 16 18:59:02.718754 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:59:02.718529    2544 memory_manager.go:368] "RemoveStaleState removing state" podUID="7ae7a71d-424c-4d16-8916-642cfa6e34b1" containerName="kube-rbac-proxy"
Sep 16 18:59:02.718754 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:59:02.718535    2544 memory_manager.go:368] "RemoveStaleState removing state" podUID="09fd6613-c8cb-4b5d-b5b5-75a2e70d9adb" containerName="kserve-container"
Sep 16 18:59:02.718754 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:59:02.718542    2544 memory_manager.go:368] "RemoveStaleState removing state" podUID="7ae7a71d-424c-4d16-8916-642cfa6e34b1" containerName="kserve-container"
Sep 16 18:59:02.718754 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:59:02.718548    2544 memory_manager.go:368] "RemoveStaleState removing state" podUID="09fd6613-c8cb-4b5d-b5b5-75a2e70d9adb" containerName="kube-rbac-proxy"
Sep 16 18:59:02.718754 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:59:02.718570    2544 memory_manager.go:368] "RemoveStaleState removing state" podUID="bb199f20-09bc-4c3f-aa96-d9a976f62292" containerName="kserve-container"
Sep 16 18:59:02.718754 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:59:02.718574    2544 memory_manager.go:368] "RemoveStaleState removing state" podUID="2985e5ca-4da7-4cd5-911e-39decc92e56d" containerName="kserve-container"
Sep 16 18:59:02.718754 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:59:02.718580    2544 memory_manager.go:368] "RemoveStaleState removing state" podUID="bb199f20-09bc-4c3f-aa96-d9a976f62292" containerName="kube-rbac-proxy"
Sep 16 18:59:02.723186 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:59:02.723166    2544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-logger-raw-16e43-predictor-84c9b8899d-2ms7b"
Sep 16 18:59:02.725731 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:59:02.725706    2544 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Sep 16 18:59:02.725870 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:59:02.725707    2544 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Sep 16 18:59:02.725870 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:59:02.725713    2544 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-logger-raw-16e43-predictor-serving-cert\""
Sep 16 18:59:02.726067 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:59:02.725940    2544 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Sep 16 18:59:02.726067 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:59:02.726033    2544 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Sep 16 18:59:02.726067 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:59:02.726036    2544 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-logger-raw-16e43-kube-rbac-proxy-sar-config\""
Sep 16 18:59:02.726425 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:59:02.726410    2544 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-zmsvt\""
Sep 16 18:59:02.737741 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:59:02.737713    2544 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-logger-raw-16e43-predictor-84c9b8899d-2ms7b"]
Sep 16 18:59:02.860000 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:59:02.859963    2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/66695b97-dadb-4454-a7e1-940f31ef3bd9-kube-rbac-proxy-sar-config\") pod \"isvc-logger-raw-16e43-predictor-84c9b8899d-2ms7b\" (UID: \"66695b97-dadb-4454-a7e1-940f31ef3bd9\") " pod="kserve-ci-e2e-test/isvc-logger-raw-16e43-predictor-84c9b8899d-2ms7b"
Sep 16 18:59:02.860000 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:59:02.860009    2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/66695b97-dadb-4454-a7e1-940f31ef3bd9-cabundle-cert\") pod \"isvc-logger-raw-16e43-predictor-84c9b8899d-2ms7b\" (UID: \"66695b97-dadb-4454-a7e1-940f31ef3bd9\") " pod="kserve-ci-e2e-test/isvc-logger-raw-16e43-predictor-84c9b8899d-2ms7b"
Sep 16 18:59:02.860219 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:59:02.860036    2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tsvzt\" (UniqueName: \"kubernetes.io/projected/66695b97-dadb-4454-a7e1-940f31ef3bd9-kube-api-access-tsvzt\") pod \"isvc-logger-raw-16e43-predictor-84c9b8899d-2ms7b\" (UID: \"66695b97-dadb-4454-a7e1-940f31ef3bd9\") " pod="kserve-ci-e2e-test/isvc-logger-raw-16e43-predictor-84c9b8899d-2ms7b"
Sep 16 18:59:02.860219 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:59:02.860093    2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/66695b97-dadb-4454-a7e1-940f31ef3bd9-proxy-tls\") pod \"isvc-logger-raw-16e43-predictor-84c9b8899d-2ms7b\" (UID: \"66695b97-dadb-4454-a7e1-940f31ef3bd9\") " pod="kserve-ci-e2e-test/isvc-logger-raw-16e43-predictor-84c9b8899d-2ms7b"
Sep 16 18:59:02.860219 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:59:02.860165    2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/66695b97-dadb-4454-a7e1-940f31ef3bd9-kserve-provision-location\") pod \"isvc-logger-raw-16e43-predictor-84c9b8899d-2ms7b\" (UID: \"66695b97-dadb-4454-a7e1-940f31ef3bd9\") " pod="kserve-ci-e2e-test/isvc-logger-raw-16e43-predictor-84c9b8899d-2ms7b"
Sep 16 18:59:02.961273 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:59:02.961236    2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tsvzt\" (UniqueName: \"kubernetes.io/projected/66695b97-dadb-4454-a7e1-940f31ef3bd9-kube-api-access-tsvzt\") pod \"isvc-logger-raw-16e43-predictor-84c9b8899d-2ms7b\" (UID: \"66695b97-dadb-4454-a7e1-940f31ef3bd9\") " pod="kserve-ci-e2e-test/isvc-logger-raw-16e43-predictor-84c9b8899d-2ms7b"
Sep 16 18:59:02.961273 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:59:02.961274    2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/66695b97-dadb-4454-a7e1-940f31ef3bd9-proxy-tls\") pod \"isvc-logger-raw-16e43-predictor-84c9b8899d-2ms7b\" (UID: \"66695b97-dadb-4454-a7e1-940f31ef3bd9\") " pod="kserve-ci-e2e-test/isvc-logger-raw-16e43-predictor-84c9b8899d-2ms7b"
Sep 16 18:59:02.961446 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:59:02.961343    2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/66695b97-dadb-4454-a7e1-940f31ef3bd9-kserve-provision-location\") pod \"isvc-logger-raw-16e43-predictor-84c9b8899d-2ms7b\" (UID: \"66695b97-dadb-4454-a7e1-940f31ef3bd9\") " pod="kserve-ci-e2e-test/isvc-logger-raw-16e43-predictor-84c9b8899d-2ms7b"
Sep 16 18:59:02.961446 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:59:02.961408    2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/66695b97-dadb-4454-a7e1-940f31ef3bd9-kube-rbac-proxy-sar-config\") pod \"isvc-logger-raw-16e43-predictor-84c9b8899d-2ms7b\" (UID: \"66695b97-dadb-4454-a7e1-940f31ef3bd9\") " pod="kserve-ci-e2e-test/isvc-logger-raw-16e43-predictor-84c9b8899d-2ms7b"
Sep 16 18:59:02.961446 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:59:02.961438    2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/66695b97-dadb-4454-a7e1-940f31ef3bd9-cabundle-cert\") pod \"isvc-logger-raw-16e43-predictor-84c9b8899d-2ms7b\" (UID: \"66695b97-dadb-4454-a7e1-940f31ef3bd9\") " pod="kserve-ci-e2e-test/isvc-logger-raw-16e43-predictor-84c9b8899d-2ms7b"
Sep 16 18:59:02.961822 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:59:02.961800    2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/66695b97-dadb-4454-a7e1-940f31ef3bd9-kserve-provision-location\") pod \"isvc-logger-raw-16e43-predictor-84c9b8899d-2ms7b\" (UID: \"66695b97-dadb-4454-a7e1-940f31ef3bd9\") " pod="kserve-ci-e2e-test/isvc-logger-raw-16e43-predictor-84c9b8899d-2ms7b"
Sep 16 18:59:02.962099 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:59:02.962082    2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/66695b97-dadb-4454-a7e1-940f31ef3bd9-kube-rbac-proxy-sar-config\") pod \"isvc-logger-raw-16e43-predictor-84c9b8899d-2ms7b\" (UID: \"66695b97-dadb-4454-a7e1-940f31ef3bd9\") " pod="kserve-ci-e2e-test/isvc-logger-raw-16e43-predictor-84c9b8899d-2ms7b"
Sep 16 18:59:02.962145 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:59:02.962105    2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/66695b97-dadb-4454-a7e1-940f31ef3bd9-cabundle-cert\") pod \"isvc-logger-raw-16e43-predictor-84c9b8899d-2ms7b\" (UID: \"66695b97-dadb-4454-a7e1-940f31ef3bd9\") " pod="kserve-ci-e2e-test/isvc-logger-raw-16e43-predictor-84c9b8899d-2ms7b"
Sep 16 18:59:02.963807 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:59:02.963791    2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/66695b97-dadb-4454-a7e1-940f31ef3bd9-proxy-tls\") pod \"isvc-logger-raw-16e43-predictor-84c9b8899d-2ms7b\" (UID: \"66695b97-dadb-4454-a7e1-940f31ef3bd9\") " pod="kserve-ci-e2e-test/isvc-logger-raw-16e43-predictor-84c9b8899d-2ms7b"
Sep 16 18:59:02.970923 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:59:02.970871    2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tsvzt\" (UniqueName: \"kubernetes.io/projected/66695b97-dadb-4454-a7e1-940f31ef3bd9-kube-api-access-tsvzt\") pod \"isvc-logger-raw-16e43-predictor-84c9b8899d-2ms7b\" (UID: \"66695b97-dadb-4454-a7e1-940f31ef3bd9\") " pod="kserve-ci-e2e-test/isvc-logger-raw-16e43-predictor-84c9b8899d-2ms7b"
Sep 16 18:59:03.033733 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:59:03.033699    2544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-logger-raw-16e43-predictor-84c9b8899d-2ms7b"
Sep 16 18:59:03.304929 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:59:03.304903    2544 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-logger-raw-16e43-predictor-84c9b8899d-2ms7b"]
Sep 16 18:59:03.307174 ip-10-0-132-80 kubenswrapper[2544]: W0916 18:59:03.307138    2544 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod66695b97_dadb_4454_a7e1_940f31ef3bd9.slice/crio-1d7ab2658f28d07f71db7770d4e461a67e6acc2d5b0cde0ff58e40eab1b2668c WatchSource:0}: Error finding container 1d7ab2658f28d07f71db7770d4e461a67e6acc2d5b0cde0ff58e40eab1b2668c: Status 404 returned error can't find the container with id 1d7ab2658f28d07f71db7770d4e461a67e6acc2d5b0cde0ff58e40eab1b2668c
Sep 16 18:59:03.899658 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:59:03.899625    2544 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-raw-16e43-predictor-84c9b8899d-2ms7b" event={"ID":"66695b97-dadb-4454-a7e1-940f31ef3bd9","Type":"ContainerStarted","Data":"a6d08c2b7d4bf591b1d2b3e0ccab0fecc4df3c6a3f98c549394e65dd4bd3c0c0"}
Sep 16 18:59:03.900082 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:59:03.899698    2544 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-raw-16e43-predictor-84c9b8899d-2ms7b" event={"ID":"66695b97-dadb-4454-a7e1-940f31ef3bd9","Type":"ContainerStarted","Data":"1d7ab2658f28d07f71db7770d4e461a67e6acc2d5b0cde0ff58e40eab1b2668c"}
Sep 16 18:59:07.917975 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:59:07.917937    2544 generic.go:356] "Generic (PLEG): container finished" podID="66695b97-dadb-4454-a7e1-940f31ef3bd9" containerID="a6d08c2b7d4bf591b1d2b3e0ccab0fecc4df3c6a3f98c549394e65dd4bd3c0c0" exitCode=0
Sep 16 18:59:07.918435 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:59:07.917978    2544 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-raw-16e43-predictor-84c9b8899d-2ms7b" event={"ID":"66695b97-dadb-4454-a7e1-940f31ef3bd9","Type":"ContainerDied","Data":"a6d08c2b7d4bf591b1d2b3e0ccab0fecc4df3c6a3f98c549394e65dd4bd3c0c0"}
Sep 16 18:59:08.924918 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:59:08.924883    2544 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-raw-16e43-predictor-84c9b8899d-2ms7b" event={"ID":"66695b97-dadb-4454-a7e1-940f31ef3bd9","Type":"ContainerStarted","Data":"22056d58e87300938cf26915b2a5d63da53497026635168b5f7b2a1f9cad6ff6"}
Sep 16 18:59:08.925298 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:59:08.924952    2544 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-raw-16e43-predictor-84c9b8899d-2ms7b" event={"ID":"66695b97-dadb-4454-a7e1-940f31ef3bd9","Type":"ContainerStarted","Data":"ea9330e75f702af4b1c9c65f24155cd8ff1ae135542a3db73ec8025a7bb7a2d5"}
Sep 16 18:59:08.925298 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:59:08.924977    2544 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-raw-16e43-predictor-84c9b8899d-2ms7b" event={"ID":"66695b97-dadb-4454-a7e1-940f31ef3bd9","Type":"ContainerStarted","Data":"d12847499eb1b07e054924ce632189c080c52816d0db02b8ba0b335a0a61e256"}
Sep 16 18:59:08.925298 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:59:08.925258    2544 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-logger-raw-16e43-predictor-84c9b8899d-2ms7b"
Sep 16 18:59:08.925470 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:59:08.925322    2544 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-logger-raw-16e43-predictor-84c9b8899d-2ms7b"
Sep 16 18:59:08.926698 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:59:08.926671    2544 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-16e43-predictor-84c9b8899d-2ms7b" podUID="66695b97-dadb-4454-a7e1-940f31ef3bd9" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.33:8080: connect: connection refused"
Sep 16 18:59:08.946627 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:59:08.946581    2544 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-logger-raw-16e43-predictor-84c9b8899d-2ms7b" podStartSLOduration=3.113810623 podStartE2EDuration="6.94654861s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.832737987s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:59:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:59:08.945709387 +0000 UTC m=+911.734711936" watchObservedRunningTime="2026-09-16 18:59:08.94654861 +0000 UTC m=+911.735551143"
Sep 16 18:59:09.930146 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:59:09.930114    2544 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-logger-raw-16e43-predictor-84c9b8899d-2ms7b"
Sep 16 18:59:09.930505 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:59:09.930274    2544 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-16e43-predictor-84c9b8899d-2ms7b" podUID="66695b97-dadb-4454-a7e1-940f31ef3bd9" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.33:8080: connect: connection refused"
Sep 16 18:59:09.931381 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:59:09.931354    2544 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-16e43-predictor-84c9b8899d-2ms7b" podUID="66695b97-dadb-4454-a7e1-940f31ef3bd9" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 16 18:59:10.933714 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:59:10.933674    2544 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-16e43-predictor-84c9b8899d-2ms7b" podUID="66695b97-dadb-4454-a7e1-940f31ef3bd9" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.33:8080: connect: connection refused"
Sep 16 18:59:10.934128 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:59:10.934100    2544 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-16e43-predictor-84c9b8899d-2ms7b" podUID="66695b97-dadb-4454-a7e1-940f31ef3bd9" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 16 18:59:15.938338 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:59:15.938309    2544 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-logger-raw-16e43-predictor-84c9b8899d-2ms7b"
Sep 16 18:59:15.938963 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:59:15.938930    2544 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-16e43-predictor-84c9b8899d-2ms7b" podUID="66695b97-dadb-4454-a7e1-940f31ef3bd9" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.33:8080: connect: connection refused"
Sep 16 18:59:15.939135 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:59:15.939103    2544 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-16e43-predictor-84c9b8899d-2ms7b" podUID="66695b97-dadb-4454-a7e1-940f31ef3bd9" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 16 18:59:25.939114 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:59:25.939077    2544 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-16e43-predictor-84c9b8899d-2ms7b" podUID="66695b97-dadb-4454-a7e1-940f31ef3bd9" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.33:8080: connect: connection refused"
Sep 16 18:59:25.939531 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:59:25.939498    2544 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-16e43-predictor-84c9b8899d-2ms7b" podUID="66695b97-dadb-4454-a7e1-940f31ef3bd9" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 16 18:59:35.939026 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:59:35.938987    2544 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-16e43-predictor-84c9b8899d-2ms7b" podUID="66695b97-dadb-4454-a7e1-940f31ef3bd9" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.33:8080: connect: connection refused"
Sep 16 18:59:35.939492 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:59:35.939469    2544 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-16e43-predictor-84c9b8899d-2ms7b" podUID="66695b97-dadb-4454-a7e1-940f31ef3bd9" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 16 18:59:45.939731 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:59:45.939690    2544 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-16e43-predictor-84c9b8899d-2ms7b" podUID="66695b97-dadb-4454-a7e1-940f31ef3bd9" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.33:8080: connect: connection refused"
Sep 16 18:59:45.942292 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:59:45.940095    2544 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-16e43-predictor-84c9b8899d-2ms7b" podUID="66695b97-dadb-4454-a7e1-940f31ef3bd9" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 16 18:59:55.938820 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:59:55.938777    2544 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-16e43-predictor-84c9b8899d-2ms7b" podUID="66695b97-dadb-4454-a7e1-940f31ef3bd9" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.33:8080: connect: connection refused"
Sep 16 18:59:55.939315 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:59:55.939290    2544 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-16e43-predictor-84c9b8899d-2ms7b" podUID="66695b97-dadb-4454-a7e1-940f31ef3bd9" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 16 18:59:57.757772 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:59:57.757741    2544 scope.go:117] "RemoveContainer" containerID="a04f3c48c8106948f06f6ca5c305bb7fabb787473e16e1d59601040a2a5946e3"
Sep 16 18:59:57.765807 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:59:57.765787    2544 scope.go:117] "RemoveContainer" containerID="942edcb2d4c852444220f19a464a876700f1fdab69de5905d7ba8a682f34ffb9"
Sep 16 18:59:57.772857 ip-10-0-132-80 kubenswrapper[2544]: I0916 18:59:57.772833    2544 scope.go:117] "RemoveContainer" containerID="f78a9ff879832ed81559f36976e1c5b4f64d160d390bb33e73735290360f66c8"
Sep 16 19:00:05.939468 ip-10-0-132-80 kubenswrapper[2544]: I0916 19:00:05.939414    2544 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-16e43-predictor-84c9b8899d-2ms7b" podUID="66695b97-dadb-4454-a7e1-940f31ef3bd9" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.33:8080: connect: connection refused"
Sep 16 19:00:05.939960 ip-10-0-132-80 kubenswrapper[2544]: I0916 19:00:05.939943    2544 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-16e43-predictor-84c9b8899d-2ms7b" podUID="66695b97-dadb-4454-a7e1-940f31ef3bd9" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 16 19:00:15.939467 ip-10-0-132-80 kubenswrapper[2544]: I0916 19:00:15.939419    2544 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-16e43-predictor-84c9b8899d-2ms7b" podUID="66695b97-dadb-4454-a7e1-940f31ef3bd9" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.33:8080: connect: connection refused"
Sep 16 19:00:15.940012 ip-10-0-132-80 kubenswrapper[2544]: I0916 19:00:15.939888    2544 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-16e43-predictor-84c9b8899d-2ms7b" podUID="66695b97-dadb-4454-a7e1-940f31ef3bd9" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 16 19:00:25.940234 ip-10-0-132-80 kubenswrapper[2544]: I0916 19:00:25.940203    2544 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-logger-raw-16e43-predictor-84c9b8899d-2ms7b"
Sep 16 19:00:25.940612 ip-10-0-132-80 kubenswrapper[2544]: I0916 19:00:25.940534    2544 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-logger-raw-16e43-predictor-84c9b8899d-2ms7b"
Sep 16 19:00:37.985900 ip-10-0-132-80 kubenswrapper[2544]: I0916 19:00:37.985861    2544 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-logger-raw-16e43-predictor-84c9b8899d-2ms7b"]
Sep 16 19:00:37.986337 ip-10-0-132-80 kubenswrapper[2544]: I0916 19:00:37.986151    2544 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-logger-raw-16e43-predictor-84c9b8899d-2ms7b" podUID="66695b97-dadb-4454-a7e1-940f31ef3bd9" containerName="kserve-container" containerID="cri-o://d12847499eb1b07e054924ce632189c080c52816d0db02b8ba0b335a0a61e256" gracePeriod=30
Sep 16 19:00:37.986337 ip-10-0-132-80 kubenswrapper[2544]: I0916 19:00:37.986157    2544 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-logger-raw-16e43-predictor-84c9b8899d-2ms7b" podUID="66695b97-dadb-4454-a7e1-940f31ef3bd9" containerName="agent" containerID="cri-o://22056d58e87300938cf26915b2a5d63da53497026635168b5f7b2a1f9cad6ff6" gracePeriod=30
Sep 16 19:00:37.986337 ip-10-0-132-80 kubenswrapper[2544]: I0916 19:00:37.986226    2544 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-logger-raw-16e43-predictor-84c9b8899d-2ms7b" podUID="66695b97-dadb-4454-a7e1-940f31ef3bd9" containerName="kube-rbac-proxy" containerID="cri-o://ea9330e75f702af4b1c9c65f24155cd8ff1ae135542a3db73ec8025a7bb7a2d5" gracePeriod=30
Sep 16 19:00:38.121387 ip-10-0-132-80 kubenswrapper[2544]: I0916 19:00:38.121349    2544 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-scale-raw-d85e8-predictor-7fbccc8dc4-gd9xw"]
Sep 16 19:00:38.124977 ip-10-0-132-80 kubenswrapper[2544]: I0916 19:00:38.124957    2544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-d85e8-predictor-7fbccc8dc4-gd9xw"
Sep 16 19:00:38.127815 ip-10-0-132-80 kubenswrapper[2544]: I0916 19:00:38.127793    2544 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-scale-raw-d85e8-predictor-serving-cert\""
Sep 16 19:00:38.127940 ip-10-0-132-80 kubenswrapper[2544]: I0916 19:00:38.127870    2544 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-scale-raw-d85e8-kube-rbac-proxy-sar-config\""
Sep 16 19:00:38.147563 ip-10-0-132-80 kubenswrapper[2544]: I0916 19:00:38.147527    2544 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-scale-raw-d85e8-predictor-7fbccc8dc4-gd9xw"]
Sep 16 19:00:38.158979 ip-10-0-132-80 kubenswrapper[2544]: I0916 19:00:38.158956    2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/dc269306-d6b2-4857-a15f-fb4579de499f-proxy-tls\") pod \"isvc-sklearn-scale-raw-d85e8-predictor-7fbccc8dc4-gd9xw\" (UID: \"dc269306-d6b2-4857-a15f-fb4579de499f\") " pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-d85e8-predictor-7fbccc8dc4-gd9xw"
Sep 16 19:00:38.159103 ip-10-0-132-80 kubenswrapper[2544]: I0916 19:00:38.159003    2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/dc269306-d6b2-4857-a15f-fb4579de499f-cabundle-cert\") pod \"isvc-sklearn-scale-raw-d85e8-predictor-7fbccc8dc4-gd9xw\" (UID: \"dc269306-d6b2-4857-a15f-fb4579de499f\") " pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-d85e8-predictor-7fbccc8dc4-gd9xw"
Sep 16 19:00:38.159103 ip-10-0-132-80 kubenswrapper[2544]: I0916 19:00:38.159029    2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/dc269306-d6b2-4857-a15f-fb4579de499f-kserve-provision-location\") pod \"isvc-sklearn-scale-raw-d85e8-predictor-7fbccc8dc4-gd9xw\" (UID: \"dc269306-d6b2-4857-a15f-fb4579de499f\") " pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-d85e8-predictor-7fbccc8dc4-gd9xw"
Sep 16 19:00:38.159232 ip-10-0-132-80 kubenswrapper[2544]: I0916 19:00:38.159120    2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qs5m5\" (UniqueName: \"kubernetes.io/projected/dc269306-d6b2-4857-a15f-fb4579de499f-kube-api-access-qs5m5\") pod \"isvc-sklearn-scale-raw-d85e8-predictor-7fbccc8dc4-gd9xw\" (UID: \"dc269306-d6b2-4857-a15f-fb4579de499f\") " pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-d85e8-predictor-7fbccc8dc4-gd9xw"
Sep 16 19:00:38.159232 ip-10-0-132-80 kubenswrapper[2544]: I0916 19:00:38.159177    2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/dc269306-d6b2-4857-a15f-fb4579de499f-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-scale-raw-d85e8-predictor-7fbccc8dc4-gd9xw\" (UID: \"dc269306-d6b2-4857-a15f-fb4579de499f\") " pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-d85e8-predictor-7fbccc8dc4-gd9xw"
Sep 16 19:00:38.259956 ip-10-0-132-80 kubenswrapper[2544]: I0916 19:00:38.259866    2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/dc269306-d6b2-4857-a15f-fb4579de499f-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-scale-raw-d85e8-predictor-7fbccc8dc4-gd9xw\" (UID: \"dc269306-d6b2-4857-a15f-fb4579de499f\") " pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-d85e8-predictor-7fbccc8dc4-gd9xw"
Sep 16 19:00:38.259956 ip-10-0-132-80 kubenswrapper[2544]: I0916 19:00:38.259908    2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/dc269306-d6b2-4857-a15f-fb4579de499f-proxy-tls\") pod \"isvc-sklearn-scale-raw-d85e8-predictor-7fbccc8dc4-gd9xw\" (UID: \"dc269306-d6b2-4857-a15f-fb4579de499f\") " pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-d85e8-predictor-7fbccc8dc4-gd9xw"
Sep 16 19:00:38.259956 ip-10-0-132-80 kubenswrapper[2544]: I0916 19:00:38.259950    2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/dc269306-d6b2-4857-a15f-fb4579de499f-cabundle-cert\") pod \"isvc-sklearn-scale-raw-d85e8-predictor-7fbccc8dc4-gd9xw\" (UID: \"dc269306-d6b2-4857-a15f-fb4579de499f\") " pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-d85e8-predictor-7fbccc8dc4-gd9xw"
Sep 16 19:00:38.260298 ip-10-0-132-80 kubenswrapper[2544]: I0916 19:00:38.259976    2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/dc269306-d6b2-4857-a15f-fb4579de499f-kserve-provision-location\") pod \"isvc-sklearn-scale-raw-d85e8-predictor-7fbccc8dc4-gd9xw\" (UID: \"dc269306-d6b2-4857-a15f-fb4579de499f\") " pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-d85e8-predictor-7fbccc8dc4-gd9xw"
Sep 16 19:00:38.260298 ip-10-0-132-80 kubenswrapper[2544]: I0916 19:00:38.260044    2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qs5m5\" (UniqueName: \"kubernetes.io/projected/dc269306-d6b2-4857-a15f-fb4579de499f-kube-api-access-qs5m5\") pod \"isvc-sklearn-scale-raw-d85e8-predictor-7fbccc8dc4-gd9xw\" (UID: \"dc269306-d6b2-4857-a15f-fb4579de499f\") " pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-d85e8-predictor-7fbccc8dc4-gd9xw"
Sep 16 19:00:38.260493 ip-10-0-132-80 kubenswrapper[2544]: I0916 19:00:38.260469    2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/dc269306-d6b2-4857-a15f-fb4579de499f-kserve-provision-location\") pod \"isvc-sklearn-scale-raw-d85e8-predictor-7fbccc8dc4-gd9xw\" (UID: \"dc269306-d6b2-4857-a15f-fb4579de499f\") " pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-d85e8-predictor-7fbccc8dc4-gd9xw"
Sep 16 19:00:38.260601 ip-10-0-132-80 kubenswrapper[2544]: I0916 19:00:38.260529    2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/dc269306-d6b2-4857-a15f-fb4579de499f-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-scale-raw-d85e8-predictor-7fbccc8dc4-gd9xw\" (UID: \"dc269306-d6b2-4857-a15f-fb4579de499f\") " pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-d85e8-predictor-7fbccc8dc4-gd9xw"
Sep 16 19:00:38.260658 ip-10-0-132-80 kubenswrapper[2544]: I0916 19:00:38.260644    2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/dc269306-d6b2-4857-a15f-fb4579de499f-cabundle-cert\") pod \"isvc-sklearn-scale-raw-d85e8-predictor-7fbccc8dc4-gd9xw\" (UID: \"dc269306-d6b2-4857-a15f-fb4579de499f\") " pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-d85e8-predictor-7fbccc8dc4-gd9xw"
Sep 16 19:00:38.262444 ip-10-0-132-80 kubenswrapper[2544]: I0916 19:00:38.262424    2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/dc269306-d6b2-4857-a15f-fb4579de499f-proxy-tls\") pod \"isvc-sklearn-scale-raw-d85e8-predictor-7fbccc8dc4-gd9xw\" (UID: \"dc269306-d6b2-4857-a15f-fb4579de499f\") " pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-d85e8-predictor-7fbccc8dc4-gd9xw"
Sep 16 19:00:38.270869 ip-10-0-132-80 kubenswrapper[2544]: I0916 19:00:38.270844    2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qs5m5\" (UniqueName: \"kubernetes.io/projected/dc269306-d6b2-4857-a15f-fb4579de499f-kube-api-access-qs5m5\") pod \"isvc-sklearn-scale-raw-d85e8-predictor-7fbccc8dc4-gd9xw\" (UID: \"dc269306-d6b2-4857-a15f-fb4579de499f\") " pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-d85e8-predictor-7fbccc8dc4-gd9xw"
Sep 16 19:00:38.305571 ip-10-0-132-80 kubenswrapper[2544]: I0916 19:00:38.305523    2544 generic.go:356] "Generic (PLEG): container finished" podID="66695b97-dadb-4454-a7e1-940f31ef3bd9" containerID="ea9330e75f702af4b1c9c65f24155cd8ff1ae135542a3db73ec8025a7bb7a2d5" exitCode=2
Sep 16 19:00:38.305698 ip-10-0-132-80 kubenswrapper[2544]: I0916 19:00:38.305592    2544 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-raw-16e43-predictor-84c9b8899d-2ms7b" event={"ID":"66695b97-dadb-4454-a7e1-940f31ef3bd9","Type":"ContainerDied","Data":"ea9330e75f702af4b1c9c65f24155cd8ff1ae135542a3db73ec8025a7bb7a2d5"}
Sep 16 19:00:38.434898 ip-10-0-132-80 kubenswrapper[2544]: I0916 19:00:38.434856    2544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-d85e8-predictor-7fbccc8dc4-gd9xw"
Sep 16 19:00:38.709342 ip-10-0-132-80 kubenswrapper[2544]: I0916 19:00:38.709317    2544 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-scale-raw-d85e8-predictor-7fbccc8dc4-gd9xw"]
Sep 16 19:00:38.712019 ip-10-0-132-80 kubenswrapper[2544]: W0916 19:00:38.711993    2544 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddc269306_d6b2_4857_a15f_fb4579de499f.slice/crio-0c351f0dcef8407a28add6d5a468123542818b23eef167bfadec66f6c503f647 WatchSource:0}: Error finding container 0c351f0dcef8407a28add6d5a468123542818b23eef167bfadec66f6c503f647: Status 404 returned error can't find the container with id 0c351f0dcef8407a28add6d5a468123542818b23eef167bfadec66f6c503f647
Sep 16 19:00:39.311223 ip-10-0-132-80 kubenswrapper[2544]: I0916 19:00:39.311181    2544 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-d85e8-predictor-7fbccc8dc4-gd9xw" event={"ID":"dc269306-d6b2-4857-a15f-fb4579de499f","Type":"ContainerStarted","Data":"cfbbde950a5c2c9726a8e17223fa2520fa118b37fb633104ee8ed4bbe174fcc2"}
Sep 16 19:00:39.311675 ip-10-0-132-80 kubenswrapper[2544]: I0916 19:00:39.311248    2544 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-d85e8-predictor-7fbccc8dc4-gd9xw" event={"ID":"dc269306-d6b2-4857-a15f-fb4579de499f","Type":"ContainerStarted","Data":"0c351f0dcef8407a28add6d5a468123542818b23eef167bfadec66f6c503f647"}
Sep 16 19:00:40.934118 ip-10-0-132-80 kubenswrapper[2544]: I0916 19:00:40.934074    2544 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-16e43-predictor-84c9b8899d-2ms7b" podUID="66695b97-dadb-4454-a7e1-940f31ef3bd9" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.33:8643/healthz\": dial tcp 10.132.0.33:8643: connect: connection refused"
Sep 16 19:00:43.335455 ip-10-0-132-80 kubenswrapper[2544]: I0916 19:00:43.335420    2544 generic.go:356] "Generic (PLEG): container finished" podID="dc269306-d6b2-4857-a15f-fb4579de499f" containerID="cfbbde950a5c2c9726a8e17223fa2520fa118b37fb633104ee8ed4bbe174fcc2" exitCode=0
Sep 16 19:00:43.335961 ip-10-0-132-80 kubenswrapper[2544]: I0916 19:00:43.335490    2544 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-d85e8-predictor-7fbccc8dc4-gd9xw" event={"ID":"dc269306-d6b2-4857-a15f-fb4579de499f","Type":"ContainerDied","Data":"cfbbde950a5c2c9726a8e17223fa2520fa118b37fb633104ee8ed4bbe174fcc2"}
Sep 16 19:00:43.338203 ip-10-0-132-80 kubenswrapper[2544]: I0916 19:00:43.338113    2544 generic.go:356] "Generic (PLEG): container finished" podID="66695b97-dadb-4454-a7e1-940f31ef3bd9" containerID="d12847499eb1b07e054924ce632189c080c52816d0db02b8ba0b335a0a61e256" exitCode=0
Sep 16 19:00:43.338203 ip-10-0-132-80 kubenswrapper[2544]: I0916 19:00:43.338172    2544 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-raw-16e43-predictor-84c9b8899d-2ms7b" event={"ID":"66695b97-dadb-4454-a7e1-940f31ef3bd9","Type":"ContainerDied","Data":"d12847499eb1b07e054924ce632189c080c52816d0db02b8ba0b335a0a61e256"}
Sep 16 19:00:44.345040 ip-10-0-132-80 kubenswrapper[2544]: I0916 19:00:44.345005    2544 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-d85e8-predictor-7fbccc8dc4-gd9xw" event={"ID":"dc269306-d6b2-4857-a15f-fb4579de499f","Type":"ContainerStarted","Data":"1819793f27225b27ea96e642266ed7db8bc85524f5085ff6034b8b9b287f747e"}
Sep 16 19:00:44.345412 ip-10-0-132-80 kubenswrapper[2544]: I0916 19:00:44.345070    2544 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-d85e8-predictor-7fbccc8dc4-gd9xw" event={"ID":"dc269306-d6b2-4857-a15f-fb4579de499f","Type":"ContainerStarted","Data":"c1e2d94261074569892b39d4b28c4e8bbd5a65c04a49931d7bd92c1240d744c2"}
Sep 16 19:00:44.345412 ip-10-0-132-80 kubenswrapper[2544]: I0916 19:00:44.345333    2544 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-d85e8-predictor-7fbccc8dc4-gd9xw"
Sep 16 19:00:44.345504 ip-10-0-132-80 kubenswrapper[2544]: I0916 19:00:44.345456    2544 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-d85e8-predictor-7fbccc8dc4-gd9xw"
Sep 16 19:00:44.346793 ip-10-0-132-80 kubenswrapper[2544]: I0916 19:00:44.346766    2544 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-d85e8-predictor-7fbccc8dc4-gd9xw" podUID="dc269306-d6b2-4857-a15f-fb4579de499f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.34:8080: connect: connection refused"
Sep 16 19:00:44.363951 ip-10-0-132-80 kubenswrapper[2544]: I0916 19:00:44.363776    2544 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-d85e8-predictor-7fbccc8dc4-gd9xw" podStartSLOduration=2.666124366 podStartE2EDuration="6.36376093s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.697636564s" isStatefulPod=false podCreationTimestamp="2026-09-16 19:00:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:00:44.363199837 +0000 UTC m=+1007.152202389" watchObservedRunningTime="2026-09-16 19:00:44.36376093 +0000 UTC m=+1007.152763465"
Sep 16 19:00:45.349930 ip-10-0-132-80 kubenswrapper[2544]: I0916 19:00:45.349892    2544 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-d85e8-predictor-7fbccc8dc4-gd9xw" podUID="dc269306-d6b2-4857-a15f-fb4579de499f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.34:8080: connect: connection refused"
Sep 16 19:00:45.934358 ip-10-0-132-80 kubenswrapper[2544]: I0916 19:00:45.934316    2544 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-16e43-predictor-84c9b8899d-2ms7b" podUID="66695b97-dadb-4454-a7e1-940f31ef3bd9" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.33:8643/healthz\": dial tcp 10.132.0.33:8643: connect: connection refused"
Sep 16 19:00:45.939785 ip-10-0-132-80 kubenswrapper[2544]: I0916 19:00:45.939753    2544 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-16e43-predictor-84c9b8899d-2ms7b" podUID="66695b97-dadb-4454-a7e1-940f31ef3bd9" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.33:8080: connect: connection refused"
Sep 16 19:00:45.940072 ip-10-0-132-80 kubenswrapper[2544]: I0916 19:00:45.940051    2544 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-16e43-predictor-84c9b8899d-2ms7b" podUID="66695b97-dadb-4454-a7e1-940f31ef3bd9" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 16 19:00:50.353996 ip-10-0-132-80 kubenswrapper[2544]: I0916 19:00:50.353966    2544 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-d85e8-predictor-7fbccc8dc4-gd9xw"
Sep 16 19:00:50.354500 ip-10-0-132-80 kubenswrapper[2544]: I0916 19:00:50.354472    2544 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-d85e8-predictor-7fbccc8dc4-gd9xw" podUID="dc269306-d6b2-4857-a15f-fb4579de499f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.34:8080: connect: connection refused"
Sep 16 19:00:50.934853 ip-10-0-132-80 kubenswrapper[2544]: I0916 19:00:50.934810    2544 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-16e43-predictor-84c9b8899d-2ms7b" podUID="66695b97-dadb-4454-a7e1-940f31ef3bd9" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.33:8643/healthz\": dial tcp 10.132.0.33:8643: connect: connection refused"
Sep 16 19:00:50.935044 ip-10-0-132-80 kubenswrapper[2544]: I0916 19:00:50.934999    2544 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-logger-raw-16e43-predictor-84c9b8899d-2ms7b"
Sep 16 19:00:55.934562 ip-10-0-132-80 kubenswrapper[2544]: I0916 19:00:55.934509    2544 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-16e43-predictor-84c9b8899d-2ms7b" podUID="66695b97-dadb-4454-a7e1-940f31ef3bd9" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.33:8643/healthz\": dial tcp 10.132.0.33:8643: connect: connection refused"
Sep 16 19:00:55.938903 ip-10-0-132-80 kubenswrapper[2544]: I0916 19:00:55.938875    2544 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-16e43-predictor-84c9b8899d-2ms7b" podUID="66695b97-dadb-4454-a7e1-940f31ef3bd9" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.33:8080: connect: connection refused"
Sep 16 19:00:55.939224 ip-10-0-132-80 kubenswrapper[2544]: I0916 19:00:55.939201    2544 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-16e43-predictor-84c9b8899d-2ms7b" podUID="66695b97-dadb-4454-a7e1-940f31ef3bd9" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 16 19:01:00.354572 ip-10-0-132-80 kubenswrapper[2544]: I0916 19:01:00.354521    2544 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-d85e8-predictor-7fbccc8dc4-gd9xw" podUID="dc269306-d6b2-4857-a15f-fb4579de499f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.34:8080: connect: connection refused"
Sep 16 19:01:00.934089 ip-10-0-132-80 kubenswrapper[2544]: I0916 19:01:00.934043    2544 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-16e43-predictor-84c9b8899d-2ms7b" podUID="66695b97-dadb-4454-a7e1-940f31ef3bd9" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.33:8643/healthz\": dial tcp 10.132.0.33:8643: connect: connection refused"
Sep 16 19:01:05.934488 ip-10-0-132-80 kubenswrapper[2544]: I0916 19:01:05.934441    2544 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-16e43-predictor-84c9b8899d-2ms7b" podUID="66695b97-dadb-4454-a7e1-940f31ef3bd9" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.33:8643/healthz\": dial tcp 10.132.0.33:8643: connect: connection refused"
Sep 16 19:01:05.938936 ip-10-0-132-80 kubenswrapper[2544]: I0916 19:01:05.938908    2544 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-16e43-predictor-84c9b8899d-2ms7b" podUID="66695b97-dadb-4454-a7e1-940f31ef3bd9" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.33:8080: connect: connection refused"
Sep 16 19:01:05.939061 ip-10-0-132-80 kubenswrapper[2544]: I0916 19:01:05.939046    2544 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-logger-raw-16e43-predictor-84c9b8899d-2ms7b"
Sep 16 19:01:05.939366 ip-10-0-132-80 kubenswrapper[2544]: I0916 19:01:05.939346    2544 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-16e43-predictor-84c9b8899d-2ms7b" podUID="66695b97-dadb-4454-a7e1-940f31ef3bd9" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 16 19:01:05.939462 ip-10-0-132-80 kubenswrapper[2544]: I0916 19:01:05.939432    2544 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-logger-raw-16e43-predictor-84c9b8899d-2ms7b"
Sep 16 19:01:08.123170 ip-10-0-132-80 kubenswrapper[2544]: I0916 19:01:08.123138    2544 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-logger-raw-16e43-predictor-84c9b8899d-2ms7b"
Sep 16 19:01:08.216301 ip-10-0-132-80 kubenswrapper[2544]: I0916 19:01:08.216267    2544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/66695b97-dadb-4454-a7e1-940f31ef3bd9-cabundle-cert\") pod \"66695b97-dadb-4454-a7e1-940f31ef3bd9\" (UID: \"66695b97-dadb-4454-a7e1-940f31ef3bd9\") "
Sep 16 19:01:08.216495 ip-10-0-132-80 kubenswrapper[2544]: I0916 19:01:08.216313    2544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/66695b97-dadb-4454-a7e1-940f31ef3bd9-proxy-tls\") pod \"66695b97-dadb-4454-a7e1-940f31ef3bd9\" (UID: \"66695b97-dadb-4454-a7e1-940f31ef3bd9\") "
Sep 16 19:01:08.216495 ip-10-0-132-80 kubenswrapper[2544]: I0916 19:01:08.216439    2544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-tsvzt\" (UniqueName: \"kubernetes.io/projected/66695b97-dadb-4454-a7e1-940f31ef3bd9-kube-api-access-tsvzt\") pod \"66695b97-dadb-4454-a7e1-940f31ef3bd9\" (UID: \"66695b97-dadb-4454-a7e1-940f31ef3bd9\") "
Sep 16 19:01:08.216495 ip-10-0-132-80 kubenswrapper[2544]: I0916 19:01:08.216471    2544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/66695b97-dadb-4454-a7e1-940f31ef3bd9-kserve-provision-location\") pod \"66695b97-dadb-4454-a7e1-940f31ef3bd9\" (UID: \"66695b97-dadb-4454-a7e1-940f31ef3bd9\") "
Sep 16 19:01:08.216713 ip-10-0-132-80 kubenswrapper[2544]: I0916 19:01:08.216512    2544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/66695b97-dadb-4454-a7e1-940f31ef3bd9-kube-rbac-proxy-sar-config\") pod \"66695b97-dadb-4454-a7e1-940f31ef3bd9\" (UID: \"66695b97-dadb-4454-a7e1-940f31ef3bd9\") "
Sep 16 19:01:08.216838 ip-10-0-132-80 kubenswrapper[2544]: I0916 19:01:08.216725    2544 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/66695b97-dadb-4454-a7e1-940f31ef3bd9-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "66695b97-dadb-4454-a7e1-940f31ef3bd9" (UID: "66695b97-dadb-4454-a7e1-940f31ef3bd9"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 19:01:08.216904 ip-10-0-132-80 kubenswrapper[2544]: I0916 19:01:08.216845    2544 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/66695b97-dadb-4454-a7e1-940f31ef3bd9-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "66695b97-dadb-4454-a7e1-940f31ef3bd9" (UID: "66695b97-dadb-4454-a7e1-940f31ef3bd9"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:01:08.216959 ip-10-0-132-80 kubenswrapper[2544]: I0916 19:01:08.216917    2544 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/66695b97-dadb-4454-a7e1-940f31ef3bd9-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "66695b97-dadb-4454-a7e1-940f31ef3bd9" (UID: "66695b97-dadb-4454-a7e1-940f31ef3bd9"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 19:01:08.218484 ip-10-0-132-80 kubenswrapper[2544]: I0916 19:01:08.218459    2544 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/66695b97-dadb-4454-a7e1-940f31ef3bd9-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "66695b97-dadb-4454-a7e1-940f31ef3bd9" (UID: "66695b97-dadb-4454-a7e1-940f31ef3bd9"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 19:01:08.218672 ip-10-0-132-80 kubenswrapper[2544]: I0916 19:01:08.218495    2544 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/66695b97-dadb-4454-a7e1-940f31ef3bd9-kube-api-access-tsvzt" (OuterVolumeSpecName: "kube-api-access-tsvzt") pod "66695b97-dadb-4454-a7e1-940f31ef3bd9" (UID: "66695b97-dadb-4454-a7e1-940f31ef3bd9"). InnerVolumeSpecName "kube-api-access-tsvzt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 19:01:08.317902 ip-10-0-132-80 kubenswrapper[2544]: I0916 19:01:08.317875    2544 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tsvzt\" (UniqueName: \"kubernetes.io/projected/66695b97-dadb-4454-a7e1-940f31ef3bd9-kube-api-access-tsvzt\") on node \"ip-10-0-132-80.ec2.internal\" DevicePath \"\""
Sep 16 19:01:08.317902 ip-10-0-132-80 kubenswrapper[2544]: I0916 19:01:08.317903    2544 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/66695b97-dadb-4454-a7e1-940f31ef3bd9-kserve-provision-location\") on node \"ip-10-0-132-80.ec2.internal\" DevicePath \"\""
Sep 16 19:01:08.318084 ip-10-0-132-80 kubenswrapper[2544]: I0916 19:01:08.317912    2544 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/66695b97-dadb-4454-a7e1-940f31ef3bd9-kube-rbac-proxy-sar-config\") on node \"ip-10-0-132-80.ec2.internal\" DevicePath \"\""
Sep 16 19:01:08.318084 ip-10-0-132-80 kubenswrapper[2544]: I0916 19:01:08.317921    2544 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/66695b97-dadb-4454-a7e1-940f31ef3bd9-cabundle-cert\") on node \"ip-10-0-132-80.ec2.internal\" DevicePath \"\""
Sep 16 19:01:08.318084 ip-10-0-132-80 kubenswrapper[2544]: I0916 19:01:08.317932    2544 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/66695b97-dadb-4454-a7e1-940f31ef3bd9-proxy-tls\") on node \"ip-10-0-132-80.ec2.internal\" DevicePath \"\""
Sep 16 19:01:08.451757 ip-10-0-132-80 kubenswrapper[2544]: I0916 19:01:08.451722    2544 generic.go:356] "Generic (PLEG): container finished" podID="66695b97-dadb-4454-a7e1-940f31ef3bd9" containerID="22056d58e87300938cf26915b2a5d63da53497026635168b5f7b2a1f9cad6ff6" exitCode=0
Sep 16 19:01:08.451757 ip-10-0-132-80 kubenswrapper[2544]: I0916 19:01:08.451764    2544 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-raw-16e43-predictor-84c9b8899d-2ms7b" event={"ID":"66695b97-dadb-4454-a7e1-940f31ef3bd9","Type":"ContainerDied","Data":"22056d58e87300938cf26915b2a5d63da53497026635168b5f7b2a1f9cad6ff6"}
Sep 16 19:01:08.451955 ip-10-0-132-80 kubenswrapper[2544]: I0916 19:01:08.451783    2544 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-logger-raw-16e43-predictor-84c9b8899d-2ms7b"
Sep 16 19:01:08.451955 ip-10-0-132-80 kubenswrapper[2544]: I0916 19:01:08.451817    2544 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-raw-16e43-predictor-84c9b8899d-2ms7b" event={"ID":"66695b97-dadb-4454-a7e1-940f31ef3bd9","Type":"ContainerDied","Data":"1d7ab2658f28d07f71db7770d4e461a67e6acc2d5b0cde0ff58e40eab1b2668c"}
Sep 16 19:01:08.451955 ip-10-0-132-80 kubenswrapper[2544]: I0916 19:01:08.451847    2544 scope.go:117] "RemoveContainer" containerID="22056d58e87300938cf26915b2a5d63da53497026635168b5f7b2a1f9cad6ff6"
Sep 16 19:01:08.460286 ip-10-0-132-80 kubenswrapper[2544]: I0916 19:01:08.460267    2544 scope.go:117] "RemoveContainer" containerID="ea9330e75f702af4b1c9c65f24155cd8ff1ae135542a3db73ec8025a7bb7a2d5"
Sep 16 19:01:08.467680 ip-10-0-132-80 kubenswrapper[2544]: I0916 19:01:08.467661    2544 scope.go:117] "RemoveContainer" containerID="d12847499eb1b07e054924ce632189c080c52816d0db02b8ba0b335a0a61e256"
Sep 16 19:01:08.474803 ip-10-0-132-80 kubenswrapper[2544]: I0916 19:01:08.474780    2544 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-logger-raw-16e43-predictor-84c9b8899d-2ms7b"]
Sep 16 19:01:08.475184 ip-10-0-132-80 kubenswrapper[2544]: I0916 19:01:08.475166    2544 scope.go:117] "RemoveContainer" containerID="a6d08c2b7d4bf591b1d2b3e0ccab0fecc4df3c6a3f98c549394e65dd4bd3c0c0"
Sep 16 19:01:08.478679 ip-10-0-132-80 kubenswrapper[2544]: I0916 19:01:08.478657    2544 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-logger-raw-16e43-predictor-84c9b8899d-2ms7b"]
Sep 16 19:01:08.482642 ip-10-0-132-80 kubenswrapper[2544]: I0916 19:01:08.482625    2544 scope.go:117] "RemoveContainer" containerID="22056d58e87300938cf26915b2a5d63da53497026635168b5f7b2a1f9cad6ff6"
Sep 16 19:01:08.482930 ip-10-0-132-80 kubenswrapper[2544]: E0916 19:01:08.482913    2544 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"22056d58e87300938cf26915b2a5d63da53497026635168b5f7b2a1f9cad6ff6\": container with ID starting with 22056d58e87300938cf26915b2a5d63da53497026635168b5f7b2a1f9cad6ff6 not found: ID does not exist" containerID="22056d58e87300938cf26915b2a5d63da53497026635168b5f7b2a1f9cad6ff6"
Sep 16 19:01:08.483021 ip-10-0-132-80 kubenswrapper[2544]: I0916 19:01:08.482941    2544 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"22056d58e87300938cf26915b2a5d63da53497026635168b5f7b2a1f9cad6ff6"} err="failed to get container status \"22056d58e87300938cf26915b2a5d63da53497026635168b5f7b2a1f9cad6ff6\": rpc error: code = NotFound desc = could not find container \"22056d58e87300938cf26915b2a5d63da53497026635168b5f7b2a1f9cad6ff6\": container with ID starting with 22056d58e87300938cf26915b2a5d63da53497026635168b5f7b2a1f9cad6ff6 not found: ID does not exist"
Sep 16 19:01:08.483021 ip-10-0-132-80 kubenswrapper[2544]: I0916 19:01:08.482964    2544 scope.go:117] "RemoveContainer" containerID="ea9330e75f702af4b1c9c65f24155cd8ff1ae135542a3db73ec8025a7bb7a2d5"
Sep 16 19:01:08.483232 ip-10-0-132-80 kubenswrapper[2544]: E0916 19:01:08.483214    2544 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ea9330e75f702af4b1c9c65f24155cd8ff1ae135542a3db73ec8025a7bb7a2d5\": container with ID starting with ea9330e75f702af4b1c9c65f24155cd8ff1ae135542a3db73ec8025a7bb7a2d5 not found: ID does not exist" containerID="ea9330e75f702af4b1c9c65f24155cd8ff1ae135542a3db73ec8025a7bb7a2d5"
Sep 16 19:01:08.483271 ip-10-0-132-80 kubenswrapper[2544]: I0916 19:01:08.483237    2544 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ea9330e75f702af4b1c9c65f24155cd8ff1ae135542a3db73ec8025a7bb7a2d5"} err="failed to get container status \"ea9330e75f702af4b1c9c65f24155cd8ff1ae135542a3db73ec8025a7bb7a2d5\": rpc error: code = NotFound desc = could not find container \"ea9330e75f702af4b1c9c65f24155cd8ff1ae135542a3db73ec8025a7bb7a2d5\": container with ID starting with ea9330e75f702af4b1c9c65f24155cd8ff1ae135542a3db73ec8025a7bb7a2d5 not found: ID does not exist"
Sep 16 19:01:08.483271 ip-10-0-132-80 kubenswrapper[2544]: I0916 19:01:08.483252    2544 scope.go:117] "RemoveContainer" containerID="d12847499eb1b07e054924ce632189c080c52816d0db02b8ba0b335a0a61e256"
Sep 16 19:01:08.483478 ip-10-0-132-80 kubenswrapper[2544]: E0916 19:01:08.483460    2544 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d12847499eb1b07e054924ce632189c080c52816d0db02b8ba0b335a0a61e256\": container with ID starting with d12847499eb1b07e054924ce632189c080c52816d0db02b8ba0b335a0a61e256 not found: ID does not exist" containerID="d12847499eb1b07e054924ce632189c080c52816d0db02b8ba0b335a0a61e256"
Sep 16 19:01:08.483600 ip-10-0-132-80 kubenswrapper[2544]: I0916 19:01:08.483485    2544 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d12847499eb1b07e054924ce632189c080c52816d0db02b8ba0b335a0a61e256"} err="failed to get container status \"d12847499eb1b07e054924ce632189c080c52816d0db02b8ba0b335a0a61e256\": rpc error: code = NotFound desc = could not find container \"d12847499eb1b07e054924ce632189c080c52816d0db02b8ba0b335a0a61e256\": container with ID starting with d12847499eb1b07e054924ce632189c080c52816d0db02b8ba0b335a0a61e256 not found: ID does not exist"
Sep 16 19:01:08.483600 ip-10-0-132-80 kubenswrapper[2544]: I0916 19:01:08.483505    2544 scope.go:117] "RemoveContainer" containerID="a6d08c2b7d4bf591b1d2b3e0ccab0fecc4df3c6a3f98c549394e65dd4bd3c0c0"
Sep 16 19:01:08.483820 ip-10-0-132-80 kubenswrapper[2544]: E0916 19:01:08.483798    2544 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a6d08c2b7d4bf591b1d2b3e0ccab0fecc4df3c6a3f98c549394e65dd4bd3c0c0\": container with ID starting with a6d08c2b7d4bf591b1d2b3e0ccab0fecc4df3c6a3f98c549394e65dd4bd3c0c0 not found: ID does not exist" containerID="a6d08c2b7d4bf591b1d2b3e0ccab0fecc4df3c6a3f98c549394e65dd4bd3c0c0"
Sep 16 19:01:08.483869 ip-10-0-132-80 kubenswrapper[2544]: I0916 19:01:08.483828    2544 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a6d08c2b7d4bf591b1d2b3e0ccab0fecc4df3c6a3f98c549394e65dd4bd3c0c0"} err="failed to get container status \"a6d08c2b7d4bf591b1d2b3e0ccab0fecc4df3c6a3f98c549394e65dd4bd3c0c0\": rpc error: code = NotFound desc = could not find container \"a6d08c2b7d4bf591b1d2b3e0ccab0fecc4df3c6a3f98c549394e65dd4bd3c0c0\": container with ID starting with a6d08c2b7d4bf591b1d2b3e0ccab0fecc4df3c6a3f98c549394e65dd4bd3c0c0 not found: ID does not exist"
Sep 16 19:01:09.721943 ip-10-0-132-80 kubenswrapper[2544]: I0916 19:01:09.721905    2544 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="66695b97-dadb-4454-a7e1-940f31ef3bd9" path="/var/lib/kubelet/pods/66695b97-dadb-4454-a7e1-940f31ef3bd9/volumes"
Sep 16 19:01:10.354372 ip-10-0-132-80 kubenswrapper[2544]: I0916 19:01:10.354335    2544 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-d85e8-predictor-7fbccc8dc4-gd9xw" podUID="dc269306-d6b2-4857-a15f-fb4579de499f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.34:8080: connect: connection refused"
Sep 16 19:01:20.354901 ip-10-0-132-80 kubenswrapper[2544]: I0916 19:01:20.354859    2544 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-d85e8-predictor-7fbccc8dc4-gd9xw" podUID="dc269306-d6b2-4857-a15f-fb4579de499f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.34:8080: connect: connection refused"
Sep 16 19:01:30.354430 ip-10-0-132-80 kubenswrapper[2544]: I0916 19:01:30.354387    2544 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-d85e8-predictor-7fbccc8dc4-gd9xw" podUID="dc269306-d6b2-4857-a15f-fb4579de499f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.34:8080: connect: connection refused"
Sep 16 19:01:40.354921 ip-10-0-132-80 kubenswrapper[2544]: I0916 19:01:40.354881    2544 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-d85e8-predictor-7fbccc8dc4-gd9xw" podUID="dc269306-d6b2-4857-a15f-fb4579de499f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.34:8080: connect: connection refused"
Sep 16 19:01:50.355460 ip-10-0-132-80 kubenswrapper[2544]: I0916 19:01:50.355373    2544 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-d85e8-predictor-7fbccc8dc4-gd9xw" podUID="dc269306-d6b2-4857-a15f-fb4579de499f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.34:8080: connect: connection refused"
Sep 16 19:02:00.354635 ip-10-0-132-80 kubenswrapper[2544]: I0916 19:02:00.354592    2544 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-d85e8-predictor-7fbccc8dc4-gd9xw" podUID="dc269306-d6b2-4857-a15f-fb4579de499f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.34:8080: connect: connection refused"
Sep 16 19:02:10.355134 ip-10-0-132-80 kubenswrapper[2544]: I0916 19:02:10.355090    2544 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-d85e8-predictor-7fbccc8dc4-gd9xw" podUID="dc269306-d6b2-4857-a15f-fb4579de499f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.34:8080: connect: connection refused"
Sep 16 19:02:20.355319 ip-10-0-132-80 kubenswrapper[2544]: I0916 19:02:20.355275    2544 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-d85e8-predictor-7fbccc8dc4-gd9xw" podUID="dc269306-d6b2-4857-a15f-fb4579de499f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.34:8080: connect: connection refused"
Sep 16 19:02:20.717121 ip-10-0-132-80 kubenswrapper[2544]: I0916 19:02:20.717028    2544 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-d85e8-predictor-7fbccc8dc4-gd9xw" podUID="dc269306-d6b2-4857-a15f-fb4579de499f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.34:8080: connect: connection refused"
Sep 16 19:02:30.717514 ip-10-0-132-80 kubenswrapper[2544]: I0916 19:02:30.717464    2544 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-d85e8-predictor-7fbccc8dc4-gd9xw" podUID="dc269306-d6b2-4857-a15f-fb4579de499f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.34:8080: connect: connection refused"
Sep 16 19:02:40.717526 ip-10-0-132-80 kubenswrapper[2544]: I0916 19:02:40.717480    2544 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-d85e8-predictor-7fbccc8dc4-gd9xw" podUID="dc269306-d6b2-4857-a15f-fb4579de499f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.34:8080: connect: connection refused"
Sep 16 19:02:50.717920 ip-10-0-132-80 kubenswrapper[2544]: I0916 19:02:50.717875    2544 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-d85e8-predictor-7fbccc8dc4-gd9xw" podUID="dc269306-d6b2-4857-a15f-fb4579de499f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.34:8080: connect: connection refused"
Sep 16 19:03:00.717271 ip-10-0-132-80 kubenswrapper[2544]: I0916 19:03:00.717227    2544 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-d85e8-predictor-7fbccc8dc4-gd9xw" podUID="dc269306-d6b2-4857-a15f-fb4579de499f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.34:8080: connect: connection refused"
Sep 16 19:03:10.717973 ip-10-0-132-80 kubenswrapper[2544]: I0916 19:03:10.717927    2544 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-d85e8-predictor-7fbccc8dc4-gd9xw" podUID="dc269306-d6b2-4857-a15f-fb4579de499f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.34:8080: connect: connection refused"
Sep 16 19:03:20.717769 ip-10-0-132-80 kubenswrapper[2544]: I0916 19:03:20.717693    2544 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-d85e8-predictor-7fbccc8dc4-gd9xw"
Sep 16 19:03:28.148067 ip-10-0-132-80 kubenswrapper[2544]: I0916 19:03:28.148032    2544 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-scale-raw-d85e8-predictor-7fbccc8dc4-gd9xw"]
Sep 16 19:03:28.148744 ip-10-0-132-80 kubenswrapper[2544]: I0916 19:03:28.148355    2544 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-d85e8-predictor-7fbccc8dc4-gd9xw" podUID="dc269306-d6b2-4857-a15f-fb4579de499f" containerName="kserve-container" containerID="cri-o://c1e2d94261074569892b39d4b28c4e8bbd5a65c04a49931d7bd92c1240d744c2" gracePeriod=30
Sep 16 19:03:28.148744 ip-10-0-132-80 kubenswrapper[2544]: I0916 19:03:28.148394    2544 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-d85e8-predictor-7fbccc8dc4-gd9xw" podUID="dc269306-d6b2-4857-a15f-fb4579de499f" containerName="kube-rbac-proxy" containerID="cri-o://1819793f27225b27ea96e642266ed7db8bc85524f5085ff6034b8b9b287f747e" gracePeriod=30
Sep 16 19:03:29.039855 ip-10-0-132-80 kubenswrapper[2544]: I0916 19:03:29.039824    2544 generic.go:356] "Generic (PLEG): container finished" podID="dc269306-d6b2-4857-a15f-fb4579de499f" containerID="1819793f27225b27ea96e642266ed7db8bc85524f5085ff6034b8b9b287f747e" exitCode=2
Sep 16 19:03:29.040045 ip-10-0-132-80 kubenswrapper[2544]: I0916 19:03:29.039899    2544 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-d85e8-predictor-7fbccc8dc4-gd9xw" event={"ID":"dc269306-d6b2-4857-a15f-fb4579de499f","Type":"ContainerDied","Data":"1819793f27225b27ea96e642266ed7db8bc85524f5085ff6034b8b9b287f747e"}
Sep 16 19:03:30.350898 ip-10-0-132-80 kubenswrapper[2544]: I0916 19:03:30.350855    2544 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-d85e8-predictor-7fbccc8dc4-gd9xw" podUID="dc269306-d6b2-4857-a15f-fb4579de499f" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.34:8643/healthz\": dial tcp 10.132.0.34:8643: connect: connection refused"
Sep 16 19:03:30.717257 ip-10-0-132-80 kubenswrapper[2544]: I0916 19:03:30.717167    2544 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-d85e8-predictor-7fbccc8dc4-gd9xw" podUID="dc269306-d6b2-4857-a15f-fb4579de499f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.34:8080: connect: connection refused"
Sep 16 19:03:33.392014 ip-10-0-132-80 kubenswrapper[2544]: I0916 19:03:33.391976    2544 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-predictor-6cc94c6f4f-z7jld"]
Sep 16 19:03:33.392391 ip-10-0-132-80 kubenswrapper[2544]: I0916 19:03:33.392315    2544 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="66695b97-dadb-4454-a7e1-940f31ef3bd9" containerName="storage-initializer"
Sep 16 19:03:33.392391 ip-10-0-132-80 kubenswrapper[2544]: I0916 19:03:33.392326    2544 state_mem.go:107] "Deleted CPUSet assignment" podUID="66695b97-dadb-4454-a7e1-940f31ef3bd9" containerName="storage-initializer"
Sep 16 19:03:33.392391 ip-10-0-132-80 kubenswrapper[2544]: I0916 19:03:33.392334    2544 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="66695b97-dadb-4454-a7e1-940f31ef3bd9" containerName="kserve-container"
Sep 16 19:03:33.392391 ip-10-0-132-80 kubenswrapper[2544]: I0916 19:03:33.392339    2544 state_mem.go:107] "Deleted CPUSet assignment" podUID="66695b97-dadb-4454-a7e1-940f31ef3bd9" containerName="kserve-container"
Sep 16 19:03:33.392391 ip-10-0-132-80 kubenswrapper[2544]: I0916 19:03:33.392349    2544 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="66695b97-dadb-4454-a7e1-940f31ef3bd9" containerName="agent"
Sep 16 19:03:33.392391 ip-10-0-132-80 kubenswrapper[2544]: I0916 19:03:33.392356    2544 state_mem.go:107] "Deleted CPUSet assignment" podUID="66695b97-dadb-4454-a7e1-940f31ef3bd9" containerName="agent"
Sep 16 19:03:33.392391 ip-10-0-132-80 kubenswrapper[2544]: I0916 19:03:33.392363    2544 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="66695b97-dadb-4454-a7e1-940f31ef3bd9" containerName="kube-rbac-proxy"
Sep 16 19:03:33.392391 ip-10-0-132-80 kubenswrapper[2544]: I0916 19:03:33.392369    2544 state_mem.go:107] "Deleted CPUSet assignment" podUID="66695b97-dadb-4454-a7e1-940f31ef3bd9" containerName="kube-rbac-proxy"
Sep 16 19:03:33.392664 ip-10-0-132-80 kubenswrapper[2544]: I0916 19:03:33.392423    2544 memory_manager.go:368] "RemoveStaleState removing state" podUID="66695b97-dadb-4454-a7e1-940f31ef3bd9" containerName="kube-rbac-proxy"
Sep 16 19:03:33.392664 ip-10-0-132-80 kubenswrapper[2544]: I0916 19:03:33.392432    2544 memory_manager.go:368] "RemoveStaleState removing state" podUID="66695b97-dadb-4454-a7e1-940f31ef3bd9" containerName="kserve-container"
Sep 16 19:03:33.392664 ip-10-0-132-80 kubenswrapper[2544]: I0916 19:03:33.392440    2544 memory_manager.go:368] "RemoveStaleState removing state" podUID="66695b97-dadb-4454-a7e1-940f31ef3bd9" containerName="agent"
Sep 16 19:03:33.395721 ip-10-0-132-80 kubenswrapper[2544]: I0916 19:03:33.395701    2544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-6cc94c6f4f-z7jld"
Sep 16 19:03:33.398012 ip-10-0-132-80 kubenswrapper[2544]: I0916 19:03:33.397983    2544 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-create-predictor-serving-cert\""
Sep 16 19:03:33.398138 ip-10-0-132-80 kubenswrapper[2544]: I0916 19:03:33.397988    2544 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-create-kube-rbac-proxy-sar-config\""
Sep 16 19:03:33.407170 ip-10-0-132-80 kubenswrapper[2544]: I0916 19:03:33.407145    2544 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-predictor-6cc94c6f4f-z7jld"]
Sep 16 19:03:33.423500 ip-10-0-132-80 kubenswrapper[2544]: I0916 19:03:33.423473    2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d68cbf45-8d2e-4c9c-a40b-6464057b3091-kserve-provision-location\") pod \"isvc-canary-create-predictor-6cc94c6f4f-z7jld\" (UID: \"d68cbf45-8d2e-4c9c-a40b-6464057b3091\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-6cc94c6f4f-z7jld"
Sep 16 19:03:33.423652 ip-10-0-132-80 kubenswrapper[2544]: I0916 19:03:33.423508    2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d68cbf45-8d2e-4c9c-a40b-6464057b3091-cabundle-cert\") pod \"isvc-canary-create-predictor-6cc94c6f4f-z7jld\" (UID: \"d68cbf45-8d2e-4c9c-a40b-6464057b3091\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-6cc94c6f4f-z7jld"
Sep 16 19:03:33.423652 ip-10-0-132-80 kubenswrapper[2544]: I0916 19:03:33.423538    2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d68cbf45-8d2e-4c9c-a40b-6464057b3091-kube-rbac-proxy-sar-config\") pod \"isvc-canary-create-predictor-6cc94c6f4f-z7jld\" (UID: \"d68cbf45-8d2e-4c9c-a40b-6464057b3091\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-6cc94c6f4f-z7jld"
Sep 16 19:03:33.423652 ip-10-0-132-80 kubenswrapper[2544]: I0916 19:03:33.423638    2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d68cbf45-8d2e-4c9c-a40b-6464057b3091-proxy-tls\") pod \"isvc-canary-create-predictor-6cc94c6f4f-z7jld\" (UID: \"d68cbf45-8d2e-4c9c-a40b-6464057b3091\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-6cc94c6f4f-z7jld"
Sep 16 19:03:33.423807 ip-10-0-132-80 kubenswrapper[2544]: I0916 19:03:33.423678    2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-slwtz\" (UniqueName: \"kubernetes.io/projected/d68cbf45-8d2e-4c9c-a40b-6464057b3091-kube-api-access-slwtz\") pod \"isvc-canary-create-predictor-6cc94c6f4f-z7jld\" (UID: \"d68cbf45-8d2e-4c9c-a40b-6464057b3091\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-6cc94c6f4f-z7jld"
Sep 16 19:03:33.524933 ip-10-0-132-80 kubenswrapper[2544]: I0916 19:03:33.524889    2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d68cbf45-8d2e-4c9c-a40b-6464057b3091-proxy-tls\") pod \"isvc-canary-create-predictor-6cc94c6f4f-z7jld\" (UID: \"d68cbf45-8d2e-4c9c-a40b-6464057b3091\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-6cc94c6f4f-z7jld"
Sep 16 19:03:33.524933 ip-10-0-132-80 kubenswrapper[2544]: I0916 19:03:33.524940    2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-slwtz\" (UniqueName: \"kubernetes.io/projected/d68cbf45-8d2e-4c9c-a40b-6464057b3091-kube-api-access-slwtz\") pod \"isvc-canary-create-predictor-6cc94c6f4f-z7jld\" (UID: \"d68cbf45-8d2e-4c9c-a40b-6464057b3091\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-6cc94c6f4f-z7jld"
Sep 16 19:03:33.525169 ip-10-0-132-80 kubenswrapper[2544]: I0916 19:03:33.525009    2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d68cbf45-8d2e-4c9c-a40b-6464057b3091-kserve-provision-location\") pod \"isvc-canary-create-predictor-6cc94c6f4f-z7jld\" (UID: \"d68cbf45-8d2e-4c9c-a40b-6464057b3091\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-6cc94c6f4f-z7jld"
Sep 16 19:03:33.525169 ip-10-0-132-80 kubenswrapper[2544]: I0916 19:03:33.525028    2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d68cbf45-8d2e-4c9c-a40b-6464057b3091-cabundle-cert\") pod \"isvc-canary-create-predictor-6cc94c6f4f-z7jld\" (UID: \"d68cbf45-8d2e-4c9c-a40b-6464057b3091\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-6cc94c6f4f-z7jld"
Sep 16 19:03:33.525169 ip-10-0-132-80 kubenswrapper[2544]: I0916 19:03:33.525057    2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d68cbf45-8d2e-4c9c-a40b-6464057b3091-kube-rbac-proxy-sar-config\") pod \"isvc-canary-create-predictor-6cc94c6f4f-z7jld\" (UID: \"d68cbf45-8d2e-4c9c-a40b-6464057b3091\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-6cc94c6f4f-z7jld"
Sep 16 19:03:33.525440 ip-10-0-132-80 kubenswrapper[2544]: I0916 19:03:33.525418    2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d68cbf45-8d2e-4c9c-a40b-6464057b3091-kserve-provision-location\") pod \"isvc-canary-create-predictor-6cc94c6f4f-z7jld\" (UID: \"d68cbf45-8d2e-4c9c-a40b-6464057b3091\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-6cc94c6f4f-z7jld"
Sep 16 19:03:33.525867 ip-10-0-132-80 kubenswrapper[2544]: I0916 19:03:33.525846    2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d68cbf45-8d2e-4c9c-a40b-6464057b3091-cabundle-cert\") pod \"isvc-canary-create-predictor-6cc94c6f4f-z7jld\" (UID: \"d68cbf45-8d2e-4c9c-a40b-6464057b3091\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-6cc94c6f4f-z7jld"
Sep 16 19:03:33.525921 ip-10-0-132-80 kubenswrapper[2544]: I0916 19:03:33.525846    2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d68cbf45-8d2e-4c9c-a40b-6464057b3091-kube-rbac-proxy-sar-config\") pod \"isvc-canary-create-predictor-6cc94c6f4f-z7jld\" (UID: \"d68cbf45-8d2e-4c9c-a40b-6464057b3091\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-6cc94c6f4f-z7jld"
Sep 16 19:03:33.527505 ip-10-0-132-80 kubenswrapper[2544]: I0916 19:03:33.527481    2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d68cbf45-8d2e-4c9c-a40b-6464057b3091-proxy-tls\") pod \"isvc-canary-create-predictor-6cc94c6f4f-z7jld\" (UID: \"d68cbf45-8d2e-4c9c-a40b-6464057b3091\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-6cc94c6f4f-z7jld"
Sep 16 19:03:33.534226 ip-10-0-132-80 kubenswrapper[2544]: I0916 19:03:33.534065    2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-slwtz\" (UniqueName: \"kubernetes.io/projected/d68cbf45-8d2e-4c9c-a40b-6464057b3091-kube-api-access-slwtz\") pod \"isvc-canary-create-predictor-6cc94c6f4f-z7jld\" (UID: \"d68cbf45-8d2e-4c9c-a40b-6464057b3091\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-6cc94c6f4f-z7jld"
Sep 16 19:03:33.705752 ip-10-0-132-80 kubenswrapper[2544]: I0916 19:03:33.705641    2544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-6cc94c6f4f-z7jld"
Sep 16 19:03:33.977619 ip-10-0-132-80 kubenswrapper[2544]: I0916 19:03:33.977594    2544 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-predictor-6cc94c6f4f-z7jld"]
Sep 16 19:03:33.979719 ip-10-0-132-80 kubenswrapper[2544]: W0916 19:03:33.979691    2544 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd68cbf45_8d2e_4c9c_a40b_6464057b3091.slice/crio-94c0c63ac71b136d8cbd622624bae3691d24790f17da8d0593ef69645e912342 WatchSource:0}: Error finding container 94c0c63ac71b136d8cbd622624bae3691d24790f17da8d0593ef69645e912342: Status 404 returned error can't find the container with id 94c0c63ac71b136d8cbd622624bae3691d24790f17da8d0593ef69645e912342
Sep 16 19:03:34.060793 ip-10-0-132-80 kubenswrapper[2544]: I0916 19:03:34.060757    2544 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-6cc94c6f4f-z7jld" event={"ID":"d68cbf45-8d2e-4c9c-a40b-6464057b3091","Type":"ContainerStarted","Data":"6f34bcb4ff357156e6476c13acf48b276cfed50174d6eeb2314ec120388468d9"}
Sep 16 19:03:34.060928 ip-10-0-132-80 kubenswrapper[2544]: I0916 19:03:34.060816    2544 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-6cc94c6f4f-z7jld" event={"ID":"d68cbf45-8d2e-4c9c-a40b-6464057b3091","Type":"ContainerStarted","Data":"94c0c63ac71b136d8cbd622624bae3691d24790f17da8d0593ef69645e912342"}
Sep 16 19:03:35.350148 ip-10-0-132-80 kubenswrapper[2544]: I0916 19:03:35.350109    2544 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-d85e8-predictor-7fbccc8dc4-gd9xw" podUID="dc269306-d6b2-4857-a15f-fb4579de499f" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.34:8643/healthz\": dial tcp 10.132.0.34:8643: connect: connection refused"
Sep 16 19:03:37.884251 ip-10-0-132-80 kubenswrapper[2544]: I0916 19:03:37.884225    2544 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-d85e8-predictor-7fbccc8dc4-gd9xw"
Sep 16 19:03:37.962254 ip-10-0-132-80 kubenswrapper[2544]: I0916 19:03:37.962159    2544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/dc269306-d6b2-4857-a15f-fb4579de499f-kube-rbac-proxy-sar-config\") pod \"dc269306-d6b2-4857-a15f-fb4579de499f\" (UID: \"dc269306-d6b2-4857-a15f-fb4579de499f\") "
Sep 16 19:03:37.962254 ip-10-0-132-80 kubenswrapper[2544]: I0916 19:03:37.962206    2544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qs5m5\" (UniqueName: \"kubernetes.io/projected/dc269306-d6b2-4857-a15f-fb4579de499f-kube-api-access-qs5m5\") pod \"dc269306-d6b2-4857-a15f-fb4579de499f\" (UID: \"dc269306-d6b2-4857-a15f-fb4579de499f\") "
Sep 16 19:03:37.962485 ip-10-0-132-80 kubenswrapper[2544]: I0916 19:03:37.962266    2544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/dc269306-d6b2-4857-a15f-fb4579de499f-proxy-tls\") pod \"dc269306-d6b2-4857-a15f-fb4579de499f\" (UID: \"dc269306-d6b2-4857-a15f-fb4579de499f\") "
Sep 16 19:03:37.962485 ip-10-0-132-80 kubenswrapper[2544]: I0916 19:03:37.962289    2544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/dc269306-d6b2-4857-a15f-fb4579de499f-kserve-provision-location\") pod \"dc269306-d6b2-4857-a15f-fb4579de499f\" (UID: \"dc269306-d6b2-4857-a15f-fb4579de499f\") "
Sep 16 19:03:37.962485 ip-10-0-132-80 kubenswrapper[2544]: I0916 19:03:37.962347    2544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/dc269306-d6b2-4857-a15f-fb4579de499f-cabundle-cert\") pod \"dc269306-d6b2-4857-a15f-fb4579de499f\" (UID: \"dc269306-d6b2-4857-a15f-fb4579de499f\") "
Sep 16 19:03:37.962685 ip-10-0-132-80 kubenswrapper[2544]: I0916 19:03:37.962656    2544 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dc269306-d6b2-4857-a15f-fb4579de499f-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "dc269306-d6b2-4857-a15f-fb4579de499f" (UID: "dc269306-d6b2-4857-a15f-fb4579de499f"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:03:37.962749 ip-10-0-132-80 kubenswrapper[2544]: I0916 19:03:37.962660    2544 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dc269306-d6b2-4857-a15f-fb4579de499f-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "dc269306-d6b2-4857-a15f-fb4579de499f" (UID: "dc269306-d6b2-4857-a15f-fb4579de499f"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 19:03:37.962790 ip-10-0-132-80 kubenswrapper[2544]: I0916 19:03:37.962763    2544 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dc269306-d6b2-4857-a15f-fb4579de499f-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "dc269306-d6b2-4857-a15f-fb4579de499f" (UID: "dc269306-d6b2-4857-a15f-fb4579de499f"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 19:03:37.964403 ip-10-0-132-80 kubenswrapper[2544]: I0916 19:03:37.964380    2544 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dc269306-d6b2-4857-a15f-fb4579de499f-kube-api-access-qs5m5" (OuterVolumeSpecName: "kube-api-access-qs5m5") pod "dc269306-d6b2-4857-a15f-fb4579de499f" (UID: "dc269306-d6b2-4857-a15f-fb4579de499f"). InnerVolumeSpecName "kube-api-access-qs5m5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 19:03:37.964491 ip-10-0-132-80 kubenswrapper[2544]: I0916 19:03:37.964381    2544 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/dc269306-d6b2-4857-a15f-fb4579de499f-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "dc269306-d6b2-4857-a15f-fb4579de499f" (UID: "dc269306-d6b2-4857-a15f-fb4579de499f"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 19:03:38.062961 ip-10-0-132-80 kubenswrapper[2544]: I0916 19:03:38.062930    2544 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/dc269306-d6b2-4857-a15f-fb4579de499f-kube-rbac-proxy-sar-config\") on node \"ip-10-0-132-80.ec2.internal\" DevicePath \"\""
Sep 16 19:03:38.062961 ip-10-0-132-80 kubenswrapper[2544]: I0916 19:03:38.062956    2544 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qs5m5\" (UniqueName: \"kubernetes.io/projected/dc269306-d6b2-4857-a15f-fb4579de499f-kube-api-access-qs5m5\") on node \"ip-10-0-132-80.ec2.internal\" DevicePath \"\""
Sep 16 19:03:38.062961 ip-10-0-132-80 kubenswrapper[2544]: I0916 19:03:38.062966    2544 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/dc269306-d6b2-4857-a15f-fb4579de499f-proxy-tls\") on node \"ip-10-0-132-80.ec2.internal\" DevicePath \"\""
Sep 16 19:03:38.063182 ip-10-0-132-80 kubenswrapper[2544]: I0916 19:03:38.062977    2544 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/dc269306-d6b2-4857-a15f-fb4579de499f-kserve-provision-location\") on node \"ip-10-0-132-80.ec2.internal\" DevicePath \"\""
Sep 16 19:03:38.063182 ip-10-0-132-80 kubenswrapper[2544]: I0916 19:03:38.062986    2544 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/dc269306-d6b2-4857-a15f-fb4579de499f-cabundle-cert\") on node \"ip-10-0-132-80.ec2.internal\" DevicePath \"\""
Sep 16 19:03:38.079206 ip-10-0-132-80 kubenswrapper[2544]: I0916 19:03:38.079177    2544 generic.go:356] "Generic (PLEG): container finished" podID="dc269306-d6b2-4857-a15f-fb4579de499f" containerID="c1e2d94261074569892b39d4b28c4e8bbd5a65c04a49931d7bd92c1240d744c2" exitCode=0
Sep 16 19:03:38.079359 ip-10-0-132-80 kubenswrapper[2544]: I0916 19:03:38.079256    2544 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-d85e8-predictor-7fbccc8dc4-gd9xw"
Sep 16 19:03:38.079359 ip-10-0-132-80 kubenswrapper[2544]: I0916 19:03:38.079254    2544 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-d85e8-predictor-7fbccc8dc4-gd9xw" event={"ID":"dc269306-d6b2-4857-a15f-fb4579de499f","Type":"ContainerDied","Data":"c1e2d94261074569892b39d4b28c4e8bbd5a65c04a49931d7bd92c1240d744c2"}
Sep 16 19:03:38.079479 ip-10-0-132-80 kubenswrapper[2544]: I0916 19:03:38.079443    2544 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-d85e8-predictor-7fbccc8dc4-gd9xw" event={"ID":"dc269306-d6b2-4857-a15f-fb4579de499f","Type":"ContainerDied","Data":"0c351f0dcef8407a28add6d5a468123542818b23eef167bfadec66f6c503f647"}
Sep 16 19:03:38.079541 ip-10-0-132-80 kubenswrapper[2544]: I0916 19:03:38.079492    2544 scope.go:117] "RemoveContainer" containerID="1819793f27225b27ea96e642266ed7db8bc85524f5085ff6034b8b9b287f747e"
Sep 16 19:03:38.080896 ip-10-0-132-80 kubenswrapper[2544]: I0916 19:03:38.080873    2544 generic.go:356] "Generic (PLEG): container finished" podID="d68cbf45-8d2e-4c9c-a40b-6464057b3091" containerID="6f34bcb4ff357156e6476c13acf48b276cfed50174d6eeb2314ec120388468d9" exitCode=0
Sep 16 19:03:38.081000 ip-10-0-132-80 kubenswrapper[2544]: I0916 19:03:38.080903    2544 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-6cc94c6f4f-z7jld" event={"ID":"d68cbf45-8d2e-4c9c-a40b-6464057b3091","Type":"ContainerDied","Data":"6f34bcb4ff357156e6476c13acf48b276cfed50174d6eeb2314ec120388468d9"}
Sep 16 19:03:38.088257 ip-10-0-132-80 kubenswrapper[2544]: I0916 19:03:38.088129    2544 scope.go:117] "RemoveContainer" containerID="c1e2d94261074569892b39d4b28c4e8bbd5a65c04a49931d7bd92c1240d744c2"
Sep 16 19:03:38.096480 ip-10-0-132-80 kubenswrapper[2544]: I0916 19:03:38.096464    2544 scope.go:117] "RemoveContainer" containerID="cfbbde950a5c2c9726a8e17223fa2520fa118b37fb633104ee8ed4bbe174fcc2"
Sep 16 19:03:38.106269 ip-10-0-132-80 kubenswrapper[2544]: I0916 19:03:38.106245    2544 scope.go:117] "RemoveContainer" containerID="1819793f27225b27ea96e642266ed7db8bc85524f5085ff6034b8b9b287f747e"
Sep 16 19:03:38.106684 ip-10-0-132-80 kubenswrapper[2544]: E0916 19:03:38.106664    2544 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1819793f27225b27ea96e642266ed7db8bc85524f5085ff6034b8b9b287f747e\": container with ID starting with 1819793f27225b27ea96e642266ed7db8bc85524f5085ff6034b8b9b287f747e not found: ID does not exist" containerID="1819793f27225b27ea96e642266ed7db8bc85524f5085ff6034b8b9b287f747e"
Sep 16 19:03:38.106746 ip-10-0-132-80 kubenswrapper[2544]: I0916 19:03:38.106694    2544 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1819793f27225b27ea96e642266ed7db8bc85524f5085ff6034b8b9b287f747e"} err="failed to get container status \"1819793f27225b27ea96e642266ed7db8bc85524f5085ff6034b8b9b287f747e\": rpc error: code = NotFound desc = could not find container \"1819793f27225b27ea96e642266ed7db8bc85524f5085ff6034b8b9b287f747e\": container with ID starting with 1819793f27225b27ea96e642266ed7db8bc85524f5085ff6034b8b9b287f747e not found: ID does not exist"
Sep 16 19:03:38.106746 ip-10-0-132-80 kubenswrapper[2544]: I0916 19:03:38.106712    2544 scope.go:117] "RemoveContainer" containerID="c1e2d94261074569892b39d4b28c4e8bbd5a65c04a49931d7bd92c1240d744c2"
Sep 16 19:03:38.106990 ip-10-0-132-80 kubenswrapper[2544]: E0916 19:03:38.106972    2544 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c1e2d94261074569892b39d4b28c4e8bbd5a65c04a49931d7bd92c1240d744c2\": container with ID starting with c1e2d94261074569892b39d4b28c4e8bbd5a65c04a49931d7bd92c1240d744c2 not found: ID does not exist" containerID="c1e2d94261074569892b39d4b28c4e8bbd5a65c04a49931d7bd92c1240d744c2"
Sep 16 19:03:38.107037 ip-10-0-132-80 kubenswrapper[2544]: I0916 19:03:38.106997    2544 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c1e2d94261074569892b39d4b28c4e8bbd5a65c04a49931d7bd92c1240d744c2"} err="failed to get container status \"c1e2d94261074569892b39d4b28c4e8bbd5a65c04a49931d7bd92c1240d744c2\": rpc error: code = NotFound desc = could not find container \"c1e2d94261074569892b39d4b28c4e8bbd5a65c04a49931d7bd92c1240d744c2\": container with ID starting with c1e2d94261074569892b39d4b28c4e8bbd5a65c04a49931d7bd92c1240d744c2 not found: ID does not exist"
Sep 16 19:03:38.107037 ip-10-0-132-80 kubenswrapper[2544]: I0916 19:03:38.107016    2544 scope.go:117] "RemoveContainer" containerID="cfbbde950a5c2c9726a8e17223fa2520fa118b37fb633104ee8ed4bbe174fcc2"
Sep 16 19:03:38.107216 ip-10-0-132-80 kubenswrapper[2544]: E0916 19:03:38.107200    2544 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cfbbde950a5c2c9726a8e17223fa2520fa118b37fb633104ee8ed4bbe174fcc2\": container with ID starting with cfbbde950a5c2c9726a8e17223fa2520fa118b37fb633104ee8ed4bbe174fcc2 not found: ID does not exist" containerID="cfbbde950a5c2c9726a8e17223fa2520fa118b37fb633104ee8ed4bbe174fcc2"
Sep 16 19:03:38.107260 ip-10-0-132-80 kubenswrapper[2544]: I0916 19:03:38.107219    2544 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cfbbde950a5c2c9726a8e17223fa2520fa118b37fb633104ee8ed4bbe174fcc2"} err="failed to get container status \"cfbbde950a5c2c9726a8e17223fa2520fa118b37fb633104ee8ed4bbe174fcc2\": rpc error: code = NotFound desc = could not find container \"cfbbde950a5c2c9726a8e17223fa2520fa118b37fb633104ee8ed4bbe174fcc2\": container with ID starting with cfbbde950a5c2c9726a8e17223fa2520fa118b37fb633104ee8ed4bbe174fcc2 not found: ID does not exist"
Sep 16 19:03:38.118524 ip-10-0-132-80 kubenswrapper[2544]: I0916 19:03:38.118498    2544 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-scale-raw-d85e8-predictor-7fbccc8dc4-gd9xw"]
Sep 16 19:03:38.125091 ip-10-0-132-80 kubenswrapper[2544]: I0916 19:03:38.125070    2544 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-scale-raw-d85e8-predictor-7fbccc8dc4-gd9xw"]
Sep 16 19:03:39.086957 ip-10-0-132-80 kubenswrapper[2544]: I0916 19:03:39.086925    2544 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-6cc94c6f4f-z7jld" event={"ID":"d68cbf45-8d2e-4c9c-a40b-6464057b3091","Type":"ContainerStarted","Data":"b368fcd10df4236463ac7afa7f1adc2e951aacd83a509e6dd868916d092107b8"}
Sep 16 19:03:39.087389 ip-10-0-132-80 kubenswrapper[2544]: I0916 19:03:39.086996    2544 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-6cc94c6f4f-z7jld" event={"ID":"d68cbf45-8d2e-4c9c-a40b-6464057b3091","Type":"ContainerStarted","Data":"e90ae18cdbfbb8b970d02d4ac91b26c49a7e64aa4e0b7a320f41e251263aafc1"}
Sep 16 19:03:39.087389 ip-10-0-132-80 kubenswrapper[2544]: I0916 19:03:39.087279    2544 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-6cc94c6f4f-z7jld"
Sep 16 19:03:39.087524 ip-10-0-132-80 kubenswrapper[2544]: I0916 19:03:39.087423    2544 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-6cc94c6f4f-z7jld"
Sep 16 19:03:39.088950 ip-10-0-132-80 kubenswrapper[2544]: I0916 19:03:39.088925    2544 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-6cc94c6f4f-z7jld" podUID="d68cbf45-8d2e-4c9c-a40b-6464057b3091" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.35:8080: connect: connection refused"
Sep 16 19:03:39.109468 ip-10-0-132-80 kubenswrapper[2544]: I0916 19:03:39.109423    2544 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-6cc94c6f4f-z7jld" podStartSLOduration=2.406544212 podStartE2EDuration="6.109411698s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.702867486s" isStatefulPod=false podCreationTimestamp="2026-09-16 19:03:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:03:39.10789531 +0000 UTC m=+1181.896897858" watchObservedRunningTime="2026-09-16 19:03:39.109411698 +0000 UTC m=+1181.898414232"
Sep 16 19:03:39.721507 ip-10-0-132-80 kubenswrapper[2544]: I0916 19:03:39.721476    2544 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="dc269306-d6b2-4857-a15f-fb4579de499f" path="/var/lib/kubelet/pods/dc269306-d6b2-4857-a15f-fb4579de499f/volumes"
Sep 16 19:03:40.091904 ip-10-0-132-80 kubenswrapper[2544]: I0916 19:03:40.091868    2544 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-6cc94c6f4f-z7jld" podUID="d68cbf45-8d2e-4c9c-a40b-6464057b3091" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.35:8080: connect: connection refused"
Sep 16 19:03:45.096912 ip-10-0-132-80 kubenswrapper[2544]: I0916 19:03:45.096881    2544 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-6cc94c6f4f-z7jld"
Sep 16 19:03:45.097328 ip-10-0-132-80 kubenswrapper[2544]: I0916 19:03:45.097306    2544 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-6cc94c6f4f-z7jld" podUID="d68cbf45-8d2e-4c9c-a40b-6464057b3091" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.35:8080: connect: connection refused"
Sep 16 19:03:55.098074 ip-10-0-132-80 kubenswrapper[2544]: I0916 19:03:55.098025    2544 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-6cc94c6f4f-z7jld" podUID="d68cbf45-8d2e-4c9c-a40b-6464057b3091" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.35:8080: connect: connection refused"
Sep 16 19:03:57.745883 ip-10-0-132-80 kubenswrapper[2544]: I0916 19:03:57.745844    2544 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-pfcb7_9e3f9e34-420e-43ef-96b4-c7cad6b2eaa9/console-operator/2.log"
Sep 16 19:03:57.747740 ip-10-0-132-80 kubenswrapper[2544]: I0916 19:03:57.747717    2544 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-pfcb7_9e3f9e34-420e-43ef-96b4-c7cad6b2eaa9/console-operator/2.log"
Sep 16 19:04:05.097770 ip-10-0-132-80 kubenswrapper[2544]: I0916 19:04:05.097733    2544 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-6cc94c6f4f-z7jld" podUID="d68cbf45-8d2e-4c9c-a40b-6464057b3091" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.35:8080: connect: connection refused"
Sep 16 19:04:15.097915 ip-10-0-132-80 kubenswrapper[2544]: I0916 19:04:15.097874    2544 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-6cc94c6f4f-z7jld" podUID="d68cbf45-8d2e-4c9c-a40b-6464057b3091" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.35:8080: connect: connection refused"
Sep 16 19:04:25.097461 ip-10-0-132-80 kubenswrapper[2544]: I0916 19:04:25.097423    2544 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-6cc94c6f4f-z7jld" podUID="d68cbf45-8d2e-4c9c-a40b-6464057b3091" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.35:8080: connect: connection refused"
Sep 16 19:04:35.097789 ip-10-0-132-80 kubenswrapper[2544]: I0916 19:04:35.097753    2544 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-6cc94c6f4f-z7jld" podUID="d68cbf45-8d2e-4c9c-a40b-6464057b3091" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.35:8080: connect: connection refused"
Sep 16 19:04:45.097758 ip-10-0-132-80 kubenswrapper[2544]: I0916 19:04:45.097718    2544 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-6cc94c6f4f-z7jld" podUID="d68cbf45-8d2e-4c9c-a40b-6464057b3091" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.35:8080: connect: connection refused"
Sep 16 19:04:55.098600 ip-10-0-132-80 kubenswrapper[2544]: I0916 19:04:55.098503    2544 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-6cc94c6f4f-z7jld"
Sep 16 19:04:56.618299 ip-10-0-132-80 kubenswrapper[2544]: I0916 19:04:56.618261    2544 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-predictor-6cc94c6f4f-z7jld"]
Sep 16 19:04:56.618737 ip-10-0-132-80 kubenswrapper[2544]: I0916 19:04:56.618529    2544 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-6cc94c6f4f-z7jld" podUID="d68cbf45-8d2e-4c9c-a40b-6464057b3091" containerName="kserve-container" containerID="cri-o://e90ae18cdbfbb8b970d02d4ac91b26c49a7e64aa4e0b7a320f41e251263aafc1" gracePeriod=30
Sep 16 19:04:56.618737 ip-10-0-132-80 kubenswrapper[2544]: I0916 19:04:56.618640    2544 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-6cc94c6f4f-z7jld" podUID="d68cbf45-8d2e-4c9c-a40b-6464057b3091" containerName="kube-rbac-proxy" containerID="cri-o://b368fcd10df4236463ac7afa7f1adc2e951aacd83a509e6dd868916d092107b8" gracePeriod=30
Sep 16 19:04:57.406393 ip-10-0-132-80 kubenswrapper[2544]: I0916 19:04:57.406360    2544 generic.go:356] "Generic (PLEG): container finished" podID="d68cbf45-8d2e-4c9c-a40b-6464057b3091" containerID="b368fcd10df4236463ac7afa7f1adc2e951aacd83a509e6dd868916d092107b8" exitCode=2
Sep 16 19:04:57.406583 ip-10-0-132-80 kubenswrapper[2544]: I0916 19:04:57.406435    2544 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-6cc94c6f4f-z7jld" event={"ID":"d68cbf45-8d2e-4c9c-a40b-6464057b3091","Type":"ContainerDied","Data":"b368fcd10df4236463ac7afa7f1adc2e951aacd83a509e6dd868916d092107b8"}
Sep 16 19:05:00.092873 ip-10-0-132-80 kubenswrapper[2544]: I0916 19:05:00.092828    2544 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-6cc94c6f4f-z7jld" podUID="d68cbf45-8d2e-4c9c-a40b-6464057b3091" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.35:8643/healthz\": dial tcp 10.132.0.35:8643: connect: connection refused"
Sep 16 19:05:01.426790 ip-10-0-132-80 kubenswrapper[2544]: I0916 19:05:01.426761    2544 generic.go:356] "Generic (PLEG): container finished" podID="d68cbf45-8d2e-4c9c-a40b-6464057b3091" containerID="e90ae18cdbfbb8b970d02d4ac91b26c49a7e64aa4e0b7a320f41e251263aafc1" exitCode=0
Sep 16 19:05:01.427124 ip-10-0-132-80 kubenswrapper[2544]: I0916 19:05:01.426832    2544 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-6cc94c6f4f-z7jld" event={"ID":"d68cbf45-8d2e-4c9c-a40b-6464057b3091","Type":"ContainerDied","Data":"e90ae18cdbfbb8b970d02d4ac91b26c49a7e64aa4e0b7a320f41e251263aafc1"}
Sep 16 19:05:01.465902 ip-10-0-132-80 kubenswrapper[2544]: I0916 19:05:01.465880    2544 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-6cc94c6f4f-z7jld"
Sep 16 19:05:01.551075 ip-10-0-132-80 kubenswrapper[2544]: I0916 19:05:01.550996    2544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d68cbf45-8d2e-4c9c-a40b-6464057b3091-proxy-tls\") pod \"d68cbf45-8d2e-4c9c-a40b-6464057b3091\" (UID: \"d68cbf45-8d2e-4c9c-a40b-6464057b3091\") "
Sep 16 19:05:01.551075 ip-10-0-132-80 kubenswrapper[2544]: I0916 19:05:01.551030    2544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-slwtz\" (UniqueName: \"kubernetes.io/projected/d68cbf45-8d2e-4c9c-a40b-6464057b3091-kube-api-access-slwtz\") pod \"d68cbf45-8d2e-4c9c-a40b-6464057b3091\" (UID: \"d68cbf45-8d2e-4c9c-a40b-6464057b3091\") "
Sep 16 19:05:01.551075 ip-10-0-132-80 kubenswrapper[2544]: I0916 19:05:01.551059    2544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d68cbf45-8d2e-4c9c-a40b-6464057b3091-cabundle-cert\") pod \"d68cbf45-8d2e-4c9c-a40b-6464057b3091\" (UID: \"d68cbf45-8d2e-4c9c-a40b-6464057b3091\") "
Sep 16 19:05:01.551353 ip-10-0-132-80 kubenswrapper[2544]: I0916 19:05:01.551253    2544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d68cbf45-8d2e-4c9c-a40b-6464057b3091-kserve-provision-location\") pod \"d68cbf45-8d2e-4c9c-a40b-6464057b3091\" (UID: \"d68cbf45-8d2e-4c9c-a40b-6464057b3091\") "
Sep 16 19:05:01.551353 ip-10-0-132-80 kubenswrapper[2544]: I0916 19:05:01.551336    2544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d68cbf45-8d2e-4c9c-a40b-6464057b3091-kube-rbac-proxy-sar-config\") pod \"d68cbf45-8d2e-4c9c-a40b-6464057b3091\" (UID: \"d68cbf45-8d2e-4c9c-a40b-6464057b3091\") "
Sep 16 19:05:01.551471 ip-10-0-132-80 kubenswrapper[2544]: I0916 19:05:01.551390    2544 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d68cbf45-8d2e-4c9c-a40b-6464057b3091-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "d68cbf45-8d2e-4c9c-a40b-6464057b3091" (UID: "d68cbf45-8d2e-4c9c-a40b-6464057b3091"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 19:05:01.551579 ip-10-0-132-80 kubenswrapper[2544]: I0916 19:05:01.551538    2544 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d68cbf45-8d2e-4c9c-a40b-6464057b3091-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "d68cbf45-8d2e-4c9c-a40b-6464057b3091" (UID: "d68cbf45-8d2e-4c9c-a40b-6464057b3091"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:05:01.551748 ip-10-0-132-80 kubenswrapper[2544]: I0916 19:05:01.551716    2544 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d68cbf45-8d2e-4c9c-a40b-6464057b3091-cabundle-cert\") on node \"ip-10-0-132-80.ec2.internal\" DevicePath \"\""
Sep 16 19:05:01.551748 ip-10-0-132-80 kubenswrapper[2544]: I0916 19:05:01.551734    2544 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d68cbf45-8d2e-4c9c-a40b-6464057b3091-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "d68cbf45-8d2e-4c9c-a40b-6464057b3091" (UID: "d68cbf45-8d2e-4c9c-a40b-6464057b3091"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 19:05:01.551886 ip-10-0-132-80 kubenswrapper[2544]: I0916 19:05:01.551747    2544 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d68cbf45-8d2e-4c9c-a40b-6464057b3091-kserve-provision-location\") on node \"ip-10-0-132-80.ec2.internal\" DevicePath \"\""
Sep 16 19:05:01.553197 ip-10-0-132-80 kubenswrapper[2544]: I0916 19:05:01.553165    2544 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d68cbf45-8d2e-4c9c-a40b-6464057b3091-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "d68cbf45-8d2e-4c9c-a40b-6464057b3091" (UID: "d68cbf45-8d2e-4c9c-a40b-6464057b3091"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 19:05:01.553197 ip-10-0-132-80 kubenswrapper[2544]: I0916 19:05:01.553195    2544 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d68cbf45-8d2e-4c9c-a40b-6464057b3091-kube-api-access-slwtz" (OuterVolumeSpecName: "kube-api-access-slwtz") pod "d68cbf45-8d2e-4c9c-a40b-6464057b3091" (UID: "d68cbf45-8d2e-4c9c-a40b-6464057b3091"). InnerVolumeSpecName "kube-api-access-slwtz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 19:05:01.652762 ip-10-0-132-80 kubenswrapper[2544]: I0916 19:05:01.652734    2544 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d68cbf45-8d2e-4c9c-a40b-6464057b3091-kube-rbac-proxy-sar-config\") on node \"ip-10-0-132-80.ec2.internal\" DevicePath \"\""
Sep 16 19:05:01.652762 ip-10-0-132-80 kubenswrapper[2544]: I0916 19:05:01.652763    2544 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d68cbf45-8d2e-4c9c-a40b-6464057b3091-proxy-tls\") on node \"ip-10-0-132-80.ec2.internal\" DevicePath \"\""
Sep 16 19:05:01.652926 ip-10-0-132-80 kubenswrapper[2544]: I0916 19:05:01.652776    2544 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-slwtz\" (UniqueName: \"kubernetes.io/projected/d68cbf45-8d2e-4c9c-a40b-6464057b3091-kube-api-access-slwtz\") on node \"ip-10-0-132-80.ec2.internal\" DevicePath \"\""
Sep 16 19:05:02.433240 ip-10-0-132-80 kubenswrapper[2544]: I0916 19:05:02.433197    2544 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-6cc94c6f4f-z7jld" event={"ID":"d68cbf45-8d2e-4c9c-a40b-6464057b3091","Type":"ContainerDied","Data":"94c0c63ac71b136d8cbd622624bae3691d24790f17da8d0593ef69645e912342"}
Sep 16 19:05:02.433240 ip-10-0-132-80 kubenswrapper[2544]: I0916 19:05:02.433230    2544 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-6cc94c6f4f-z7jld"
Sep 16 19:05:02.433695 ip-10-0-132-80 kubenswrapper[2544]: I0916 19:05:02.433295    2544 scope.go:117] "RemoveContainer" containerID="b368fcd10df4236463ac7afa7f1adc2e951aacd83a509e6dd868916d092107b8"
Sep 16 19:05:02.441528 ip-10-0-132-80 kubenswrapper[2544]: I0916 19:05:02.441503    2544 scope.go:117] "RemoveContainer" containerID="e90ae18cdbfbb8b970d02d4ac91b26c49a7e64aa4e0b7a320f41e251263aafc1"
Sep 16 19:05:02.448623 ip-10-0-132-80 kubenswrapper[2544]: I0916 19:05:02.448598    2544 scope.go:117] "RemoveContainer" containerID="6f34bcb4ff357156e6476c13acf48b276cfed50174d6eeb2314ec120388468d9"
Sep 16 19:05:02.453873 ip-10-0-132-80 kubenswrapper[2544]: I0916 19:05:02.453734    2544 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-predictor-6cc94c6f4f-z7jld"]
Sep 16 19:05:02.458201 ip-10-0-132-80 kubenswrapper[2544]: I0916 19:05:02.458145    2544 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-predictor-6cc94c6f4f-z7jld"]
Sep 16 19:05:03.721475 ip-10-0-132-80 kubenswrapper[2544]: I0916 19:05:03.721443    2544 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d68cbf45-8d2e-4c9c-a40b-6464057b3091" path="/var/lib/kubelet/pods/d68cbf45-8d2e-4c9c-a40b-6464057b3091/volumes"
Sep 16 19:05:08.661995 ip-10-0-132-80 kubenswrapper[2544]: I0916 19:05:08.661962    2544 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-predictor-b9b98bcd9-x6xcs"]
Sep 16 19:05:08.662369 ip-10-0-132-80 kubenswrapper[2544]: I0916 19:05:08.662305    2544 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="dc269306-d6b2-4857-a15f-fb4579de499f" containerName="storage-initializer"
Sep 16 19:05:08.662369 ip-10-0-132-80 kubenswrapper[2544]: I0916 19:05:08.662316    2544 state_mem.go:107] "Deleted CPUSet assignment" podUID="dc269306-d6b2-4857-a15f-fb4579de499f" containerName="storage-initializer"
Sep 16 19:05:08.662369 ip-10-0-132-80 kubenswrapper[2544]: I0916 19:05:08.662324    2544 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d68cbf45-8d2e-4c9c-a40b-6464057b3091" containerName="storage-initializer"
Sep 16 19:05:08.662369 ip-10-0-132-80 kubenswrapper[2544]: I0916 19:05:08.662330    2544 state_mem.go:107] "Deleted CPUSet assignment" podUID="d68cbf45-8d2e-4c9c-a40b-6464057b3091" containerName="storage-initializer"
Sep 16 19:05:08.662369 ip-10-0-132-80 kubenswrapper[2544]: I0916 19:05:08.662337    2544 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d68cbf45-8d2e-4c9c-a40b-6464057b3091" containerName="kube-rbac-proxy"
Sep 16 19:05:08.662369 ip-10-0-132-80 kubenswrapper[2544]: I0916 19:05:08.662343    2544 state_mem.go:107] "Deleted CPUSet assignment" podUID="d68cbf45-8d2e-4c9c-a40b-6464057b3091" containerName="kube-rbac-proxy"
Sep 16 19:05:08.662369 ip-10-0-132-80 kubenswrapper[2544]: I0916 19:05:08.662349    2544 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="dc269306-d6b2-4857-a15f-fb4579de499f" containerName="kserve-container"
Sep 16 19:05:08.662369 ip-10-0-132-80 kubenswrapper[2544]: I0916 19:05:08.662355    2544 state_mem.go:107] "Deleted CPUSet assignment" podUID="dc269306-d6b2-4857-a15f-fb4579de499f" containerName="kserve-container"
Sep 16 19:05:08.662369 ip-10-0-132-80 kubenswrapper[2544]: I0916 19:05:08.662362    2544 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="dc269306-d6b2-4857-a15f-fb4579de499f" containerName="kube-rbac-proxy"
Sep 16 19:05:08.662369 ip-10-0-132-80 kubenswrapper[2544]: I0916 19:05:08.662368    2544 state_mem.go:107] "Deleted CPUSet assignment" podUID="dc269306-d6b2-4857-a15f-fb4579de499f" containerName="kube-rbac-proxy"
Sep 16 19:05:08.662369 ip-10-0-132-80 kubenswrapper[2544]: I0916 19:05:08.662374    2544 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d68cbf45-8d2e-4c9c-a40b-6464057b3091" containerName="kserve-container"
Sep 16 19:05:08.662804 ip-10-0-132-80 kubenswrapper[2544]: I0916 19:05:08.662380    2544 state_mem.go:107] "Deleted CPUSet assignment" podUID="d68cbf45-8d2e-4c9c-a40b-6464057b3091" containerName="kserve-container"
Sep 16 19:05:08.662804 ip-10-0-132-80 kubenswrapper[2544]: I0916 19:05:08.662433    2544 memory_manager.go:368] "RemoveStaleState removing state" podUID="dc269306-d6b2-4857-a15f-fb4579de499f" containerName="kserve-container"
Sep 16 19:05:08.662804 ip-10-0-132-80 kubenswrapper[2544]: I0916 19:05:08.662445    2544 memory_manager.go:368] "RemoveStaleState removing state" podUID="dc269306-d6b2-4857-a15f-fb4579de499f" containerName="kube-rbac-proxy"
Sep 16 19:05:08.662804 ip-10-0-132-80 kubenswrapper[2544]: I0916 19:05:08.662451    2544 memory_manager.go:368] "RemoveStaleState removing state" podUID="d68cbf45-8d2e-4c9c-a40b-6464057b3091" containerName="kube-rbac-proxy"
Sep 16 19:05:08.662804 ip-10-0-132-80 kubenswrapper[2544]: I0916 19:05:08.662457    2544 memory_manager.go:368] "RemoveStaleState removing state" podUID="d68cbf45-8d2e-4c9c-a40b-6464057b3091" containerName="kserve-container"
Sep 16 19:05:08.667194 ip-10-0-132-80 kubenswrapper[2544]: I0916 19:05:08.667174    2544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-b9b98bcd9-x6xcs"
Sep 16 19:05:08.669713 ip-10-0-132-80 kubenswrapper[2544]: I0916 19:05:08.669672    2544 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Sep 16 19:05:08.669860 ip-10-0-132-80 kubenswrapper[2544]: I0916 19:05:08.669719    2544 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Sep 16 19:05:08.669860 ip-10-0-132-80 kubenswrapper[2544]: I0916 19:05:08.669750    2544 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-promote-kube-rbac-proxy-sar-config\""
Sep 16 19:05:08.670303 ip-10-0-132-80 kubenswrapper[2544]: I0916 19:05:08.670285    2544 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Sep 16 19:05:08.670401 ip-10-0-132-80 kubenswrapper[2544]: I0916 19:05:08.670288    2544 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-zmsvt\""
Sep 16 19:05:08.670401 ip-10-0-132-80 kubenswrapper[2544]: I0916 19:05:08.670312    2544 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-promote-predictor-serving-cert\""
Sep 16 19:05:08.670401 ip-10-0-132-80 kubenswrapper[2544]: I0916 19:05:08.670376    2544 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Sep 16 19:05:08.679984 ip-10-0-132-80 kubenswrapper[2544]: I0916 19:05:08.679962    2544 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-predictor-b9b98bcd9-x6xcs"]
Sep 16 19:05:08.809273 ip-10-0-132-80 kubenswrapper[2544]: I0916 19:05:08.809217    2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/0a24ca20-5b1d-4dab-b80f-c3e98a84ead5-proxy-tls\") pod \"isvc-canary-promote-predictor-b9b98bcd9-x6xcs\" (UID: \"0a24ca20-5b1d-4dab-b80f-c3e98a84ead5\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-b9b98bcd9-x6xcs"
Sep 16 19:05:08.809273 ip-10-0-132-80 kubenswrapper[2544]: I0916 19:05:08.809276    2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/0a24ca20-5b1d-4dab-b80f-c3e98a84ead5-cabundle-cert\") pod \"isvc-canary-promote-predictor-b9b98bcd9-x6xcs\" (UID: \"0a24ca20-5b1d-4dab-b80f-c3e98a84ead5\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-b9b98bcd9-x6xcs"
Sep 16 19:05:08.809592 ip-10-0-132-80 kubenswrapper[2544]: I0916 19:05:08.809323    2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zgvkk\" (UniqueName: \"kubernetes.io/projected/0a24ca20-5b1d-4dab-b80f-c3e98a84ead5-kube-api-access-zgvkk\") pod \"isvc-canary-promote-predictor-b9b98bcd9-x6xcs\" (UID: \"0a24ca20-5b1d-4dab-b80f-c3e98a84ead5\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-b9b98bcd9-x6xcs"
Sep 16 19:05:08.809592 ip-10-0-132-80 kubenswrapper[2544]: I0916 19:05:08.809352    2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/0a24ca20-5b1d-4dab-b80f-c3e98a84ead5-kube-rbac-proxy-sar-config\") pod \"isvc-canary-promote-predictor-b9b98bcd9-x6xcs\" (UID: \"0a24ca20-5b1d-4dab-b80f-c3e98a84ead5\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-b9b98bcd9-x6xcs"
Sep 16 19:05:08.809592 ip-10-0-132-80 kubenswrapper[2544]: I0916 19:05:08.809466    2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/0a24ca20-5b1d-4dab-b80f-c3e98a84ead5-kserve-provision-location\") pod \"isvc-canary-promote-predictor-b9b98bcd9-x6xcs\" (UID: \"0a24ca20-5b1d-4dab-b80f-c3e98a84ead5\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-b9b98bcd9-x6xcs"
Sep 16 19:05:08.910331 ip-10-0-132-80 kubenswrapper[2544]: I0916 19:05:08.910293    2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/0a24ca20-5b1d-4dab-b80f-c3e98a84ead5-proxy-tls\") pod \"isvc-canary-promote-predictor-b9b98bcd9-x6xcs\" (UID: \"0a24ca20-5b1d-4dab-b80f-c3e98a84ead5\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-b9b98bcd9-x6xcs"
Sep 16 19:05:08.910331 ip-10-0-132-80 kubenswrapper[2544]: I0916 19:05:08.910329    2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/0a24ca20-5b1d-4dab-b80f-c3e98a84ead5-cabundle-cert\") pod \"isvc-canary-promote-predictor-b9b98bcd9-x6xcs\" (UID: \"0a24ca20-5b1d-4dab-b80f-c3e98a84ead5\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-b9b98bcd9-x6xcs"
Sep 16 19:05:08.910701 ip-10-0-132-80 kubenswrapper[2544]: I0916 19:05:08.910351    2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zgvkk\" (UniqueName: \"kubernetes.io/projected/0a24ca20-5b1d-4dab-b80f-c3e98a84ead5-kube-api-access-zgvkk\") pod \"isvc-canary-promote-predictor-b9b98bcd9-x6xcs\" (UID: \"0a24ca20-5b1d-4dab-b80f-c3e98a84ead5\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-b9b98bcd9-x6xcs"
Sep 16 19:05:08.910701 ip-10-0-132-80 kubenswrapper[2544]: E0916 19:05:08.910469    2544 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-canary-promote-predictor-serving-cert: secret "isvc-canary-promote-predictor-serving-cert" not found
Sep 16 19:05:08.910701 ip-10-0-132-80 kubenswrapper[2544]: I0916 19:05:08.910474    2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/0a24ca20-5b1d-4dab-b80f-c3e98a84ead5-kube-rbac-proxy-sar-config\") pod \"isvc-canary-promote-predictor-b9b98bcd9-x6xcs\" (UID: \"0a24ca20-5b1d-4dab-b80f-c3e98a84ead5\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-b9b98bcd9-x6xcs"
Sep 16 19:05:08.910701 ip-10-0-132-80 kubenswrapper[2544]: E0916 19:05:08.910587    2544 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0a24ca20-5b1d-4dab-b80f-c3e98a84ead5-proxy-tls podName:0a24ca20-5b1d-4dab-b80f-c3e98a84ead5 nodeName:}" failed. No retries permitted until 2026-09-16 19:05:09.410538819 +0000 UTC m=+1272.199541345 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/0a24ca20-5b1d-4dab-b80f-c3e98a84ead5-proxy-tls") pod "isvc-canary-promote-predictor-b9b98bcd9-x6xcs" (UID: "0a24ca20-5b1d-4dab-b80f-c3e98a84ead5") : secret "isvc-canary-promote-predictor-serving-cert" not found
Sep 16 19:05:08.910701 ip-10-0-132-80 kubenswrapper[2544]: I0916 19:05:08.910693    2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/0a24ca20-5b1d-4dab-b80f-c3e98a84ead5-kserve-provision-location\") pod \"isvc-canary-promote-predictor-b9b98bcd9-x6xcs\" (UID: \"0a24ca20-5b1d-4dab-b80f-c3e98a84ead5\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-b9b98bcd9-x6xcs"
Sep 16 19:05:08.911172 ip-10-0-132-80 kubenswrapper[2544]: I0916 19:05:08.911155    2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/0a24ca20-5b1d-4dab-b80f-c3e98a84ead5-kserve-provision-location\") pod \"isvc-canary-promote-predictor-b9b98bcd9-x6xcs\" (UID: \"0a24ca20-5b1d-4dab-b80f-c3e98a84ead5\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-b9b98bcd9-x6xcs"
Sep 16 19:05:08.911241 ip-10-0-132-80 kubenswrapper[2544]: I0916 19:05:08.911169    2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/0a24ca20-5b1d-4dab-b80f-c3e98a84ead5-cabundle-cert\") pod \"isvc-canary-promote-predictor-b9b98bcd9-x6xcs\" (UID: \"0a24ca20-5b1d-4dab-b80f-c3e98a84ead5\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-b9b98bcd9-x6xcs"
Sep 16 19:05:08.911524 ip-10-0-132-80 kubenswrapper[2544]: I0916 19:05:08.911503    2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/0a24ca20-5b1d-4dab-b80f-c3e98a84ead5-kube-rbac-proxy-sar-config\") pod \"isvc-canary-promote-predictor-b9b98bcd9-x6xcs\" (UID: \"0a24ca20-5b1d-4dab-b80f-c3e98a84ead5\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-b9b98bcd9-x6xcs"
Sep 16 19:05:08.921788 ip-10-0-132-80 kubenswrapper[2544]: I0916 19:05:08.921735    2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zgvkk\" (UniqueName: \"kubernetes.io/projected/0a24ca20-5b1d-4dab-b80f-c3e98a84ead5-kube-api-access-zgvkk\") pod \"isvc-canary-promote-predictor-b9b98bcd9-x6xcs\" (UID: \"0a24ca20-5b1d-4dab-b80f-c3e98a84ead5\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-b9b98bcd9-x6xcs"
Sep 16 19:05:09.414794 ip-10-0-132-80 kubenswrapper[2544]: I0916 19:05:09.414755    2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/0a24ca20-5b1d-4dab-b80f-c3e98a84ead5-proxy-tls\") pod \"isvc-canary-promote-predictor-b9b98bcd9-x6xcs\" (UID: \"0a24ca20-5b1d-4dab-b80f-c3e98a84ead5\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-b9b98bcd9-x6xcs"
Sep 16 19:05:09.417128 ip-10-0-132-80 kubenswrapper[2544]: I0916 19:05:09.417110    2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/0a24ca20-5b1d-4dab-b80f-c3e98a84ead5-proxy-tls\") pod \"isvc-canary-promote-predictor-b9b98bcd9-x6xcs\" (UID: \"0a24ca20-5b1d-4dab-b80f-c3e98a84ead5\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-b9b98bcd9-x6xcs"
Sep 16 19:05:09.577251 ip-10-0-132-80 kubenswrapper[2544]: I0916 19:05:09.577217    2544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-b9b98bcd9-x6xcs"
Sep 16 19:05:09.851754 ip-10-0-132-80 kubenswrapper[2544]: I0916 19:05:09.851724    2544 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-predictor-b9b98bcd9-x6xcs"]
Sep 16 19:05:09.854814 ip-10-0-132-80 kubenswrapper[2544]: W0916 19:05:09.854786    2544 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0a24ca20_5b1d_4dab_b80f_c3e98a84ead5.slice/crio-bc27ef9032891ca5a398964016390182caa280b25cbd343c39b44cdeb1e73f0e WatchSource:0}: Error finding container bc27ef9032891ca5a398964016390182caa280b25cbd343c39b44cdeb1e73f0e: Status 404 returned error can't find the container with id bc27ef9032891ca5a398964016390182caa280b25cbd343c39b44cdeb1e73f0e
Sep 16 19:05:10.471719 ip-10-0-132-80 kubenswrapper[2544]: I0916 19:05:10.471682    2544 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-b9b98bcd9-x6xcs" event={"ID":"0a24ca20-5b1d-4dab-b80f-c3e98a84ead5","Type":"ContainerStarted","Data":"5abc73906eefd357d407078b04541f2949bcdfbce5a3fd3d4fd616caa771c956"}
Sep 16 19:05:10.471900 ip-10-0-132-80 kubenswrapper[2544]: I0916 19:05:10.471744    2544 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-b9b98bcd9-x6xcs" event={"ID":"0a24ca20-5b1d-4dab-b80f-c3e98a84ead5","Type":"ContainerStarted","Data":"bc27ef9032891ca5a398964016390182caa280b25cbd343c39b44cdeb1e73f0e"}
Sep 16 19:05:14.489195 ip-10-0-132-80 kubenswrapper[2544]: I0916 19:05:14.489164    2544 generic.go:356] "Generic (PLEG): container finished" podID="0a24ca20-5b1d-4dab-b80f-c3e98a84ead5" containerID="5abc73906eefd357d407078b04541f2949bcdfbce5a3fd3d4fd616caa771c956" exitCode=0
Sep 16 19:05:14.489645 ip-10-0-132-80 kubenswrapper[2544]: I0916 19:05:14.489235    2544 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-b9b98bcd9-x6xcs" event={"ID":"0a24ca20-5b1d-4dab-b80f-c3e98a84ead5","Type":"ContainerDied","Data":"5abc73906eefd357d407078b04541f2949bcdfbce5a3fd3d4fd616caa771c956"}
Sep 16 19:05:15.495916 ip-10-0-132-80 kubenswrapper[2544]: I0916 19:05:15.495878    2544 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-b9b98bcd9-x6xcs" event={"ID":"0a24ca20-5b1d-4dab-b80f-c3e98a84ead5","Type":"ContainerStarted","Data":"b0c5ecc6b350ddf2cd89e9612d1e2ec86e664ff256169196e38a1d12de47e19c"}
Sep 16 19:05:15.496284 ip-10-0-132-80 kubenswrapper[2544]: I0916 19:05:15.495971    2544 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-b9b98bcd9-x6xcs" event={"ID":"0a24ca20-5b1d-4dab-b80f-c3e98a84ead5","Type":"ContainerStarted","Data":"a566249e25a6e3f1df44e7beb06f6e8a0a2cb90e5625f2d012c0501dec0fd95d"}
Sep 16 19:05:15.496284 ip-10-0-132-80 kubenswrapper[2544]: I0916 19:05:15.496175    2544 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-b9b98bcd9-x6xcs"
Sep 16 19:05:15.532910 ip-10-0-132-80 kubenswrapper[2544]: I0916 19:05:15.532862    2544 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-b9b98bcd9-x6xcs" podStartSLOduration=3.954762144 podStartE2EDuration="7.53284737s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.578085226s" isStatefulPod=false podCreationTimestamp="2026-09-16 19:05:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:05:15.532122203 +0000 UTC m=+1278.321124780" watchObservedRunningTime="2026-09-16 19:05:15.53284737 +0000 UTC m=+1278.321849904"
Sep 16 19:05:16.501594 ip-10-0-132-80 kubenswrapper[2544]: I0916 19:05:16.501546    2544 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-b9b98bcd9-x6xcs"
Sep 16 19:05:16.502985 ip-10-0-132-80 kubenswrapper[2544]: I0916 19:05:16.502953    2544 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-b9b98bcd9-x6xcs" podUID="0a24ca20-5b1d-4dab-b80f-c3e98a84ead5" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.36:8080: connect: connection refused"
Sep 16 19:05:17.506044 ip-10-0-132-80 kubenswrapper[2544]: I0916 19:05:17.506002    2544 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-b9b98bcd9-x6xcs" podUID="0a24ca20-5b1d-4dab-b80f-c3e98a84ead5" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.36:8080: connect: connection refused"
Sep 16 19:05:22.510835 ip-10-0-132-80 kubenswrapper[2544]: I0916 19:05:22.510805    2544 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-b9b98bcd9-x6xcs"
Sep 16 19:05:22.511477 ip-10-0-132-80 kubenswrapper[2544]: I0916 19:05:22.511447    2544 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-b9b98bcd9-x6xcs" podUID="0a24ca20-5b1d-4dab-b80f-c3e98a84ead5" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.36:8080: connect: connection refused"
Sep 16 19:05:32.511435 ip-10-0-132-80 kubenswrapper[2544]: I0916 19:05:32.511395    2544 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-b9b98bcd9-x6xcs" podUID="0a24ca20-5b1d-4dab-b80f-c3e98a84ead5" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.36:8080: connect: connection refused"
Sep 16 19:05:42.512350 ip-10-0-132-80 kubenswrapper[2544]: I0916 19:05:42.512310    2544 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-b9b98bcd9-x6xcs" podUID="0a24ca20-5b1d-4dab-b80f-c3e98a84ead5" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.36:8080: connect: connection refused"
Sep 16 19:05:52.511473 ip-10-0-132-80 kubenswrapper[2544]: I0916 19:05:52.511426    2544 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-b9b98bcd9-x6xcs" podUID="0a24ca20-5b1d-4dab-b80f-c3e98a84ead5" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.36:8080: connect: connection refused"
Sep 16 19:06:02.512141 ip-10-0-132-80 kubenswrapper[2544]: I0916 19:06:02.512096    2544 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-b9b98bcd9-x6xcs" podUID="0a24ca20-5b1d-4dab-b80f-c3e98a84ead5" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.36:8080: connect: connection refused"
Sep 16 19:06:12.512195 ip-10-0-132-80 kubenswrapper[2544]: I0916 19:06:12.512150    2544 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-b9b98bcd9-x6xcs" podUID="0a24ca20-5b1d-4dab-b80f-c3e98a84ead5" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.36:8080: connect: connection refused"
Sep 16 19:06:22.511844 ip-10-0-132-80 kubenswrapper[2544]: I0916 19:06:22.511800    2544 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-b9b98bcd9-x6xcs" podUID="0a24ca20-5b1d-4dab-b80f-c3e98a84ead5" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.36:8080: connect: connection refused"
Sep 16 19:06:31.072034 ip-10-0-132-80 kubenswrapper[2544]: I0916 19:06:31.071983    2544 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-predictor-b9b98bcd9-x6xcs"]
Sep 16 19:06:31.072483 ip-10-0-132-80 kubenswrapper[2544]: I0916 19:06:31.072435    2544 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-b9b98bcd9-x6xcs" podUID="0a24ca20-5b1d-4dab-b80f-c3e98a84ead5" containerName="kserve-container" containerID="cri-o://a566249e25a6e3f1df44e7beb06f6e8a0a2cb90e5625f2d012c0501dec0fd95d" gracePeriod=30
Sep 16 19:06:31.072706 ip-10-0-132-80 kubenswrapper[2544]: I0916 19:06:31.072669    2544 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-b9b98bcd9-x6xcs" podUID="0a24ca20-5b1d-4dab-b80f-c3e98a84ead5" containerName="kube-rbac-proxy" containerID="cri-o://b0c5ecc6b350ddf2cd89e9612d1e2ec86e664ff256169196e38a1d12de47e19c" gracePeriod=30
Sep 16 19:06:31.077638 ip-10-0-132-80 kubenswrapper[2544]: I0916 19:06:31.073860    2544 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-b9b98bcd9-x6xcs"
Sep 16 19:06:31.811316 ip-10-0-132-80 kubenswrapper[2544]: I0916 19:06:31.811283    2544 generic.go:356] "Generic (PLEG): container finished" podID="0a24ca20-5b1d-4dab-b80f-c3e98a84ead5" containerID="b0c5ecc6b350ddf2cd89e9612d1e2ec86e664ff256169196e38a1d12de47e19c" exitCode=2
Sep 16 19:06:31.811498 ip-10-0-132-80 kubenswrapper[2544]: I0916 19:06:31.811356    2544 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-b9b98bcd9-x6xcs" event={"ID":"0a24ca20-5b1d-4dab-b80f-c3e98a84ead5","Type":"ContainerDied","Data":"b0c5ecc6b350ddf2cd89e9612d1e2ec86e664ff256169196e38a1d12de47e19c"}
Sep 16 19:06:32.507019 ip-10-0-132-80 kubenswrapper[2544]: I0916 19:06:32.506969    2544 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-b9b98bcd9-x6xcs" podUID="0a24ca20-5b1d-4dab-b80f-c3e98a84ead5" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.36:8643/healthz\": dial tcp 10.132.0.36:8643: connect: connection refused"
Sep 16 19:06:35.824224 ip-10-0-132-80 kubenswrapper[2544]: I0916 19:06:35.824199    2544 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-b9b98bcd9-x6xcs"
Sep 16 19:06:35.828472 ip-10-0-132-80 kubenswrapper[2544]: I0916 19:06:35.828447    2544 generic.go:356] "Generic (PLEG): container finished" podID="0a24ca20-5b1d-4dab-b80f-c3e98a84ead5" containerID="a566249e25a6e3f1df44e7beb06f6e8a0a2cb90e5625f2d012c0501dec0fd95d" exitCode=0
Sep 16 19:06:35.828617 ip-10-0-132-80 kubenswrapper[2544]: I0916 19:06:35.828508    2544 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-b9b98bcd9-x6xcs"
Sep 16 19:06:35.828617 ip-10-0-132-80 kubenswrapper[2544]: I0916 19:06:35.828527    2544 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-b9b98bcd9-x6xcs" event={"ID":"0a24ca20-5b1d-4dab-b80f-c3e98a84ead5","Type":"ContainerDied","Data":"a566249e25a6e3f1df44e7beb06f6e8a0a2cb90e5625f2d012c0501dec0fd95d"}
Sep 16 19:06:35.828617 ip-10-0-132-80 kubenswrapper[2544]: I0916 19:06:35.828615    2544 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-b9b98bcd9-x6xcs" event={"ID":"0a24ca20-5b1d-4dab-b80f-c3e98a84ead5","Type":"ContainerDied","Data":"bc27ef9032891ca5a398964016390182caa280b25cbd343c39b44cdeb1e73f0e"}
Sep 16 19:06:35.828782 ip-10-0-132-80 kubenswrapper[2544]: I0916 19:06:35.828646    2544 scope.go:117] "RemoveContainer" containerID="b0c5ecc6b350ddf2cd89e9612d1e2ec86e664ff256169196e38a1d12de47e19c"
Sep 16 19:06:35.836248 ip-10-0-132-80 kubenswrapper[2544]: I0916 19:06:35.836229    2544 scope.go:117] "RemoveContainer" containerID="a566249e25a6e3f1df44e7beb06f6e8a0a2cb90e5625f2d012c0501dec0fd95d"
Sep 16 19:06:35.843352 ip-10-0-132-80 kubenswrapper[2544]: I0916 19:06:35.843335    2544 scope.go:117] "RemoveContainer" containerID="5abc73906eefd357d407078b04541f2949bcdfbce5a3fd3d4fd616caa771c956"
Sep 16 19:06:35.852190 ip-10-0-132-80 kubenswrapper[2544]: I0916 19:06:35.852173    2544 scope.go:117] "RemoveContainer" containerID="b0c5ecc6b350ddf2cd89e9612d1e2ec86e664ff256169196e38a1d12de47e19c"
Sep 16 19:06:35.852468 ip-10-0-132-80 kubenswrapper[2544]: E0916 19:06:35.852448    2544 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b0c5ecc6b350ddf2cd89e9612d1e2ec86e664ff256169196e38a1d12de47e19c\": container with ID starting with b0c5ecc6b350ddf2cd89e9612d1e2ec86e664ff256169196e38a1d12de47e19c not found: ID does not exist" containerID="b0c5ecc6b350ddf2cd89e9612d1e2ec86e664ff256169196e38a1d12de47e19c"
Sep 16 19:06:35.852571 ip-10-0-132-80 kubenswrapper[2544]: I0916 19:06:35.852476    2544 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b0c5ecc6b350ddf2cd89e9612d1e2ec86e664ff256169196e38a1d12de47e19c"} err="failed to get container status \"b0c5ecc6b350ddf2cd89e9612d1e2ec86e664ff256169196e38a1d12de47e19c\": rpc error: code = NotFound desc = could not find container \"b0c5ecc6b350ddf2cd89e9612d1e2ec86e664ff256169196e38a1d12de47e19c\": container with ID starting with b0c5ecc6b350ddf2cd89e9612d1e2ec86e664ff256169196e38a1d12de47e19c not found: ID does not exist"
Sep 16 19:06:35.852571 ip-10-0-132-80 kubenswrapper[2544]: I0916 19:06:35.852495    2544 scope.go:117] "RemoveContainer" containerID="a566249e25a6e3f1df44e7beb06f6e8a0a2cb90e5625f2d012c0501dec0fd95d"
Sep 16 19:06:35.852775 ip-10-0-132-80 kubenswrapper[2544]: E0916 19:06:35.852761    2544 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a566249e25a6e3f1df44e7beb06f6e8a0a2cb90e5625f2d012c0501dec0fd95d\": container with ID starting with a566249e25a6e3f1df44e7beb06f6e8a0a2cb90e5625f2d012c0501dec0fd95d not found: ID does not exist" containerID="a566249e25a6e3f1df44e7beb06f6e8a0a2cb90e5625f2d012c0501dec0fd95d"
Sep 16 19:06:35.852838 ip-10-0-132-80 kubenswrapper[2544]: I0916 19:06:35.852779    2544 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a566249e25a6e3f1df44e7beb06f6e8a0a2cb90e5625f2d012c0501dec0fd95d"} err="failed to get container status \"a566249e25a6e3f1df44e7beb06f6e8a0a2cb90e5625f2d012c0501dec0fd95d\": rpc error: code = NotFound desc = could not find container \"a566249e25a6e3f1df44e7beb06f6e8a0a2cb90e5625f2d012c0501dec0fd95d\": container with ID starting with a566249e25a6e3f1df44e7beb06f6e8a0a2cb90e5625f2d012c0501dec0fd95d not found: ID does not exist"
Sep 16 19:06:35.852838 ip-10-0-132-80 kubenswrapper[2544]: I0916 19:06:35.852792    2544 scope.go:117] "RemoveContainer" containerID="5abc73906eefd357d407078b04541f2949bcdfbce5a3fd3d4fd616caa771c956"
Sep 16 19:06:35.853059 ip-10-0-132-80 kubenswrapper[2544]: E0916 19:06:35.853023    2544 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5abc73906eefd357d407078b04541f2949bcdfbce5a3fd3d4fd616caa771c956\": container with ID starting with 5abc73906eefd357d407078b04541f2949bcdfbce5a3fd3d4fd616caa771c956 not found: ID does not exist" containerID="5abc73906eefd357d407078b04541f2949bcdfbce5a3fd3d4fd616caa771c956"
Sep 16 19:06:35.853155 ip-10-0-132-80 kubenswrapper[2544]: I0916 19:06:35.853064    2544 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5abc73906eefd357d407078b04541f2949bcdfbce5a3fd3d4fd616caa771c956"} err="failed to get container status \"5abc73906eefd357d407078b04541f2949bcdfbce5a3fd3d4fd616caa771c956\": rpc error: code = NotFound desc = could not find container \"5abc73906eefd357d407078b04541f2949bcdfbce5a3fd3d4fd616caa771c956\": container with ID starting with 5abc73906eefd357d407078b04541f2949bcdfbce5a3fd3d4fd616caa771c956 not found: ID does not exist"
Sep 16 19:06:35.937589 ip-10-0-132-80 kubenswrapper[2544]: I0916 19:06:35.937545    2544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/0a24ca20-5b1d-4dab-b80f-c3e98a84ead5-proxy-tls\") pod \"0a24ca20-5b1d-4dab-b80f-c3e98a84ead5\" (UID: \"0a24ca20-5b1d-4dab-b80f-c3e98a84ead5\") "
Sep 16 19:06:35.937589 ip-10-0-132-80 kubenswrapper[2544]: I0916 19:06:35.937600    2544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/0a24ca20-5b1d-4dab-b80f-c3e98a84ead5-cabundle-cert\") pod \"0a24ca20-5b1d-4dab-b80f-c3e98a84ead5\" (UID: \"0a24ca20-5b1d-4dab-b80f-c3e98a84ead5\") "
Sep 16 19:06:35.937825 ip-10-0-132-80 kubenswrapper[2544]: I0916 19:06:35.937645    2544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zgvkk\" (UniqueName: \"kubernetes.io/projected/0a24ca20-5b1d-4dab-b80f-c3e98a84ead5-kube-api-access-zgvkk\") pod \"0a24ca20-5b1d-4dab-b80f-c3e98a84ead5\" (UID: \"0a24ca20-5b1d-4dab-b80f-c3e98a84ead5\") "
Sep 16 19:06:35.937825 ip-10-0-132-80 kubenswrapper[2544]: I0916 19:06:35.937700    2544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/0a24ca20-5b1d-4dab-b80f-c3e98a84ead5-kube-rbac-proxy-sar-config\") pod \"0a24ca20-5b1d-4dab-b80f-c3e98a84ead5\" (UID: \"0a24ca20-5b1d-4dab-b80f-c3e98a84ead5\") "
Sep 16 19:06:35.937825 ip-10-0-132-80 kubenswrapper[2544]: I0916 19:06:35.937744    2544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/0a24ca20-5b1d-4dab-b80f-c3e98a84ead5-kserve-provision-location\") pod \"0a24ca20-5b1d-4dab-b80f-c3e98a84ead5\" (UID: \"0a24ca20-5b1d-4dab-b80f-c3e98a84ead5\") "
Sep 16 19:06:35.938031 ip-10-0-132-80 kubenswrapper[2544]: I0916 19:06:35.938006    2544 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0a24ca20-5b1d-4dab-b80f-c3e98a84ead5-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "0a24ca20-5b1d-4dab-b80f-c3e98a84ead5" (UID: "0a24ca20-5b1d-4dab-b80f-c3e98a84ead5"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 19:06:35.938031 ip-10-0-132-80 kubenswrapper[2544]: I0916 19:06:35.938020    2544 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0a24ca20-5b1d-4dab-b80f-c3e98a84ead5-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "0a24ca20-5b1d-4dab-b80f-c3e98a84ead5" (UID: "0a24ca20-5b1d-4dab-b80f-c3e98a84ead5"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:06:35.938149 ip-10-0-132-80 kubenswrapper[2544]: I0916 19:06:35.938072    2544 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0a24ca20-5b1d-4dab-b80f-c3e98a84ead5-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "0a24ca20-5b1d-4dab-b80f-c3e98a84ead5" (UID: "0a24ca20-5b1d-4dab-b80f-c3e98a84ead5"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 19:06:35.939525 ip-10-0-132-80 kubenswrapper[2544]: I0916 19:06:35.939501    2544 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0a24ca20-5b1d-4dab-b80f-c3e98a84ead5-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "0a24ca20-5b1d-4dab-b80f-c3e98a84ead5" (UID: "0a24ca20-5b1d-4dab-b80f-c3e98a84ead5"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 19:06:35.939655 ip-10-0-132-80 kubenswrapper[2544]: I0916 19:06:35.939631    2544 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0a24ca20-5b1d-4dab-b80f-c3e98a84ead5-kube-api-access-zgvkk" (OuterVolumeSpecName: "kube-api-access-zgvkk") pod "0a24ca20-5b1d-4dab-b80f-c3e98a84ead5" (UID: "0a24ca20-5b1d-4dab-b80f-c3e98a84ead5"). InnerVolumeSpecName "kube-api-access-zgvkk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 19:06:36.039191 ip-10-0-132-80 kubenswrapper[2544]: I0916 19:06:36.039160    2544 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/0a24ca20-5b1d-4dab-b80f-c3e98a84ead5-kserve-provision-location\") on node \"ip-10-0-132-80.ec2.internal\" DevicePath \"\""
Sep 16 19:06:36.039191 ip-10-0-132-80 kubenswrapper[2544]: I0916 19:06:36.039188    2544 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/0a24ca20-5b1d-4dab-b80f-c3e98a84ead5-proxy-tls\") on node \"ip-10-0-132-80.ec2.internal\" DevicePath \"\""
Sep 16 19:06:36.039356 ip-10-0-132-80 kubenswrapper[2544]: I0916 19:06:36.039199    2544 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/0a24ca20-5b1d-4dab-b80f-c3e98a84ead5-cabundle-cert\") on node \"ip-10-0-132-80.ec2.internal\" DevicePath \"\""
Sep 16 19:06:36.039356 ip-10-0-132-80 kubenswrapper[2544]: I0916 19:06:36.039207    2544 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zgvkk\" (UniqueName: \"kubernetes.io/projected/0a24ca20-5b1d-4dab-b80f-c3e98a84ead5-kube-api-access-zgvkk\") on node \"ip-10-0-132-80.ec2.internal\" DevicePath \"\""
Sep 16 19:06:36.039356 ip-10-0-132-80 kubenswrapper[2544]: I0916 19:06:36.039215    2544 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/0a24ca20-5b1d-4dab-b80f-c3e98a84ead5-kube-rbac-proxy-sar-config\") on node \"ip-10-0-132-80.ec2.internal\" DevicePath \"\""
Sep 16 19:06:36.152396 ip-10-0-132-80 kubenswrapper[2544]: I0916 19:06:36.152348    2544 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-predictor-b9b98bcd9-x6xcs"]
Sep 16 19:06:36.153865 ip-10-0-132-80 kubenswrapper[2544]: I0916 19:06:36.153839    2544 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-predictor-b9b98bcd9-x6xcs"]
Sep 16 19:06:37.721890 ip-10-0-132-80 kubenswrapper[2544]: I0916 19:06:37.721847    2544 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0a24ca20-5b1d-4dab-b80f-c3e98a84ead5" path="/var/lib/kubelet/pods/0a24ca20-5b1d-4dab-b80f-c3e98a84ead5/volumes"
Sep 16 19:06:38.973274 ip-10-0-132-80 kubenswrapper[2544]: I0916 19:06:38.973236    2544 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-7cc6db9b5d-hk7r5"]
Sep 16 19:06:38.973652 ip-10-0-132-80 kubenswrapper[2544]: I0916 19:06:38.973572    2544 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0a24ca20-5b1d-4dab-b80f-c3e98a84ead5" containerName="storage-initializer"
Sep 16 19:06:38.973652 ip-10-0-132-80 kubenswrapper[2544]: I0916 19:06:38.973588    2544 state_mem.go:107] "Deleted CPUSet assignment" podUID="0a24ca20-5b1d-4dab-b80f-c3e98a84ead5" containerName="storage-initializer"
Sep 16 19:06:38.973652 ip-10-0-132-80 kubenswrapper[2544]: I0916 19:06:38.973606    2544 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0a24ca20-5b1d-4dab-b80f-c3e98a84ead5" containerName="kserve-container"
Sep 16 19:06:38.973652 ip-10-0-132-80 kubenswrapper[2544]: I0916 19:06:38.973614    2544 state_mem.go:107] "Deleted CPUSet assignment" podUID="0a24ca20-5b1d-4dab-b80f-c3e98a84ead5" containerName="kserve-container"
Sep 16 19:06:38.973652 ip-10-0-132-80 kubenswrapper[2544]: I0916 19:06:38.973625    2544 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0a24ca20-5b1d-4dab-b80f-c3e98a84ead5" containerName="kube-rbac-proxy"
Sep 16 19:06:38.973652 ip-10-0-132-80 kubenswrapper[2544]: I0916 19:06:38.973631    2544 state_mem.go:107] "Deleted CPUSet assignment" podUID="0a24ca20-5b1d-4dab-b80f-c3e98a84ead5" containerName="kube-rbac-proxy"
Sep 16 19:06:38.973850 ip-10-0-132-80 kubenswrapper[2544]: I0916 19:06:38.973692    2544 memory_manager.go:368] "RemoveStaleState removing state" podUID="0a24ca20-5b1d-4dab-b80f-c3e98a84ead5" containerName="kserve-container"
Sep 16 19:06:38.973850 ip-10-0-132-80 kubenswrapper[2544]: I0916 19:06:38.973702    2544 memory_manager.go:368] "RemoveStaleState removing state" podUID="0a24ca20-5b1d-4dab-b80f-c3e98a84ead5" containerName="kube-rbac-proxy"
Sep 16 19:06:38.978626 ip-10-0-132-80 kubenswrapper[2544]: I0916 19:06:38.978608    2544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-7cc6db9b5d-hk7r5"
Sep 16 19:06:38.983331 ip-10-0-132-80 kubenswrapper[2544]: I0916 19:06:38.983307    2544 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-promote-v2-predictor-serving-cert\""
Sep 16 19:06:38.983788 ip-10-0-132-80 kubenswrapper[2544]: I0916 19:06:38.983765    2544 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-promote-kube-rbac-proxy-sar-config\""
Sep 16 19:06:38.983788 ip-10-0-132-80 kubenswrapper[2544]: I0916 19:06:38.983798    2544 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-zmsvt\""
Sep 16 19:06:38.984130 ip-10-0-132-80 kubenswrapper[2544]: I0916 19:06:38.983929    2544 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Sep 16 19:06:38.984195 ip-10-0-132-80 kubenswrapper[2544]: I0916 19:06:38.984176    2544 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Sep 16 19:06:38.984235 ip-10-0-132-80 kubenswrapper[2544]: I0916 19:06:38.984195    2544 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Sep 16 19:06:38.984285 ip-10-0-132-80 kubenswrapper[2544]: I0916 19:06:38.984261    2544 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Sep 16 19:06:38.990621 ip-10-0-132-80 kubenswrapper[2544]: I0916 19:06:38.990599    2544 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-7cc6db9b5d-hk7r5"]
Sep 16 19:06:39.065656 ip-10-0-132-80 kubenswrapper[2544]: I0916 19:06:39.065630    2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/5afdf766-d550-41e7-b687-9af59c2823de-cabundle-cert\") pod \"isvc-canary-promote-v2-predictor-7cc6db9b5d-hk7r5\" (UID: \"5afdf766-d550-41e7-b687-9af59c2823de\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-7cc6db9b5d-hk7r5"
Sep 16 19:06:39.065798 ip-10-0-132-80 kubenswrapper[2544]: I0916 19:06:39.065672    2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/5afdf766-d550-41e7-b687-9af59c2823de-proxy-tls\") pod \"isvc-canary-promote-v2-predictor-7cc6db9b5d-hk7r5\" (UID: \"5afdf766-d550-41e7-b687-9af59c2823de\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-7cc6db9b5d-hk7r5"
Sep 16 19:06:39.065798 ip-10-0-132-80 kubenswrapper[2544]: I0916 19:06:39.065754    2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/5afdf766-d550-41e7-b687-9af59c2823de-kube-rbac-proxy-sar-config\") pod \"isvc-canary-promote-v2-predictor-7cc6db9b5d-hk7r5\" (UID: \"5afdf766-d550-41e7-b687-9af59c2823de\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-7cc6db9b5d-hk7r5"
Sep 16 19:06:39.065798 ip-10-0-132-80 kubenswrapper[2544]: I0916 19:06:39.065778    2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f9xsk\" (UniqueName: \"kubernetes.io/projected/5afdf766-d550-41e7-b687-9af59c2823de-kube-api-access-f9xsk\") pod \"isvc-canary-promote-v2-predictor-7cc6db9b5d-hk7r5\" (UID: \"5afdf766-d550-41e7-b687-9af59c2823de\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-7cc6db9b5d-hk7r5"
Sep 16 19:06:39.065907 ip-10-0-132-80 kubenswrapper[2544]: I0916 19:06:39.065807    2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/5afdf766-d550-41e7-b687-9af59c2823de-kserve-provision-location\") pod \"isvc-canary-promote-v2-predictor-7cc6db9b5d-hk7r5\" (UID: \"5afdf766-d550-41e7-b687-9af59c2823de\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-7cc6db9b5d-hk7r5"
Sep 16 19:06:39.167219 ip-10-0-132-80 kubenswrapper[2544]: I0916 19:06:39.167181    2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/5afdf766-d550-41e7-b687-9af59c2823de-kube-rbac-proxy-sar-config\") pod \"isvc-canary-promote-v2-predictor-7cc6db9b5d-hk7r5\" (UID: \"5afdf766-d550-41e7-b687-9af59c2823de\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-7cc6db9b5d-hk7r5"
Sep 16 19:06:39.167219 ip-10-0-132-80 kubenswrapper[2544]: I0916 19:06:39.167220    2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f9xsk\" (UniqueName: \"kubernetes.io/projected/5afdf766-d550-41e7-b687-9af59c2823de-kube-api-access-f9xsk\") pod \"isvc-canary-promote-v2-predictor-7cc6db9b5d-hk7r5\" (UID: \"5afdf766-d550-41e7-b687-9af59c2823de\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-7cc6db9b5d-hk7r5"
Sep 16 19:06:39.167439 ip-10-0-132-80 kubenswrapper[2544]: I0916 19:06:39.167245    2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/5afdf766-d550-41e7-b687-9af59c2823de-kserve-provision-location\") pod \"isvc-canary-promote-v2-predictor-7cc6db9b5d-hk7r5\" (UID: \"5afdf766-d550-41e7-b687-9af59c2823de\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-7cc6db9b5d-hk7r5"
Sep 16 19:06:39.167496 ip-10-0-132-80 kubenswrapper[2544]: I0916 19:06:39.167449    2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/5afdf766-d550-41e7-b687-9af59c2823de-cabundle-cert\") pod \"isvc-canary-promote-v2-predictor-7cc6db9b5d-hk7r5\" (UID: \"5afdf766-d550-41e7-b687-9af59c2823de\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-7cc6db9b5d-hk7r5"
Sep 16 19:06:39.167548 ip-10-0-132-80 kubenswrapper[2544]: I0916 19:06:39.167501    2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/5afdf766-d550-41e7-b687-9af59c2823de-proxy-tls\") pod \"isvc-canary-promote-v2-predictor-7cc6db9b5d-hk7r5\" (UID: \"5afdf766-d550-41e7-b687-9af59c2823de\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-7cc6db9b5d-hk7r5"
Sep 16 19:06:39.167745 ip-10-0-132-80 kubenswrapper[2544]: I0916 19:06:39.167723    2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/5afdf766-d550-41e7-b687-9af59c2823de-kserve-provision-location\") pod \"isvc-canary-promote-v2-predictor-7cc6db9b5d-hk7r5\" (UID: \"5afdf766-d550-41e7-b687-9af59c2823de\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-7cc6db9b5d-hk7r5"
Sep 16 19:06:39.167973 ip-10-0-132-80 kubenswrapper[2544]: I0916 19:06:39.167956    2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/5afdf766-d550-41e7-b687-9af59c2823de-kube-rbac-proxy-sar-config\") pod \"isvc-canary-promote-v2-predictor-7cc6db9b5d-hk7r5\" (UID: \"5afdf766-d550-41e7-b687-9af59c2823de\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-7cc6db9b5d-hk7r5"
Sep 16 19:06:39.168037 ip-10-0-132-80 kubenswrapper[2544]: I0916 19:06:39.168025    2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/5afdf766-d550-41e7-b687-9af59c2823de-cabundle-cert\") pod \"isvc-canary-promote-v2-predictor-7cc6db9b5d-hk7r5\" (UID: \"5afdf766-d550-41e7-b687-9af59c2823de\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-7cc6db9b5d-hk7r5"
Sep 16 19:06:39.169894 ip-10-0-132-80 kubenswrapper[2544]: I0916 19:06:39.169875    2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/5afdf766-d550-41e7-b687-9af59c2823de-proxy-tls\") pod \"isvc-canary-promote-v2-predictor-7cc6db9b5d-hk7r5\" (UID: \"5afdf766-d550-41e7-b687-9af59c2823de\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-7cc6db9b5d-hk7r5"
Sep 16 19:06:39.175921 ip-10-0-132-80 kubenswrapper[2544]: I0916 19:06:39.175894    2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-f9xsk\" (UniqueName: \"kubernetes.io/projected/5afdf766-d550-41e7-b687-9af59c2823de-kube-api-access-f9xsk\") pod \"isvc-canary-promote-v2-predictor-7cc6db9b5d-hk7r5\" (UID: \"5afdf766-d550-41e7-b687-9af59c2823de\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-7cc6db9b5d-hk7r5"
Sep 16 19:06:39.288852 ip-10-0-132-80 kubenswrapper[2544]: I0916 19:06:39.288748    2544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-7cc6db9b5d-hk7r5"
Sep 16 19:06:39.564682 ip-10-0-132-80 kubenswrapper[2544]: I0916 19:06:39.564647    2544 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-7cc6db9b5d-hk7r5"]
Sep 16 19:06:39.567773 ip-10-0-132-80 kubenswrapper[2544]: W0916 19:06:39.567745    2544 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5afdf766_d550_41e7_b687_9af59c2823de.slice/crio-336dd885674fb8f44d89ba96d55634d7753c0dc39e7ac2567653909c6afb5a2e WatchSource:0}: Error finding container 336dd885674fb8f44d89ba96d55634d7753c0dc39e7ac2567653909c6afb5a2e: Status 404 returned error can't find the container with id 336dd885674fb8f44d89ba96d55634d7753c0dc39e7ac2567653909c6afb5a2e
Sep 16 19:06:39.846670 ip-10-0-132-80 kubenswrapper[2544]: I0916 19:06:39.846577    2544 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-7cc6db9b5d-hk7r5" event={"ID":"5afdf766-d550-41e7-b687-9af59c2823de","Type":"ContainerStarted","Data":"b4718b3d303dd95628b54bd39ffd842b2b3aab16bd9e3224ca352f42e7a79025"}
Sep 16 19:06:39.846670 ip-10-0-132-80 kubenswrapper[2544]: I0916 19:06:39.846633    2544 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-7cc6db9b5d-hk7r5" event={"ID":"5afdf766-d550-41e7-b687-9af59c2823de","Type":"ContainerStarted","Data":"336dd885674fb8f44d89ba96d55634d7753c0dc39e7ac2567653909c6afb5a2e"}
Sep 16 19:06:43.863694 ip-10-0-132-80 kubenswrapper[2544]: I0916 19:06:43.863663    2544 generic.go:356] "Generic (PLEG): container finished" podID="5afdf766-d550-41e7-b687-9af59c2823de" containerID="b4718b3d303dd95628b54bd39ffd842b2b3aab16bd9e3224ca352f42e7a79025" exitCode=0
Sep 16 19:06:43.864109 ip-10-0-132-80 kubenswrapper[2544]: I0916 19:06:43.863741    2544 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-7cc6db9b5d-hk7r5" event={"ID":"5afdf766-d550-41e7-b687-9af59c2823de","Type":"ContainerDied","Data":"b4718b3d303dd95628b54bd39ffd842b2b3aab16bd9e3224ca352f42e7a79025"}
Sep 16 19:06:44.870104 ip-10-0-132-80 kubenswrapper[2544]: I0916 19:06:44.870072    2544 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-7cc6db9b5d-hk7r5" event={"ID":"5afdf766-d550-41e7-b687-9af59c2823de","Type":"ContainerStarted","Data":"73d12176d1aa02f5265ae793073ff533daff8b862a4a869262099abb5bd173d3"}
Sep 16 19:06:44.870584 ip-10-0-132-80 kubenswrapper[2544]: I0916 19:06:44.870142    2544 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-7cc6db9b5d-hk7r5" event={"ID":"5afdf766-d550-41e7-b687-9af59c2823de","Type":"ContainerStarted","Data":"1af8ec04d1b16c6d145bc0a19a89684eb6381c14adb7a5672046488127acd580"}
Sep 16 19:06:44.870584 ip-10-0-132-80 kubenswrapper[2544]: I0916 19:06:44.870365    2544 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-7cc6db9b5d-hk7r5"
Sep 16 19:06:44.892000 ip-10-0-132-80 kubenswrapper[2544]: I0916 19:06:44.891939    2544 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-7cc6db9b5d-hk7r5" podStartSLOduration=3.166651483 podStartE2EDuration="6.891917864s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.725266381s" isStatefulPod=false podCreationTimestamp="2026-09-16 19:06:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:06:44.89144689 +0000 UTC m=+1367.680449445" watchObservedRunningTime="2026-09-16 19:06:44.891917864 +0000 UTC m=+1367.680920397"
Sep 16 19:06:45.874393 ip-10-0-132-80 kubenswrapper[2544]: I0916 19:06:45.874360    2544 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-7cc6db9b5d-hk7r5"
Sep 16 19:06:45.875417 ip-10-0-132-80 kubenswrapper[2544]: I0916 19:06:45.875390    2544 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-7cc6db9b5d-hk7r5" podUID="5afdf766-d550-41e7-b687-9af59c2823de" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.37:8080: connect: connection refused"
Sep 16 19:06:46.878113 ip-10-0-132-80 kubenswrapper[2544]: I0916 19:06:46.878070    2544 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-7cc6db9b5d-hk7r5" podUID="5afdf766-d550-41e7-b687-9af59c2823de" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.37:8080: connect: connection refused"
Sep 16 19:06:51.882852 ip-10-0-132-80 kubenswrapper[2544]: I0916 19:06:51.882823    2544 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-7cc6db9b5d-hk7r5"
Sep 16 19:06:51.883324 ip-10-0-132-80 kubenswrapper[2544]: I0916 19:06:51.883297    2544 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-7cc6db9b5d-hk7r5" podUID="5afdf766-d550-41e7-b687-9af59c2823de" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.37:8080: connect: connection refused"
Sep 16 19:07:01.883520 ip-10-0-132-80 kubenswrapper[2544]: I0916 19:07:01.883472    2544 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-7cc6db9b5d-hk7r5" podUID="5afdf766-d550-41e7-b687-9af59c2823de" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.37:8080: connect: connection refused"
Sep 16 19:07:11.884136 ip-10-0-132-80 kubenswrapper[2544]: I0916 19:07:11.884100    2544 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-7cc6db9b5d-hk7r5" podUID="5afdf766-d550-41e7-b687-9af59c2823de" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.37:8080: connect: connection refused"
Sep 16 19:07:21.883982 ip-10-0-132-80 kubenswrapper[2544]: I0916 19:07:21.883941    2544 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-7cc6db9b5d-hk7r5" podUID="5afdf766-d550-41e7-b687-9af59c2823de" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.37:8080: connect: connection refused"
Sep 16 19:07:31.884215 ip-10-0-132-80 kubenswrapper[2544]: I0916 19:07:31.884172    2544 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-7cc6db9b5d-hk7r5" podUID="5afdf766-d550-41e7-b687-9af59c2823de" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.37:8080: connect: connection refused"
Sep 16 19:07:41.883833 ip-10-0-132-80 kubenswrapper[2544]: I0916 19:07:41.883791    2544 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-7cc6db9b5d-hk7r5" podUID="5afdf766-d550-41e7-b687-9af59c2823de" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.37:8080: connect: connection refused"
Sep 16 19:07:51.883830 ip-10-0-132-80 kubenswrapper[2544]: I0916 19:07:51.883739    2544 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-7cc6db9b5d-hk7r5" podUID="5afdf766-d550-41e7-b687-9af59c2823de" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.37:8080: connect: connection refused"
Sep 16 19:08:01.884706 ip-10-0-132-80 kubenswrapper[2544]: I0916 19:08:01.884671    2544 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-7cc6db9b5d-hk7r5"
Sep 16 19:08:08.978518 ip-10-0-132-80 kubenswrapper[2544]: I0916 19:08:08.978485    2544 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-7cc6db9b5d-hk7r5"]
Sep 16 19:08:08.979008 ip-10-0-132-80 kubenswrapper[2544]: I0916 19:08:08.978736    2544 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-7cc6db9b5d-hk7r5" podUID="5afdf766-d550-41e7-b687-9af59c2823de" containerName="kserve-container" containerID="cri-o://1af8ec04d1b16c6d145bc0a19a89684eb6381c14adb7a5672046488127acd580" gracePeriod=30
Sep 16 19:08:08.979008 ip-10-0-132-80 kubenswrapper[2544]: I0916 19:08:08.978787    2544 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-7cc6db9b5d-hk7r5" podUID="5afdf766-d550-41e7-b687-9af59c2823de" containerName="kube-rbac-proxy" containerID="cri-o://73d12176d1aa02f5265ae793073ff533daff8b862a4a869262099abb5bd173d3" gracePeriod=30
Sep 16 19:08:09.228740 ip-10-0-132-80 kubenswrapper[2544]: I0916 19:08:09.228657    2544 generic.go:356] "Generic (PLEG): container finished" podID="5afdf766-d550-41e7-b687-9af59c2823de" containerID="73d12176d1aa02f5265ae793073ff533daff8b862a4a869262099abb5bd173d3" exitCode=2
Sep 16 19:08:09.228740 ip-10-0-132-80 kubenswrapper[2544]: I0916 19:08:09.228695    2544 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-7cc6db9b5d-hk7r5" event={"ID":"5afdf766-d550-41e7-b687-9af59c2823de","Type":"ContainerDied","Data":"73d12176d1aa02f5265ae793073ff533daff8b862a4a869262099abb5bd173d3"}
Sep 16 19:08:11.878787 ip-10-0-132-80 kubenswrapper[2544]: I0916 19:08:11.878740    2544 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-7cc6db9b5d-hk7r5" podUID="5afdf766-d550-41e7-b687-9af59c2823de" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.37:8643/healthz\": dial tcp 10.132.0.37:8643: connect: connection refused"
Sep 16 19:08:11.883916 ip-10-0-132-80 kubenswrapper[2544]: I0916 19:08:11.883892    2544 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-7cc6db9b5d-hk7r5" podUID="5afdf766-d550-41e7-b687-9af59c2823de" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.37:8080: connect: connection refused"
Sep 16 19:08:14.023082 ip-10-0-132-80 kubenswrapper[2544]: I0916 19:08:14.023054    2544 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-7cc6db9b5d-hk7r5"
Sep 16 19:08:14.144390 ip-10-0-132-80 kubenswrapper[2544]: I0916 19:08:14.144323    2544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/5afdf766-d550-41e7-b687-9af59c2823de-kube-rbac-proxy-sar-config\") pod \"5afdf766-d550-41e7-b687-9af59c2823de\" (UID: \"5afdf766-d550-41e7-b687-9af59c2823de\") "
Sep 16 19:08:14.144390 ip-10-0-132-80 kubenswrapper[2544]: I0916 19:08:14.144360    2544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/5afdf766-d550-41e7-b687-9af59c2823de-proxy-tls\") pod \"5afdf766-d550-41e7-b687-9af59c2823de\" (UID: \"5afdf766-d550-41e7-b687-9af59c2823de\") "
Sep 16 19:08:14.144390 ip-10-0-132-80 kubenswrapper[2544]: I0916 19:08:14.144384    2544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/5afdf766-d550-41e7-b687-9af59c2823de-cabundle-cert\") pod \"5afdf766-d550-41e7-b687-9af59c2823de\" (UID: \"5afdf766-d550-41e7-b687-9af59c2823de\") "
Sep 16 19:08:14.144685 ip-10-0-132-80 kubenswrapper[2544]: I0916 19:08:14.144412    2544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/5afdf766-d550-41e7-b687-9af59c2823de-kserve-provision-location\") pod \"5afdf766-d550-41e7-b687-9af59c2823de\" (UID: \"5afdf766-d550-41e7-b687-9af59c2823de\") "
Sep 16 19:08:14.144685 ip-10-0-132-80 kubenswrapper[2544]: I0916 19:08:14.144467    2544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-f9xsk\" (UniqueName: \"kubernetes.io/projected/5afdf766-d550-41e7-b687-9af59c2823de-kube-api-access-f9xsk\") pod \"5afdf766-d550-41e7-b687-9af59c2823de\" (UID: \"5afdf766-d550-41e7-b687-9af59c2823de\") "
Sep 16 19:08:14.144798 ip-10-0-132-80 kubenswrapper[2544]: I0916 19:08:14.144692    2544 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5afdf766-d550-41e7-b687-9af59c2823de-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "5afdf766-d550-41e7-b687-9af59c2823de" (UID: "5afdf766-d550-41e7-b687-9af59c2823de"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 19:08:14.144798 ip-10-0-132-80 kubenswrapper[2544]: I0916 19:08:14.144780    2544 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5afdf766-d550-41e7-b687-9af59c2823de-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "5afdf766-d550-41e7-b687-9af59c2823de" (UID: "5afdf766-d550-41e7-b687-9af59c2823de"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 19:08:14.144895 ip-10-0-132-80 kubenswrapper[2544]: I0916 19:08:14.144823    2544 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5afdf766-d550-41e7-b687-9af59c2823de-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "5afdf766-d550-41e7-b687-9af59c2823de" (UID: "5afdf766-d550-41e7-b687-9af59c2823de"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:08:14.146521 ip-10-0-132-80 kubenswrapper[2544]: I0916 19:08:14.146500    2544 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5afdf766-d550-41e7-b687-9af59c2823de-kube-api-access-f9xsk" (OuterVolumeSpecName: "kube-api-access-f9xsk") pod "5afdf766-d550-41e7-b687-9af59c2823de" (UID: "5afdf766-d550-41e7-b687-9af59c2823de"). InnerVolumeSpecName "kube-api-access-f9xsk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 19:08:14.146629 ip-10-0-132-80 kubenswrapper[2544]: I0916 19:08:14.146537    2544 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5afdf766-d550-41e7-b687-9af59c2823de-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "5afdf766-d550-41e7-b687-9af59c2823de" (UID: "5afdf766-d550-41e7-b687-9af59c2823de"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 19:08:14.245722 ip-10-0-132-80 kubenswrapper[2544]: I0916 19:08:14.245697    2544 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/5afdf766-d550-41e7-b687-9af59c2823de-kube-rbac-proxy-sar-config\") on node \"ip-10-0-132-80.ec2.internal\" DevicePath \"\""
Sep 16 19:08:14.245722 ip-10-0-132-80 kubenswrapper[2544]: I0916 19:08:14.245717    2544 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/5afdf766-d550-41e7-b687-9af59c2823de-proxy-tls\") on node \"ip-10-0-132-80.ec2.internal\" DevicePath \"\""
Sep 16 19:08:14.245845 ip-10-0-132-80 kubenswrapper[2544]: I0916 19:08:14.245732    2544 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/5afdf766-d550-41e7-b687-9af59c2823de-cabundle-cert\") on node \"ip-10-0-132-80.ec2.internal\" DevicePath \"\""
Sep 16 19:08:14.245845 ip-10-0-132-80 kubenswrapper[2544]: I0916 19:08:14.245745    2544 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/5afdf766-d550-41e7-b687-9af59c2823de-kserve-provision-location\") on node \"ip-10-0-132-80.ec2.internal\" DevicePath \"\""
Sep 16 19:08:14.245845 ip-10-0-132-80 kubenswrapper[2544]: I0916 19:08:14.245752    2544 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f9xsk\" (UniqueName: \"kubernetes.io/projected/5afdf766-d550-41e7-b687-9af59c2823de-kube-api-access-f9xsk\") on node \"ip-10-0-132-80.ec2.internal\" DevicePath \"\""
Sep 16 19:08:14.251181 ip-10-0-132-80 kubenswrapper[2544]: I0916 19:08:14.251159    2544 generic.go:356] "Generic (PLEG): container finished" podID="5afdf766-d550-41e7-b687-9af59c2823de" containerID="1af8ec04d1b16c6d145bc0a19a89684eb6381c14adb7a5672046488127acd580" exitCode=0
Sep 16 19:08:14.251269 ip-10-0-132-80 kubenswrapper[2544]: I0916 19:08:14.251217    2544 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-7cc6db9b5d-hk7r5"
Sep 16 19:08:14.251269 ip-10-0-132-80 kubenswrapper[2544]: I0916 19:08:14.251242    2544 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-7cc6db9b5d-hk7r5" event={"ID":"5afdf766-d550-41e7-b687-9af59c2823de","Type":"ContainerDied","Data":"1af8ec04d1b16c6d145bc0a19a89684eb6381c14adb7a5672046488127acd580"}
Sep 16 19:08:14.251345 ip-10-0-132-80 kubenswrapper[2544]: I0916 19:08:14.251302    2544 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-7cc6db9b5d-hk7r5" event={"ID":"5afdf766-d550-41e7-b687-9af59c2823de","Type":"ContainerDied","Data":"336dd885674fb8f44d89ba96d55634d7753c0dc39e7ac2567653909c6afb5a2e"}
Sep 16 19:08:14.251345 ip-10-0-132-80 kubenswrapper[2544]: I0916 19:08:14.251328    2544 scope.go:117] "RemoveContainer" containerID="73d12176d1aa02f5265ae793073ff533daff8b862a4a869262099abb5bd173d3"
Sep 16 19:08:14.259362 ip-10-0-132-80 kubenswrapper[2544]: I0916 19:08:14.259346    2544 scope.go:117] "RemoveContainer" containerID="1af8ec04d1b16c6d145bc0a19a89684eb6381c14adb7a5672046488127acd580"
Sep 16 19:08:14.266449 ip-10-0-132-80 kubenswrapper[2544]: I0916 19:08:14.266431    2544 scope.go:117] "RemoveContainer" containerID="b4718b3d303dd95628b54bd39ffd842b2b3aab16bd9e3224ca352f42e7a79025"
Sep 16 19:08:14.272786 ip-10-0-132-80 kubenswrapper[2544]: I0916 19:08:14.272765    2544 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-7cc6db9b5d-hk7r5"]
Sep 16 19:08:14.273750 ip-10-0-132-80 kubenswrapper[2544]: I0916 19:08:14.273733    2544 scope.go:117] "RemoveContainer" containerID="73d12176d1aa02f5265ae793073ff533daff8b862a4a869262099abb5bd173d3"
Sep 16 19:08:14.274040 ip-10-0-132-80 kubenswrapper[2544]: E0916 19:08:14.274012    2544 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"73d12176d1aa02f5265ae793073ff533daff8b862a4a869262099abb5bd173d3\": container with ID starting with 73d12176d1aa02f5265ae793073ff533daff8b862a4a869262099abb5bd173d3 not found: ID does not exist" containerID="73d12176d1aa02f5265ae793073ff533daff8b862a4a869262099abb5bd173d3"
Sep 16 19:08:14.274158 ip-10-0-132-80 kubenswrapper[2544]: I0916 19:08:14.274048    2544 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"73d12176d1aa02f5265ae793073ff533daff8b862a4a869262099abb5bd173d3"} err="failed to get container status \"73d12176d1aa02f5265ae793073ff533daff8b862a4a869262099abb5bd173d3\": rpc error: code = NotFound desc = could not find container \"73d12176d1aa02f5265ae793073ff533daff8b862a4a869262099abb5bd173d3\": container with ID starting with 73d12176d1aa02f5265ae793073ff533daff8b862a4a869262099abb5bd173d3 not found: ID does not exist"
Sep 16 19:08:14.274158 ip-10-0-132-80 kubenswrapper[2544]: I0916 19:08:14.274072    2544 scope.go:117] "RemoveContainer" containerID="1af8ec04d1b16c6d145bc0a19a89684eb6381c14adb7a5672046488127acd580"
Sep 16 19:08:14.274341 ip-10-0-132-80 kubenswrapper[2544]: E0916 19:08:14.274320    2544 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1af8ec04d1b16c6d145bc0a19a89684eb6381c14adb7a5672046488127acd580\": container with ID starting with 1af8ec04d1b16c6d145bc0a19a89684eb6381c14adb7a5672046488127acd580 not found: ID does not exist" containerID="1af8ec04d1b16c6d145bc0a19a89684eb6381c14adb7a5672046488127acd580"
Sep 16 19:08:14.274420 ip-10-0-132-80 kubenswrapper[2544]: I0916 19:08:14.274350    2544 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1af8ec04d1b16c6d145bc0a19a89684eb6381c14adb7a5672046488127acd580"} err="failed to get container status \"1af8ec04d1b16c6d145bc0a19a89684eb6381c14adb7a5672046488127acd580\": rpc error: code = NotFound desc = could not find container \"1af8ec04d1b16c6d145bc0a19a89684eb6381c14adb7a5672046488127acd580\": container with ID starting with 1af8ec04d1b16c6d145bc0a19a89684eb6381c14adb7a5672046488127acd580 not found: ID does not exist"
Sep 16 19:08:14.274420 ip-10-0-132-80 kubenswrapper[2544]: I0916 19:08:14.274370    2544 scope.go:117] "RemoveContainer" containerID="b4718b3d303dd95628b54bd39ffd842b2b3aab16bd9e3224ca352f42e7a79025"
Sep 16 19:08:14.274710 ip-10-0-132-80 kubenswrapper[2544]: E0916 19:08:14.274683    2544 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b4718b3d303dd95628b54bd39ffd842b2b3aab16bd9e3224ca352f42e7a79025\": container with ID starting with b4718b3d303dd95628b54bd39ffd842b2b3aab16bd9e3224ca352f42e7a79025 not found: ID does not exist" containerID="b4718b3d303dd95628b54bd39ffd842b2b3aab16bd9e3224ca352f42e7a79025"
Sep 16 19:08:14.274822 ip-10-0-132-80 kubenswrapper[2544]: I0916 19:08:14.274717    2544 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b4718b3d303dd95628b54bd39ffd842b2b3aab16bd9e3224ca352f42e7a79025"} err="failed to get container status \"b4718b3d303dd95628b54bd39ffd842b2b3aab16bd9e3224ca352f42e7a79025\": rpc error: code = NotFound desc = could not find container \"b4718b3d303dd95628b54bd39ffd842b2b3aab16bd9e3224ca352f42e7a79025\": container with ID starting with b4718b3d303dd95628b54bd39ffd842b2b3aab16bd9e3224ca352f42e7a79025 not found: ID does not exist"
Sep 16 19:08:14.276579 ip-10-0-132-80 kubenswrapper[2544]: I0916 19:08:14.276546    2544 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-7cc6db9b5d-hk7r5"]
Sep 16 19:08:15.722214 ip-10-0-132-80 kubenswrapper[2544]: I0916 19:08:15.722180    2544 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5afdf766-d550-41e7-b687-9af59c2823de" path="/var/lib/kubelet/pods/5afdf766-d550-41e7-b687-9af59c2823de/volumes"
Sep 16 19:08:33.172660 ip-10-0-132-80 kubenswrapper[2544]: I0916 19:08:33.172620    2544 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-647756665c-q8rlw"]
Sep 16 19:08:33.173060 ip-10-0-132-80 kubenswrapper[2544]: I0916 19:08:33.172942    2544 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5afdf766-d550-41e7-b687-9af59c2823de" containerName="storage-initializer"
Sep 16 19:08:33.173060 ip-10-0-132-80 kubenswrapper[2544]: I0916 19:08:33.172954    2544 state_mem.go:107] "Deleted CPUSet assignment" podUID="5afdf766-d550-41e7-b687-9af59c2823de" containerName="storage-initializer"
Sep 16 19:08:33.173060 ip-10-0-132-80 kubenswrapper[2544]: I0916 19:08:33.172965    2544 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5afdf766-d550-41e7-b687-9af59c2823de" containerName="kserve-container"
Sep 16 19:08:33.173060 ip-10-0-132-80 kubenswrapper[2544]: I0916 19:08:33.172971    2544 state_mem.go:107] "Deleted CPUSet assignment" podUID="5afdf766-d550-41e7-b687-9af59c2823de" containerName="kserve-container"
Sep 16 19:08:33.173060 ip-10-0-132-80 kubenswrapper[2544]: I0916 19:08:33.172984    2544 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5afdf766-d550-41e7-b687-9af59c2823de" containerName="kube-rbac-proxy"
Sep 16 19:08:33.173060 ip-10-0-132-80 kubenswrapper[2544]: I0916 19:08:33.172990    2544 state_mem.go:107] "Deleted CPUSet assignment" podUID="5afdf766-d550-41e7-b687-9af59c2823de" containerName="kube-rbac-proxy"
Sep 16 19:08:33.173060 ip-10-0-132-80 kubenswrapper[2544]: I0916 19:08:33.173050    2544 memory_manager.go:368] "RemoveStaleState removing state" podUID="5afdf766-d550-41e7-b687-9af59c2823de" containerName="kserve-container"
Sep 16 19:08:33.173060 ip-10-0-132-80 kubenswrapper[2544]: I0916 19:08:33.173057    2544 memory_manager.go:368] "RemoveStaleState removing state" podUID="5afdf766-d550-41e7-b687-9af59c2823de" containerName="kube-rbac-proxy"
Sep 16 19:08:33.176253 ip-10-0-132-80 kubenswrapper[2544]: I0916 19:08:33.176235    2544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-647756665c-q8rlw"
Sep 16 19:08:33.178437 ip-10-0-132-80 kubenswrapper[2544]: I0916 19:08:33.178409    2544 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-logger-raw-cipn-kube-rbac-proxy-sar-config\""
Sep 16 19:08:33.178598 ip-10-0-132-80 kubenswrapper[2544]: I0916 19:08:33.178466    2544 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Sep 16 19:08:33.178598 ip-10-0-132-80 kubenswrapper[2544]: I0916 19:08:33.178534    2544 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Sep 16 19:08:33.178736 ip-10-0-132-80 kubenswrapper[2544]: I0916 19:08:33.178719    2544 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-logger-raw-cipn-predictor-serving-cert\""
Sep 16 19:08:33.178787 ip-10-0-132-80 kubenswrapper[2544]: I0916 19:08:33.178762    2544 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Sep 16 19:08:33.178834 ip-10-0-132-80 kubenswrapper[2544]: I0916 19:08:33.178787    2544 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-zmsvt\""
Sep 16 19:08:33.178834 ip-10-0-132-80 kubenswrapper[2544]: I0916 19:08:33.178722    2544 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Sep 16 19:08:33.187593 ip-10-0-132-80 kubenswrapper[2544]: I0916 19:08:33.187568    2544 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-647756665c-q8rlw"]
Sep 16 19:08:33.303454 ip-10-0-132-80 kubenswrapper[2544]: I0916 19:08:33.303420    2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zjrzh\" (UniqueName: \"kubernetes.io/projected/fb1507b9-4934-42df-9691-0f4de73beb3a-kube-api-access-zjrzh\") pod \"isvc-logger-raw-cipn-predictor-647756665c-q8rlw\" (UID: \"fb1507b9-4934-42df-9691-0f4de73beb3a\") " pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-647756665c-q8rlw"
Sep 16 19:08:33.303454 ip-10-0-132-80 kubenswrapper[2544]: I0916 19:08:33.303451    2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/fb1507b9-4934-42df-9691-0f4de73beb3a-kserve-provision-location\") pod \"isvc-logger-raw-cipn-predictor-647756665c-q8rlw\" (UID: \"fb1507b9-4934-42df-9691-0f4de73beb3a\") " pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-647756665c-q8rlw"
Sep 16 19:08:33.303753 ip-10-0-132-80 kubenswrapper[2544]: I0916 19:08:33.303582    2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/fb1507b9-4934-42df-9691-0f4de73beb3a-cabundle-cert\") pod \"isvc-logger-raw-cipn-predictor-647756665c-q8rlw\" (UID: \"fb1507b9-4934-42df-9691-0f4de73beb3a\") " pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-647756665c-q8rlw"
Sep 16 19:08:33.303753 ip-10-0-132-80 kubenswrapper[2544]: I0916 19:08:33.303640    2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/fb1507b9-4934-42df-9691-0f4de73beb3a-proxy-tls\") pod \"isvc-logger-raw-cipn-predictor-647756665c-q8rlw\" (UID: \"fb1507b9-4934-42df-9691-0f4de73beb3a\") " pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-647756665c-q8rlw"
Sep 16 19:08:33.303753 ip-10-0-132-80 kubenswrapper[2544]: I0916 19:08:33.303675    2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/fb1507b9-4934-42df-9691-0f4de73beb3a-kube-rbac-proxy-sar-config\") pod \"isvc-logger-raw-cipn-predictor-647756665c-q8rlw\" (UID: \"fb1507b9-4934-42df-9691-0f4de73beb3a\") " pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-647756665c-q8rlw"
Sep 16 19:08:33.404404 ip-10-0-132-80 kubenswrapper[2544]: I0916 19:08:33.404367    2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zjrzh\" (UniqueName: \"kubernetes.io/projected/fb1507b9-4934-42df-9691-0f4de73beb3a-kube-api-access-zjrzh\") pod \"isvc-logger-raw-cipn-predictor-647756665c-q8rlw\" (UID: \"fb1507b9-4934-42df-9691-0f4de73beb3a\") " pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-647756665c-q8rlw"
Sep 16 19:08:33.404404 ip-10-0-132-80 kubenswrapper[2544]: I0916 19:08:33.404400    2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/fb1507b9-4934-42df-9691-0f4de73beb3a-kserve-provision-location\") pod \"isvc-logger-raw-cipn-predictor-647756665c-q8rlw\" (UID: \"fb1507b9-4934-42df-9691-0f4de73beb3a\") " pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-647756665c-q8rlw"
Sep 16 19:08:33.404673 ip-10-0-132-80 kubenswrapper[2544]: I0916 19:08:33.404595    2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/fb1507b9-4934-42df-9691-0f4de73beb3a-cabundle-cert\") pod \"isvc-logger-raw-cipn-predictor-647756665c-q8rlw\" (UID: \"fb1507b9-4934-42df-9691-0f4de73beb3a\") " pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-647756665c-q8rlw"
Sep 16 19:08:33.404673 ip-10-0-132-80 kubenswrapper[2544]: I0916 19:08:33.404649    2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/fb1507b9-4934-42df-9691-0f4de73beb3a-proxy-tls\") pod \"isvc-logger-raw-cipn-predictor-647756665c-q8rlw\" (UID: \"fb1507b9-4934-42df-9691-0f4de73beb3a\") " pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-647756665c-q8rlw"
Sep 16 19:08:33.404802 ip-10-0-132-80 kubenswrapper[2544]: I0916 19:08:33.404754    2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/fb1507b9-4934-42df-9691-0f4de73beb3a-kube-rbac-proxy-sar-config\") pod \"isvc-logger-raw-cipn-predictor-647756665c-q8rlw\" (UID: \"fb1507b9-4934-42df-9691-0f4de73beb3a\") " pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-647756665c-q8rlw"
Sep 16 19:08:33.404868 ip-10-0-132-80 kubenswrapper[2544]: I0916 19:08:33.404805    2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/fb1507b9-4934-42df-9691-0f4de73beb3a-kserve-provision-location\") pod \"isvc-logger-raw-cipn-predictor-647756665c-q8rlw\" (UID: \"fb1507b9-4934-42df-9691-0f4de73beb3a\") " pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-647756665c-q8rlw"
Sep 16 19:08:33.405183 ip-10-0-132-80 kubenswrapper[2544]: I0916 19:08:33.405166    2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/fb1507b9-4934-42df-9691-0f4de73beb3a-cabundle-cert\") pod \"isvc-logger-raw-cipn-predictor-647756665c-q8rlw\" (UID: \"fb1507b9-4934-42df-9691-0f4de73beb3a\") " pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-647756665c-q8rlw"
Sep 16 19:08:33.405301 ip-10-0-132-80 kubenswrapper[2544]: I0916 19:08:33.405282    2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/fb1507b9-4934-42df-9691-0f4de73beb3a-kube-rbac-proxy-sar-config\") pod \"isvc-logger-raw-cipn-predictor-647756665c-q8rlw\" (UID: \"fb1507b9-4934-42df-9691-0f4de73beb3a\") " pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-647756665c-q8rlw"
Sep 16 19:08:33.407095 ip-10-0-132-80 kubenswrapper[2544]: I0916 19:08:33.407074    2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/fb1507b9-4934-42df-9691-0f4de73beb3a-proxy-tls\") pod \"isvc-logger-raw-cipn-predictor-647756665c-q8rlw\" (UID: \"fb1507b9-4934-42df-9691-0f4de73beb3a\") " pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-647756665c-q8rlw"
Sep 16 19:08:33.411930 ip-10-0-132-80 kubenswrapper[2544]: I0916 19:08:33.411909    2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zjrzh\" (UniqueName: \"kubernetes.io/projected/fb1507b9-4934-42df-9691-0f4de73beb3a-kube-api-access-zjrzh\") pod \"isvc-logger-raw-cipn-predictor-647756665c-q8rlw\" (UID: \"fb1507b9-4934-42df-9691-0f4de73beb3a\") " pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-647756665c-q8rlw"
Sep 16 19:08:33.486763 ip-10-0-132-80 kubenswrapper[2544]: I0916 19:08:33.486694    2544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-647756665c-q8rlw"
Sep 16 19:08:33.760854 ip-10-0-132-80 kubenswrapper[2544]: I0916 19:08:33.760826    2544 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-647756665c-q8rlw"]
Sep 16 19:08:33.762545 ip-10-0-132-80 kubenswrapper[2544]: W0916 19:08:33.762518    2544 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfb1507b9_4934_42df_9691_0f4de73beb3a.slice/crio-045c21889a3f2873f48dac2389f359700034a28ddb2b2b57ddb5189eee6d1f5d WatchSource:0}: Error finding container 045c21889a3f2873f48dac2389f359700034a28ddb2b2b57ddb5189eee6d1f5d: Status 404 returned error can't find the container with id 045c21889a3f2873f48dac2389f359700034a28ddb2b2b57ddb5189eee6d1f5d
Sep 16 19:08:34.333588 ip-10-0-132-80 kubenswrapper[2544]: I0916 19:08:34.333533    2544 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-647756665c-q8rlw" event={"ID":"fb1507b9-4934-42df-9691-0f4de73beb3a","Type":"ContainerStarted","Data":"ff777f50caea24f2b6b3fde0f848bcdb84ec9b757466dc02f65742ac7226a616"}
Sep 16 19:08:34.334057 ip-10-0-132-80 kubenswrapper[2544]: I0916 19:08:34.333631    2544 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-647756665c-q8rlw" event={"ID":"fb1507b9-4934-42df-9691-0f4de73beb3a","Type":"ContainerStarted","Data":"045c21889a3f2873f48dac2389f359700034a28ddb2b2b57ddb5189eee6d1f5d"}
Sep 16 19:08:38.352206 ip-10-0-132-80 kubenswrapper[2544]: I0916 19:08:38.352170    2544 generic.go:356] "Generic (PLEG): container finished" podID="fb1507b9-4934-42df-9691-0f4de73beb3a" containerID="ff777f50caea24f2b6b3fde0f848bcdb84ec9b757466dc02f65742ac7226a616" exitCode=0
Sep 16 19:08:38.352603 ip-10-0-132-80 kubenswrapper[2544]: I0916 19:08:38.352242    2544 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-647756665c-q8rlw" event={"ID":"fb1507b9-4934-42df-9691-0f4de73beb3a","Type":"ContainerDied","Data":"ff777f50caea24f2b6b3fde0f848bcdb84ec9b757466dc02f65742ac7226a616"}
Sep 16 19:08:39.358343 ip-10-0-132-80 kubenswrapper[2544]: I0916 19:08:39.358309    2544 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-647756665c-q8rlw" event={"ID":"fb1507b9-4934-42df-9691-0f4de73beb3a","Type":"ContainerStarted","Data":"62f5ceb79e5922a9c49132c0867962404c5eb8633f32feff550b09e1b89c44ce"}
Sep 16 19:08:39.358760 ip-10-0-132-80 kubenswrapper[2544]: I0916 19:08:39.358380    2544 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-647756665c-q8rlw" event={"ID":"fb1507b9-4934-42df-9691-0f4de73beb3a","Type":"ContainerStarted","Data":"f4c044a7c11f4ffa931b6bcb791e20c9f6a8c62776937a02d8b43c58c2dce6cc"}
Sep 16 19:08:39.358760 ip-10-0-132-80 kubenswrapper[2544]: I0916 19:08:39.358404    2544 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-647756665c-q8rlw" event={"ID":"fb1507b9-4934-42df-9691-0f4de73beb3a","Type":"ContainerStarted","Data":"57bdd26fbf1c706f9591d87e69e7f25dbba4e86bb7239fd3d82a497fce419bfb"}
Sep 16 19:08:39.358760 ip-10-0-132-80 kubenswrapper[2544]: I0916 19:08:39.358711    2544 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-647756665c-q8rlw"
Sep 16 19:08:39.358897 ip-10-0-132-80 kubenswrapper[2544]: I0916 19:08:39.358777    2544 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-647756665c-q8rlw"
Sep 16 19:08:39.358897 ip-10-0-132-80 kubenswrapper[2544]: I0916 19:08:39.358817    2544 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-647756665c-q8rlw"
Sep 16 19:08:39.360094 ip-10-0-132-80 kubenswrapper[2544]: I0916 19:08:39.360071    2544 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-647756665c-q8rlw" podUID="fb1507b9-4934-42df-9691-0f4de73beb3a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.38:8080: connect: connection refused"
Sep 16 19:08:39.360743 ip-10-0-132-80 kubenswrapper[2544]: I0916 19:08:39.360721    2544 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-647756665c-q8rlw" podUID="fb1507b9-4934-42df-9691-0f4de73beb3a" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 16 19:08:39.382489 ip-10-0-132-80 kubenswrapper[2544]: I0916 19:08:39.382448    2544 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-647756665c-q8rlw" podStartSLOduration=2.777528957 podStartE2EDuration="6.382429884s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.604900927s" isStatefulPod=false podCreationTimestamp="2026-09-16 19:08:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:08:39.380402633 +0000 UTC m=+1482.169405167" watchObservedRunningTime="2026-09-16 19:08:39.382429884 +0000 UTC m=+1482.171432418"
Sep 16 19:08:40.362938 ip-10-0-132-80 kubenswrapper[2544]: I0916 19:08:40.362891    2544 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-647756665c-q8rlw" podUID="fb1507b9-4934-42df-9691-0f4de73beb3a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.38:8080: connect: connection refused"
Sep 16 19:08:40.363478 ip-10-0-132-80 kubenswrapper[2544]: I0916 19:08:40.363337    2544 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-647756665c-q8rlw" podUID="fb1507b9-4934-42df-9691-0f4de73beb3a" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 16 19:08:45.367659 ip-10-0-132-80 kubenswrapper[2544]: I0916 19:08:45.367625    2544 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-647756665c-q8rlw"
Sep 16 19:08:45.368303 ip-10-0-132-80 kubenswrapper[2544]: I0916 19:08:45.368264    2544 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-647756665c-q8rlw" podUID="fb1507b9-4934-42df-9691-0f4de73beb3a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.38:8080: connect: connection refused"
Sep 16 19:08:45.368609 ip-10-0-132-80 kubenswrapper[2544]: I0916 19:08:45.368574    2544 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-647756665c-q8rlw" podUID="fb1507b9-4934-42df-9691-0f4de73beb3a" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 16 19:08:55.368676 ip-10-0-132-80 kubenswrapper[2544]: I0916 19:08:55.368628    2544 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-647756665c-q8rlw" podUID="fb1507b9-4934-42df-9691-0f4de73beb3a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.38:8080: connect: connection refused"
Sep 16 19:08:55.369143 ip-10-0-132-80 kubenswrapper[2544]: I0916 19:08:55.369122    2544 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-647756665c-q8rlw" podUID="fb1507b9-4934-42df-9691-0f4de73beb3a" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 16 19:08:57.774871 ip-10-0-132-80 kubenswrapper[2544]: I0916 19:08:57.774843    2544 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-pfcb7_9e3f9e34-420e-43ef-96b4-c7cad6b2eaa9/console-operator/2.log"
Sep 16 19:08:57.780679 ip-10-0-132-80 kubenswrapper[2544]: I0916 19:08:57.780659    2544 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-pfcb7_9e3f9e34-420e-43ef-96b4-c7cad6b2eaa9/console-operator/2.log"
Sep 16 19:09:05.368789 ip-10-0-132-80 kubenswrapper[2544]: I0916 19:09:05.368743    2544 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-647756665c-q8rlw" podUID="fb1507b9-4934-42df-9691-0f4de73beb3a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.38:8080: connect: connection refused"
Sep 16 19:09:05.369212 ip-10-0-132-80 kubenswrapper[2544]: I0916 19:09:05.369164    2544 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-647756665c-q8rlw" podUID="fb1507b9-4934-42df-9691-0f4de73beb3a" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 16 19:09:15.368531 ip-10-0-132-80 kubenswrapper[2544]: I0916 19:09:15.368474    2544 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-647756665c-q8rlw" podUID="fb1507b9-4934-42df-9691-0f4de73beb3a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.38:8080: connect: connection refused"
Sep 16 19:09:15.368991 ip-10-0-132-80 kubenswrapper[2544]: I0916 19:09:15.368951    2544 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-647756665c-q8rlw" podUID="fb1507b9-4934-42df-9691-0f4de73beb3a" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 16 19:09:25.368706 ip-10-0-132-80 kubenswrapper[2544]: I0916 19:09:25.368641    2544 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-647756665c-q8rlw" podUID="fb1507b9-4934-42df-9691-0f4de73beb3a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.38:8080: connect: connection refused"
Sep 16 19:09:25.369163 ip-10-0-132-80 kubenswrapper[2544]: I0916 19:09:25.369019    2544 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-647756665c-q8rlw" podUID="fb1507b9-4934-42df-9691-0f4de73beb3a" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 16 19:09:35.369017 ip-10-0-132-80 kubenswrapper[2544]: I0916 19:09:35.368967    2544 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-647756665c-q8rlw" podUID="fb1507b9-4934-42df-9691-0f4de73beb3a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.38:8080: connect: connection refused"
Sep 16 19:09:35.369506 ip-10-0-132-80 kubenswrapper[2544]: I0916 19:09:35.369380    2544 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-647756665c-q8rlw" podUID="fb1507b9-4934-42df-9691-0f4de73beb3a" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 16 19:09:45.368795 ip-10-0-132-80 kubenswrapper[2544]: I0916 19:09:45.368742    2544 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-647756665c-q8rlw" podUID="fb1507b9-4934-42df-9691-0f4de73beb3a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.38:8080: connect: connection refused"
Sep 16 19:09:45.369281 ip-10-0-132-80 kubenswrapper[2544]: I0916 19:09:45.369205    2544 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-647756665c-q8rlw" podUID="fb1507b9-4934-42df-9691-0f4de73beb3a" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 16 19:09:55.368803 ip-10-0-132-80 kubenswrapper[2544]: I0916 19:09:55.368771    2544 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-647756665c-q8rlw"
Sep 16 19:09:55.371082 ip-10-0-132-80 kubenswrapper[2544]: I0916 19:09:55.369009    2544 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-647756665c-q8rlw"
Sep 16 19:10:08.418516 ip-10-0-132-80 kubenswrapper[2544]: I0916 19:10:08.418476    2544 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-647756665c-q8rlw"]
Sep 16 19:10:08.419048 ip-10-0-132-80 kubenswrapper[2544]: I0916 19:10:08.418994    2544 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-647756665c-q8rlw" podUID="fb1507b9-4934-42df-9691-0f4de73beb3a" containerName="kserve-container" containerID="cri-o://57bdd26fbf1c706f9591d87e69e7f25dbba4e86bb7239fd3d82a497fce419bfb" gracePeriod=30
Sep 16 19:10:08.419306 ip-10-0-132-80 kubenswrapper[2544]: I0916 19:10:08.419267    2544 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-647756665c-q8rlw" podUID="fb1507b9-4934-42df-9691-0f4de73beb3a" containerName="agent" containerID="cri-o://62f5ceb79e5922a9c49132c0867962404c5eb8633f32feff550b09e1b89c44ce" gracePeriod=30
Sep 16 19:10:08.419400 ip-10-0-132-80 kubenswrapper[2544]: I0916 19:10:08.419286    2544 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-647756665c-q8rlw" podUID="fb1507b9-4934-42df-9691-0f4de73beb3a" containerName="kube-rbac-proxy" containerID="cri-o://f4c044a7c11f4ffa931b6bcb791e20c9f6a8c62776937a02d8b43c58c2dce6cc" gracePeriod=30
Sep 16 19:10:08.726746 ip-10-0-132-80 kubenswrapper[2544]: I0916 19:10:08.726656    2544 generic.go:356] "Generic (PLEG): container finished" podID="fb1507b9-4934-42df-9691-0f4de73beb3a" containerID="f4c044a7c11f4ffa931b6bcb791e20c9f6a8c62776937a02d8b43c58c2dce6cc" exitCode=2
Sep 16 19:10:08.726896 ip-10-0-132-80 kubenswrapper[2544]: I0916 19:10:08.726731    2544 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-647756665c-q8rlw" event={"ID":"fb1507b9-4934-42df-9691-0f4de73beb3a","Type":"ContainerDied","Data":"f4c044a7c11f4ffa931b6bcb791e20c9f6a8c62776937a02d8b43c58c2dce6cc"}
Sep 16 19:10:09.439039 ip-10-0-132-80 kubenswrapper[2544]: I0916 19:10:09.439001    2544 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-rzx9t/must-gather-p56tg"]
Sep 16 19:10:09.442670 ip-10-0-132-80 kubenswrapper[2544]: I0916 19:10:09.442644    2544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-rzx9t/must-gather-p56tg"
Sep 16 19:10:09.444948 ip-10-0-132-80 kubenswrapper[2544]: I0916 19:10:09.444928    2544 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-rzx9t\"/\"kube-root-ca.crt\""
Sep 16 19:10:09.445063 ip-10-0-132-80 kubenswrapper[2544]: I0916 19:10:09.444960    2544 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-rzx9t\"/\"openshift-service-ca.crt\""
Sep 16 19:10:09.445118 ip-10-0-132-80 kubenswrapper[2544]: I0916 19:10:09.445097    2544 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-rzx9t\"/\"default-dockercfg-szrmf\""
Sep 16 19:10:09.453065 ip-10-0-132-80 kubenswrapper[2544]: I0916 19:10:09.453044    2544 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-rzx9t/must-gather-p56tg"]
Sep 16 19:10:09.515385 ip-10-0-132-80 kubenswrapper[2544]: I0916 19:10:09.515354    2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/d3df97df-afe0-4e9a-95b1-a3957218e67f-must-gather-output\") pod \"must-gather-p56tg\" (UID: \"d3df97df-afe0-4e9a-95b1-a3957218e67f\") " pod="openshift-must-gather-rzx9t/must-gather-p56tg"
Sep 16 19:10:09.515593 ip-10-0-132-80 kubenswrapper[2544]: I0916 19:10:09.515415    2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9smpx\" (UniqueName: \"kubernetes.io/projected/d3df97df-afe0-4e9a-95b1-a3957218e67f-kube-api-access-9smpx\") pod \"must-gather-p56tg\" (UID: \"d3df97df-afe0-4e9a-95b1-a3957218e67f\") " pod="openshift-must-gather-rzx9t/must-gather-p56tg"
Sep 16 19:10:09.616036 ip-10-0-132-80 kubenswrapper[2544]: I0916 19:10:09.615986    2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/d3df97df-afe0-4e9a-95b1-a3957218e67f-must-gather-output\") pod \"must-gather-p56tg\" (UID: \"d3df97df-afe0-4e9a-95b1-a3957218e67f\") " pod="openshift-must-gather-rzx9t/must-gather-p56tg"
Sep 16 19:10:09.616223 ip-10-0-132-80 kubenswrapper[2544]: I0916 19:10:09.616079    2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9smpx\" (UniqueName: \"kubernetes.io/projected/d3df97df-afe0-4e9a-95b1-a3957218e67f-kube-api-access-9smpx\") pod \"must-gather-p56tg\" (UID: \"d3df97df-afe0-4e9a-95b1-a3957218e67f\") " pod="openshift-must-gather-rzx9t/must-gather-p56tg"
Sep 16 19:10:09.616343 ip-10-0-132-80 kubenswrapper[2544]: I0916 19:10:09.616321    2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/d3df97df-afe0-4e9a-95b1-a3957218e67f-must-gather-output\") pod \"must-gather-p56tg\" (UID: \"d3df97df-afe0-4e9a-95b1-a3957218e67f\") " pod="openshift-must-gather-rzx9t/must-gather-p56tg"
Sep 16 19:10:09.627882 ip-10-0-132-80 kubenswrapper[2544]: I0916 19:10:09.627855    2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9smpx\" (UniqueName: \"kubernetes.io/projected/d3df97df-afe0-4e9a-95b1-a3957218e67f-kube-api-access-9smpx\") pod \"must-gather-p56tg\" (UID: \"d3df97df-afe0-4e9a-95b1-a3957218e67f\") " pod="openshift-must-gather-rzx9t/must-gather-p56tg"
Sep 16 19:10:09.751871 ip-10-0-132-80 kubenswrapper[2544]: I0916 19:10:09.751782    2544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-rzx9t/must-gather-p56tg"
Sep 16 19:10:10.022438 ip-10-0-132-80 kubenswrapper[2544]: I0916 19:10:10.022367    2544 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-rzx9t/must-gather-p56tg"]
Sep 16 19:10:10.025299 ip-10-0-132-80 kubenswrapper[2544]: W0916 19:10:10.025258    2544 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podd3df97df_afe0_4e9a_95b1_a3957218e67f.slice/crio-6057fe3a72a7a43694247d6f4ed6352ca2a2eba22b41d9bc9b70dad6ba08f4be WatchSource:0}: Error finding container 6057fe3a72a7a43694247d6f4ed6352ca2a2eba22b41d9bc9b70dad6ba08f4be: Status 404 returned error can't find the container with id 6057fe3a72a7a43694247d6f4ed6352ca2a2eba22b41d9bc9b70dad6ba08f4be
Sep 16 19:10:10.027211 ip-10-0-132-80 kubenswrapper[2544]: I0916 19:10:10.027189    2544 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 16 19:10:10.363406 ip-10-0-132-80 kubenswrapper[2544]: I0916 19:10:10.363367    2544 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-647756665c-q8rlw" podUID="fb1507b9-4934-42df-9691-0f4de73beb3a" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.38:8643/healthz\": dial tcp 10.132.0.38:8643: connect: connection refused"
Sep 16 19:10:10.750659 ip-10-0-132-80 kubenswrapper[2544]: I0916 19:10:10.750568    2544 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-rzx9t/must-gather-p56tg" event={"ID":"d3df97df-afe0-4e9a-95b1-a3957218e67f","Type":"ContainerStarted","Data":"6057fe3a72a7a43694247d6f4ed6352ca2a2eba22b41d9bc9b70dad6ba08f4be"}
Sep 16 19:10:13.768867 ip-10-0-132-80 kubenswrapper[2544]: I0916 19:10:13.768837    2544 generic.go:356] "Generic (PLEG): container finished" podID="fb1507b9-4934-42df-9691-0f4de73beb3a" containerID="57bdd26fbf1c706f9591d87e69e7f25dbba4e86bb7239fd3d82a497fce419bfb" exitCode=0
Sep 16 19:10:13.769318 ip-10-0-132-80 kubenswrapper[2544]: I0916 19:10:13.768880    2544 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-647756665c-q8rlw" event={"ID":"fb1507b9-4934-42df-9691-0f4de73beb3a","Type":"ContainerDied","Data":"57bdd26fbf1c706f9591d87e69e7f25dbba4e86bb7239fd3d82a497fce419bfb"}
Sep 16 19:10:15.363487 ip-10-0-132-80 kubenswrapper[2544]: I0916 19:10:15.363447    2544 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-647756665c-q8rlw" podUID="fb1507b9-4934-42df-9691-0f4de73beb3a" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.38:8643/healthz\": dial tcp 10.132.0.38:8643: connect: connection refused"
Sep 16 19:10:15.368850 ip-10-0-132-80 kubenswrapper[2544]: I0916 19:10:15.368816    2544 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-647756665c-q8rlw" podUID="fb1507b9-4934-42df-9691-0f4de73beb3a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.38:8080: connect: connection refused"
Sep 16 19:10:15.369229 ip-10-0-132-80 kubenswrapper[2544]: I0916 19:10:15.369194    2544 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-647756665c-q8rlw" podUID="fb1507b9-4934-42df-9691-0f4de73beb3a" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 16 19:10:16.787494 ip-10-0-132-80 kubenswrapper[2544]: I0916 19:10:16.787351    2544 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-rzx9t/must-gather-p56tg" event={"ID":"d3df97df-afe0-4e9a-95b1-a3957218e67f","Type":"ContainerStarted","Data":"2d82616f5953445931c629b4dd3e23897a12fdabc07dfcf6624accb5045a4389"}
Sep 16 19:10:16.787494 ip-10-0-132-80 kubenswrapper[2544]: I0916 19:10:16.787405    2544 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-rzx9t/must-gather-p56tg" event={"ID":"d3df97df-afe0-4e9a-95b1-a3957218e67f","Type":"ContainerStarted","Data":"0a7419adbbc1f3a8b19816a64e82233960586064345b57ebe3d04ca9eb70bec8"}
Sep 16 19:10:16.805872 ip-10-0-132-80 kubenswrapper[2544]: I0916 19:10:16.805610    2544 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-rzx9t/must-gather-p56tg" podStartSLOduration=1.71417413 podStartE2EDuration="7.805589658s" totalImagesPullingTime="6.091415528s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 19:10:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:10:16.804697523 +0000 UTC m=+1579.593700141" watchObservedRunningTime="2026-09-16 19:10:16.805589658 +0000 UTC m=+1579.594592193"
Sep 16 19:10:20.363641 ip-10-0-132-80 kubenswrapper[2544]: I0916 19:10:20.363590    2544 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-647756665c-q8rlw" podUID="fb1507b9-4934-42df-9691-0f4de73beb3a" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.38:8643/healthz\": dial tcp 10.132.0.38:8643: connect: connection refused"
Sep 16 19:10:20.364087 ip-10-0-132-80 kubenswrapper[2544]: I0916 19:10:20.363747    2544 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-647756665c-q8rlw"
Sep 16 19:10:24.995702 ip-10-0-132-80 kubenswrapper[2544]: I0916 19:10:24.995666    2544 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-647756665c-q8rlw_fb1507b9-4934-42df-9691-0f4de73beb3a/kserve-container/0.log"
Sep 16 19:10:25.027189 ip-10-0-132-80 kubenswrapper[2544]: I0916 19:10:25.027159    2544 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-647756665c-q8rlw_fb1507b9-4934-42df-9691-0f4de73beb3a/kube-rbac-proxy/0.log"
Sep 16 19:10:25.044652 ip-10-0-132-80 kubenswrapper[2544]: I0916 19:10:25.044620    2544 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-647756665c-q8rlw_fb1507b9-4934-42df-9691-0f4de73beb3a/agent/0.log"
Sep 16 19:10:25.057031 ip-10-0-132-80 kubenswrapper[2544]: I0916 19:10:25.057014    2544 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-647756665c-q8rlw_fb1507b9-4934-42df-9691-0f4de73beb3a/storage-initializer/0.log"
Sep 16 19:10:25.364047 ip-10-0-132-80 kubenswrapper[2544]: I0916 19:10:25.364001    2544 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-647756665c-q8rlw" podUID="fb1507b9-4934-42df-9691-0f4de73beb3a" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.38:8643/healthz\": dial tcp 10.132.0.38:8643: connect: connection refused"
Sep 16 19:10:25.368542 ip-10-0-132-80 kubenswrapper[2544]: I0916 19:10:25.368452    2544 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-647756665c-q8rlw" podUID="fb1507b9-4934-42df-9691-0f4de73beb3a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.38:8080: connect: connection refused"
Sep 16 19:10:25.369107 ip-10-0-132-80 kubenswrapper[2544]: I0916 19:10:25.369076    2544 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-647756665c-q8rlw" podUID="fb1507b9-4934-42df-9691-0f4de73beb3a" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 16 19:10:26.026014 ip-10-0-132-80 kubenswrapper[2544]: I0916 19:10:26.025988    2544 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-647756665c-q8rlw_fb1507b9-4934-42df-9691-0f4de73beb3a/kserve-container/0.log"
Sep 16 19:10:26.039438 ip-10-0-132-80 kubenswrapper[2544]: I0916 19:10:26.039399    2544 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-647756665c-q8rlw_fb1507b9-4934-42df-9691-0f4de73beb3a/kube-rbac-proxy/0.log"
Sep 16 19:10:26.050770 ip-10-0-132-80 kubenswrapper[2544]: I0916 19:10:26.050726    2544 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-647756665c-q8rlw_fb1507b9-4934-42df-9691-0f4de73beb3a/agent/0.log"
Sep 16 19:10:26.063512 ip-10-0-132-80 kubenswrapper[2544]: I0916 19:10:26.063457    2544 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-647756665c-q8rlw_fb1507b9-4934-42df-9691-0f4de73beb3a/storage-initializer/0.log"
Sep 16 19:10:26.910706 ip-10-0-132-80 kubenswrapper[2544]: I0916 19:10:26.910674    2544 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-647756665c-q8rlw_fb1507b9-4934-42df-9691-0f4de73beb3a/kserve-container/0.log"
Sep 16 19:10:26.929235 ip-10-0-132-80 kubenswrapper[2544]: I0916 19:10:26.929207    2544 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-647756665c-q8rlw_fb1507b9-4934-42df-9691-0f4de73beb3a/kube-rbac-proxy/0.log"
Sep 16 19:10:26.942504 ip-10-0-132-80 kubenswrapper[2544]: I0916 19:10:26.942476    2544 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-647756665c-q8rlw_fb1507b9-4934-42df-9691-0f4de73beb3a/agent/0.log"
Sep 16 19:10:26.955034 ip-10-0-132-80 kubenswrapper[2544]: I0916 19:10:26.955007    2544 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-647756665c-q8rlw_fb1507b9-4934-42df-9691-0f4de73beb3a/storage-initializer/0.log"
Sep 16 19:10:27.802036 ip-10-0-132-80 kubenswrapper[2544]: I0916 19:10:27.802005    2544 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-647756665c-q8rlw_fb1507b9-4934-42df-9691-0f4de73beb3a/kserve-container/0.log"
Sep 16 19:10:27.815217 ip-10-0-132-80 kubenswrapper[2544]: I0916 19:10:27.815193    2544 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-647756665c-q8rlw_fb1507b9-4934-42df-9691-0f4de73beb3a/kube-rbac-proxy/0.log"
Sep 16 19:10:27.827410 ip-10-0-132-80 kubenswrapper[2544]: I0916 19:10:27.827385    2544 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-647756665c-q8rlw_fb1507b9-4934-42df-9691-0f4de73beb3a/agent/0.log"
Sep 16 19:10:27.838986 ip-10-0-132-80 kubenswrapper[2544]: I0916 19:10:27.838962    2544 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-647756665c-q8rlw_fb1507b9-4934-42df-9691-0f4de73beb3a/storage-initializer/0.log"
Sep 16 19:10:28.665470 ip-10-0-132-80 kubenswrapper[2544]: I0916 19:10:28.665437    2544 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-647756665c-q8rlw_fb1507b9-4934-42df-9691-0f4de73beb3a/kserve-container/0.log"
Sep 16 19:10:28.680369 ip-10-0-132-80 kubenswrapper[2544]: I0916 19:10:28.680321    2544 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-647756665c-q8rlw_fb1507b9-4934-42df-9691-0f4de73beb3a/kube-rbac-proxy/0.log"
Sep 16 19:10:28.691177 ip-10-0-132-80 kubenswrapper[2544]: I0916 19:10:28.691156    2544 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-647756665c-q8rlw_fb1507b9-4934-42df-9691-0f4de73beb3a/agent/0.log"
Sep 16 19:10:28.702493 ip-10-0-132-80 kubenswrapper[2544]: I0916 19:10:28.702468    2544 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-647756665c-q8rlw_fb1507b9-4934-42df-9691-0f4de73beb3a/storage-initializer/0.log"
Sep 16 19:10:29.547930 ip-10-0-132-80 kubenswrapper[2544]: I0916 19:10:29.547901    2544 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-647756665c-q8rlw_fb1507b9-4934-42df-9691-0f4de73beb3a/kserve-container/0.log"
Sep 16 19:10:29.561879 ip-10-0-132-80 kubenswrapper[2544]: I0916 19:10:29.561859    2544 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-647756665c-q8rlw_fb1507b9-4934-42df-9691-0f4de73beb3a/kube-rbac-proxy/0.log"
Sep 16 19:10:29.574279 ip-10-0-132-80 kubenswrapper[2544]: I0916 19:10:29.574259    2544 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-647756665c-q8rlw_fb1507b9-4934-42df-9691-0f4de73beb3a/agent/0.log"
Sep 16 19:10:29.586860 ip-10-0-132-80 kubenswrapper[2544]: I0916 19:10:29.586836    2544 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-647756665c-q8rlw_fb1507b9-4934-42df-9691-0f4de73beb3a/storage-initializer/0.log"
Sep 16 19:10:30.363737 ip-10-0-132-80 kubenswrapper[2544]: I0916 19:10:30.363697    2544 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-647756665c-q8rlw" podUID="fb1507b9-4934-42df-9691-0f4de73beb3a" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.38:8643/healthz\": dial tcp 10.132.0.38:8643: connect: connection refused"
Sep 16 19:10:30.428592 ip-10-0-132-80 kubenswrapper[2544]: I0916 19:10:30.428546    2544 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-647756665c-q8rlw_fb1507b9-4934-42df-9691-0f4de73beb3a/kserve-container/0.log"
Sep 16 19:10:30.442919 ip-10-0-132-80 kubenswrapper[2544]: I0916 19:10:30.442892    2544 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-647756665c-q8rlw_fb1507b9-4934-42df-9691-0f4de73beb3a/kube-rbac-proxy/0.log"
Sep 16 19:10:30.454419 ip-10-0-132-80 kubenswrapper[2544]: I0916 19:10:30.454397    2544 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-647756665c-q8rlw_fb1507b9-4934-42df-9691-0f4de73beb3a/agent/0.log"
Sep 16 19:10:30.472849 ip-10-0-132-80 kubenswrapper[2544]: I0916 19:10:30.472822    2544 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-647756665c-q8rlw_fb1507b9-4934-42df-9691-0f4de73beb3a/storage-initializer/0.log"
Sep 16 19:10:31.323739 ip-10-0-132-80 kubenswrapper[2544]: I0916 19:10:31.323709    2544 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-647756665c-q8rlw_fb1507b9-4934-42df-9691-0f4de73beb3a/kserve-container/0.log"
Sep 16 19:10:31.341754 ip-10-0-132-80 kubenswrapper[2544]: I0916 19:10:31.341731    2544 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-647756665c-q8rlw_fb1507b9-4934-42df-9691-0f4de73beb3a/kube-rbac-proxy/0.log"
Sep 16 19:10:31.356197 ip-10-0-132-80 kubenswrapper[2544]: I0916 19:10:31.356179    2544 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-647756665c-q8rlw_fb1507b9-4934-42df-9691-0f4de73beb3a/agent/0.log"
Sep 16 19:10:31.374345 ip-10-0-132-80 kubenswrapper[2544]: I0916 19:10:31.374323    2544 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-647756665c-q8rlw_fb1507b9-4934-42df-9691-0f4de73beb3a/storage-initializer/0.log"
Sep 16 19:10:32.240737 ip-10-0-132-80 kubenswrapper[2544]: I0916 19:10:32.240706    2544 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-647756665c-q8rlw_fb1507b9-4934-42df-9691-0f4de73beb3a/kserve-container/0.log"
Sep 16 19:10:32.258749 ip-10-0-132-80 kubenswrapper[2544]: I0916 19:10:32.258724    2544 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-647756665c-q8rlw_fb1507b9-4934-42df-9691-0f4de73beb3a/kube-rbac-proxy/0.log"
Sep 16 19:10:32.279268 ip-10-0-132-80 kubenswrapper[2544]: I0916 19:10:32.279239    2544 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-647756665c-q8rlw_fb1507b9-4934-42df-9691-0f4de73beb3a/agent/0.log"
Sep 16 19:10:32.302967 ip-10-0-132-80 kubenswrapper[2544]: I0916 19:10:32.302948    2544 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-647756665c-q8rlw_fb1507b9-4934-42df-9691-0f4de73beb3a/storage-initializer/0.log"
Sep 16 19:10:33.126703 ip-10-0-132-80 kubenswrapper[2544]: I0916 19:10:33.126677    2544 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-647756665c-q8rlw_fb1507b9-4934-42df-9691-0f4de73beb3a/kserve-container/0.log"
Sep 16 19:10:33.145939 ip-10-0-132-80 kubenswrapper[2544]: I0916 19:10:33.145913    2544 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-647756665c-q8rlw_fb1507b9-4934-42df-9691-0f4de73beb3a/kube-rbac-proxy/0.log"
Sep 16 19:10:33.164769 ip-10-0-132-80 kubenswrapper[2544]: I0916 19:10:33.164747    2544 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-647756665c-q8rlw_fb1507b9-4934-42df-9691-0f4de73beb3a/agent/0.log"
Sep 16 19:10:33.180377 ip-10-0-132-80 kubenswrapper[2544]: I0916 19:10:33.180352    2544 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-647756665c-q8rlw_fb1507b9-4934-42df-9691-0f4de73beb3a/storage-initializer/0.log"
Sep 16 19:10:34.024616 ip-10-0-132-80 kubenswrapper[2544]: I0916 19:10:34.024584    2544 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-647756665c-q8rlw_fb1507b9-4934-42df-9691-0f4de73beb3a/kserve-container/0.log"
Sep 16 19:10:34.041953 ip-10-0-132-80 kubenswrapper[2544]: I0916 19:10:34.041910    2544 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-647756665c-q8rlw_fb1507b9-4934-42df-9691-0f4de73beb3a/kube-rbac-proxy/0.log"
Sep 16 19:10:34.056326 ip-10-0-132-80 kubenswrapper[2544]: I0916 19:10:34.056302    2544 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-647756665c-q8rlw_fb1507b9-4934-42df-9691-0f4de73beb3a/agent/0.log"
Sep 16 19:10:34.070025 ip-10-0-132-80 kubenswrapper[2544]: I0916 19:10:34.070006    2544 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-647756665c-q8rlw_fb1507b9-4934-42df-9691-0f4de73beb3a/storage-initializer/0.log"
Sep 16 19:10:34.906141 ip-10-0-132-80 kubenswrapper[2544]: I0916 19:10:34.906114    2544 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-647756665c-q8rlw_fb1507b9-4934-42df-9691-0f4de73beb3a/kserve-container/0.log"
Sep 16 19:10:34.918353 ip-10-0-132-80 kubenswrapper[2544]: I0916 19:10:34.918332    2544 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-647756665c-q8rlw_fb1507b9-4934-42df-9691-0f4de73beb3a/kube-rbac-proxy/0.log"
Sep 16 19:10:34.928943 ip-10-0-132-80 kubenswrapper[2544]: I0916 19:10:34.928927    2544 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-647756665c-q8rlw_fb1507b9-4934-42df-9691-0f4de73beb3a/agent/0.log"
Sep 16 19:10:34.941013 ip-10-0-132-80 kubenswrapper[2544]: I0916 19:10:34.940992    2544 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-647756665c-q8rlw_fb1507b9-4934-42df-9691-0f4de73beb3a/storage-initializer/0.log"
Sep 16 19:10:35.363429 ip-10-0-132-80 kubenswrapper[2544]: I0916 19:10:35.363387    2544 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-647756665c-q8rlw" podUID="fb1507b9-4934-42df-9691-0f4de73beb3a" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.38:8643/healthz\": dial tcp 10.132.0.38:8643: connect: connection refused"
Sep 16 19:10:35.368867 ip-10-0-132-80 kubenswrapper[2544]: I0916 19:10:35.368834    2544 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-647756665c-q8rlw" podUID="fb1507b9-4934-42df-9691-0f4de73beb3a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.38:8080: connect: connection refused"
Sep 16 19:10:35.368985 ip-10-0-132-80 kubenswrapper[2544]: I0916 19:10:35.368938    2544 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-647756665c-q8rlw"
Sep 16 19:10:35.369400 ip-10-0-132-80 kubenswrapper[2544]: I0916 19:10:35.369377    2544 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-647756665c-q8rlw" podUID="fb1507b9-4934-42df-9691-0f4de73beb3a" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 16 19:10:35.369491 ip-10-0-132-80 kubenswrapper[2544]: I0916 19:10:35.369469    2544 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-647756665c-q8rlw"
Sep 16 19:10:35.783314 ip-10-0-132-80 kubenswrapper[2544]: I0916 19:10:35.783256    2544 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-647756665c-q8rlw_fb1507b9-4934-42df-9691-0f4de73beb3a/kserve-container/0.log"
Sep 16 19:10:35.795977 ip-10-0-132-80 kubenswrapper[2544]: I0916 19:10:35.795952    2544 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-647756665c-q8rlw_fb1507b9-4934-42df-9691-0f4de73beb3a/kube-rbac-proxy/0.log"
Sep 16 19:10:35.806830 ip-10-0-132-80 kubenswrapper[2544]: I0916 19:10:35.806805    2544 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-647756665c-q8rlw_fb1507b9-4934-42df-9691-0f4de73beb3a/agent/0.log"
Sep 16 19:10:35.818238 ip-10-0-132-80 kubenswrapper[2544]: I0916 19:10:35.818221    2544 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-647756665c-q8rlw_fb1507b9-4934-42df-9691-0f4de73beb3a/storage-initializer/0.log"
Sep 16 19:10:36.696910 ip-10-0-132-80 kubenswrapper[2544]: I0916 19:10:36.696879    2544 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-647756665c-q8rlw_fb1507b9-4934-42df-9691-0f4de73beb3a/kserve-container/0.log"
Sep 16 19:10:36.716954 ip-10-0-132-80 kubenswrapper[2544]: I0916 19:10:36.716923    2544 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-647756665c-q8rlw_fb1507b9-4934-42df-9691-0f4de73beb3a/kube-rbac-proxy/0.log"
Sep 16 19:10:36.735216 ip-10-0-132-80 kubenswrapper[2544]: I0916 19:10:36.735193    2544 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-647756665c-q8rlw_fb1507b9-4934-42df-9691-0f4de73beb3a/agent/0.log"
Sep 16 19:10:36.754294 ip-10-0-132-80 kubenswrapper[2544]: I0916 19:10:36.754274    2544 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-647756665c-q8rlw_fb1507b9-4934-42df-9691-0f4de73beb3a/storage-initializer/0.log"
Sep 16 19:10:37.687687 ip-10-0-132-80 kubenswrapper[2544]: I0916 19:10:37.687657    2544 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-647756665c-q8rlw_fb1507b9-4934-42df-9691-0f4de73beb3a/kserve-container/0.log"
Sep 16 19:10:37.704055 ip-10-0-132-80 kubenswrapper[2544]: I0916 19:10:37.704029    2544 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-647756665c-q8rlw_fb1507b9-4934-42df-9691-0f4de73beb3a/kube-rbac-proxy/0.log"
Sep 16 19:10:37.720920 ip-10-0-132-80 kubenswrapper[2544]: I0916 19:10:37.720892    2544 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-647756665c-q8rlw_fb1507b9-4934-42df-9691-0f4de73beb3a/agent/0.log"
Sep 16 19:10:37.734044 ip-10-0-132-80 kubenswrapper[2544]: I0916 19:10:37.734022    2544 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-647756665c-q8rlw_fb1507b9-4934-42df-9691-0f4de73beb3a/storage-initializer/0.log"
Sep 16 19:10:38.896234 ip-10-0-132-80 kubenswrapper[2544]: I0916 19:10:38.896196    2544 generic.go:356] "Generic (PLEG): container finished" podID="fb1507b9-4934-42df-9691-0f4de73beb3a" containerID="62f5ceb79e5922a9c49132c0867962404c5eb8633f32feff550b09e1b89c44ce" exitCode=0
Sep 16 19:10:38.896643 ip-10-0-132-80 kubenswrapper[2544]: I0916 19:10:38.896262    2544 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-647756665c-q8rlw" event={"ID":"fb1507b9-4934-42df-9691-0f4de73beb3a","Type":"ContainerDied","Data":"62f5ceb79e5922a9c49132c0867962404c5eb8633f32feff550b09e1b89c44ce"}
Sep 16 19:10:38.898011 ip-10-0-132-80 kubenswrapper[2544]: I0916 19:10:38.897990    2544 generic.go:356] "Generic (PLEG): container finished" podID="d3df97df-afe0-4e9a-95b1-a3957218e67f" containerID="0a7419adbbc1f3a8b19816a64e82233960586064345b57ebe3d04ca9eb70bec8" exitCode=0
Sep 16 19:10:38.898118 ip-10-0-132-80 kubenswrapper[2544]: I0916 19:10:38.898058    2544 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-rzx9t/must-gather-p56tg" event={"ID":"d3df97df-afe0-4e9a-95b1-a3957218e67f","Type":"ContainerDied","Data":"0a7419adbbc1f3a8b19816a64e82233960586064345b57ebe3d04ca9eb70bec8"}
Sep 16 19:10:38.898405 ip-10-0-132-80 kubenswrapper[2544]: I0916 19:10:38.898390    2544 scope.go:117] "RemoveContainer" containerID="0a7419adbbc1f3a8b19816a64e82233960586064345b57ebe3d04ca9eb70bec8"
Sep 16 19:10:39.071545 ip-10-0-132-80 kubenswrapper[2544]: I0916 19:10:39.071523    2544 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-647756665c-q8rlw"
Sep 16 19:10:39.187841 ip-10-0-132-80 kubenswrapper[2544]: I0916 19:10:39.187755    2544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/fb1507b9-4934-42df-9691-0f4de73beb3a-proxy-tls\") pod \"fb1507b9-4934-42df-9691-0f4de73beb3a\" (UID: \"fb1507b9-4934-42df-9691-0f4de73beb3a\") "
Sep 16 19:10:39.187841 ip-10-0-132-80 kubenswrapper[2544]: I0916 19:10:39.187799    2544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/fb1507b9-4934-42df-9691-0f4de73beb3a-cabundle-cert\") pod \"fb1507b9-4934-42df-9691-0f4de73beb3a\" (UID: \"fb1507b9-4934-42df-9691-0f4de73beb3a\") "
Sep 16 19:10:39.188073 ip-10-0-132-80 kubenswrapper[2544]: I0916 19:10:39.187908    2544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/fb1507b9-4934-42df-9691-0f4de73beb3a-kube-rbac-proxy-sar-config\") pod \"fb1507b9-4934-42df-9691-0f4de73beb3a\" (UID: \"fb1507b9-4934-42df-9691-0f4de73beb3a\") "
Sep 16 19:10:39.188073 ip-10-0-132-80 kubenswrapper[2544]: I0916 19:10:39.187947    2544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/fb1507b9-4934-42df-9691-0f4de73beb3a-kserve-provision-location\") pod \"fb1507b9-4934-42df-9691-0f4de73beb3a\" (UID: \"fb1507b9-4934-42df-9691-0f4de73beb3a\") "
Sep 16 19:10:39.188073 ip-10-0-132-80 kubenswrapper[2544]: I0916 19:10:39.187970    2544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zjrzh\" (UniqueName: \"kubernetes.io/projected/fb1507b9-4934-42df-9691-0f4de73beb3a-kube-api-access-zjrzh\") pod \"fb1507b9-4934-42df-9691-0f4de73beb3a\" (UID: \"fb1507b9-4934-42df-9691-0f4de73beb3a\") "
Sep 16 19:10:39.188268 ip-10-0-132-80 kubenswrapper[2544]: I0916 19:10:39.188242    2544 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fb1507b9-4934-42df-9691-0f4de73beb3a-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "fb1507b9-4934-42df-9691-0f4de73beb3a" (UID: "fb1507b9-4934-42df-9691-0f4de73beb3a"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:10:39.188328 ip-10-0-132-80 kubenswrapper[2544]: I0916 19:10:39.188244    2544 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fb1507b9-4934-42df-9691-0f4de73beb3a-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "fb1507b9-4934-42df-9691-0f4de73beb3a" (UID: "fb1507b9-4934-42df-9691-0f4de73beb3a"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 19:10:39.188373 ip-10-0-132-80 kubenswrapper[2544]: I0916 19:10:39.188330    2544 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fb1507b9-4934-42df-9691-0f4de73beb3a-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "fb1507b9-4934-42df-9691-0f4de73beb3a" (UID: "fb1507b9-4934-42df-9691-0f4de73beb3a"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 19:10:39.189935 ip-10-0-132-80 kubenswrapper[2544]: I0916 19:10:39.189914    2544 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fb1507b9-4934-42df-9691-0f4de73beb3a-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "fb1507b9-4934-42df-9691-0f4de73beb3a" (UID: "fb1507b9-4934-42df-9691-0f4de73beb3a"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 19:10:39.190017 ip-10-0-132-80 kubenswrapper[2544]: I0916 19:10:39.189946    2544 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fb1507b9-4934-42df-9691-0f4de73beb3a-kube-api-access-zjrzh" (OuterVolumeSpecName: "kube-api-access-zjrzh") pod "fb1507b9-4934-42df-9691-0f4de73beb3a" (UID: "fb1507b9-4934-42df-9691-0f4de73beb3a"). InnerVolumeSpecName "kube-api-access-zjrzh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 19:10:39.289436 ip-10-0-132-80 kubenswrapper[2544]: I0916 19:10:39.289394    2544 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/fb1507b9-4934-42df-9691-0f4de73beb3a-cabundle-cert\") on node \"ip-10-0-132-80.ec2.internal\" DevicePath \"\""
Sep 16 19:10:39.289436 ip-10-0-132-80 kubenswrapper[2544]: I0916 19:10:39.289426    2544 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/fb1507b9-4934-42df-9691-0f4de73beb3a-kube-rbac-proxy-sar-config\") on node \"ip-10-0-132-80.ec2.internal\" DevicePath \"\""
Sep 16 19:10:39.289436 ip-10-0-132-80 kubenswrapper[2544]: I0916 19:10:39.289439    2544 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/fb1507b9-4934-42df-9691-0f4de73beb3a-kserve-provision-location\") on node \"ip-10-0-132-80.ec2.internal\" DevicePath \"\""
Sep 16 19:10:39.289436 ip-10-0-132-80 kubenswrapper[2544]: I0916 19:10:39.289450    2544 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zjrzh\" (UniqueName: \"kubernetes.io/projected/fb1507b9-4934-42df-9691-0f4de73beb3a-kube-api-access-zjrzh\") on node \"ip-10-0-132-80.ec2.internal\" DevicePath \"\""
Sep 16 19:10:39.289749 ip-10-0-132-80 kubenswrapper[2544]: I0916 19:10:39.289462    2544 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/fb1507b9-4934-42df-9691-0f4de73beb3a-proxy-tls\") on node \"ip-10-0-132-80.ec2.internal\" DevicePath \"\""
Sep 16 19:10:39.505908 ip-10-0-132-80 kubenswrapper[2544]: I0916 19:10:39.505792    2544 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-must-gather-rzx9t_must-gather-p56tg_d3df97df-afe0-4e9a-95b1-a3957218e67f/gather/0.log"
Sep 16 19:10:39.904373 ip-10-0-132-80 kubenswrapper[2544]: I0916 19:10:39.904344    2544 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-647756665c-q8rlw"
Sep 16 19:10:39.904814 ip-10-0-132-80 kubenswrapper[2544]: I0916 19:10:39.904343    2544 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-647756665c-q8rlw" event={"ID":"fb1507b9-4934-42df-9691-0f4de73beb3a","Type":"ContainerDied","Data":"045c21889a3f2873f48dac2389f359700034a28ddb2b2b57ddb5189eee6d1f5d"}
Sep 16 19:10:39.904814 ip-10-0-132-80 kubenswrapper[2544]: I0916 19:10:39.904486    2544 scope.go:117] "RemoveContainer" containerID="62f5ceb79e5922a9c49132c0867962404c5eb8633f32feff550b09e1b89c44ce"
Sep 16 19:10:39.914134 ip-10-0-132-80 kubenswrapper[2544]: I0916 19:10:39.914113    2544 scope.go:117] "RemoveContainer" containerID="f4c044a7c11f4ffa931b6bcb791e20c9f6a8c62776937a02d8b43c58c2dce6cc"
Sep 16 19:10:39.921999 ip-10-0-132-80 kubenswrapper[2544]: I0916 19:10:39.921981    2544 scope.go:117] "RemoveContainer" containerID="57bdd26fbf1c706f9591d87e69e7f25dbba4e86bb7239fd3d82a497fce419bfb"
Sep 16 19:10:39.927799 ip-10-0-132-80 kubenswrapper[2544]: I0916 19:10:39.927776    2544 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-647756665c-q8rlw"]
Sep 16 19:10:39.929794 ip-10-0-132-80 kubenswrapper[2544]: I0916 19:10:39.929777    2544 scope.go:117] "RemoveContainer" containerID="ff777f50caea24f2b6b3fde0f848bcdb84ec9b757466dc02f65742ac7226a616"
Sep 16 19:10:39.932388 ip-10-0-132-80 kubenswrapper[2544]: I0916 19:10:39.932365    2544 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-647756665c-q8rlw"]
Sep 16 19:10:40.185297 ip-10-0-132-80 kubenswrapper[2544]: I0916 19:10:40.185206    2544 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-txqpj/must-gather-jcbh5"]
Sep 16 19:10:40.185657 ip-10-0-132-80 kubenswrapper[2544]: I0916 19:10:40.185638    2544 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fb1507b9-4934-42df-9691-0f4de73beb3a" containerName="kserve-container"
Sep 16 19:10:40.185657 ip-10-0-132-80 kubenswrapper[2544]: I0916 19:10:40.185658    2544 state_mem.go:107] "Deleted CPUSet assignment" podUID="fb1507b9-4934-42df-9691-0f4de73beb3a" containerName="kserve-container"
Sep 16 19:10:40.185818 ip-10-0-132-80 kubenswrapper[2544]: I0916 19:10:40.185682    2544 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fb1507b9-4934-42df-9691-0f4de73beb3a" containerName="storage-initializer"
Sep 16 19:10:40.185818 ip-10-0-132-80 kubenswrapper[2544]: I0916 19:10:40.185689    2544 state_mem.go:107] "Deleted CPUSet assignment" podUID="fb1507b9-4934-42df-9691-0f4de73beb3a" containerName="storage-initializer"
Sep 16 19:10:40.185818 ip-10-0-132-80 kubenswrapper[2544]: I0916 19:10:40.185698    2544 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fb1507b9-4934-42df-9691-0f4de73beb3a" containerName="agent"
Sep 16 19:10:40.185818 ip-10-0-132-80 kubenswrapper[2544]: I0916 19:10:40.185706    2544 state_mem.go:107] "Deleted CPUSet assignment" podUID="fb1507b9-4934-42df-9691-0f4de73beb3a" containerName="agent"
Sep 16 19:10:40.185818 ip-10-0-132-80 kubenswrapper[2544]: I0916 19:10:40.185729    2544 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fb1507b9-4934-42df-9691-0f4de73beb3a" containerName="kube-rbac-proxy"
Sep 16 19:10:40.185818 ip-10-0-132-80 kubenswrapper[2544]: I0916 19:10:40.185740    2544 state_mem.go:107] "Deleted CPUSet assignment" podUID="fb1507b9-4934-42df-9691-0f4de73beb3a" containerName="kube-rbac-proxy"
Sep 16 19:10:40.186112 ip-10-0-132-80 kubenswrapper[2544]: I0916 19:10:40.185820    2544 memory_manager.go:368] "RemoveStaleState removing state" podUID="fb1507b9-4934-42df-9691-0f4de73beb3a" containerName="kube-rbac-proxy"
Sep 16 19:10:40.186112 ip-10-0-132-80 kubenswrapper[2544]: I0916 19:10:40.185833    2544 memory_manager.go:368] "RemoveStaleState removing state" podUID="fb1507b9-4934-42df-9691-0f4de73beb3a" containerName="kserve-container"
Sep 16 19:10:40.186112 ip-10-0-132-80 kubenswrapper[2544]: I0916 19:10:40.185843    2544 memory_manager.go:368] "RemoveStaleState removing state" podUID="fb1507b9-4934-42df-9691-0f4de73beb3a" containerName="agent"
Sep 16 19:10:40.190491 ip-10-0-132-80 kubenswrapper[2544]: I0916 19:10:40.190471    2544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-txqpj/must-gather-jcbh5"
Sep 16 19:10:40.194234 ip-10-0-132-80 kubenswrapper[2544]: I0916 19:10:40.194217    2544 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-txqpj\"/\"openshift-service-ca.crt\""
Sep 16 19:10:40.195009 ip-10-0-132-80 kubenswrapper[2544]: I0916 19:10:40.194991    2544 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-txqpj\"/\"kube-root-ca.crt\""
Sep 16 19:10:40.195340 ip-10-0-132-80 kubenswrapper[2544]: I0916 19:10:40.195323    2544 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-txqpj\"/\"default-dockercfg-hx8r2\""
Sep 16 19:10:40.199438 ip-10-0-132-80 kubenswrapper[2544]: I0916 19:10:40.199418    2544 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-txqpj/must-gather-jcbh5"]
Sep 16 19:10:40.298903 ip-10-0-132-80 kubenswrapper[2544]: I0916 19:10:40.298861    2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/30eb1676-0ca7-49e6-8425-a056d5c754b6-must-gather-output\") pod \"must-gather-jcbh5\" (UID: \"30eb1676-0ca7-49e6-8425-a056d5c754b6\") " pod="openshift-must-gather-txqpj/must-gather-jcbh5"
Sep 16 19:10:40.299089 ip-10-0-132-80 kubenswrapper[2544]: I0916 19:10:40.298916    2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8smbw\" (UniqueName: \"kubernetes.io/projected/30eb1676-0ca7-49e6-8425-a056d5c754b6-kube-api-access-8smbw\") pod \"must-gather-jcbh5\" (UID: \"30eb1676-0ca7-49e6-8425-a056d5c754b6\") " pod="openshift-must-gather-txqpj/must-gather-jcbh5"
Sep 16 19:10:40.400132 ip-10-0-132-80 kubenswrapper[2544]: I0916 19:10:40.400094    2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/30eb1676-0ca7-49e6-8425-a056d5c754b6-must-gather-output\") pod \"must-gather-jcbh5\" (UID: \"30eb1676-0ca7-49e6-8425-a056d5c754b6\") " pod="openshift-must-gather-txqpj/must-gather-jcbh5"
Sep 16 19:10:40.400307 ip-10-0-132-80 kubenswrapper[2544]: I0916 19:10:40.400157    2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8smbw\" (UniqueName: \"kubernetes.io/projected/30eb1676-0ca7-49e6-8425-a056d5c754b6-kube-api-access-8smbw\") pod \"must-gather-jcbh5\" (UID: \"30eb1676-0ca7-49e6-8425-a056d5c754b6\") " pod="openshift-must-gather-txqpj/must-gather-jcbh5"
Sep 16 19:10:40.400493 ip-10-0-132-80 kubenswrapper[2544]: I0916 19:10:40.400471    2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/30eb1676-0ca7-49e6-8425-a056d5c754b6-must-gather-output\") pod \"must-gather-jcbh5\" (UID: \"30eb1676-0ca7-49e6-8425-a056d5c754b6\") " pod="openshift-must-gather-txqpj/must-gather-jcbh5"
Sep 16 19:10:40.408823 ip-10-0-132-80 kubenswrapper[2544]: I0916 19:10:40.408792    2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8smbw\" (UniqueName: \"kubernetes.io/projected/30eb1676-0ca7-49e6-8425-a056d5c754b6-kube-api-access-8smbw\") pod \"must-gather-jcbh5\" (UID: \"30eb1676-0ca7-49e6-8425-a056d5c754b6\") " pod="openshift-must-gather-txqpj/must-gather-jcbh5"
Sep 16 19:10:40.499725 ip-10-0-132-80 kubenswrapper[2544]: I0916 19:10:40.499647    2544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-txqpj/must-gather-jcbh5"
Sep 16 19:10:40.770651 ip-10-0-132-80 kubenswrapper[2544]: I0916 19:10:40.770548    2544 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-txqpj/must-gather-jcbh5"]
Sep 16 19:10:40.773312 ip-10-0-132-80 kubenswrapper[2544]: W0916 19:10:40.773281    2544 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod30eb1676_0ca7_49e6_8425_a056d5c754b6.slice/crio-8bacf3236263f333b2fac9b9693c83a42559aebadd7fcc10227a36681745cbc7 WatchSource:0}: Error finding container 8bacf3236263f333b2fac9b9693c83a42559aebadd7fcc10227a36681745cbc7: Status 404 returned error can't find the container with id 8bacf3236263f333b2fac9b9693c83a42559aebadd7fcc10227a36681745cbc7
Sep 16 19:10:40.910380 ip-10-0-132-80 kubenswrapper[2544]: I0916 19:10:40.910355    2544 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-txqpj/must-gather-jcbh5" event={"ID":"30eb1676-0ca7-49e6-8425-a056d5c754b6","Type":"ContainerStarted","Data":"8bacf3236263f333b2fac9b9693c83a42559aebadd7fcc10227a36681745cbc7"}
Sep 16 19:10:41.722266 ip-10-0-132-80 kubenswrapper[2544]: I0916 19:10:41.722234    2544 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="fb1507b9-4934-42df-9691-0f4de73beb3a" path="/var/lib/kubelet/pods/fb1507b9-4934-42df-9691-0f4de73beb3a/volumes"
Sep 16 19:10:41.920226 ip-10-0-132-80 kubenswrapper[2544]: I0916 19:10:41.920183    2544 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-txqpj/must-gather-jcbh5" event={"ID":"30eb1676-0ca7-49e6-8425-a056d5c754b6","Type":"ContainerStarted","Data":"9cffe83c8db42639e15b4ed6e577cb74bb5c784793da916399cbedb8229534cf"}
Sep 16 19:10:41.920802 ip-10-0-132-80 kubenswrapper[2544]: I0916 19:10:41.920773    2544 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-txqpj/must-gather-jcbh5" event={"ID":"30eb1676-0ca7-49e6-8425-a056d5c754b6","Type":"ContainerStarted","Data":"63890a50978ff2656db5040a3ce79bf9cc42a07f9ca1fa52426eb63a8a0df435"}
Sep 16 19:10:41.938311 ip-10-0-132-80 kubenswrapper[2544]: I0916 19:10:41.938261    2544 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-txqpj/must-gather-jcbh5" podStartSLOduration=1.156526071 podStartE2EDuration="1.938246726s" totalImagesPullingTime="781.720655ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 19:10:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:10:41.937153249 +0000 UTC m=+1604.726155786" watchObservedRunningTime="2026-09-16 19:10:41.938246726 +0000 UTC m=+1604.727249260"
Sep 16 19:10:43.127736 ip-10-0-132-80 kubenswrapper[2544]: I0916 19:10:43.127703    2544 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-pg4kf_2bbf45e9-df8e-4a67-b777-0c629428fca2/global-pull-secret-syncer/0.log"
Sep 16 19:10:43.202848 ip-10-0-132-80 kubenswrapper[2544]: I0916 19:10:43.202818    2544 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-pzpb9_0494be2a-d0e5-4570-b05c-0c595a0b494a/konnectivity-agent/0.log"
Sep 16 19:10:43.325272 ip-10-0-132-80 kubenswrapper[2544]: I0916 19:10:43.325238    2544 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-132-80.ec2.internal_3b41cfe1537c43ad1e53b07ff984e93b/haproxy/0.log"
Sep 16 19:10:45.022571 ip-10-0-132-80 kubenswrapper[2544]: I0916 19:10:45.022005    2544 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-must-gather-rzx9t/must-gather-p56tg"]
Sep 16 19:10:45.022571 ip-10-0-132-80 kubenswrapper[2544]: I0916 19:10:45.022320    2544 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-must-gather-rzx9t/must-gather-p56tg" podUID="d3df97df-afe0-4e9a-95b1-a3957218e67f" containerName="copy" containerID="cri-o://2d82616f5953445931c629b4dd3e23897a12fdabc07dfcf6624accb5045a4389" gracePeriod=2
Sep 16 19:10:45.037632 ip-10-0-132-80 kubenswrapper[2544]: E0916 19:10:45.025037    2544 status_manager.go:1018] "Failed to get status for pod" err="pods \"must-gather-p56tg\" is forbidden: User \"system:node:ip-10-0-132-80.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"openshift-must-gather-rzx9t\": no relationship found between node 'ip-10-0-132-80.ec2.internal' and this object" podUID="d3df97df-afe0-4e9a-95b1-a3957218e67f" pod="openshift-must-gather-rzx9t/must-gather-p56tg"
Sep 16 19:10:45.037632 ip-10-0-132-80 kubenswrapper[2544]: I0916 19:10:45.025606    2544 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-must-gather-rzx9t/must-gather-p56tg"]
Sep 16 19:10:45.428970 ip-10-0-132-80 kubenswrapper[2544]: I0916 19:10:45.428941    2544 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-must-gather-rzx9t_must-gather-p56tg_d3df97df-afe0-4e9a-95b1-a3957218e67f/copy/0.log"
Sep 16 19:10:45.440230 ip-10-0-132-80 kubenswrapper[2544]: I0916 19:10:45.436706    2544 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-rzx9t/must-gather-p56tg"
Sep 16 19:10:45.443574 ip-10-0-132-80 kubenswrapper[2544]: E0916 19:10:45.440788    2544 status_manager.go:1018] "Failed to get status for pod" err="pods \"must-gather-p56tg\" is forbidden: User \"system:node:ip-10-0-132-80.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"openshift-must-gather-rzx9t\": no relationship found between node 'ip-10-0-132-80.ec2.internal' and this object" podUID="d3df97df-afe0-4e9a-95b1-a3957218e67f" pod="openshift-must-gather-rzx9t/must-gather-p56tg"
Sep 16 19:10:45.546925 ip-10-0-132-80 kubenswrapper[2544]: I0916 19:10:45.546880    2544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/d3df97df-afe0-4e9a-95b1-a3957218e67f-must-gather-output\") pod \"d3df97df-afe0-4e9a-95b1-a3957218e67f\" (UID: \"d3df97df-afe0-4e9a-95b1-a3957218e67f\") "
Sep 16 19:10:45.547108 ip-10-0-132-80 kubenswrapper[2544]: I0916 19:10:45.546990    2544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9smpx\" (UniqueName: \"kubernetes.io/projected/d3df97df-afe0-4e9a-95b1-a3957218e67f-kube-api-access-9smpx\") pod \"d3df97df-afe0-4e9a-95b1-a3957218e67f\" (UID: \"d3df97df-afe0-4e9a-95b1-a3957218e67f\") "
Sep 16 19:10:45.549338 ip-10-0-132-80 kubenswrapper[2544]: I0916 19:10:45.549300    2544 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d3df97df-afe0-4e9a-95b1-a3957218e67f-must-gather-output" (OuterVolumeSpecName: "must-gather-output") pod "d3df97df-afe0-4e9a-95b1-a3957218e67f" (UID: "d3df97df-afe0-4e9a-95b1-a3957218e67f"). InnerVolumeSpecName "must-gather-output". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:10:45.552648 ip-10-0-132-80 kubenswrapper[2544]: I0916 19:10:45.552618    2544 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d3df97df-afe0-4e9a-95b1-a3957218e67f-kube-api-access-9smpx" (OuterVolumeSpecName: "kube-api-access-9smpx") pod "d3df97df-afe0-4e9a-95b1-a3957218e67f" (UID: "d3df97df-afe0-4e9a-95b1-a3957218e67f"). InnerVolumeSpecName "kube-api-access-9smpx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 19:10:45.648033 ip-10-0-132-80 kubenswrapper[2544]: I0916 19:10:45.647915    2544 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9smpx\" (UniqueName: \"kubernetes.io/projected/d3df97df-afe0-4e9a-95b1-a3957218e67f-kube-api-access-9smpx\") on node \"ip-10-0-132-80.ec2.internal\" DevicePath \"\""
Sep 16 19:10:45.648033 ip-10-0-132-80 kubenswrapper[2544]: I0916 19:10:45.647956    2544 reconciler_common.go:299] "Volume detached for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/d3df97df-afe0-4e9a-95b1-a3957218e67f-must-gather-output\") on node \"ip-10-0-132-80.ec2.internal\" DevicePath \"\""
Sep 16 19:10:45.727753 ip-10-0-132-80 kubenswrapper[2544]: I0916 19:10:45.726049    2544 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d3df97df-afe0-4e9a-95b1-a3957218e67f" path="/var/lib/kubelet/pods/d3df97df-afe0-4e9a-95b1-a3957218e67f/volumes"
Sep 16 19:10:45.943962 ip-10-0-132-80 kubenswrapper[2544]: I0916 19:10:45.943873    2544 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-must-gather-rzx9t_must-gather-p56tg_d3df97df-afe0-4e9a-95b1-a3957218e67f/copy/0.log"
Sep 16 19:10:45.944331 ip-10-0-132-80 kubenswrapper[2544]: I0916 19:10:45.944302    2544 generic.go:356] "Generic (PLEG): container finished" podID="d3df97df-afe0-4e9a-95b1-a3957218e67f" containerID="2d82616f5953445931c629b4dd3e23897a12fdabc07dfcf6624accb5045a4389" exitCode=143
Sep 16 19:10:45.944473 ip-10-0-132-80 kubenswrapper[2544]: I0916 19:10:45.944460    2544 scope.go:117] "RemoveContainer" containerID="2d82616f5953445931c629b4dd3e23897a12fdabc07dfcf6624accb5045a4389"
Sep 16 19:10:45.944651 ip-10-0-132-80 kubenswrapper[2544]: I0916 19:10:45.944635    2544 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-rzx9t/must-gather-p56tg"
Sep 16 19:10:45.979886 ip-10-0-132-80 kubenswrapper[2544]: I0916 19:10:45.979735    2544 scope.go:117] "RemoveContainer" containerID="0a7419adbbc1f3a8b19816a64e82233960586064345b57ebe3d04ca9eb70bec8"
Sep 16 19:10:46.000329 ip-10-0-132-80 kubenswrapper[2544]: I0916 19:10:46.000291    2544 scope.go:117] "RemoveContainer" containerID="2d82616f5953445931c629b4dd3e23897a12fdabc07dfcf6624accb5045a4389"
Sep 16 19:10:46.001008 ip-10-0-132-80 kubenswrapper[2544]: E0916 19:10:46.000909    2544 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2d82616f5953445931c629b4dd3e23897a12fdabc07dfcf6624accb5045a4389\": container with ID starting with 2d82616f5953445931c629b4dd3e23897a12fdabc07dfcf6624accb5045a4389 not found: ID does not exist" containerID="2d82616f5953445931c629b4dd3e23897a12fdabc07dfcf6624accb5045a4389"
Sep 16 19:10:46.001008 ip-10-0-132-80 kubenswrapper[2544]: I0916 19:10:46.000959    2544 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2d82616f5953445931c629b4dd3e23897a12fdabc07dfcf6624accb5045a4389"} err="failed to get container status \"2d82616f5953445931c629b4dd3e23897a12fdabc07dfcf6624accb5045a4389\": rpc error: code = NotFound desc = could not find container \"2d82616f5953445931c629b4dd3e23897a12fdabc07dfcf6624accb5045a4389\": container with ID starting with 2d82616f5953445931c629b4dd3e23897a12fdabc07dfcf6624accb5045a4389 not found: ID does not exist"
Sep 16 19:10:46.001008 ip-10-0-132-80 kubenswrapper[2544]: I0916 19:10:46.000985    2544 scope.go:117] "RemoveContainer" containerID="0a7419adbbc1f3a8b19816a64e82233960586064345b57ebe3d04ca9eb70bec8"
Sep 16 19:10:46.001657 ip-10-0-132-80 kubenswrapper[2544]: E0916 19:10:46.001595    2544 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0a7419adbbc1f3a8b19816a64e82233960586064345b57ebe3d04ca9eb70bec8\": container with ID starting with 0a7419adbbc1f3a8b19816a64e82233960586064345b57ebe3d04ca9eb70bec8 not found: ID does not exist" containerID="0a7419adbbc1f3a8b19816a64e82233960586064345b57ebe3d04ca9eb70bec8"
Sep 16 19:10:46.001657 ip-10-0-132-80 kubenswrapper[2544]: I0916 19:10:46.001622    2544 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0a7419adbbc1f3a8b19816a64e82233960586064345b57ebe3d04ca9eb70bec8"} err="failed to get container status \"0a7419adbbc1f3a8b19816a64e82233960586064345b57ebe3d04ca9eb70bec8\": rpc error: code = NotFound desc = could not find container \"0a7419adbbc1f3a8b19816a64e82233960586064345b57ebe3d04ca9eb70bec8\": container with ID starting with 0a7419adbbc1f3a8b19816a64e82233960586064345b57ebe3d04ca9eb70bec8 not found: ID does not exist"
Sep 16 19:10:47.335768 ip-10-0-132-80 kubenswrapper[2544]: I0916 19:10:47.335738    2544 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_c1055a71-f078-4e9d-a1d4-7320f07566c8/alertmanager/0.log"
Sep 16 19:10:47.572711 ip-10-0-132-80 kubenswrapper[2544]: I0916 19:10:47.572682    2544 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_c1055a71-f078-4e9d-a1d4-7320f07566c8/config-reloader/0.log"
Sep 16 19:10:47.705305 ip-10-0-132-80 kubenswrapper[2544]: I0916 19:10:47.705209    2544 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_c1055a71-f078-4e9d-a1d4-7320f07566c8/kube-rbac-proxy-web/0.log"
Sep 16 19:10:47.932684 ip-10-0-132-80 kubenswrapper[2544]: I0916 19:10:47.932663    2544 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_c1055a71-f078-4e9d-a1d4-7320f07566c8/kube-rbac-proxy/0.log"
Sep 16 19:10:48.010545 ip-10-0-132-80 kubenswrapper[2544]: I0916 19:10:48.010462    2544 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_c1055a71-f078-4e9d-a1d4-7320f07566c8/kube-rbac-proxy-metric/0.log"
Sep 16 19:10:48.081945 ip-10-0-132-80 kubenswrapper[2544]: I0916 19:10:48.081910    2544 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_c1055a71-f078-4e9d-a1d4-7320f07566c8/prom-label-proxy/0.log"
Sep 16 19:10:48.179256 ip-10-0-132-80 kubenswrapper[2544]: I0916 19:10:48.179218    2544 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_c1055a71-f078-4e9d-a1d4-7320f07566c8/init-config-reloader/0.log"
Sep 16 19:10:48.317687 ip-10-0-132-80 kubenswrapper[2544]: I0916 19:10:48.317659    2544 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-59fbdd77d-2fwf6_2b942120-fb59-49f7-a96f-3f7165b19110/kube-state-metrics/0.log"
Sep 16 19:10:48.384771 ip-10-0-132-80 kubenswrapper[2544]: I0916 19:10:48.384737    2544 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-59fbdd77d-2fwf6_2b942120-fb59-49f7-a96f-3f7165b19110/kube-rbac-proxy-main/0.log"
Sep 16 19:10:48.456671 ip-10-0-132-80 kubenswrapper[2544]: I0916 19:10:48.456637    2544 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-59fbdd77d-2fwf6_2b942120-fb59-49f7-a96f-3f7165b19110/kube-rbac-proxy-self/0.log"
Sep 16 19:10:48.504367 ip-10-0-132-80 kubenswrapper[2544]: I0916 19:10:48.504338    2544 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_metrics-server-7544b7b8b8-fk7dc_e96b6d09-3c97-41ed-860d-02ffed52f284/metrics-server/0.log"
Sep 16 19:10:48.886460 ip-10-0-132-80 kubenswrapper[2544]: I0916 19:10:48.886412    2544 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-sljjd_8b245d92-ed58-4240-8b0d-7dab57750690/node-exporter/0.log"
Sep 16 19:10:48.941874 ip-10-0-132-80 kubenswrapper[2544]: I0916 19:10:48.941778    2544 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-sljjd_8b245d92-ed58-4240-8b0d-7dab57750690/kube-rbac-proxy/0.log"
Sep 16 19:10:48.996691 ip-10-0-132-80 kubenswrapper[2544]: I0916 19:10:48.996665    2544 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-sljjd_8b245d92-ed58-4240-8b0d-7dab57750690/init-textfile/0.log"
Sep 16 19:10:49.068452 ip-10-0-132-80 kubenswrapper[2544]: I0916 19:10:49.068425    2544 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-67dcf8c979-694l8_ce1ddaa9-0566-4f7a-bc19-e7a7b958ae22/kube-rbac-proxy-main/0.log"
Sep 16 19:10:49.117130 ip-10-0-132-80 kubenswrapper[2544]: I0916 19:10:49.117068    2544 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-67dcf8c979-694l8_ce1ddaa9-0566-4f7a-bc19-e7a7b958ae22/kube-rbac-proxy-self/0.log"
Sep 16 19:10:49.144044 ip-10-0-132-80 kubenswrapper[2544]: I0916 19:10:49.143974    2544 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-67dcf8c979-694l8_ce1ddaa9-0566-4f7a-bc19-e7a7b958ae22/openshift-state-metrics/0.log"
Sep 16 19:10:49.217478 ip-10-0-132-80 kubenswrapper[2544]: I0916 19:10:49.217445    2544 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_c650a6db-a02b-40ca-a614-b2d37d2b2173/prometheus/0.log"
Sep 16 19:10:49.276883 ip-10-0-132-80 kubenswrapper[2544]: I0916 19:10:49.276848    2544 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_c650a6db-a02b-40ca-a614-b2d37d2b2173/config-reloader/0.log"
Sep 16 19:10:49.311955 ip-10-0-132-80 kubenswrapper[2544]: I0916 19:10:49.311927    2544 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_c650a6db-a02b-40ca-a614-b2d37d2b2173/thanos-sidecar/0.log"
Sep 16 19:10:49.346077 ip-10-0-132-80 kubenswrapper[2544]: I0916 19:10:49.346002    2544 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_c650a6db-a02b-40ca-a614-b2d37d2b2173/kube-rbac-proxy-web/0.log"
Sep 16 19:10:49.389002 ip-10-0-132-80 kubenswrapper[2544]: I0916 19:10:49.388975    2544 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_c650a6db-a02b-40ca-a614-b2d37d2b2173/kube-rbac-proxy/0.log"
Sep 16 19:10:49.422359 ip-10-0-132-80 kubenswrapper[2544]: I0916 19:10:49.422171    2544 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_c650a6db-a02b-40ca-a614-b2d37d2b2173/kube-rbac-proxy-thanos/0.log"
Sep 16 19:10:49.451774 ip-10-0-132-80 kubenswrapper[2544]: I0916 19:10:49.451746    2544 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_c650a6db-a02b-40ca-a614-b2d37d2b2173/init-config-reloader/0.log"
Sep 16 19:10:49.536357 ip-10-0-132-80 kubenswrapper[2544]: I0916 19:10:49.536324    2544 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-admission-webhook-5864cd99d7-pk89k_837cae96-cf48-488d-8a96-e698f41de5df/prometheus-operator-admission-webhook/0.log"
Sep 16 19:10:49.571366 ip-10-0-132-80 kubenswrapper[2544]: I0916 19:10:49.571335    2544 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-598cbd87f8-kmf7z_ac291abc-ad47-431c-89be-9de4e0f33e88/telemeter-client/0.log"
Sep 16 19:10:49.608255 ip-10-0-132-80 kubenswrapper[2544]: I0916 19:10:49.608227    2544 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-598cbd87f8-kmf7z_ac291abc-ad47-431c-89be-9de4e0f33e88/reload/0.log"
Sep 16 19:10:49.639907 ip-10-0-132-80 kubenswrapper[2544]: I0916 19:10:49.639875    2544 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-598cbd87f8-kmf7z_ac291abc-ad47-431c-89be-9de4e0f33e88/kube-rbac-proxy/0.log"
Sep 16 19:10:50.890402 ip-10-0-132-80 kubenswrapper[2544]: I0916 19:10:50.890334    2544 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-console_networking-console-plugin-5dcbc7df6b-q4dbv_aa574cca-15b0-43dd-b30c-58d124d977d2/networking-console-plugin/0.log"
Sep 16 19:10:50.944603 ip-10-0-132-80 kubenswrapper[2544]: I0916 19:10:50.944570    2544 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-txqpj/perf-node-gather-daemonset-cpzwj"]
Sep 16 19:10:50.945323 ip-10-0-132-80 kubenswrapper[2544]: I0916 19:10:50.945293    2544 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d3df97df-afe0-4e9a-95b1-a3957218e67f" containerName="copy"
Sep 16 19:10:50.945472 ip-10-0-132-80 kubenswrapper[2544]: I0916 19:10:50.945459    2544 state_mem.go:107] "Deleted CPUSet assignment" podUID="d3df97df-afe0-4e9a-95b1-a3957218e67f" containerName="copy"
Sep 16 19:10:50.945614 ip-10-0-132-80 kubenswrapper[2544]: I0916 19:10:50.945599    2544 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d3df97df-afe0-4e9a-95b1-a3957218e67f" containerName="gather"
Sep 16 19:10:50.945726 ip-10-0-132-80 kubenswrapper[2544]: I0916 19:10:50.945713    2544 state_mem.go:107] "Deleted CPUSet assignment" podUID="d3df97df-afe0-4e9a-95b1-a3957218e67f" containerName="gather"
Sep 16 19:10:50.945900 ip-10-0-132-80 kubenswrapper[2544]: I0916 19:10:50.945886    2544 memory_manager.go:368] "RemoveStaleState removing state" podUID="d3df97df-afe0-4e9a-95b1-a3957218e67f" containerName="copy"
Sep 16 19:10:50.946001 ip-10-0-132-80 kubenswrapper[2544]: I0916 19:10:50.945989    2544 memory_manager.go:368] "RemoveStaleState removing state" podUID="d3df97df-afe0-4e9a-95b1-a3957218e67f" containerName="gather"
Sep 16 19:10:50.950740 ip-10-0-132-80 kubenswrapper[2544]: I0916 19:10:50.950708    2544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-txqpj/perf-node-gather-daemonset-cpzwj"
Sep 16 19:10:50.968704 ip-10-0-132-80 kubenswrapper[2544]: I0916 19:10:50.968673    2544 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-txqpj/perf-node-gather-daemonset-cpzwj"]
Sep 16 19:10:51.001612 ip-10-0-132-80 kubenswrapper[2544]: I0916 19:10:51.001492    2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/459188d6-685d-4796-b50d-08f624e03d40-lib-modules\") pod \"perf-node-gather-daemonset-cpzwj\" (UID: \"459188d6-685d-4796-b50d-08f624e03d40\") " pod="openshift-must-gather-txqpj/perf-node-gather-daemonset-cpzwj"
Sep 16 19:10:51.001612 ip-10-0-132-80 kubenswrapper[2544]: I0916 19:10:51.001568    2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x9vj8\" (UniqueName: \"kubernetes.io/projected/459188d6-685d-4796-b50d-08f624e03d40-kube-api-access-x9vj8\") pod \"perf-node-gather-daemonset-cpzwj\" (UID: \"459188d6-685d-4796-b50d-08f624e03d40\") " pod="openshift-must-gather-txqpj/perf-node-gather-daemonset-cpzwj"
Sep 16 19:10:51.001849 ip-10-0-132-80 kubenswrapper[2544]: I0916 19:10:51.001696    2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/459188d6-685d-4796-b50d-08f624e03d40-proc\") pod \"perf-node-gather-daemonset-cpzwj\" (UID: \"459188d6-685d-4796-b50d-08f624e03d40\") " pod="openshift-must-gather-txqpj/perf-node-gather-daemonset-cpzwj"
Sep 16 19:10:51.001849 ip-10-0-132-80 kubenswrapper[2544]: I0916 19:10:51.001741    2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/459188d6-685d-4796-b50d-08f624e03d40-sys\") pod \"perf-node-gather-daemonset-cpzwj\" (UID: \"459188d6-685d-4796-b50d-08f624e03d40\") " pod="openshift-must-gather-txqpj/perf-node-gather-daemonset-cpzwj"
Sep 16 19:10:51.001849 ip-10-0-132-80 kubenswrapper[2544]: I0916 19:10:51.001771    2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/459188d6-685d-4796-b50d-08f624e03d40-podres\") pod \"perf-node-gather-daemonset-cpzwj\" (UID: \"459188d6-685d-4796-b50d-08f624e03d40\") " pod="openshift-must-gather-txqpj/perf-node-gather-daemonset-cpzwj"
Sep 16 19:10:51.102834 ip-10-0-132-80 kubenswrapper[2544]: I0916 19:10:51.102797    2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/459188d6-685d-4796-b50d-08f624e03d40-podres\") pod \"perf-node-gather-daemonset-cpzwj\" (UID: \"459188d6-685d-4796-b50d-08f624e03d40\") " pod="openshift-must-gather-txqpj/perf-node-gather-daemonset-cpzwj"
Sep 16 19:10:51.103092 ip-10-0-132-80 kubenswrapper[2544]: I0916 19:10:51.103071    2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/459188d6-685d-4796-b50d-08f624e03d40-lib-modules\") pod \"perf-node-gather-daemonset-cpzwj\" (UID: \"459188d6-685d-4796-b50d-08f624e03d40\") " pod="openshift-must-gather-txqpj/perf-node-gather-daemonset-cpzwj"
Sep 16 19:10:51.103204 ip-10-0-132-80 kubenswrapper[2544]: I0916 19:10:51.103178    2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/459188d6-685d-4796-b50d-08f624e03d40-lib-modules\") pod \"perf-node-gather-daemonset-cpzwj\" (UID: \"459188d6-685d-4796-b50d-08f624e03d40\") " pod="openshift-must-gather-txqpj/perf-node-gather-daemonset-cpzwj"
Sep 16 19:10:51.103373 ip-10-0-132-80 kubenswrapper[2544]: I0916 19:10:51.103222    2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x9vj8\" (UniqueName: \"kubernetes.io/projected/459188d6-685d-4796-b50d-08f624e03d40-kube-api-access-x9vj8\") pod \"perf-node-gather-daemonset-cpzwj\" (UID: \"459188d6-685d-4796-b50d-08f624e03d40\") " pod="openshift-must-gather-txqpj/perf-node-gather-daemonset-cpzwj"
Sep 16 19:10:51.103481 ip-10-0-132-80 kubenswrapper[2544]: I0916 19:10:51.103468    2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/459188d6-685d-4796-b50d-08f624e03d40-proc\") pod \"perf-node-gather-daemonset-cpzwj\" (UID: \"459188d6-685d-4796-b50d-08f624e03d40\") " pod="openshift-must-gather-txqpj/perf-node-gather-daemonset-cpzwj"
Sep 16 19:10:51.103535 ip-10-0-132-80 kubenswrapper[2544]: I0916 19:10:51.103514    2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/459188d6-685d-4796-b50d-08f624e03d40-sys\") pod \"perf-node-gather-daemonset-cpzwj\" (UID: \"459188d6-685d-4796-b50d-08f624e03d40\") " pod="openshift-must-gather-txqpj/perf-node-gather-daemonset-cpzwj"
Sep 16 19:10:51.103653 ip-10-0-132-80 kubenswrapper[2544]: I0916 19:10:51.103621    2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/459188d6-685d-4796-b50d-08f624e03d40-proc\") pod \"perf-node-gather-daemonset-cpzwj\" (UID: \"459188d6-685d-4796-b50d-08f624e03d40\") " pod="openshift-must-gather-txqpj/perf-node-gather-daemonset-cpzwj"
Sep 16 19:10:51.103711 ip-10-0-132-80 kubenswrapper[2544]: I0916 19:10:51.103625    2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/459188d6-685d-4796-b50d-08f624e03d40-sys\") pod \"perf-node-gather-daemonset-cpzwj\" (UID: \"459188d6-685d-4796-b50d-08f624e03d40\") " pod="openshift-must-gather-txqpj/perf-node-gather-daemonset-cpzwj"
Sep 16 19:10:51.103711 ip-10-0-132-80 kubenswrapper[2544]: I0916 19:10:51.102983    2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/459188d6-685d-4796-b50d-08f624e03d40-podres\") pod \"perf-node-gather-daemonset-cpzwj\" (UID: \"459188d6-685d-4796-b50d-08f624e03d40\") " pod="openshift-must-gather-txqpj/perf-node-gather-daemonset-cpzwj"
Sep 16 19:10:51.112107 ip-10-0-132-80 kubenswrapper[2544]: I0916 19:10:51.112084    2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-x9vj8\" (UniqueName: \"kubernetes.io/projected/459188d6-685d-4796-b50d-08f624e03d40-kube-api-access-x9vj8\") pod \"perf-node-gather-daemonset-cpzwj\" (UID: \"459188d6-685d-4796-b50d-08f624e03d40\") " pod="openshift-must-gather-txqpj/perf-node-gather-daemonset-cpzwj"
Sep 16 19:10:51.264717 ip-10-0-132-80 kubenswrapper[2544]: I0916 19:10:51.264640    2544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-txqpj/perf-node-gather-daemonset-cpzwj"
Sep 16 19:10:51.433991 ip-10-0-132-80 kubenswrapper[2544]: I0916 19:10:51.433960    2544 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-pfcb7_9e3f9e34-420e-43ef-96b4-c7cad6b2eaa9/console-operator/2.log"
Sep 16 19:10:51.438904 ip-10-0-132-80 kubenswrapper[2544]: I0916 19:10:51.438880    2544 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-pfcb7_9e3f9e34-420e-43ef-96b4-c7cad6b2eaa9/console-operator/3.log"
Sep 16 19:10:51.546530 ip-10-0-132-80 kubenswrapper[2544]: I0916 19:10:51.546456    2544 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-txqpj/perf-node-gather-daemonset-cpzwj"]
Sep 16 19:10:51.549491 ip-10-0-132-80 kubenswrapper[2544]: W0916 19:10:51.549462    2544 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod459188d6_685d_4796_b50d_08f624e03d40.slice/crio-981da2ccc5343e488e9bcffb936aff0d42810a61eda938e6f93aa1decbc77154 WatchSource:0}: Error finding container 981da2ccc5343e488e9bcffb936aff0d42810a61eda938e6f93aa1decbc77154: Status 404 returned error can't find the container with id 981da2ccc5343e488e9bcffb936aff0d42810a61eda938e6f93aa1decbc77154
Sep 16 19:10:51.932683 ip-10-0-132-80 kubenswrapper[2544]: I0916 19:10:51.932649    2544 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_downloads-76f99584f4-m4dhw_e03463ea-f247-46f9-b8a8-57ccef5477d0/download-server/0.log"
Sep 16 19:10:51.987138 ip-10-0-132-80 kubenswrapper[2544]: I0916 19:10:51.987104    2544 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-txqpj/perf-node-gather-daemonset-cpzwj" event={"ID":"459188d6-685d-4796-b50d-08f624e03d40","Type":"ContainerStarted","Data":"defe0b8cf3af924216729e9a8c751d2bb1bfd612cd7d7961581f503cbfb66e0f"}
Sep 16 19:10:51.987328 ip-10-0-132-80 kubenswrapper[2544]: I0916 19:10:51.987163    2544 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-txqpj/perf-node-gather-daemonset-cpzwj" event={"ID":"459188d6-685d-4796-b50d-08f624e03d40","Type":"ContainerStarted","Data":"981da2ccc5343e488e9bcffb936aff0d42810a61eda938e6f93aa1decbc77154"}
Sep 16 19:10:51.987328 ip-10-0-132-80 kubenswrapper[2544]: I0916 19:10:51.987187    2544 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-txqpj/perf-node-gather-daemonset-cpzwj"
Sep 16 19:10:52.007579 ip-10-0-132-80 kubenswrapper[2544]: I0916 19:10:52.007497    2544 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-txqpj/perf-node-gather-daemonset-cpzwj" podStartSLOduration=2.007474646 podStartE2EDuration="2.007474646s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-16 19:10:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:10:52.005268749 +0000 UTC m=+1614.794271283" watchObservedRunningTime="2026-09-16 19:10:52.007474646 +0000 UTC m=+1614.796477180"
Sep 16 19:10:53.555898 ip-10-0-132-80 kubenswrapper[2544]: I0916 19:10:53.555851    2544 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-n8djq_b291cc26-ff6e-4adb-b219-8bd70a9aa3b2/dns/0.log"
Sep 16 19:10:53.584334 ip-10-0-132-80 kubenswrapper[2544]: I0916 19:10:53.584271    2544 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-n8djq_b291cc26-ff6e-4adb-b219-8bd70a9aa3b2/kube-rbac-proxy/0.log"
Sep 16 19:10:53.799675 ip-10-0-132-80 kubenswrapper[2544]: I0916 19:10:53.799640    2544 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-sxbgl_524324ca-6ba2-4885-9ed6-570fa75665e7/dns-node-resolver/0.log"
Sep 16 19:10:54.285046 ip-10-0-132-80 kubenswrapper[2544]: I0916 19:10:54.285017    2544 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-2hdh4_273fb2b8-0579-4f86-abc5-2801f6f47bcd/node-ca/0.log"
Sep 16 19:10:55.574476 ip-10-0-132-80 kubenswrapper[2544]: I0916 19:10:55.574451    2544 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-jmnkv_ee8d40ee-7c8a-496a-a4e4-56cdc3e0aaa2/serve-healthcheck-canary/0.log"
Sep 16 19:10:56.147787 ip-10-0-132-80 kubenswrapper[2544]: I0916 19:10:56.147756    2544 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-xhj2z_3e46e7aa-10f9-4c7d-bfa5-62504dc45cfd/kube-rbac-proxy/0.log"
Sep 16 19:10:56.169025 ip-10-0-132-80 kubenswrapper[2544]: I0916 19:10:56.168988    2544 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-xhj2z_3e46e7aa-10f9-4c7d-bfa5-62504dc45cfd/exporter/0.log"
Sep 16 19:10:56.190092 ip-10-0-132-80 kubenswrapper[2544]: I0916 19:10:56.190063    2544 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-xhj2z_3e46e7aa-10f9-4c7d-bfa5-62504dc45cfd/extractor/0.log"
Sep 16 19:10:58.003437 ip-10-0-132-80 kubenswrapper[2544]: I0916 19:10:58.003404    2544 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-txqpj/perf-node-gather-daemonset-cpzwj"
Sep 16 19:10:58.923581 ip-10-0-132-80 kubenswrapper[2544]: I0916 19:10:58.923538    2544 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_s3-init-ljlnm_d2391c30-9a0d-44b8-9f6e-b9d433288c96/s3-init/0.log"
Sep 16 19:10:58.941748 ip-10-0-132-80 kubenswrapper[2544]: I0916 19:10:58.941723    2544 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_s3-init-ljlnm_d2391c30-9a0d-44b8-9f6e-b9d433288c96/download-hf-model/0.log"
Sep 16 19:11:03.569240 ip-10-0-132-80 kubenswrapper[2544]: I0916 19:11:03.569215    2544 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-8b4d697c5-mm4qq_358d96b5-9e80-4d99-9478-81be76c4db64/migrator/0.log"
Sep 16 19:11:03.594004 ip-10-0-132-80 kubenswrapper[2544]: I0916 19:11:03.593982    2544 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-8b4d697c5-mm4qq_358d96b5-9e80-4d99-9478-81be76c4db64/graceful-termination/0.log"
Sep 16 19:11:05.524326 ip-10-0-132-80 kubenswrapper[2544]: I0916 19:11:05.524292    2544 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-znf6p_38dd6a59-da17-47cd-b10f-540f2fff9a89/kube-multus-additional-cni-plugins/0.log"
Sep 16 19:11:05.548345 ip-10-0-132-80 kubenswrapper[2544]: I0916 19:11:05.548317    2544 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-znf6p_38dd6a59-da17-47cd-b10f-540f2fff9a89/egress-router-binary-copy/0.log"
Sep 16 19:11:05.576338 ip-10-0-132-80 kubenswrapper[2544]: I0916 19:11:05.576311    2544 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-znf6p_38dd6a59-da17-47cd-b10f-540f2fff9a89/cni-plugins/0.log"
Sep 16 19:11:05.617441 ip-10-0-132-80 kubenswrapper[2544]: I0916 19:11:05.617415    2544 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-znf6p_38dd6a59-da17-47cd-b10f-540f2fff9a89/bond-cni-plugin/0.log"
Sep 16 19:11:05.657435 ip-10-0-132-80 kubenswrapper[2544]: I0916 19:11:05.657409    2544 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-znf6p_38dd6a59-da17-47cd-b10f-540f2fff9a89/routeoverride-cni/0.log"
Sep 16 19:11:05.700435 ip-10-0-132-80 kubenswrapper[2544]: I0916 19:11:05.700407    2544 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-znf6p_38dd6a59-da17-47cd-b10f-540f2fff9a89/whereabouts-cni-bincopy/0.log"
Sep 16 19:11:05.734196 ip-10-0-132-80 kubenswrapper[2544]: I0916 19:11:05.734173    2544 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-znf6p_38dd6a59-da17-47cd-b10f-540f2fff9a89/whereabouts-cni/0.log"
Sep 16 19:11:05.781246 ip-10-0-132-80 kubenswrapper[2544]: I0916 19:11:05.781179    2544 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-dbwbz_bef94c9f-6884-4c22-b616-32f4e38a4772/kube-multus/0.log"
Sep 16 19:11:05.949364 ip-10-0-132-80 kubenswrapper[2544]: I0916 19:11:05.949330    2544 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-pdfpq_db1e3b32-114c-4bd9-b28c-21a2a9d6efaa/network-metrics-daemon/0.log"
Sep 16 19:11:05.989381 ip-10-0-132-80 kubenswrapper[2544]: I0916 19:11:05.989352    2544 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-pdfpq_db1e3b32-114c-4bd9-b28c-21a2a9d6efaa/kube-rbac-proxy/0.log"
Sep 16 19:11:07.138243 ip-10-0-132-80 kubenswrapper[2544]: I0916 19:11:07.138215    2544 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-nqcpx_7cefdcfe-d218-4ac0-a3d8-811df8c043c2/ovn-controller/0.log"
Sep 16 19:11:07.158796 ip-10-0-132-80 kubenswrapper[2544]: I0916 19:11:07.158771    2544 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-nqcpx_7cefdcfe-d218-4ac0-a3d8-811df8c043c2/ovn-acl-logging/0.log"
Sep 16 19:11:07.185360 ip-10-0-132-80 kubenswrapper[2544]: I0916 19:11:07.185330    2544 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-nqcpx_7cefdcfe-d218-4ac0-a3d8-811df8c043c2/kube-rbac-proxy-node/0.log"
Sep 16 19:11:07.216467 ip-10-0-132-80 kubenswrapper[2544]: I0916 19:11:07.216439    2544 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-nqcpx_7cefdcfe-d218-4ac0-a3d8-811df8c043c2/kube-rbac-proxy-ovn-metrics/0.log"
Sep 16 19:11:07.245306 ip-10-0-132-80 kubenswrapper[2544]: I0916 19:11:07.245279    2544 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-nqcpx_7cefdcfe-d218-4ac0-a3d8-811df8c043c2/northd/0.log"
Sep 16 19:11:07.271879 ip-10-0-132-80 kubenswrapper[2544]: I0916 19:11:07.271850    2544 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-nqcpx_7cefdcfe-d218-4ac0-a3d8-811df8c043c2/nbdb/0.log"
Sep 16 19:11:07.296882 ip-10-0-132-80 kubenswrapper[2544]: I0916 19:11:07.296852    2544 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-nqcpx_7cefdcfe-d218-4ac0-a3d8-811df8c043c2/sbdb/0.log"
Sep 16 19:11:07.342468 ip-10-0-132-80 kubenswrapper[2544]: I0916 19:11:07.342441    2544 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-nqcpx_7cefdcfe-d218-4ac0-a3d8-811df8c043c2/ovnkube-controller/0.log"