Sep 16 18:45:30.074495 ip-10-0-138-127 systemd[1]: Starting Kubernetes Kubelet...
Sep 16 18:45:30.484224 ip-10-0-138-127 kubenswrapper[2581]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 16 18:45:30.484224 ip-10-0-138-127 kubenswrapper[2581]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Sep 16 18:45:30.484224 ip-10-0-138-127 kubenswrapper[2581]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 16 18:45:30.484224 ip-10-0-138-127 kubenswrapper[2581]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI.
Sep 16 18:45:30.484224 ip-10-0-138-127 kubenswrapper[2581]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 16 18:45:30.487331 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:45:30.487188    2581 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime"
Sep 16 18:45:30.492684 ip-10-0-138-127 kubenswrapper[2581]: W0916 18:45:30.492653    2581 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 16 18:45:30.492684 ip-10-0-138-127 kubenswrapper[2581]: W0916 18:45:30.492681    2581 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 16 18:45:30.492684 ip-10-0-138-127 kubenswrapper[2581]: W0916 18:45:30.492687    2581 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 16 18:45:30.492684 ip-10-0-138-127 kubenswrapper[2581]: W0916 18:45:30.492693    2581 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 16 18:45:30.492923 ip-10-0-138-127 kubenswrapper[2581]: W0916 18:45:30.492699    2581 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 16 18:45:30.492923 ip-10-0-138-127 kubenswrapper[2581]: W0916 18:45:30.492709    2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 16 18:45:30.492923 ip-10-0-138-127 kubenswrapper[2581]: W0916 18:45:30.492714    2581 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 16 18:45:30.492923 ip-10-0-138-127 kubenswrapper[2581]: W0916 18:45:30.492718    2581 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 16 18:45:30.492923 ip-10-0-138-127 kubenswrapper[2581]: W0916 18:45:30.492722    2581 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 16 18:45:30.492923 ip-10-0-138-127 kubenswrapper[2581]: W0916 18:45:30.492729    2581 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 16 18:45:30.492923 ip-10-0-138-127 kubenswrapper[2581]: W0916 18:45:30.492735    2581 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 16 18:45:30.492923 ip-10-0-138-127 kubenswrapper[2581]: W0916 18:45:30.492739    2581 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 16 18:45:30.492923 ip-10-0-138-127 kubenswrapper[2581]: W0916 18:45:30.492745    2581 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 16 18:45:30.492923 ip-10-0-138-127 kubenswrapper[2581]: W0916 18:45:30.492749    2581 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 16 18:45:30.492923 ip-10-0-138-127 kubenswrapper[2581]: W0916 18:45:30.492753    2581 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 16 18:45:30.492923 ip-10-0-138-127 kubenswrapper[2581]: W0916 18:45:30.492757    2581 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 16 18:45:30.492923 ip-10-0-138-127 kubenswrapper[2581]: W0916 18:45:30.492762    2581 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 16 18:45:30.492923 ip-10-0-138-127 kubenswrapper[2581]: W0916 18:45:30.492771    2581 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 16 18:45:30.492923 ip-10-0-138-127 kubenswrapper[2581]: W0916 18:45:30.492776    2581 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 16 18:45:30.492923 ip-10-0-138-127 kubenswrapper[2581]: W0916 18:45:30.492780    2581 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 16 18:45:30.492923 ip-10-0-138-127 kubenswrapper[2581]: W0916 18:45:30.492784    2581 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 16 18:45:30.492923 ip-10-0-138-127 kubenswrapper[2581]: W0916 18:45:30.492789    2581 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 16 18:45:30.492923 ip-10-0-138-127 kubenswrapper[2581]: W0916 18:45:30.492793    2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 16 18:45:30.493468 ip-10-0-138-127 kubenswrapper[2581]: W0916 18:45:30.492798    2581 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 16 18:45:30.493468 ip-10-0-138-127 kubenswrapper[2581]: W0916 18:45:30.492801    2581 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 16 18:45:30.493468 ip-10-0-138-127 kubenswrapper[2581]: W0916 18:45:30.492804    2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 16 18:45:30.493468 ip-10-0-138-127 kubenswrapper[2581]: W0916 18:45:30.492807    2581 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 16 18:45:30.493468 ip-10-0-138-127 kubenswrapper[2581]: W0916 18:45:30.492810    2581 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 16 18:45:30.493468 ip-10-0-138-127 kubenswrapper[2581]: W0916 18:45:30.492812    2581 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 16 18:45:30.493468 ip-10-0-138-127 kubenswrapper[2581]: W0916 18:45:30.492815    2581 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 16 18:45:30.493468 ip-10-0-138-127 kubenswrapper[2581]: W0916 18:45:30.492821    2581 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 16 18:45:30.493468 ip-10-0-138-127 kubenswrapper[2581]: W0916 18:45:30.492823    2581 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 16 18:45:30.493468 ip-10-0-138-127 kubenswrapper[2581]: W0916 18:45:30.492828    2581 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 16 18:45:30.493468 ip-10-0-138-127 kubenswrapper[2581]: W0916 18:45:30.492832    2581 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 16 18:45:30.493468 ip-10-0-138-127 kubenswrapper[2581]: W0916 18:45:30.492835    2581 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 16 18:45:30.493468 ip-10-0-138-127 kubenswrapper[2581]: W0916 18:45:30.492838    2581 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 16 18:45:30.493468 ip-10-0-138-127 kubenswrapper[2581]: W0916 18:45:30.492841    2581 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 16 18:45:30.493468 ip-10-0-138-127 kubenswrapper[2581]: W0916 18:45:30.492845    2581 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 16 18:45:30.493468 ip-10-0-138-127 kubenswrapper[2581]: W0916 18:45:30.492848    2581 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 16 18:45:30.493468 ip-10-0-138-127 kubenswrapper[2581]: W0916 18:45:30.492850    2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 16 18:45:30.493468 ip-10-0-138-127 kubenswrapper[2581]: W0916 18:45:30.492853    2581 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 16 18:45:30.493468 ip-10-0-138-127 kubenswrapper[2581]: W0916 18:45:30.492856    2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 16 18:45:30.493468 ip-10-0-138-127 kubenswrapper[2581]: W0916 18:45:30.492861    2581 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 16 18:45:30.493950 ip-10-0-138-127 kubenswrapper[2581]: W0916 18:45:30.492864    2581 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 16 18:45:30.493950 ip-10-0-138-127 kubenswrapper[2581]: W0916 18:45:30.492868    2581 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 16 18:45:30.493950 ip-10-0-138-127 kubenswrapper[2581]: W0916 18:45:30.492874    2581 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 16 18:45:30.493950 ip-10-0-138-127 kubenswrapper[2581]: W0916 18:45:30.492877    2581 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 16 18:45:30.493950 ip-10-0-138-127 kubenswrapper[2581]: W0916 18:45:30.492880    2581 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 16 18:45:30.493950 ip-10-0-138-127 kubenswrapper[2581]: W0916 18:45:30.492883    2581 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 16 18:45:30.493950 ip-10-0-138-127 kubenswrapper[2581]: W0916 18:45:30.492886    2581 feature_gate.go:328] unrecognized feature gate: Example
Sep 16 18:45:30.493950 ip-10-0-138-127 kubenswrapper[2581]: W0916 18:45:30.492889    2581 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 16 18:45:30.493950 ip-10-0-138-127 kubenswrapper[2581]: W0916 18:45:30.492891    2581 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 16 18:45:30.493950 ip-10-0-138-127 kubenswrapper[2581]: W0916 18:45:30.492894    2581 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 16 18:45:30.493950 ip-10-0-138-127 kubenswrapper[2581]: W0916 18:45:30.492897    2581 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 16 18:45:30.493950 ip-10-0-138-127 kubenswrapper[2581]: W0916 18:45:30.492900    2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 16 18:45:30.493950 ip-10-0-138-127 kubenswrapper[2581]: W0916 18:45:30.492904    2581 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 16 18:45:30.493950 ip-10-0-138-127 kubenswrapper[2581]: W0916 18:45:30.492919    2581 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 16 18:45:30.493950 ip-10-0-138-127 kubenswrapper[2581]: W0916 18:45:30.493076    2581 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 16 18:45:30.493950 ip-10-0-138-127 kubenswrapper[2581]: W0916 18:45:30.493082    2581 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 16 18:45:30.493950 ip-10-0-138-127 kubenswrapper[2581]: W0916 18:45:30.493086    2581 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 16 18:45:30.493950 ip-10-0-138-127 kubenswrapper[2581]: W0916 18:45:30.493090    2581 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 16 18:45:30.493950 ip-10-0-138-127 kubenswrapper[2581]: W0916 18:45:30.493099    2581 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 16 18:45:30.493950 ip-10-0-138-127 kubenswrapper[2581]: W0916 18:45:30.493102    2581 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 16 18:45:30.494436 ip-10-0-138-127 kubenswrapper[2581]: W0916 18:45:30.493106    2581 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 16 18:45:30.494436 ip-10-0-138-127 kubenswrapper[2581]: W0916 18:45:30.493110    2581 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 16 18:45:30.494436 ip-10-0-138-127 kubenswrapper[2581]: W0916 18:45:30.493114    2581 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 16 18:45:30.494436 ip-10-0-138-127 kubenswrapper[2581]: W0916 18:45:30.493118    2581 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 16 18:45:30.494436 ip-10-0-138-127 kubenswrapper[2581]: W0916 18:45:30.493121    2581 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 16 18:45:30.494436 ip-10-0-138-127 kubenswrapper[2581]: W0916 18:45:30.493124    2581 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 16 18:45:30.494436 ip-10-0-138-127 kubenswrapper[2581]: W0916 18:45:30.493127    2581 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 16 18:45:30.494436 ip-10-0-138-127 kubenswrapper[2581]: W0916 18:45:30.493130    2581 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 16 18:45:30.494436 ip-10-0-138-127 kubenswrapper[2581]: W0916 18:45:30.493133    2581 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 16 18:45:30.494436 ip-10-0-138-127 kubenswrapper[2581]: W0916 18:45:30.493136    2581 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 16 18:45:30.494436 ip-10-0-138-127 kubenswrapper[2581]: W0916 18:45:30.493139    2581 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 16 18:45:30.494436 ip-10-0-138-127 kubenswrapper[2581]: W0916 18:45:30.493142    2581 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 16 18:45:30.494436 ip-10-0-138-127 kubenswrapper[2581]: W0916 18:45:30.493144    2581 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 16 18:45:30.494436 ip-10-0-138-127 kubenswrapper[2581]: W0916 18:45:30.493147    2581 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 16 18:45:30.494436 ip-10-0-138-127 kubenswrapper[2581]: W0916 18:45:30.493149    2581 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 16 18:45:30.494436 ip-10-0-138-127 kubenswrapper[2581]: W0916 18:45:30.493152    2581 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 16 18:45:30.494436 ip-10-0-138-127 kubenswrapper[2581]: W0916 18:45:30.493157    2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 16 18:45:30.494436 ip-10-0-138-127 kubenswrapper[2581]: W0916 18:45:30.493160    2581 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 16 18:45:30.494436 ip-10-0-138-127 kubenswrapper[2581]: W0916 18:45:30.493163    2581 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 16 18:45:30.494887 ip-10-0-138-127 kubenswrapper[2581]: W0916 18:45:30.493166    2581 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 16 18:45:30.494887 ip-10-0-138-127 kubenswrapper[2581]: W0916 18:45:30.493168    2581 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 16 18:45:30.494887 ip-10-0-138-127 kubenswrapper[2581]: W0916 18:45:30.493171    2581 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 16 18:45:30.494887 ip-10-0-138-127 kubenswrapper[2581]: W0916 18:45:30.493174    2581 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 16 18:45:30.494887 ip-10-0-138-127 kubenswrapper[2581]: W0916 18:45:30.493176    2581 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 16 18:45:30.494887 ip-10-0-138-127 kubenswrapper[2581]: W0916 18:45:30.493179    2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 16 18:45:30.494887 ip-10-0-138-127 kubenswrapper[2581]: W0916 18:45:30.493181    2581 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 16 18:45:30.494887 ip-10-0-138-127 kubenswrapper[2581]: W0916 18:45:30.493184    2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 16 18:45:30.494887 ip-10-0-138-127 kubenswrapper[2581]: W0916 18:45:30.493187    2581 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 16 18:45:30.494887 ip-10-0-138-127 kubenswrapper[2581]: W0916 18:45:30.493203    2581 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 16 18:45:30.494887 ip-10-0-138-127 kubenswrapper[2581]: W0916 18:45:30.493207    2581 feature_gate.go:328] unrecognized feature gate: Example2
Sep 16 18:45:30.494887 ip-10-0-138-127 kubenswrapper[2581]: W0916 18:45:30.493210    2581 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 16 18:45:30.494887 ip-10-0-138-127 kubenswrapper[2581]: W0916 18:45:30.493213    2581 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 16 18:45:30.494887 ip-10-0-138-127 kubenswrapper[2581]: W0916 18:45:30.493216    2581 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 16 18:45:30.494887 ip-10-0-138-127 kubenswrapper[2581]: W0916 18:45:30.493218    2581 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 16 18:45:30.494887 ip-10-0-138-127 kubenswrapper[2581]: W0916 18:45:30.493221    2581 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 16 18:45:30.494887 ip-10-0-138-127 kubenswrapper[2581]: W0916 18:45:30.493224    2581 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 16 18:45:30.494887 ip-10-0-138-127 kubenswrapper[2581]: W0916 18:45:30.493962    2581 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 16 18:45:30.494887 ip-10-0-138-127 kubenswrapper[2581]: W0916 18:45:30.493968    2581 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 16 18:45:30.494887 ip-10-0-138-127 kubenswrapper[2581]: W0916 18:45:30.493971    2581 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 16 18:45:30.495402 ip-10-0-138-127 kubenswrapper[2581]: W0916 18:45:30.493974    2581 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 16 18:45:30.495402 ip-10-0-138-127 kubenswrapper[2581]: W0916 18:45:30.493977    2581 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 16 18:45:30.495402 ip-10-0-138-127 kubenswrapper[2581]: W0916 18:45:30.493980    2581 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 16 18:45:30.495402 ip-10-0-138-127 kubenswrapper[2581]: W0916 18:45:30.493982    2581 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 16 18:45:30.495402 ip-10-0-138-127 kubenswrapper[2581]: W0916 18:45:30.493985    2581 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 16 18:45:30.495402 ip-10-0-138-127 kubenswrapper[2581]: W0916 18:45:30.493988    2581 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 16 18:45:30.495402 ip-10-0-138-127 kubenswrapper[2581]: W0916 18:45:30.493991    2581 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 16 18:45:30.495402 ip-10-0-138-127 kubenswrapper[2581]: W0916 18:45:30.493993    2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 16 18:45:30.495402 ip-10-0-138-127 kubenswrapper[2581]: W0916 18:45:30.493996    2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 16 18:45:30.495402 ip-10-0-138-127 kubenswrapper[2581]: W0916 18:45:30.493998    2581 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 16 18:45:30.495402 ip-10-0-138-127 kubenswrapper[2581]: W0916 18:45:30.494001    2581 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 16 18:45:30.495402 ip-10-0-138-127 kubenswrapper[2581]: W0916 18:45:30.494004    2581 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 16 18:45:30.495402 ip-10-0-138-127 kubenswrapper[2581]: W0916 18:45:30.494006    2581 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 16 18:45:30.495402 ip-10-0-138-127 kubenswrapper[2581]: W0916 18:45:30.494009    2581 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 16 18:45:30.495402 ip-10-0-138-127 kubenswrapper[2581]: W0916 18:45:30.494012    2581 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 16 18:45:30.495402 ip-10-0-138-127 kubenswrapper[2581]: W0916 18:45:30.494014    2581 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 16 18:45:30.495402 ip-10-0-138-127 kubenswrapper[2581]: W0916 18:45:30.494019    2581 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 16 18:45:30.495402 ip-10-0-138-127 kubenswrapper[2581]: W0916 18:45:30.494022    2581 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 16 18:45:30.495402 ip-10-0-138-127 kubenswrapper[2581]: W0916 18:45:30.494025    2581 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 16 18:45:30.495869 ip-10-0-138-127 kubenswrapper[2581]: W0916 18:45:30.494029    2581 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 16 18:45:30.495869 ip-10-0-138-127 kubenswrapper[2581]: W0916 18:45:30.494032    2581 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 16 18:45:30.495869 ip-10-0-138-127 kubenswrapper[2581]: W0916 18:45:30.494035    2581 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 16 18:45:30.495869 ip-10-0-138-127 kubenswrapper[2581]: W0916 18:45:30.494038    2581 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 16 18:45:30.495869 ip-10-0-138-127 kubenswrapper[2581]: W0916 18:45:30.494040    2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 16 18:45:30.495869 ip-10-0-138-127 kubenswrapper[2581]: W0916 18:45:30.494043    2581 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 16 18:45:30.495869 ip-10-0-138-127 kubenswrapper[2581]: W0916 18:45:30.494045    2581 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 16 18:45:30.495869 ip-10-0-138-127 kubenswrapper[2581]: W0916 18:45:30.494048    2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 16 18:45:30.495869 ip-10-0-138-127 kubenswrapper[2581]: W0916 18:45:30.494050    2581 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 16 18:45:30.495869 ip-10-0-138-127 kubenswrapper[2581]: W0916 18:45:30.494053    2581 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 16 18:45:30.495869 ip-10-0-138-127 kubenswrapper[2581]: W0916 18:45:30.494055    2581 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 16 18:45:30.495869 ip-10-0-138-127 kubenswrapper[2581]: W0916 18:45:30.494058    2581 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 16 18:45:30.495869 ip-10-0-138-127 kubenswrapper[2581]: W0916 18:45:30.494060    2581 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 16 18:45:30.495869 ip-10-0-138-127 kubenswrapper[2581]: W0916 18:45:30.494062    2581 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 16 18:45:30.495869 ip-10-0-138-127 kubenswrapper[2581]: W0916 18:45:30.494065    2581 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 16 18:45:30.495869 ip-10-0-138-127 kubenswrapper[2581]: W0916 18:45:30.494067    2581 feature_gate.go:328] unrecognized feature gate: Example2
Sep 16 18:45:30.495869 ip-10-0-138-127 kubenswrapper[2581]: W0916 18:45:30.494070    2581 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 16 18:45:30.495869 ip-10-0-138-127 kubenswrapper[2581]: W0916 18:45:30.494073    2581 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 16 18:45:30.495869 ip-10-0-138-127 kubenswrapper[2581]: W0916 18:45:30.494075    2581 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 16 18:45:30.495869 ip-10-0-138-127 kubenswrapper[2581]: W0916 18:45:30.494079    2581 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 16 18:45:30.496379 ip-10-0-138-127 kubenswrapper[2581]: W0916 18:45:30.494081    2581 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 16 18:45:30.496379 ip-10-0-138-127 kubenswrapper[2581]: W0916 18:45:30.494084    2581 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 16 18:45:30.496379 ip-10-0-138-127 kubenswrapper[2581]: W0916 18:45:30.494086    2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 16 18:45:30.496379 ip-10-0-138-127 kubenswrapper[2581]: W0916 18:45:30.494089    2581 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 16 18:45:30.496379 ip-10-0-138-127 kubenswrapper[2581]: W0916 18:45:30.494092    2581 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 16 18:45:30.496379 ip-10-0-138-127 kubenswrapper[2581]: W0916 18:45:30.494094    2581 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 16 18:45:30.496379 ip-10-0-138-127 kubenswrapper[2581]: W0916 18:45:30.494097    2581 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 16 18:45:30.496379 ip-10-0-138-127 kubenswrapper[2581]: W0916 18:45:30.494099    2581 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 16 18:45:30.496379 ip-10-0-138-127 kubenswrapper[2581]: W0916 18:45:30.494103    2581 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 16 18:45:30.496379 ip-10-0-138-127 kubenswrapper[2581]: W0916 18:45:30.494105    2581 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 16 18:45:30.496379 ip-10-0-138-127 kubenswrapper[2581]: W0916 18:45:30.494108    2581 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 16 18:45:30.496379 ip-10-0-138-127 kubenswrapper[2581]: W0916 18:45:30.494110    2581 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 16 18:45:30.496379 ip-10-0-138-127 kubenswrapper[2581]: W0916 18:45:30.494113    2581 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 16 18:45:30.496379 ip-10-0-138-127 kubenswrapper[2581]: W0916 18:45:30.494116    2581 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 16 18:45:30.496379 ip-10-0-138-127 kubenswrapper[2581]: W0916 18:45:30.494118    2581 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 16 18:45:30.496379 ip-10-0-138-127 kubenswrapper[2581]: W0916 18:45:30.494121    2581 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 16 18:45:30.496379 ip-10-0-138-127 kubenswrapper[2581]: W0916 18:45:30.494123    2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 16 18:45:30.496379 ip-10-0-138-127 kubenswrapper[2581]: W0916 18:45:30.494126    2581 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 16 18:45:30.496379 ip-10-0-138-127 kubenswrapper[2581]: W0916 18:45:30.494128    2581 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 16 18:45:30.496379 ip-10-0-138-127 kubenswrapper[2581]: W0916 18:45:30.494131    2581 feature_gate.go:328] unrecognized feature gate: Example
Sep 16 18:45:30.496861 ip-10-0-138-127 kubenswrapper[2581]: W0916 18:45:30.494133    2581 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 16 18:45:30.496861 ip-10-0-138-127 kubenswrapper[2581]: W0916 18:45:30.494135    2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 16 18:45:30.496861 ip-10-0-138-127 kubenswrapper[2581]: W0916 18:45:30.494138    2581 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 16 18:45:30.496861 ip-10-0-138-127 kubenswrapper[2581]: W0916 18:45:30.494140    2581 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 16 18:45:30.496861 ip-10-0-138-127 kubenswrapper[2581]: W0916 18:45:30.494143    2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 16 18:45:30.496861 ip-10-0-138-127 kubenswrapper[2581]: W0916 18:45:30.494145    2581 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 16 18:45:30.496861 ip-10-0-138-127 kubenswrapper[2581]: W0916 18:45:30.494148    2581 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 16 18:45:30.496861 ip-10-0-138-127 kubenswrapper[2581]: W0916 18:45:30.494150    2581 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 16 18:45:30.496861 ip-10-0-138-127 kubenswrapper[2581]: W0916 18:45:30.494153    2581 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 16 18:45:30.496861 ip-10-0-138-127 kubenswrapper[2581]: W0916 18:45:30.494155    2581 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 16 18:45:30.496861 ip-10-0-138-127 kubenswrapper[2581]: W0916 18:45:30.494158    2581 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 16 18:45:30.496861 ip-10-0-138-127 kubenswrapper[2581]: W0916 18:45:30.494162    2581 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 16 18:45:30.496861 ip-10-0-138-127 kubenswrapper[2581]: W0916 18:45:30.494164    2581 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 16 18:45:30.496861 ip-10-0-138-127 kubenswrapper[2581]: W0916 18:45:30.494167    2581 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 16 18:45:30.496861 ip-10-0-138-127 kubenswrapper[2581]: W0916 18:45:30.494169    2581 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 16 18:45:30.496861 ip-10-0-138-127 kubenswrapper[2581]: W0916 18:45:30.494171    2581 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 16 18:45:30.496861 ip-10-0-138-127 kubenswrapper[2581]: W0916 18:45:30.494175    2581 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 16 18:45:30.496861 ip-10-0-138-127 kubenswrapper[2581]: W0916 18:45:30.494178    2581 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 16 18:45:30.496861 ip-10-0-138-127 kubenswrapper[2581]: W0916 18:45:30.494180    2581 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 16 18:45:30.496861 ip-10-0-138-127 kubenswrapper[2581]: W0916 18:45:30.494183    2581 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 16 18:45:30.497380 ip-10-0-138-127 kubenswrapper[2581]: W0916 18:45:30.494185    2581 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 16 18:45:30.497380 ip-10-0-138-127 kubenswrapper[2581]: W0916 18:45:30.494188    2581 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 16 18:45:30.497380 ip-10-0-138-127 kubenswrapper[2581]: W0916 18:45:30.494208    2581 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 16 18:45:30.497380 ip-10-0-138-127 kubenswrapper[2581]: W0916 18:45:30.494211    2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 16 18:45:30.497380 ip-10-0-138-127 kubenswrapper[2581]: W0916 18:45:30.494214    2581 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 16 18:45:30.497380 ip-10-0-138-127 kubenswrapper[2581]: W0916 18:45:30.494217    2581 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 16 18:45:30.497380 ip-10-0-138-127 kubenswrapper[2581]: W0916 18:45:30.494221    2581 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 16 18:45:30.497380 ip-10-0-138-127 kubenswrapper[2581]: W0916 18:45:30.494224    2581 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 16 18:45:30.497380 ip-10-0-138-127 kubenswrapper[2581]: W0916 18:45:30.494227    2581 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 16 18:45:30.497380 ip-10-0-138-127 kubenswrapper[2581]: W0916 18:45:30.494231    2581 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 16 18:45:30.497380 ip-10-0-138-127 kubenswrapper[2581]: W0916 18:45:30.494235    2581 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 16 18:45:30.497380 ip-10-0-138-127 kubenswrapper[2581]: W0916 18:45:30.494238    2581 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 16 18:45:30.497380 ip-10-0-138-127 kubenswrapper[2581]: W0916 18:45:30.494241    2581 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 16 18:45:30.497380 ip-10-0-138-127 kubenswrapper[2581]: W0916 18:45:30.494244    2581 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 16 18:45:30.497380 ip-10-0-138-127 kubenswrapper[2581]: W0916 18:45:30.494246    2581 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 16 18:45:30.497380 ip-10-0-138-127 kubenswrapper[2581]: W0916 18:45:30.494249    2581 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 16 18:45:30.497380 ip-10-0-138-127 kubenswrapper[2581]: W0916 18:45:30.494251    2581 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 16 18:45:30.497380 ip-10-0-138-127 kubenswrapper[2581]: W0916 18:45:30.494670    2581 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 16 18:45:30.497380 ip-10-0-138-127 kubenswrapper[2581]: W0916 18:45:30.494676    2581 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 16 18:45:30.497839 ip-10-0-138-127 kubenswrapper[2581]: W0916 18:45:30.494679    2581 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 16 18:45:30.497839 ip-10-0-138-127 kubenswrapper[2581]: W0916 18:45:30.494682    2581 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 16 18:45:30.497839 ip-10-0-138-127 kubenswrapper[2581]: W0916 18:45:30.494685    2581 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 16 18:45:30.497839 ip-10-0-138-127 kubenswrapper[2581]: W0916 18:45:30.494688    2581 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 16 18:45:30.497839 ip-10-0-138-127 kubenswrapper[2581]: W0916 18:45:30.494691    2581 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 16 18:45:30.497839 ip-10-0-138-127 kubenswrapper[2581]: W0916 18:45:30.494694    2581 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 16 18:45:30.497839 ip-10-0-138-127 kubenswrapper[2581]: W0916 18:45:30.494696    2581 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 16 18:45:30.497839 ip-10-0-138-127 kubenswrapper[2581]: W0916 18:45:30.494699    2581 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 16 18:45:30.497839 ip-10-0-138-127 kubenswrapper[2581]: W0916 18:45:30.494701    2581 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 16 18:45:30.497839 ip-10-0-138-127 kubenswrapper[2581]: W0916 18:45:30.494704    2581 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 16 18:45:30.497839 ip-10-0-138-127 kubenswrapper[2581]: W0916 18:45:30.494707    2581 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 16 18:45:30.497839 ip-10-0-138-127 kubenswrapper[2581]: W0916 18:45:30.494710    2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 16 18:45:30.497839 ip-10-0-138-127 kubenswrapper[2581]: W0916 18:45:30.494712    2581 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 16 18:45:30.497839 ip-10-0-138-127 kubenswrapper[2581]: W0916 18:45:30.494715    2581 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 16 18:45:30.497839 ip-10-0-138-127 kubenswrapper[2581]: W0916 18:45:30.494718    2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 16 18:45:30.497839 ip-10-0-138-127 kubenswrapper[2581]: W0916 18:45:30.494721    2581 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 16 18:45:30.497839 ip-10-0-138-127 kubenswrapper[2581]: W0916 18:45:30.494724    2581 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 16 18:45:30.497839 ip-10-0-138-127 kubenswrapper[2581]: W0916 18:45:30.494726    2581 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 16 18:45:30.497839 ip-10-0-138-127 kubenswrapper[2581]: W0916 18:45:30.494730    2581 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 16 18:45:30.497839 ip-10-0-138-127 kubenswrapper[2581]: W0916 18:45:30.494732    2581 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 16 18:45:30.498346 ip-10-0-138-127 kubenswrapper[2581]: W0916 18:45:30.494735    2581 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 16 18:45:30.498346 ip-10-0-138-127 kubenswrapper[2581]: W0916 18:45:30.494737    2581 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 16 18:45:30.498346 ip-10-0-138-127 kubenswrapper[2581]: W0916 18:45:30.494741    2581 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 16 18:45:30.498346 ip-10-0-138-127 kubenswrapper[2581]: W0916 18:45:30.494745    2581 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 16 18:45:30.498346 ip-10-0-138-127 kubenswrapper[2581]: W0916 18:45:30.494748    2581 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 16 18:45:30.498346 ip-10-0-138-127 kubenswrapper[2581]: W0916 18:45:30.494751    2581 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 16 18:45:30.498346 ip-10-0-138-127 kubenswrapper[2581]: W0916 18:45:30.494754    2581 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 16 18:45:30.498346 ip-10-0-138-127 kubenswrapper[2581]: W0916 18:45:30.494757    2581 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 16 18:45:30.498346 ip-10-0-138-127 kubenswrapper[2581]: W0916 18:45:30.494760    2581 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 16 18:45:30.498346 ip-10-0-138-127 kubenswrapper[2581]: W0916 18:45:30.494762    2581 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 16 18:45:30.498346 ip-10-0-138-127 kubenswrapper[2581]: W0916 18:45:30.494764    2581 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 16 18:45:30.498346 ip-10-0-138-127 kubenswrapper[2581]: W0916 18:45:30.494767    2581 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 16 18:45:30.498346 ip-10-0-138-127 kubenswrapper[2581]: W0916 18:45:30.494770    2581 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 16 18:45:30.498346 ip-10-0-138-127 kubenswrapper[2581]: W0916 18:45:30.494772    2581 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 16 18:45:30.498346 ip-10-0-138-127 kubenswrapper[2581]: W0916 18:45:30.494775    2581 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 16 18:45:30.498346 ip-10-0-138-127 kubenswrapper[2581]: W0916 18:45:30.494777    2581 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 16 18:45:30.498346 ip-10-0-138-127 kubenswrapper[2581]: W0916 18:45:30.494780    2581 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 16 18:45:30.498346 ip-10-0-138-127 kubenswrapper[2581]: W0916 18:45:30.494782    2581 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 16 18:45:30.498346 ip-10-0-138-127 kubenswrapper[2581]: W0916 18:45:30.494785    2581 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 16 18:45:30.498831 ip-10-0-138-127 kubenswrapper[2581]: W0916 18:45:30.494787    2581 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 16 18:45:30.498831 ip-10-0-138-127 kubenswrapper[2581]: W0916 18:45:30.494790    2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 16 18:45:30.498831 ip-10-0-138-127 kubenswrapper[2581]: W0916 18:45:30.494792    2581 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 16 18:45:30.498831 ip-10-0-138-127 kubenswrapper[2581]: W0916 18:45:30.494795    2581 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 16 18:45:30.498831 ip-10-0-138-127 kubenswrapper[2581]: W0916 18:45:30.494797    2581 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 16 18:45:30.498831 ip-10-0-138-127 kubenswrapper[2581]: W0916 18:45:30.494799    2581 feature_gate.go:328] unrecognized feature gate: Example
Sep 16 18:45:30.498831 ip-10-0-138-127 kubenswrapper[2581]: W0916 18:45:30.494802    2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 16 18:45:30.498831 ip-10-0-138-127 kubenswrapper[2581]: W0916 18:45:30.494804    2581 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 16 18:45:30.498831 ip-10-0-138-127 kubenswrapper[2581]: W0916 18:45:30.494808    2581 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 16 18:45:30.498831 ip-10-0-138-127 kubenswrapper[2581]: W0916 18:45:30.494812    2581 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 16 18:45:30.498831 ip-10-0-138-127 kubenswrapper[2581]: W0916 18:45:30.494815    2581 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 16 18:45:30.498831 ip-10-0-138-127 kubenswrapper[2581]: W0916 18:45:30.494818    2581 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 16 18:45:30.498831 ip-10-0-138-127 kubenswrapper[2581]: W0916 18:45:30.494821    2581 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 16 18:45:30.498831 ip-10-0-138-127 kubenswrapper[2581]: W0916 18:45:30.494824    2581 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 16 18:45:30.498831 ip-10-0-138-127 kubenswrapper[2581]: W0916 18:45:30.494826    2581 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 16 18:45:30.498831 ip-10-0-138-127 kubenswrapper[2581]: W0916 18:45:30.494829    2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 16 18:45:30.498831 ip-10-0-138-127 kubenswrapper[2581]: W0916 18:45:30.494832    2581 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 16 18:45:30.498831 ip-10-0-138-127 kubenswrapper[2581]: W0916 18:45:30.494834    2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 16 18:45:30.498831 ip-10-0-138-127 kubenswrapper[2581]: W0916 18:45:30.494838    2581 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 16 18:45:30.499314 ip-10-0-138-127 kubenswrapper[2581]: W0916 18:45:30.494840    2581 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 16 18:45:30.499314 ip-10-0-138-127 kubenswrapper[2581]: W0916 18:45:30.494843    2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 16 18:45:30.499314 ip-10-0-138-127 kubenswrapper[2581]: W0916 18:45:30.494845    2581 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 16 18:45:30.499314 ip-10-0-138-127 kubenswrapper[2581]: W0916 18:45:30.494848    2581 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 16 18:45:30.499314 ip-10-0-138-127 kubenswrapper[2581]: W0916 18:45:30.494851    2581 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 16 18:45:30.499314 ip-10-0-138-127 kubenswrapper[2581]: W0916 18:45:30.494853    2581 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 16 18:45:30.499314 ip-10-0-138-127 kubenswrapper[2581]: W0916 18:45:30.494856    2581 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 16 18:45:30.499314 ip-10-0-138-127 kubenswrapper[2581]: W0916 18:45:30.494858    2581 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 16 18:45:30.499314 ip-10-0-138-127 kubenswrapper[2581]: W0916 18:45:30.494860    2581 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 16 18:45:30.499314 ip-10-0-138-127 kubenswrapper[2581]: W0916 18:45:30.494863    2581 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 16 18:45:30.499314 ip-10-0-138-127 kubenswrapper[2581]: W0916 18:45:30.494865    2581 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 16 18:45:30.499314 ip-10-0-138-127 kubenswrapper[2581]: W0916 18:45:30.494868    2581 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 16 18:45:30.499314 ip-10-0-138-127 kubenswrapper[2581]: W0916 18:45:30.494870    2581 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 16 18:45:30.499314 ip-10-0-138-127 kubenswrapper[2581]: W0916 18:45:30.494873    2581 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 16 18:45:30.499314 ip-10-0-138-127 kubenswrapper[2581]: W0916 18:45:30.494875    2581 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 16 18:45:30.499314 ip-10-0-138-127 kubenswrapper[2581]: W0916 18:45:30.494877    2581 feature_gate.go:328] unrecognized feature gate: Example2
Sep 16 18:45:30.499314 ip-10-0-138-127 kubenswrapper[2581]: W0916 18:45:30.494880    2581 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 16 18:45:30.499314 ip-10-0-138-127 kubenswrapper[2581]: W0916 18:45:30.494882    2581 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 16 18:45:30.499314 ip-10-0-138-127 kubenswrapper[2581]: W0916 18:45:30.494885    2581 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 16 18:45:30.499314 ip-10-0-138-127 kubenswrapper[2581]: W0916 18:45:30.494888    2581 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 16 18:45:30.499800 ip-10-0-138-127 kubenswrapper[2581]: W0916 18:45:30.494891    2581 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 16 18:45:30.499800 ip-10-0-138-127 kubenswrapper[2581]: W0916 18:45:30.494895    2581 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 16 18:45:30.499800 ip-10-0-138-127 kubenswrapper[2581]: W0916 18:45:30.494897    2581 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 16 18:45:30.499800 ip-10-0-138-127 kubenswrapper[2581]: W0916 18:45:30.494901    2581 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 16 18:45:30.499800 ip-10-0-138-127 kubenswrapper[2581]: W0916 18:45:30.494905    2581 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 16 18:45:30.499800 ip-10-0-138-127 kubenswrapper[2581]: W0916 18:45:30.494908    2581 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 16 18:45:30.499800 ip-10-0-138-127 kubenswrapper[2581]: W0916 18:45:30.494921    2581 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 16 18:45:30.499800 ip-10-0-138-127 kubenswrapper[2581]: W0916 18:45:30.494924    2581 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 16 18:45:30.499800 ip-10-0-138-127 kubenswrapper[2581]: W0916 18:45:30.494927    2581 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 16 18:45:30.499800 ip-10-0-138-127 kubenswrapper[2581]: W0916 18:45:30.494929    2581 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 16 18:45:30.499800 ip-10-0-138-127 kubenswrapper[2581]: W0916 18:45:30.494932    2581 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 16 18:45:30.499800 ip-10-0-138-127 kubenswrapper[2581]: W0916 18:45:30.494935    2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 16 18:45:30.499800 ip-10-0-138-127 kubenswrapper[2581]: W0916 18:45:30.494938    2581 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 16 18:45:30.499800 ip-10-0-138-127 kubenswrapper[2581]: W0916 18:45:30.494940    2581 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 16 18:45:30.499800 ip-10-0-138-127 kubenswrapper[2581]: W0916 18:45:30.494943    2581 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 16 18:45:30.499800 ip-10-0-138-127 kubenswrapper[2581]: W0916 18:45:30.494945    2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 16 18:45:30.499800 ip-10-0-138-127 kubenswrapper[2581]: W0916 18:45:30.494948    2581 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 16 18:45:30.499800 ip-10-0-138-127 kubenswrapper[2581]: W0916 18:45:30.494950    2581 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 16 18:45:30.499800 ip-10-0-138-127 kubenswrapper[2581]: W0916 18:45:30.494953    2581 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 16 18:45:30.499800 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:45:30.495032    2581 flags.go:64] FLAG: --address="0.0.0.0"
Sep 16 18:45:30.500366 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:45:30.495069    2581 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Sep 16 18:45:30.500366 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:45:30.495075    2581 flags.go:64] FLAG: --anonymous-auth="true"
Sep 16 18:45:30.500366 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:45:30.495080    2581 flags.go:64] FLAG: --application-metrics-count-limit="100"
Sep 16 18:45:30.500366 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:45:30.495084    2581 flags.go:64] FLAG: --authentication-token-webhook="false"
Sep 16 18:45:30.500366 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:45:30.495087    2581 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Sep 16 18:45:30.500366 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:45:30.495092    2581 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Sep 16 18:45:30.500366 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:45:30.495097    2581 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Sep 16 18:45:30.500366 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:45:30.495100    2581 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Sep 16 18:45:30.500366 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:45:30.495103    2581 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Sep 16 18:45:30.500366 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:45:30.495107    2581 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Sep 16 18:45:30.500366 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:45:30.495110    2581 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Sep 16 18:45:30.500366 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:45:30.495113    2581 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Sep 16 18:45:30.500366 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:45:30.495116    2581 flags.go:64] FLAG: --cgroup-root=""
Sep 16 18:45:30.500366 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:45:30.495119    2581 flags.go:64] FLAG: --cgroups-per-qos="true"
Sep 16 18:45:30.500366 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:45:30.495122    2581 flags.go:64] FLAG: --client-ca-file=""
Sep 16 18:45:30.500366 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:45:30.495127    2581 flags.go:64] FLAG: --cloud-provider="external"
Sep 16 18:45:30.500366 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:45:30.495130    2581 flags.go:64] FLAG: --cluster-dns="[]"
Sep 16 18:45:30.500366 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:45:30.495134    2581 flags.go:64] FLAG: --cluster-domain=""
Sep 16 18:45:30.500366 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:45:30.495136    2581 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Sep 16 18:45:30.500366 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:45:30.495139    2581 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Sep 16 18:45:30.500366 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:45:30.495142    2581 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Sep 16 18:45:30.500366 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:45:30.495146    2581 flags.go:64] FLAG: --container-log-max-files="5"
Sep 16 18:45:30.500366 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:45:30.495149    2581 flags.go:64] FLAG: --container-log-max-size="10Mi"
Sep 16 18:45:30.500366 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:45:30.495155    2581 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Sep 16 18:45:30.500936 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:45:30.495158    2581 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Sep 16 18:45:30.500936 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:45:30.495161    2581 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Sep 16 18:45:30.500936 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:45:30.495164    2581 flags.go:64] FLAG: --contention-profiling="false"
Sep 16 18:45:30.500936 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:45:30.495167    2581 flags.go:64] FLAG: --cpu-cfs-quota="true"
Sep 16 18:45:30.500936 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:45:30.495170    2581 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Sep 16 18:45:30.500936 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:45:30.495173    2581 flags.go:64] FLAG: --cpu-manager-policy="none"
Sep 16 18:45:30.500936 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:45:30.495175    2581 flags.go:64] FLAG: --cpu-manager-policy-options=""
Sep 16 18:45:30.500936 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:45:30.495179    2581 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Sep 16 18:45:30.500936 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:45:30.495182    2581 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Sep 16 18:45:30.500936 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:45:30.495185    2581 flags.go:64] FLAG: --enable-debugging-handlers="true"
Sep 16 18:45:30.500936 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:45:30.495188    2581 flags.go:64] FLAG: --enable-load-reader="false"
Sep 16 18:45:30.500936 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:45:30.495205    2581 flags.go:64] FLAG: --enable-server="true"
Sep 16 18:45:30.500936 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:45:30.495208    2581 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Sep 16 18:45:30.500936 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:45:30.495213    2581 flags.go:64] FLAG: --event-burst="100"
Sep 16 18:45:30.500936 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:45:30.495217    2581 flags.go:64] FLAG: --event-qps="50"
Sep 16 18:45:30.500936 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:45:30.495220    2581 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Sep 16 18:45:30.500936 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:45:30.495223    2581 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Sep 16 18:45:30.500936 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:45:30.495226    2581 flags.go:64] FLAG: --eviction-hard=""
Sep 16 18:45:30.500936 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:45:30.495230    2581 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Sep 16 18:45:30.500936 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:45:30.495233    2581 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Sep 16 18:45:30.500936 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:45:30.495235    2581 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Sep 16 18:45:30.500936 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:45:30.495239    2581 flags.go:64] FLAG: --eviction-soft=""
Sep 16 18:45:30.500936 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:45:30.495241    2581 flags.go:64] FLAG: --eviction-soft-grace-period=""
Sep 16 18:45:30.500936 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:45:30.495244    2581 flags.go:64] FLAG: --exit-on-lock-contention="false"
Sep 16 18:45:30.500936 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:45:30.495247    2581 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Sep 16 18:45:30.501577 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:45:30.495250    2581 flags.go:64] FLAG: --experimental-mounter-path=""
Sep 16 18:45:30.501577 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:45:30.495253    2581 flags.go:64] FLAG: --fail-cgroupv1="false"
Sep 16 18:45:30.501577 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:45:30.495256    2581 flags.go:64] FLAG: --fail-swap-on="true"
Sep 16 18:45:30.501577 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:45:30.495259    2581 flags.go:64] FLAG: --feature-gates=""
Sep 16 18:45:30.501577 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:45:30.495262    2581 flags.go:64] FLAG: --file-check-frequency="20s"
Sep 16 18:45:30.501577 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:45:30.495265    2581 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Sep 16 18:45:30.501577 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:45:30.495270    2581 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Sep 16 18:45:30.501577 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:45:30.495273    2581 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Sep 16 18:45:30.501577 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:45:30.495276    2581 flags.go:64] FLAG: --healthz-port="10248"
Sep 16 18:45:30.501577 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:45:30.495279    2581 flags.go:64] FLAG: --help="false"
Sep 16 18:45:30.501577 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:45:30.495281    2581 flags.go:64] FLAG: --hostname-override="ip-10-0-138-127.ec2.internal"
Sep 16 18:45:30.501577 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:45:30.495284    2581 flags.go:64] FLAG: --housekeeping-interval="10s"
Sep 16 18:45:30.501577 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:45:30.495287    2581 flags.go:64] FLAG: --http-check-frequency="20s"
Sep 16 18:45:30.501577 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:45:30.495290    2581 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Sep 16 18:45:30.501577 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:45:30.495293    2581 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Sep 16 18:45:30.501577 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:45:30.495296    2581 flags.go:64] FLAG: --image-gc-high-threshold="85"
Sep 16 18:45:30.501577 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:45:30.495299    2581 flags.go:64] FLAG: --image-gc-low-threshold="80"
Sep 16 18:45:30.501577 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:45:30.495302    2581 flags.go:64] FLAG: --image-service-endpoint=""
Sep 16 18:45:30.501577 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:45:30.495305    2581 flags.go:64] FLAG: --kernel-memcg-notification="false"
Sep 16 18:45:30.501577 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:45:30.495307    2581 flags.go:64] FLAG: --kube-api-burst="100"
Sep 16 18:45:30.501577 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:45:30.495310    2581 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Sep 16 18:45:30.501577 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:45:30.495313    2581 flags.go:64] FLAG: --kube-api-qps="50"
Sep 16 18:45:30.501577 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:45:30.495316    2581 flags.go:64] FLAG: --kube-reserved=""
Sep 16 18:45:30.501577 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:45:30.495318    2581 flags.go:64] FLAG: --kube-reserved-cgroup=""
Sep 16 18:45:30.502294 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:45:30.495321    2581 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Sep 16 18:45:30.502294 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:45:30.495324    2581 flags.go:64] FLAG: --kubelet-cgroups=""
Sep 16 18:45:30.502294 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:45:30.495331    2581 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Sep 16 18:45:30.502294 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:45:30.495333    2581 flags.go:64] FLAG: --lock-file=""
Sep 16 18:45:30.502294 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:45:30.495336    2581 flags.go:64] FLAG: --log-cadvisor-usage="false"
Sep 16 18:45:30.502294 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:45:30.495339    2581 flags.go:64] FLAG: --log-flush-frequency="5s"
Sep 16 18:45:30.502294 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:45:30.495342    2581 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Sep 16 18:45:30.502294 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:45:30.495347    2581 flags.go:64] FLAG: --log-json-split-stream="false"
Sep 16 18:45:30.502294 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:45:30.495350    2581 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Sep 16 18:45:30.502294 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:45:30.495353    2581 flags.go:64] FLAG: --log-text-split-stream="false"
Sep 16 18:45:30.502294 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:45:30.495356    2581 flags.go:64] FLAG: --logging-format="text"
Sep 16 18:45:30.502294 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:45:30.495359    2581 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Sep 16 18:45:30.502294 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:45:30.495362    2581 flags.go:64] FLAG: --make-iptables-util-chains="true"
Sep 16 18:45:30.502294 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:45:30.495364    2581 flags.go:64] FLAG: --manifest-url=""
Sep 16 18:45:30.502294 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:45:30.495369    2581 flags.go:64] FLAG: --manifest-url-header=""
Sep 16 18:45:30.502294 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:45:30.495373    2581 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Sep 16 18:45:30.502294 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:45:30.495376    2581 flags.go:64] FLAG: --max-open-files="1000000"
Sep 16 18:45:30.502294 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:45:30.495380    2581 flags.go:64] FLAG: --max-pods="110"
Sep 16 18:45:30.502294 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:45:30.495383    2581 flags.go:64] FLAG: --maximum-dead-containers="-1"
Sep 16 18:45:30.502294 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:45:30.495386    2581 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Sep 16 18:45:30.502294 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:45:30.495389    2581 flags.go:64] FLAG: --memory-manager-policy="None"
Sep 16 18:45:30.502294 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:45:30.495392    2581 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Sep 16 18:45:30.502294 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:45:30.495394    2581 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Sep 16 18:45:30.502294 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:45:30.495397    2581 flags.go:64] FLAG: --node-ip="0.0.0.0"
Sep 16 18:45:30.502294 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:45:30.495400    2581 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Sep 16 18:45:30.502933 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:45:30.495407    2581 flags.go:64] FLAG: --node-status-max-images="50"
Sep 16 18:45:30.502933 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:45:30.495410    2581 flags.go:64] FLAG: --node-status-update-frequency="10s"
Sep 16 18:45:30.502933 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:45:30.495413    2581 flags.go:64] FLAG: --oom-score-adj="-999"
Sep 16 18:45:30.502933 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:45:30.495416    2581 flags.go:64] FLAG: --pod-cidr=""
Sep 16 18:45:30.502933 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:45:30.495419    2581 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:eee392dbe3ac2ed4a38bca29ae3bd486f5b7780406c62c600154cd1906871b10"
Sep 16 18:45:30.502933 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:45:30.495424    2581 flags.go:64] FLAG: --pod-manifest-path=""
Sep 16 18:45:30.502933 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:45:30.495427    2581 flags.go:64] FLAG: --pod-max-pids="-1"
Sep 16 18:45:30.502933 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:45:30.495430    2581 flags.go:64] FLAG: --pods-per-core="0"
Sep 16 18:45:30.502933 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:45:30.495433    2581 flags.go:64] FLAG: --port="10250"
Sep 16 18:45:30.502933 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:45:30.495436    2581 flags.go:64] FLAG: --protect-kernel-defaults="false"
Sep 16 18:45:30.502933 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:45:30.495440    2581 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-0e3f8ef7a35a9bbdb"
Sep 16 18:45:30.502933 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:45:30.495443    2581 flags.go:64] FLAG: --qos-reserved=""
Sep 16 18:45:30.502933 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:45:30.495446    2581 flags.go:64] FLAG: --read-only-port="10255"
Sep 16 18:45:30.502933 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:45:30.495449    2581 flags.go:64] FLAG: --register-node="true"
Sep 16 18:45:30.502933 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:45:30.495452    2581 flags.go:64] FLAG: --register-with-taints=""
Sep 16 18:45:30.502933 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:45:30.495455    2581 flags.go:64] FLAG: --registry-burst="10"
Sep 16 18:45:30.502933 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:45:30.495459    2581 flags.go:64] FLAG: --registry-qps="5"
Sep 16 18:45:30.502933 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:45:30.495461    2581 flags.go:64] FLAG: --reserved-cpus=""
Sep 16 18:45:30.502933 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:45:30.495464    2581 flags.go:64] FLAG: --reserved-memory=""
Sep 16 18:45:30.502933 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:45:30.495468    2581 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Sep 16 18:45:30.502933 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:45:30.495470    2581 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Sep 16 18:45:30.502933 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:45:30.495475    2581 flags.go:64] FLAG: --rotate-certificates="false"
Sep 16 18:45:30.502933 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:45:30.495478    2581 flags.go:64] FLAG: --rotate-server-certificates="false"
Sep 16 18:45:30.502933 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:45:30.495480    2581 flags.go:64] FLAG: --runonce="false"
Sep 16 18:45:30.502933 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:45:30.495483    2581 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Sep 16 18:45:30.503604 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:45:30.495486    2581 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Sep 16 18:45:30.503604 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:45:30.495489    2581 flags.go:64] FLAG: --seccomp-default="false"
Sep 16 18:45:30.503604 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:45:30.495491    2581 flags.go:64] FLAG: --serialize-image-pulls="true"
Sep 16 18:45:30.503604 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:45:30.495494    2581 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Sep 16 18:45:30.503604 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:45:30.495497    2581 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Sep 16 18:45:30.503604 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:45:30.495500    2581 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Sep 16 18:45:30.503604 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:45:30.495503    2581 flags.go:64] FLAG: --storage-driver-password="root"
Sep 16 18:45:30.503604 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:45:30.495505    2581 flags.go:64] FLAG: --storage-driver-secure="false"
Sep 16 18:45:30.503604 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:45:30.495508    2581 flags.go:64] FLAG: --storage-driver-table="stats"
Sep 16 18:45:30.503604 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:45:30.495511    2581 flags.go:64] FLAG: --storage-driver-user="root"
Sep 16 18:45:30.503604 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:45:30.495514    2581 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Sep 16 18:45:30.503604 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:45:30.495517    2581 flags.go:64] FLAG: --sync-frequency="1m0s"
Sep 16 18:45:30.503604 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:45:30.495519    2581 flags.go:64] FLAG: --system-cgroups=""
Sep 16 18:45:30.503604 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:45:30.495522    2581 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Sep 16 18:45:30.503604 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:45:30.495527    2581 flags.go:64] FLAG: --system-reserved-cgroup=""
Sep 16 18:45:30.503604 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:45:30.495530    2581 flags.go:64] FLAG: --tls-cert-file=""
Sep 16 18:45:30.503604 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:45:30.495533    2581 flags.go:64] FLAG: --tls-cipher-suites="[]"
Sep 16 18:45:30.503604 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:45:30.495538    2581 flags.go:64] FLAG: --tls-min-version=""
Sep 16 18:45:30.503604 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:45:30.495541    2581 flags.go:64] FLAG: --tls-private-key-file=""
Sep 16 18:45:30.503604 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:45:30.495544    2581 flags.go:64] FLAG: --topology-manager-policy="none"
Sep 16 18:45:30.503604 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:45:30.495546    2581 flags.go:64] FLAG: --topology-manager-policy-options=""
Sep 16 18:45:30.503604 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:45:30.495549    2581 flags.go:64] FLAG: --topology-manager-scope="container"
Sep 16 18:45:30.503604 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:45:30.495552    2581 flags.go:64] FLAG: --v="2"
Sep 16 18:45:30.503604 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:45:30.495556    2581 flags.go:64] FLAG: --version="false"
Sep 16 18:45:30.503604 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:45:30.495561    2581 flags.go:64] FLAG: --vmodule=""
Sep 16 18:45:30.504232 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:45:30.495564    2581 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Sep 16 18:45:30.504232 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:45:30.495567    2581 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Sep 16 18:45:30.504232 ip-10-0-138-127 kubenswrapper[2581]: W0916 18:45:30.495680    2581 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 16 18:45:30.504232 ip-10-0-138-127 kubenswrapper[2581]: W0916 18:45:30.495685    2581 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 16 18:45:30.504232 ip-10-0-138-127 kubenswrapper[2581]: W0916 18:45:30.495688    2581 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 16 18:45:30.504232 ip-10-0-138-127 kubenswrapper[2581]: W0916 18:45:30.495691    2581 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 16 18:45:30.504232 ip-10-0-138-127 kubenswrapper[2581]: W0916 18:45:30.495693    2581 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 16 18:45:30.504232 ip-10-0-138-127 kubenswrapper[2581]: W0916 18:45:30.495696    2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 16 18:45:30.504232 ip-10-0-138-127 kubenswrapper[2581]: W0916 18:45:30.495698    2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 16 18:45:30.504232 ip-10-0-138-127 kubenswrapper[2581]: W0916 18:45:30.495701    2581 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 16 18:45:30.504232 ip-10-0-138-127 kubenswrapper[2581]: W0916 18:45:30.495703    2581 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 16 18:45:30.504232 ip-10-0-138-127 kubenswrapper[2581]: W0916 18:45:30.495706    2581 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 16 18:45:30.504232 ip-10-0-138-127 kubenswrapper[2581]: W0916 18:45:30.495708    2581 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 16 18:45:30.504232 ip-10-0-138-127 kubenswrapper[2581]: W0916 18:45:30.495710    2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 16 18:45:30.504232 ip-10-0-138-127 kubenswrapper[2581]: W0916 18:45:30.495713    2581 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 16 18:45:30.504232 ip-10-0-138-127 kubenswrapper[2581]: W0916 18:45:30.495716    2581 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 16 18:45:30.504232 ip-10-0-138-127 kubenswrapper[2581]: W0916 18:45:30.495718    2581 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 16 18:45:30.504232 ip-10-0-138-127 kubenswrapper[2581]: W0916 18:45:30.495721    2581 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 16 18:45:30.504232 ip-10-0-138-127 kubenswrapper[2581]: W0916 18:45:30.495723    2581 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 16 18:45:30.504232 ip-10-0-138-127 kubenswrapper[2581]: W0916 18:45:30.495726    2581 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 16 18:45:30.504724 ip-10-0-138-127 kubenswrapper[2581]: W0916 18:45:30.495728    2581 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 16 18:45:30.504724 ip-10-0-138-127 kubenswrapper[2581]: W0916 18:45:30.495731    2581 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 16 18:45:30.504724 ip-10-0-138-127 kubenswrapper[2581]: W0916 18:45:30.495734    2581 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 16 18:45:30.504724 ip-10-0-138-127 kubenswrapper[2581]: W0916 18:45:30.495736    2581 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 16 18:45:30.504724 ip-10-0-138-127 kubenswrapper[2581]: W0916 18:45:30.495740    2581 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 16 18:45:30.504724 ip-10-0-138-127 kubenswrapper[2581]: W0916 18:45:30.495743    2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 16 18:45:30.504724 ip-10-0-138-127 kubenswrapper[2581]: W0916 18:45:30.495745    2581 feature_gate.go:328] unrecognized feature gate: Example2
Sep 16 18:45:30.504724 ip-10-0-138-127 kubenswrapper[2581]: W0916 18:45:30.495748    2581 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 16 18:45:30.504724 ip-10-0-138-127 kubenswrapper[2581]: W0916 18:45:30.495751    2581 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 16 18:45:30.504724 ip-10-0-138-127 kubenswrapper[2581]: W0916 18:45:30.495753    2581 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 16 18:45:30.504724 ip-10-0-138-127 kubenswrapper[2581]: W0916 18:45:30.495756    2581 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 16 18:45:30.504724 ip-10-0-138-127 kubenswrapper[2581]: W0916 18:45:30.495758    2581 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 16 18:45:30.504724 ip-10-0-138-127 kubenswrapper[2581]: W0916 18:45:30.495762    2581 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 16 18:45:30.504724 ip-10-0-138-127 kubenswrapper[2581]: W0916 18:45:30.495764    2581 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 16 18:45:30.504724 ip-10-0-138-127 kubenswrapper[2581]: W0916 18:45:30.495767    2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 16 18:45:30.504724 ip-10-0-138-127 kubenswrapper[2581]: W0916 18:45:30.495771    2581 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 16 18:45:30.504724 ip-10-0-138-127 kubenswrapper[2581]: W0916 18:45:30.495773    2581 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 16 18:45:30.504724 ip-10-0-138-127 kubenswrapper[2581]: W0916 18:45:30.495776    2581 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 16 18:45:30.504724 ip-10-0-138-127 kubenswrapper[2581]: W0916 18:45:30.495778    2581 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 16 18:45:30.504724 ip-10-0-138-127 kubenswrapper[2581]: W0916 18:45:30.495781    2581 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 16 18:45:30.505227 ip-10-0-138-127 kubenswrapper[2581]: W0916 18:45:30.495785    2581 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 16 18:45:30.505227 ip-10-0-138-127 kubenswrapper[2581]: W0916 18:45:30.495788    2581 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 16 18:45:30.505227 ip-10-0-138-127 kubenswrapper[2581]: W0916 18:45:30.495791    2581 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 16 18:45:30.505227 ip-10-0-138-127 kubenswrapper[2581]: W0916 18:45:30.495794    2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 16 18:45:30.505227 ip-10-0-138-127 kubenswrapper[2581]: W0916 18:45:30.495797    2581 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 16 18:45:30.505227 ip-10-0-138-127 kubenswrapper[2581]: W0916 18:45:30.495800    2581 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 16 18:45:30.505227 ip-10-0-138-127 kubenswrapper[2581]: W0916 18:45:30.495803    2581 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 16 18:45:30.505227 ip-10-0-138-127 kubenswrapper[2581]: W0916 18:45:30.495806    2581 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 16 18:45:30.505227 ip-10-0-138-127 kubenswrapper[2581]: W0916 18:45:30.495809    2581 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 16 18:45:30.505227 ip-10-0-138-127 kubenswrapper[2581]: W0916 18:45:30.495811    2581 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 16 18:45:30.505227 ip-10-0-138-127 kubenswrapper[2581]: W0916 18:45:30.495814    2581 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 16 18:45:30.505227 ip-10-0-138-127 kubenswrapper[2581]: W0916 18:45:30.495817    2581 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 16 18:45:30.505227 ip-10-0-138-127 kubenswrapper[2581]: W0916 18:45:30.495821    2581 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 16 18:45:30.505227 ip-10-0-138-127 kubenswrapper[2581]: W0916 18:45:30.495824    2581 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 16 18:45:30.505227 ip-10-0-138-127 kubenswrapper[2581]: W0916 18:45:30.495827    2581 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 16 18:45:30.505227 ip-10-0-138-127 kubenswrapper[2581]: W0916 18:45:30.495831    2581 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 16 18:45:30.505227 ip-10-0-138-127 kubenswrapper[2581]: W0916 18:45:30.495836    2581 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 16 18:45:30.505227 ip-10-0-138-127 kubenswrapper[2581]: W0916 18:45:30.495839    2581 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 16 18:45:30.505701 ip-10-0-138-127 kubenswrapper[2581]: W0916 18:45:30.495843    2581 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 16 18:45:30.505701 ip-10-0-138-127 kubenswrapper[2581]: W0916 18:45:30.495845    2581 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 16 18:45:30.505701 ip-10-0-138-127 kubenswrapper[2581]: W0916 18:45:30.495848    2581 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 16 18:45:30.505701 ip-10-0-138-127 kubenswrapper[2581]: W0916 18:45:30.495851    2581 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 16 18:45:30.505701 ip-10-0-138-127 kubenswrapper[2581]: W0916 18:45:30.495853    2581 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 16 18:45:30.505701 ip-10-0-138-127 kubenswrapper[2581]: W0916 18:45:30.495856    2581 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 16 18:45:30.505701 ip-10-0-138-127 kubenswrapper[2581]: W0916 18:45:30.495858    2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 16 18:45:30.505701 ip-10-0-138-127 kubenswrapper[2581]: W0916 18:45:30.495861    2581 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 16 18:45:30.505701 ip-10-0-138-127 kubenswrapper[2581]: W0916 18:45:30.495864    2581 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 16 18:45:30.505701 ip-10-0-138-127 kubenswrapper[2581]: W0916 18:45:30.495867    2581 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 16 18:45:30.505701 ip-10-0-138-127 kubenswrapper[2581]: W0916 18:45:30.495871    2581 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 16 18:45:30.505701 ip-10-0-138-127 kubenswrapper[2581]: W0916 18:45:30.495873    2581 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 16 18:45:30.505701 ip-10-0-138-127 kubenswrapper[2581]: W0916 18:45:30.495876    2581 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 16 18:45:30.505701 ip-10-0-138-127 kubenswrapper[2581]: W0916 18:45:30.495878    2581 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 16 18:45:30.505701 ip-10-0-138-127 kubenswrapper[2581]: W0916 18:45:30.495881    2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 16 18:45:30.505701 ip-10-0-138-127 kubenswrapper[2581]: W0916 18:45:30.495884    2581 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 16 18:45:30.505701 ip-10-0-138-127 kubenswrapper[2581]: W0916 18:45:30.495886    2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 16 18:45:30.505701 ip-10-0-138-127 kubenswrapper[2581]: W0916 18:45:30.495889    2581 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 16 18:45:30.505701 ip-10-0-138-127 kubenswrapper[2581]: W0916 18:45:30.495891    2581 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 16 18:45:30.506173 ip-10-0-138-127 kubenswrapper[2581]: W0916 18:45:30.495893    2581 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 16 18:45:30.506173 ip-10-0-138-127 kubenswrapper[2581]: W0916 18:45:30.495896    2581 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 16 18:45:30.506173 ip-10-0-138-127 kubenswrapper[2581]: W0916 18:45:30.495898    2581 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 16 18:45:30.506173 ip-10-0-138-127 kubenswrapper[2581]: W0916 18:45:30.495901    2581 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 16 18:45:30.506173 ip-10-0-138-127 kubenswrapper[2581]: W0916 18:45:30.495903    2581 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 16 18:45:30.506173 ip-10-0-138-127 kubenswrapper[2581]: W0916 18:45:30.495906    2581 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 16 18:45:30.506173 ip-10-0-138-127 kubenswrapper[2581]: W0916 18:45:30.495908    2581 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 16 18:45:30.506173 ip-10-0-138-127 kubenswrapper[2581]: W0916 18:45:30.495911    2581 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 16 18:45:30.506173 ip-10-0-138-127 kubenswrapper[2581]: W0916 18:45:30.495913    2581 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 16 18:45:30.506173 ip-10-0-138-127 kubenswrapper[2581]: W0916 18:45:30.495915    2581 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 16 18:45:30.506173 ip-10-0-138-127 kubenswrapper[2581]: W0916 18:45:30.495918    2581 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 16 18:45:30.506173 ip-10-0-138-127 kubenswrapper[2581]: W0916 18:45:30.495922    2581 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 16 18:45:30.506173 ip-10-0-138-127 kubenswrapper[2581]: W0916 18:45:30.495924    2581 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 16 18:45:30.506173 ip-10-0-138-127 kubenswrapper[2581]: W0916 18:45:30.495926    2581 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 16 18:45:30.506173 ip-10-0-138-127 kubenswrapper[2581]: W0916 18:45:30.495929    2581 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 16 18:45:30.506173 ip-10-0-138-127 kubenswrapper[2581]: W0916 18:45:30.495931    2581 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 16 18:45:30.506173 ip-10-0-138-127 kubenswrapper[2581]: W0916 18:45:30.495934    2581 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 16 18:45:30.506173 ip-10-0-138-127 kubenswrapper[2581]: W0916 18:45:30.495936    2581 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 16 18:45:30.506173 ip-10-0-138-127 kubenswrapper[2581]: W0916 18:45:30.495939    2581 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 16 18:45:30.506173 ip-10-0-138-127 kubenswrapper[2581]: W0916 18:45:30.495941    2581 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 16 18:45:30.506682 ip-10-0-138-127 kubenswrapper[2581]: W0916 18:45:30.495944    2581 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 16 18:45:30.506682 ip-10-0-138-127 kubenswrapper[2581]: W0916 18:45:30.495953    2581 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 16 18:45:30.506682 ip-10-0-138-127 kubenswrapper[2581]: W0916 18:45:30.495957    2581 feature_gate.go:328] unrecognized feature gate: Example
Sep 16 18:45:30.506682 ip-10-0-138-127 kubenswrapper[2581]: W0916 18:45:30.495960    2581 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 16 18:45:30.506682 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:45:30.495965    2581 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 16 18:45:30.506682 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:45:30.504013    2581 server.go:529] "Kubelet version" kubeletVersion="v1.34.9"
Sep 16 18:45:30.506682 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:45:30.504032    2581 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Sep 16 18:45:30.506682 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:45:30.504052    2581 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Sep 16 18:45:30.506682 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:45:30.504058    2581 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Sep 16 18:45:30.506682 ip-10-0-138-127 kubenswrapper[2581]: W0916 18:45:30.504123    2581 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 16 18:45:30.506682 ip-10-0-138-127 kubenswrapper[2581]: W0916 18:45:30.504129    2581 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 16 18:45:30.506682 ip-10-0-138-127 kubenswrapper[2581]: W0916 18:45:30.504132    2581 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 16 18:45:30.506682 ip-10-0-138-127 kubenswrapper[2581]: W0916 18:45:30.504135    2581 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 16 18:45:30.506682 ip-10-0-138-127 kubenswrapper[2581]: W0916 18:45:30.504138    2581 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 16 18:45:30.506682 ip-10-0-138-127 kubenswrapper[2581]: W0916 18:45:30.504141    2581 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 16 18:45:30.507109 ip-10-0-138-127 kubenswrapper[2581]: W0916 18:45:30.504144    2581 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 16 18:45:30.507109 ip-10-0-138-127 kubenswrapper[2581]: W0916 18:45:30.504146    2581 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 16 18:45:30.507109 ip-10-0-138-127 kubenswrapper[2581]: W0916 18:45:30.504149    2581 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 16 18:45:30.507109 ip-10-0-138-127 kubenswrapper[2581]: W0916 18:45:30.504152    2581 feature_gate.go:328] unrecognized feature gate: Example2
Sep 16 18:45:30.507109 ip-10-0-138-127 kubenswrapper[2581]: W0916 18:45:30.504154    2581 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 16 18:45:30.507109 ip-10-0-138-127 kubenswrapper[2581]: W0916 18:45:30.504157    2581 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 16 18:45:30.507109 ip-10-0-138-127 kubenswrapper[2581]: W0916 18:45:30.504159    2581 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 16 18:45:30.507109 ip-10-0-138-127 kubenswrapper[2581]: W0916 18:45:30.504162    2581 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 16 18:45:30.507109 ip-10-0-138-127 kubenswrapper[2581]: W0916 18:45:30.504164    2581 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 16 18:45:30.507109 ip-10-0-138-127 kubenswrapper[2581]: W0916 18:45:30.504167    2581 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 16 18:45:30.507109 ip-10-0-138-127 kubenswrapper[2581]: W0916 18:45:30.504169    2581 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 16 18:45:30.507109 ip-10-0-138-127 kubenswrapper[2581]: W0916 18:45:30.504172    2581 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 16 18:45:30.507109 ip-10-0-138-127 kubenswrapper[2581]: W0916 18:45:30.504174    2581 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 16 18:45:30.507109 ip-10-0-138-127 kubenswrapper[2581]: W0916 18:45:30.504177    2581 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 16 18:45:30.507109 ip-10-0-138-127 kubenswrapper[2581]: W0916 18:45:30.504179    2581 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 16 18:45:30.507109 ip-10-0-138-127 kubenswrapper[2581]: W0916 18:45:30.504182    2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 16 18:45:30.507109 ip-10-0-138-127 kubenswrapper[2581]: W0916 18:45:30.504184    2581 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 16 18:45:30.507109 ip-10-0-138-127 kubenswrapper[2581]: W0916 18:45:30.504187    2581 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 16 18:45:30.507109 ip-10-0-138-127 kubenswrapper[2581]: W0916 18:45:30.504189    2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 16 18:45:30.507613 ip-10-0-138-127 kubenswrapper[2581]: W0916 18:45:30.504208    2581 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 16 18:45:30.507613 ip-10-0-138-127 kubenswrapper[2581]: W0916 18:45:30.504212    2581 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 16 18:45:30.507613 ip-10-0-138-127 kubenswrapper[2581]: W0916 18:45:30.504215    2581 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 16 18:45:30.507613 ip-10-0-138-127 kubenswrapper[2581]: W0916 18:45:30.504218    2581 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 16 18:45:30.507613 ip-10-0-138-127 kubenswrapper[2581]: W0916 18:45:30.504223    2581 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 16 18:45:30.507613 ip-10-0-138-127 kubenswrapper[2581]: W0916 18:45:30.504226    2581 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 16 18:45:30.507613 ip-10-0-138-127 kubenswrapper[2581]: W0916 18:45:30.504229    2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 16 18:45:30.507613 ip-10-0-138-127 kubenswrapper[2581]: W0916 18:45:30.504232    2581 feature_gate.go:328] unrecognized feature gate: Example
Sep 16 18:45:30.507613 ip-10-0-138-127 kubenswrapper[2581]: W0916 18:45:30.504235    2581 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 16 18:45:30.507613 ip-10-0-138-127 kubenswrapper[2581]: W0916 18:45:30.504238    2581 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 16 18:45:30.507613 ip-10-0-138-127 kubenswrapper[2581]: W0916 18:45:30.504240    2581 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 16 18:45:30.507613 ip-10-0-138-127 kubenswrapper[2581]: W0916 18:45:30.504243    2581 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 16 18:45:30.507613 ip-10-0-138-127 kubenswrapper[2581]: W0916 18:45:30.504246    2581 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 16 18:45:30.507613 ip-10-0-138-127 kubenswrapper[2581]: W0916 18:45:30.504248    2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 16 18:45:30.507613 ip-10-0-138-127 kubenswrapper[2581]: W0916 18:45:30.504251    2581 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 16 18:45:30.507613 ip-10-0-138-127 kubenswrapper[2581]: W0916 18:45:30.504253    2581 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 16 18:45:30.507613 ip-10-0-138-127 kubenswrapper[2581]: W0916 18:45:30.504256    2581 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 16 18:45:30.507613 ip-10-0-138-127 kubenswrapper[2581]: W0916 18:45:30.504259    2581 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 16 18:45:30.507613 ip-10-0-138-127 kubenswrapper[2581]: W0916 18:45:30.504262    2581 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 16 18:45:30.507613 ip-10-0-138-127 kubenswrapper[2581]: W0916 18:45:30.504264    2581 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 16 18:45:30.508092 ip-10-0-138-127 kubenswrapper[2581]: W0916 18:45:30.504267    2581 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 16 18:45:30.508092 ip-10-0-138-127 kubenswrapper[2581]: W0916 18:45:30.504269    2581 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 16 18:45:30.508092 ip-10-0-138-127 kubenswrapper[2581]: W0916 18:45:30.504272    2581 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 16 18:45:30.508092 ip-10-0-138-127 kubenswrapper[2581]: W0916 18:45:30.504275    2581 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 16 18:45:30.508092 ip-10-0-138-127 kubenswrapper[2581]: W0916 18:45:30.504277    2581 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 16 18:45:30.508092 ip-10-0-138-127 kubenswrapper[2581]: W0916 18:45:30.504280    2581 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 16 18:45:30.508092 ip-10-0-138-127 kubenswrapper[2581]: W0916 18:45:30.504282    2581 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 16 18:45:30.508092 ip-10-0-138-127 kubenswrapper[2581]: W0916 18:45:30.504285    2581 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 16 18:45:30.508092 ip-10-0-138-127 kubenswrapper[2581]: W0916 18:45:30.504287    2581 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 16 18:45:30.508092 ip-10-0-138-127 kubenswrapper[2581]: W0916 18:45:30.504290    2581 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 16 18:45:30.508092 ip-10-0-138-127 kubenswrapper[2581]: W0916 18:45:30.504292    2581 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 16 18:45:30.508092 ip-10-0-138-127 kubenswrapper[2581]: W0916 18:45:30.504295    2581 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 16 18:45:30.508092 ip-10-0-138-127 kubenswrapper[2581]: W0916 18:45:30.504297    2581 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 16 18:45:30.508092 ip-10-0-138-127 kubenswrapper[2581]: W0916 18:45:30.504300    2581 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 16 18:45:30.508092 ip-10-0-138-127 kubenswrapper[2581]: W0916 18:45:30.504302    2581 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 16 18:45:30.508092 ip-10-0-138-127 kubenswrapper[2581]: W0916 18:45:30.504305    2581 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 16 18:45:30.508092 ip-10-0-138-127 kubenswrapper[2581]: W0916 18:45:30.504308    2581 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 16 18:45:30.508092 ip-10-0-138-127 kubenswrapper[2581]: W0916 18:45:30.504311    2581 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 16 18:45:30.508092 ip-10-0-138-127 kubenswrapper[2581]: W0916 18:45:30.504315    2581 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 16 18:45:30.508092 ip-10-0-138-127 kubenswrapper[2581]: W0916 18:45:30.504319    2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 16 18:45:30.508610 ip-10-0-138-127 kubenswrapper[2581]: W0916 18:45:30.504323    2581 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 16 18:45:30.508610 ip-10-0-138-127 kubenswrapper[2581]: W0916 18:45:30.504326    2581 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 16 18:45:30.508610 ip-10-0-138-127 kubenswrapper[2581]: W0916 18:45:30.504328    2581 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 16 18:45:30.508610 ip-10-0-138-127 kubenswrapper[2581]: W0916 18:45:30.504331    2581 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 16 18:45:30.508610 ip-10-0-138-127 kubenswrapper[2581]: W0916 18:45:30.504333    2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 16 18:45:30.508610 ip-10-0-138-127 kubenswrapper[2581]: W0916 18:45:30.504336    2581 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 16 18:45:30.508610 ip-10-0-138-127 kubenswrapper[2581]: W0916 18:45:30.504339    2581 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 16 18:45:30.508610 ip-10-0-138-127 kubenswrapper[2581]: W0916 18:45:30.504342    2581 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 16 18:45:30.508610 ip-10-0-138-127 kubenswrapper[2581]: W0916 18:45:30.504345    2581 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 16 18:45:30.508610 ip-10-0-138-127 kubenswrapper[2581]: W0916 18:45:30.504347    2581 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 16 18:45:30.508610 ip-10-0-138-127 kubenswrapper[2581]: W0916 18:45:30.504350    2581 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 16 18:45:30.508610 ip-10-0-138-127 kubenswrapper[2581]: W0916 18:45:30.504352    2581 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 16 18:45:30.508610 ip-10-0-138-127 kubenswrapper[2581]: W0916 18:45:30.504355    2581 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 16 18:45:30.508610 ip-10-0-138-127 kubenswrapper[2581]: W0916 18:45:30.504358    2581 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 16 18:45:30.508610 ip-10-0-138-127 kubenswrapper[2581]: W0916 18:45:30.504362    2581 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 16 18:45:30.508610 ip-10-0-138-127 kubenswrapper[2581]: W0916 18:45:30.504365    2581 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 16 18:45:30.508610 ip-10-0-138-127 kubenswrapper[2581]: W0916 18:45:30.504368    2581 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 16 18:45:30.508610 ip-10-0-138-127 kubenswrapper[2581]: W0916 18:45:30.504371    2581 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 16 18:45:30.508610 ip-10-0-138-127 kubenswrapper[2581]: W0916 18:45:30.504374    2581 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 16 18:45:30.509075 ip-10-0-138-127 kubenswrapper[2581]: W0916 18:45:30.504376    2581 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 16 18:45:30.509075 ip-10-0-138-127 kubenswrapper[2581]: W0916 18:45:30.504379    2581 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 16 18:45:30.509075 ip-10-0-138-127 kubenswrapper[2581]: W0916 18:45:30.504382    2581 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 16 18:45:30.509075 ip-10-0-138-127 kubenswrapper[2581]: W0916 18:45:30.504385    2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 16 18:45:30.509075 ip-10-0-138-127 kubenswrapper[2581]: W0916 18:45:30.504387    2581 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 16 18:45:30.509075 ip-10-0-138-127 kubenswrapper[2581]: W0916 18:45:30.504390    2581 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 16 18:45:30.509075 ip-10-0-138-127 kubenswrapper[2581]: W0916 18:45:30.504392    2581 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 16 18:45:30.509075 ip-10-0-138-127 kubenswrapper[2581]: W0916 18:45:30.504395    2581 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 16 18:45:30.509075 ip-10-0-138-127 kubenswrapper[2581]: W0916 18:45:30.504397    2581 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 16 18:45:30.509075 ip-10-0-138-127 kubenswrapper[2581]: W0916 18:45:30.504401    2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 16 18:45:30.509075 ip-10-0-138-127 kubenswrapper[2581]: W0916 18:45:30.504404    2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 16 18:45:30.509075 ip-10-0-138-127 kubenswrapper[2581]: W0916 18:45:30.504407    2581 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 16 18:45:30.509075 ip-10-0-138-127 kubenswrapper[2581]: W0916 18:45:30.504410    2581 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 16 18:45:30.509075 ip-10-0-138-127 kubenswrapper[2581]: W0916 18:45:30.504412    2581 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 16 18:45:30.509075 ip-10-0-138-127 kubenswrapper[2581]: W0916 18:45:30.504415    2581 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 16 18:45:30.509471 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:45:30.504420    2581 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 16 18:45:30.509471 ip-10-0-138-127 kubenswrapper[2581]: W0916 18:45:30.504549    2581 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 16 18:45:30.509471 ip-10-0-138-127 kubenswrapper[2581]: W0916 18:45:30.504555    2581 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 16 18:45:30.509471 ip-10-0-138-127 kubenswrapper[2581]: W0916 18:45:30.504557    2581 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 16 18:45:30.509471 ip-10-0-138-127 kubenswrapper[2581]: W0916 18:45:30.504560    2581 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 16 18:45:30.509471 ip-10-0-138-127 kubenswrapper[2581]: W0916 18:45:30.504563    2581 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 16 18:45:30.509471 ip-10-0-138-127 kubenswrapper[2581]: W0916 18:45:30.504566    2581 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 16 18:45:30.509471 ip-10-0-138-127 kubenswrapper[2581]: W0916 18:45:30.504569    2581 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 16 18:45:30.509471 ip-10-0-138-127 kubenswrapper[2581]: W0916 18:45:30.504572    2581 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 16 18:45:30.509471 ip-10-0-138-127 kubenswrapper[2581]: W0916 18:45:30.504574    2581 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 16 18:45:30.509471 ip-10-0-138-127 kubenswrapper[2581]: W0916 18:45:30.504577    2581 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 16 18:45:30.509471 ip-10-0-138-127 kubenswrapper[2581]: W0916 18:45:30.504579    2581 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 16 18:45:30.509471 ip-10-0-138-127 kubenswrapper[2581]: W0916 18:45:30.504583    2581 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 16 18:45:30.509471 ip-10-0-138-127 kubenswrapper[2581]: W0916 18:45:30.504585    2581 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 16 18:45:30.509471 ip-10-0-138-127 kubenswrapper[2581]: W0916 18:45:30.504588    2581 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 16 18:45:30.509847 ip-10-0-138-127 kubenswrapper[2581]: W0916 18:45:30.504591    2581 feature_gate.go:328] unrecognized feature gate: Example2
Sep 16 18:45:30.509847 ip-10-0-138-127 kubenswrapper[2581]: W0916 18:45:30.504593    2581 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 16 18:45:30.509847 ip-10-0-138-127 kubenswrapper[2581]: W0916 18:45:30.504596    2581 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 16 18:45:30.509847 ip-10-0-138-127 kubenswrapper[2581]: W0916 18:45:30.504599    2581 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 16 18:45:30.509847 ip-10-0-138-127 kubenswrapper[2581]: W0916 18:45:30.504601    2581 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 16 18:45:30.509847 ip-10-0-138-127 kubenswrapper[2581]: W0916 18:45:30.504604    2581 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 16 18:45:30.509847 ip-10-0-138-127 kubenswrapper[2581]: W0916 18:45:30.504607    2581 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 16 18:45:30.509847 ip-10-0-138-127 kubenswrapper[2581]: W0916 18:45:30.504610    2581 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 16 18:45:30.509847 ip-10-0-138-127 kubenswrapper[2581]: W0916 18:45:30.504612    2581 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 16 18:45:30.509847 ip-10-0-138-127 kubenswrapper[2581]: W0916 18:45:30.504615    2581 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 16 18:45:30.509847 ip-10-0-138-127 kubenswrapper[2581]: W0916 18:45:30.504617    2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 16 18:45:30.509847 ip-10-0-138-127 kubenswrapper[2581]: W0916 18:45:30.504620    2581 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 16 18:45:30.509847 ip-10-0-138-127 kubenswrapper[2581]: W0916 18:45:30.504623    2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 16 18:45:30.509847 ip-10-0-138-127 kubenswrapper[2581]: W0916 18:45:30.504625    2581 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 16 18:45:30.509847 ip-10-0-138-127 kubenswrapper[2581]: W0916 18:45:30.504628    2581 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 16 18:45:30.509847 ip-10-0-138-127 kubenswrapper[2581]: W0916 18:45:30.504631    2581 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 16 18:45:30.509847 ip-10-0-138-127 kubenswrapper[2581]: W0916 18:45:30.504634    2581 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 16 18:45:30.509847 ip-10-0-138-127 kubenswrapper[2581]: W0916 18:45:30.504636    2581 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 16 18:45:30.509847 ip-10-0-138-127 kubenswrapper[2581]: W0916 18:45:30.504639    2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 16 18:45:30.509847 ip-10-0-138-127 kubenswrapper[2581]: W0916 18:45:30.504641    2581 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 16 18:45:30.510355 ip-10-0-138-127 kubenswrapper[2581]: W0916 18:45:30.504644    2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 16 18:45:30.510355 ip-10-0-138-127 kubenswrapper[2581]: W0916 18:45:30.504646    2581 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 16 18:45:30.510355 ip-10-0-138-127 kubenswrapper[2581]: W0916 18:45:30.504649    2581 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 16 18:45:30.510355 ip-10-0-138-127 kubenswrapper[2581]: W0916 18:45:30.504651    2581 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 16 18:45:30.510355 ip-10-0-138-127 kubenswrapper[2581]: W0916 18:45:30.504654    2581 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 16 18:45:30.510355 ip-10-0-138-127 kubenswrapper[2581]: W0916 18:45:30.504656    2581 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 16 18:45:30.510355 ip-10-0-138-127 kubenswrapper[2581]: W0916 18:45:30.504659    2581 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 16 18:45:30.510355 ip-10-0-138-127 kubenswrapper[2581]: W0916 18:45:30.504661    2581 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 16 18:45:30.510355 ip-10-0-138-127 kubenswrapper[2581]: W0916 18:45:30.504664    2581 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 16 18:45:30.510355 ip-10-0-138-127 kubenswrapper[2581]: W0916 18:45:30.504667    2581 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 16 18:45:30.510355 ip-10-0-138-127 kubenswrapper[2581]: W0916 18:45:30.504669    2581 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 16 18:45:30.510355 ip-10-0-138-127 kubenswrapper[2581]: W0916 18:45:30.504672    2581 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 16 18:45:30.510355 ip-10-0-138-127 kubenswrapper[2581]: W0916 18:45:30.504675    2581 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 16 18:45:30.510355 ip-10-0-138-127 kubenswrapper[2581]: W0916 18:45:30.504677    2581 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 16 18:45:30.510355 ip-10-0-138-127 kubenswrapper[2581]: W0916 18:45:30.504680    2581 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 16 18:45:30.510355 ip-10-0-138-127 kubenswrapper[2581]: W0916 18:45:30.504684    2581 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 16 18:45:30.510355 ip-10-0-138-127 kubenswrapper[2581]: W0916 18:45:30.504687    2581 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 16 18:45:30.510355 ip-10-0-138-127 kubenswrapper[2581]: W0916 18:45:30.504691    2581 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 16 18:45:30.510355 ip-10-0-138-127 kubenswrapper[2581]: W0916 18:45:30.504695    2581 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 16 18:45:30.510815 ip-10-0-138-127 kubenswrapper[2581]: W0916 18:45:30.504697    2581 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 16 18:45:30.510815 ip-10-0-138-127 kubenswrapper[2581]: W0916 18:45:30.504700    2581 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 16 18:45:30.510815 ip-10-0-138-127 kubenswrapper[2581]: W0916 18:45:30.504703    2581 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 16 18:45:30.510815 ip-10-0-138-127 kubenswrapper[2581]: W0916 18:45:30.504706    2581 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 16 18:45:30.510815 ip-10-0-138-127 kubenswrapper[2581]: W0916 18:45:30.504709    2581 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 16 18:45:30.510815 ip-10-0-138-127 kubenswrapper[2581]: W0916 18:45:30.504712    2581 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 16 18:45:30.510815 ip-10-0-138-127 kubenswrapper[2581]: W0916 18:45:30.504715    2581 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 16 18:45:30.510815 ip-10-0-138-127 kubenswrapper[2581]: W0916 18:45:30.504717    2581 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 16 18:45:30.510815 ip-10-0-138-127 kubenswrapper[2581]: W0916 18:45:30.504721    2581 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 16 18:45:30.510815 ip-10-0-138-127 kubenswrapper[2581]: W0916 18:45:30.504724    2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 16 18:45:30.510815 ip-10-0-138-127 kubenswrapper[2581]: W0916 18:45:30.504727    2581 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 16 18:45:30.510815 ip-10-0-138-127 kubenswrapper[2581]: W0916 18:45:30.504730    2581 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 16 18:45:30.510815 ip-10-0-138-127 kubenswrapper[2581]: W0916 18:45:30.504732    2581 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 16 18:45:30.510815 ip-10-0-138-127 kubenswrapper[2581]: W0916 18:45:30.504735    2581 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 16 18:45:30.510815 ip-10-0-138-127 kubenswrapper[2581]: W0916 18:45:30.504738    2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 16 18:45:30.510815 ip-10-0-138-127 kubenswrapper[2581]: W0916 18:45:30.504740    2581 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 16 18:45:30.510815 ip-10-0-138-127 kubenswrapper[2581]: W0916 18:45:30.504743    2581 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 16 18:45:30.510815 ip-10-0-138-127 kubenswrapper[2581]: W0916 18:45:30.504745    2581 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 16 18:45:30.510815 ip-10-0-138-127 kubenswrapper[2581]: W0916 18:45:30.504747    2581 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 16 18:45:30.510815 ip-10-0-138-127 kubenswrapper[2581]: W0916 18:45:30.504750    2581 feature_gate.go:328] unrecognized feature gate: Example
Sep 16 18:45:30.511350 ip-10-0-138-127 kubenswrapper[2581]: W0916 18:45:30.504752    2581 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 16 18:45:30.511350 ip-10-0-138-127 kubenswrapper[2581]: W0916 18:45:30.504755    2581 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 16 18:45:30.511350 ip-10-0-138-127 kubenswrapper[2581]: W0916 18:45:30.504757    2581 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 16 18:45:30.511350 ip-10-0-138-127 kubenswrapper[2581]: W0916 18:45:30.504760    2581 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 16 18:45:30.511350 ip-10-0-138-127 kubenswrapper[2581]: W0916 18:45:30.504762    2581 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 16 18:45:30.511350 ip-10-0-138-127 kubenswrapper[2581]: W0916 18:45:30.504765    2581 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 16 18:45:30.511350 ip-10-0-138-127 kubenswrapper[2581]: W0916 18:45:30.504767    2581 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 16 18:45:30.511350 ip-10-0-138-127 kubenswrapper[2581]: W0916 18:45:30.504770    2581 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 16 18:45:30.511350 ip-10-0-138-127 kubenswrapper[2581]: W0916 18:45:30.504773    2581 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 16 18:45:30.511350 ip-10-0-138-127 kubenswrapper[2581]: W0916 18:45:30.504776    2581 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 16 18:45:30.511350 ip-10-0-138-127 kubenswrapper[2581]: W0916 18:45:30.504779    2581 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 16 18:45:30.511350 ip-10-0-138-127 kubenswrapper[2581]: W0916 18:45:30.504782    2581 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 16 18:45:30.511350 ip-10-0-138-127 kubenswrapper[2581]: W0916 18:45:30.504785    2581 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 16 18:45:30.511350 ip-10-0-138-127 kubenswrapper[2581]: W0916 18:45:30.504788    2581 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 16 18:45:30.511350 ip-10-0-138-127 kubenswrapper[2581]: W0916 18:45:30.504790    2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 16 18:45:30.511350 ip-10-0-138-127 kubenswrapper[2581]: W0916 18:45:30.504793    2581 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 16 18:45:30.511350 ip-10-0-138-127 kubenswrapper[2581]: W0916 18:45:30.504796    2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 16 18:45:30.511350 ip-10-0-138-127 kubenswrapper[2581]: W0916 18:45:30.504799    2581 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 16 18:45:30.511350 ip-10-0-138-127 kubenswrapper[2581]: W0916 18:45:30.504802    2581 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 16 18:45:30.511831 ip-10-0-138-127 kubenswrapper[2581]: W0916 18:45:30.504805    2581 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 16 18:45:30.511831 ip-10-0-138-127 kubenswrapper[2581]: W0916 18:45:30.504807    2581 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 16 18:45:30.511831 ip-10-0-138-127 kubenswrapper[2581]: W0916 18:45:30.504810    2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 16 18:45:30.511831 ip-10-0-138-127 kubenswrapper[2581]: W0916 18:45:30.504813    2581 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 16 18:45:30.511831 ip-10-0-138-127 kubenswrapper[2581]: W0916 18:45:30.504815    2581 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 16 18:45:30.511831 ip-10-0-138-127 kubenswrapper[2581]: W0916 18:45:30.504817    2581 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 16 18:45:30.511831 ip-10-0-138-127 kubenswrapper[2581]: W0916 18:45:30.504820    2581 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 16 18:45:30.511831 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:45:30.504825    2581 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 16 18:45:30.511831 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:45:30.504942    2581 server.go:962] "Client rotation is on, will bootstrap in background"
Sep 16 18:45:30.511831 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:45:30.507302    2581 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir"
Sep 16 18:45:30.511831 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:45:30.508437    2581 server.go:1019] "Starting client certificate rotation"
Sep 16 18:45:30.511831 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:45:30.508537    2581 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Sep 16 18:45:30.511831 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:45:30.508578    2581 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet"
Sep 16 18:45:30.512143 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:45:30.511497    2581 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Sep 16 18:45:30.515739 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:45:30.515721    2581 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Sep 16 18:45:30.526934 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:45:30.526908    2581 log.go:25] "Validated CRI v1 runtime API"
Sep 16 18:45:30.530719 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:45:30.530703    2581 log.go:25] "Validated CRI v1 image API"
Sep 16 18:45:30.531613 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:45:30.531593    2581 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Sep 16 18:45:30.536759 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:45:30.536738    2581 fs.go:135] Filesystem UUIDs: map[050b18d7-d727-4e4b-a33b-5595d2e70f3e:/dev/nvme0n1p4 7B77-95E7:/dev/nvme0n1p2 eae4861b-5b50-46b0-87f3-bf0cd5a22f50:/dev/nvme0n1p3]
Sep 16 18:45:30.536813 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:45:30.536758    2581 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}]
Sep 16 18:45:30.542894 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:45:30.542770    2581 manager.go:216] Machine: {Timestamp:2026-09-16 18:45:30.541332023 +0000 UTC m=+0.360745924 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3063781 MemoryCapacity:32812154880 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2c0c6ff04c2103f086822b43129753 SystemUUID:ec2c0c6f-f04c-2103-f086-822b43129753 BootID:b97cd242-54c7-43c7-bc77-bbc6040ebce3 Filesystems:[{Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16406077440 Type:vfs Inodes:4005390 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6562430976 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6111232 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16406077440 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:ff:f6:03:7d:f5 Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:ff:f6:03:7d:f5 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:2a:15:ef:7a:5c:72 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:32812154880 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:34603008 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Sep 16 18:45:30.542894 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:45:30.542879    2581 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Sep 16 18:45:30.543064 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:45:30.542979    2581 manager.go:232] Version: {KernelVersion:5.14.0-570.140.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260908-1 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Sep 16 18:45:30.544270 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:45:30.544243    2581 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Sep 16 18:45:30.546454 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:45:30.546424    2581 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Sep 16 18:45:30.546609 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:45:30.546459    2581 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-138-127.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Sep 16 18:45:30.548005 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:45:30.547994    2581 topology_manager.go:138] "Creating topology manager with none policy"
Sep 16 18:45:30.548045 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:45:30.548007    2581 container_manager_linux.go:309] "Creating device plugin manager"
Sep 16 18:45:30.548045 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:45:30.548015    2581 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Sep 16 18:45:30.548045 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:45:30.548029    2581 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Sep 16 18:45:30.549339 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:45:30.549328    2581 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager"
Sep 16 18:45:30.554110 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:45:30.554095    2581 state_mem.go:36] "Initialized new in-memory state store"
Sep 16 18:45:30.554253 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:45:30.554243    2581 server.go:1240] "Using root directory" path="/var/lib/kubelet"
Sep 16 18:45:30.554294 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:45:30.554276    2581 kubelet.go:486] "Attempting to sync node with API server"
Sep 16 18:45:30.554294 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:45:30.554286    2581 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests"
Sep 16 18:45:30.554352 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:45:30.554301    2581 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Sep 16 18:45:30.554352 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:45:30.554312    2581 kubelet.go:398] "Adding apiserver pod source"
Sep 16 18:45:30.554352 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:45:30.554327    2581 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Sep 16 18:45:30.554857 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:45:30.554707    2581 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Sep 16 18:45:30.554931 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:45:30.554863    2581 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Sep 16 18:45:30.557530 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:45:30.557506    2581 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.12-5.rhaos4.21.gitd1ec63c.el9" apiVersion="v1"
Sep 16 18:45:30.560525 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:45:30.560510    2581 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Sep 16 18:45:30.560570 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:45:30.560542    2581 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Sep 16 18:45:30.563362 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:45:30.563345    2581 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Sep 16 18:45:30.563362 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:45:30.563364    2581 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Sep 16 18:45:30.563470 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:45:30.563370    2581 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Sep 16 18:45:30.563470 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:45:30.563375    2581 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Sep 16 18:45:30.563470 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:45:30.563383    2581 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Sep 16 18:45:30.563470 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:45:30.563388    2581 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Sep 16 18:45:30.563470 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:45:30.563401    2581 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Sep 16 18:45:30.563470 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:45:30.563406    2581 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Sep 16 18:45:30.563470 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:45:30.563414    2581 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Sep 16 18:45:30.563470 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:45:30.563420    2581 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Sep 16 18:45:30.563470 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:45:30.563428    2581 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Sep 16 18:45:30.563470 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:45:30.563433    2581 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Sep 16 18:45:30.564313 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:45:30.564294    2581 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Sep 16 18:45:30.564313 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:45:30.564305    2581 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Sep 16 18:45:30.568029 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:45:30.567984    2581 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-138-127.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Sep 16 18:45:30.568123 ip-10-0-138-127 kubenswrapper[2581]: E0916 18:45:30.568030    2581 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-138-127.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node"
Sep 16 18:45:30.568123 ip-10-0-138-127 kubenswrapper[2581]: E0916 18:45:30.568030    2581 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service"
Sep 16 18:45:30.568411 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:45:30.568399    2581 server.go:1268] "Started kubelet"
Sep 16 18:45:30.568461 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:45:30.568438    2581 server.go:180] "Starting to listen" address="0.0.0.0" port=10250
Sep 16 18:45:30.568526 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:45:30.568476    2581 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Sep 16 18:45:30.568577 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:45:30.568553    2581 server_v1.go:49] "podresources" method="list" useActivePods=true
Sep 16 18:45:30.569465 ip-10-0-138-127 systemd[1]: Started Kubernetes Kubelet.
Sep 16 18:45:30.572947 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:45:30.572904    2581 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Sep 16 18:45:30.574794 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:45:30.574772    2581 server.go:310] "Adding debug handlers to kubelet server"
Sep 16 18:45:30.577580 ip-10-0-138-127 kubenswrapper[2581]: E0916 18:45:30.577561    2581 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache"
Sep 16 18:45:30.580083 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:45:30.580061    2581 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Sep 16 18:45:30.580275 ip-10-0-138-127 kubenswrapper[2581]: E0916 18:45:30.579138    2581 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-138-127.ec2.internal.18d5e1778f21ed6a  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-138-127.ec2.internal,UID:ip-10-0-138-127.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-138-127.ec2.internal,},FirstTimestamp:2026-09-16 18:45:30.56836337 +0000 UTC m=+0.387777270,LastTimestamp:2026-09-16 18:45:30.56836337 +0000 UTC m=+0.387777270,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-138-127.ec2.internal,}"
Sep 16 18:45:30.580669 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:45:30.580652    2581 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer"
Sep 16 18:45:30.580813 ip-10-0-138-127 kubenswrapper[2581]: E0916 18:45:30.580796    2581 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-138-127.ec2.internal\" not found"
Sep 16 18:45:30.581433 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:45:30.581417    2581 factory.go:55] Registering systemd factory
Sep 16 18:45:30.581506 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:45:30.581437    2581 factory.go:223] Registration of the systemd container factory successfully
Sep 16 18:45:30.581579 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:45:30.581542    2581 volume_manager.go:311] "The desired_state_of_world populator starts"
Sep 16 18:45:30.581579 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:45:30.581562    2581 volume_manager.go:313] "Starting Kubelet Volume Manager"
Sep 16 18:45:30.581676 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:45:30.581541    2581 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Sep 16 18:45:30.581723 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:45:30.581694    2581 reconstruct.go:97] "Volume reconstruction finished"
Sep 16 18:45:30.581723 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:45:30.581704    2581 reconciler.go:29] "Reconciler: start to sync state"
Sep 16 18:45:30.581835 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:45:30.581821    2581 factory.go:153] Registering CRI-O factory
Sep 16 18:45:30.581871 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:45:30.581838    2581 factory.go:223] Registration of the crio container factory successfully
Sep 16 18:45:30.581902 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:45:30.581893    2581 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Sep 16 18:45:30.581949 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:45:30.581911    2581 factory.go:103] Registering Raw factory
Sep 16 18:45:30.581982 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:45:30.581953    2581 manager.go:1196] Started watching for new ooms in manager
Sep 16 18:45:30.582447 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:45:30.582434    2581 manager.go:349] Starting recovery of all containers
Sep 16 18:45:30.589638 ip-10-0-138-127 kubenswrapper[2581]: E0916 18:45:30.589599    2581 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-138-127.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms"
Sep 16 18:45:30.589638 ip-10-0-138-127 kubenswrapper[2581]: E0916 18:45:30.589603    2581 reflector.go:205] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver"
Sep 16 18:45:30.593623 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:45:30.593607    2581 manager.go:354] Recovery completed
Sep 16 18:45:30.598548 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:45:30.598534    2581 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 16 18:45:30.603398 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:45:30.603371    2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-127.ec2.internal" event="NodeHasSufficientMemory"
Sep 16 18:45:30.603464 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:45:30.603434    2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-127.ec2.internal" event="NodeHasNoDiskPressure"
Sep 16 18:45:30.603464 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:45:30.603447    2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-127.ec2.internal" event="NodeHasSufficientPID"
Sep 16 18:45:30.604116 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:45:30.604084    2581 cpu_manager.go:222] "Starting CPU manager" policy="none"
Sep 16 18:45:30.604180 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:45:30.604114    2581 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 16 18:45:30.604235 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:45:30.604116    2581 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s"
Sep 16 18:45:30.604268 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:45:30.604249    2581 state_mem.go:36] "Initialized new in-memory state store"
Sep 16 18:45:30.604996 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:45:30.604979    2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-127.ec2.internal" event="NodeHasSufficientMemory"
Sep 16 18:45:30.605063 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:45:30.605009    2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-127.ec2.internal" event="NodeHasNoDiskPressure"
Sep 16 18:45:30.605063 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:45:30.605019    2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-127.ec2.internal" event="NodeHasSufficientPID"
Sep 16 18:45:30.605460 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:45:30.605444    2581 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-gxxcc"
Sep 16 18:45:30.606576 ip-10-0-138-127 kubenswrapper[2581]: E0916 18:45:30.606501    2581 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-138-127.ec2.internal.18d5e17791388011  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-138-127.ec2.internal,UID:ip-10-0-138-127.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-138-127.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-138-127.ec2.internal,},FirstTimestamp:2026-09-16 18:45:30.603397137 +0000 UTC m=+0.422811037,LastTimestamp:2026-09-16 18:45:30.603397137 +0000 UTC m=+0.422811037,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-138-127.ec2.internal,}"
Sep 16 18:45:30.606684 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:45:30.606564    2581 policy_none.go:49] "None policy: Start"
Sep 16 18:45:30.606684 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:45:30.606653    2581 memory_manager.go:187] "Starting memorymanager" policy="None"
Sep 16 18:45:30.606684 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:45:30.606669    2581 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Sep 16 18:45:30.608783 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:45:30.608773    2581 policy_none.go:47] "Start"
Sep 16 18:45:30.615703 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:45:30.615676    2581 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-gxxcc"
Sep 16 18:45:30.616672 ip-10-0-138-127 kubenswrapper[2581]: E0916 18:45:30.616603    2581 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-138-127.ec2.internal.18d5e17791392eee  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-138-127.ec2.internal,UID:ip-10-0-138-127.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasNoDiskPressure,Message:Node ip-10-0-138-127.ec2.internal status is now: NodeHasNoDiskPressure,Source:EventSource{Component:kubelet,Host:ip-10-0-138-127.ec2.internal,},FirstTimestamp:2026-09-16 18:45:30.603441902 +0000 UTC m=+0.422855803,LastTimestamp:2026-09-16 18:45:30.603441902 +0000 UTC m=+0.422855803,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-138-127.ec2.internal,}"
Sep 16 18:45:30.660133 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:45:30.646795    2581 manager.go:337] "Starting Device Plugin manager"
Sep 16 18:45:30.660133 ip-10-0-138-127 kubenswrapper[2581]: E0916 18:45:30.646832    2581 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Sep 16 18:45:30.660133 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:45:30.646842    2581 server.go:87] "Starting device plugin registration server"
Sep 16 18:45:30.660133 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:45:30.647188    2581 eviction_manager.go:189] "Eviction manager: starting control loop"
Sep 16 18:45:30.660133 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:45:30.647220    2581 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Sep 16 18:45:30.660133 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:45:30.647333    2581 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Sep 16 18:45:30.660133 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:45:30.647425    2581 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts"
Sep 16 18:45:30.660133 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:45:30.647434    2581 plugin_manager.go:118] "Starting Kubelet Plugin Manager"
Sep 16 18:45:30.660133 ip-10-0-138-127 kubenswrapper[2581]: E0916 18:45:30.647932    2581 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Sep 16 18:45:30.660133 ip-10-0-138-127 kubenswrapper[2581]: E0916 18:45:30.647987    2581 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-138-127.ec2.internal\" not found"
Sep 16 18:45:30.720050 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:45:30.720004    2581 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Sep 16 18:45:30.721509 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:45:30.721491    2581 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Sep 16 18:45:30.721594 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:45:30.721560    2581 status_manager.go:244] "Starting to sync pod status with apiserver"
Sep 16 18:45:30.721594 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:45:30.721591    2581 kubelet.go:2448] "Starting kubelet main sync loop"
Sep 16 18:45:30.721690 ip-10-0-138-127 kubenswrapper[2581]: E0916 18:45:30.721631    2581 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful"
Sep 16 18:45:30.724276 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:45:30.724252    2581 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 16 18:45:30.748406 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:45:30.748313    2581 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 16 18:45:30.749822 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:45:30.749805    2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-127.ec2.internal" event="NodeHasSufficientMemory"
Sep 16 18:45:30.749872 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:45:30.749840    2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-127.ec2.internal" event="NodeHasNoDiskPressure"
Sep 16 18:45:30.749872 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:45:30.749852    2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-127.ec2.internal" event="NodeHasSufficientPID"
Sep 16 18:45:30.749945 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:45:30.749876    2581 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-138-127.ec2.internal"
Sep 16 18:45:30.758492 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:45:30.758474    2581 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-138-127.ec2.internal"
Sep 16 18:45:30.758546 ip-10-0-138-127 kubenswrapper[2581]: E0916 18:45:30.758500    2581 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-138-127.ec2.internal\": node \"ip-10-0-138-127.ec2.internal\" not found"
Sep 16 18:45:30.774498 ip-10-0-138-127 kubenswrapper[2581]: E0916 18:45:30.774465    2581 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-138-127.ec2.internal\" not found"
Sep 16 18:45:30.822784 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:45:30.822723    2581 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["kube-system/kube-apiserver-proxy-ip-10-0-138-127.ec2.internal","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-127.ec2.internal"]
Sep 16 18:45:30.822952 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:45:30.822838    2581 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 16 18:45:30.823870 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:45:30.823853    2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-127.ec2.internal" event="NodeHasSufficientMemory"
Sep 16 18:45:30.823961 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:45:30.823890    2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-127.ec2.internal" event="NodeHasNoDiskPressure"
Sep 16 18:45:30.823961 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:45:30.823905    2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-127.ec2.internal" event="NodeHasSufficientPID"
Sep 16 18:45:30.826337 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:45:30.826320    2581 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 16 18:45:30.826437 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:45:30.826418    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-138-127.ec2.internal"
Sep 16 18:45:30.826491 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:45:30.826450    2581 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 16 18:45:30.827164 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:45:30.827143    2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-127.ec2.internal" event="NodeHasSufficientMemory"
Sep 16 18:45:30.827284 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:45:30.827181    2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-127.ec2.internal" event="NodeHasNoDiskPressure"
Sep 16 18:45:30.827284 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:45:30.827212    2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-127.ec2.internal" event="NodeHasSufficientPID"
Sep 16 18:45:30.827284 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:45:30.827216    2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-127.ec2.internal" event="NodeHasSufficientMemory"
Sep 16 18:45:30.827284 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:45:30.827238    2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-127.ec2.internal" event="NodeHasNoDiskPressure"
Sep 16 18:45:30.827284 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:45:30.827258    2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-127.ec2.internal" event="NodeHasSufficientPID"
Sep 16 18:45:30.829537 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:45:30.829520    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-127.ec2.internal"
Sep 16 18:45:30.829628 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:45:30.829553    2581 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 16 18:45:30.830294 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:45:30.830279    2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-127.ec2.internal" event="NodeHasSufficientMemory"
Sep 16 18:45:30.830360 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:45:30.830308    2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-127.ec2.internal" event="NodeHasNoDiskPressure"
Sep 16 18:45:30.830360 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:45:30.830318    2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-127.ec2.internal" event="NodeHasSufficientPID"
Sep 16 18:45:30.860153 ip-10-0-138-127 kubenswrapper[2581]: E0916 18:45:30.860125    2581 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-138-127.ec2.internal\" not found" node="ip-10-0-138-127.ec2.internal"
Sep 16 18:45:30.864713 ip-10-0-138-127 kubenswrapper[2581]: E0916 18:45:30.864695    2581 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-138-127.ec2.internal\" not found" node="ip-10-0-138-127.ec2.internal"
Sep 16 18:45:30.875036 ip-10-0-138-127 kubenswrapper[2581]: E0916 18:45:30.875013    2581 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-138-127.ec2.internal\" not found"
Sep 16 18:45:30.883468 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:45:30.883401    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/37df0dc0824bc6d3c2498649ba68632f-config\") pod \"kube-apiserver-proxy-ip-10-0-138-127.ec2.internal\" (UID: \"37df0dc0824bc6d3c2498649ba68632f\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-138-127.ec2.internal"
Sep 16 18:45:30.883571 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:45:30.883475    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/3f19801b96f1829134dfd0b39bce2048-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-138-127.ec2.internal\" (UID: \"3f19801b96f1829134dfd0b39bce2048\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-127.ec2.internal"
Sep 16 18:45:30.883571 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:45:30.883502    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/3f19801b96f1829134dfd0b39bce2048-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-138-127.ec2.internal\" (UID: \"3f19801b96f1829134dfd0b39bce2048\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-127.ec2.internal"
Sep 16 18:45:30.975627 ip-10-0-138-127 kubenswrapper[2581]: E0916 18:45:30.975590    2581 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-138-127.ec2.internal\" not found"
Sep 16 18:45:30.983643 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:45:30.983619    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/37df0dc0824bc6d3c2498649ba68632f-config\") pod \"kube-apiserver-proxy-ip-10-0-138-127.ec2.internal\" (UID: \"37df0dc0824bc6d3c2498649ba68632f\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-138-127.ec2.internal"
Sep 16 18:45:30.983746 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:45:30.983648    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/3f19801b96f1829134dfd0b39bce2048-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-138-127.ec2.internal\" (UID: \"3f19801b96f1829134dfd0b39bce2048\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-127.ec2.internal"
Sep 16 18:45:30.983746 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:45:30.983665    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/3f19801b96f1829134dfd0b39bce2048-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-138-127.ec2.internal\" (UID: \"3f19801b96f1829134dfd0b39bce2048\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-127.ec2.internal"
Sep 16 18:45:30.983746 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:45:30.983718    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/3f19801b96f1829134dfd0b39bce2048-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-138-127.ec2.internal\" (UID: \"3f19801b96f1829134dfd0b39bce2048\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-127.ec2.internal"
Sep 16 18:45:30.983746 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:45:30.983733    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/37df0dc0824bc6d3c2498649ba68632f-config\") pod \"kube-apiserver-proxy-ip-10-0-138-127.ec2.internal\" (UID: \"37df0dc0824bc6d3c2498649ba68632f\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-138-127.ec2.internal"
Sep 16 18:45:30.983875 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:45:30.983740    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/3f19801b96f1829134dfd0b39bce2048-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-138-127.ec2.internal\" (UID: \"3f19801b96f1829134dfd0b39bce2048\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-127.ec2.internal"
Sep 16 18:45:31.076744 ip-10-0-138-127 kubenswrapper[2581]: E0916 18:45:31.076659    2581 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-138-127.ec2.internal\" not found"
Sep 16 18:45:31.163247 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:45:31.163215    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-138-127.ec2.internal"
Sep 16 18:45:31.168445 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:45:31.168250    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-127.ec2.internal"
Sep 16 18:45:31.177208 ip-10-0-138-127 kubenswrapper[2581]: E0916 18:45:31.177162    2581 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-138-127.ec2.internal\" not found"
Sep 16 18:45:31.277810 ip-10-0-138-127 kubenswrapper[2581]: E0916 18:45:31.277749    2581 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-138-127.ec2.internal\" not found"
Sep 16 18:45:31.378337 ip-10-0-138-127 kubenswrapper[2581]: E0916 18:45:31.378255    2581 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-138-127.ec2.internal\" not found"
Sep 16 18:45:31.476907 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:45:31.476881    2581 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 16 18:45:31.478986 ip-10-0-138-127 kubenswrapper[2581]: E0916 18:45:31.478967    2581 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-138-127.ec2.internal\" not found"
Sep 16 18:45:31.508494 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:45:31.508459    2581 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials"
Sep 16 18:45:31.508909 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:45:31.508615    2581 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Sep 16 18:45:31.508909 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:45:31.508650    2581 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Sep 16 18:45:31.579522 ip-10-0-138-127 kubenswrapper[2581]: E0916 18:45:31.579486    2581 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-138-127.ec2.internal\" not found"
Sep 16 18:45:31.621352 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:45:31.621312    2581 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-09-15 18:40:31 +0000 UTC" deadline="2028-05-24 08:33:23.09494331 +0000 UTC"
Sep 16 18:45:31.621352 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:45:31.621348    2581 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="14773h47m51.473598893s"
Sep 16 18:45:31.679627 ip-10-0-138-127 kubenswrapper[2581]: E0916 18:45:31.679556    2581 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-138-127.ec2.internal\" not found"
Sep 16 18:45:31.724065 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:45:31.724035    2581 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 16 18:45:31.725278 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:45:31.725258    2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-127.ec2.internal" event="NodeHasSufficientMemory"
Sep 16 18:45:31.725391 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:45:31.725291    2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-127.ec2.internal" event="NodeHasNoDiskPressure"
Sep 16 18:45:31.725391 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:45:31.725306    2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-127.ec2.internal" event="NodeHasSufficientPID"
Sep 16 18:45:31.748846 ip-10-0-138-127 kubenswrapper[2581]: W0916 18:45:31.748807    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3f19801b96f1829134dfd0b39bce2048.slice/crio-380e8f99a62a4bf3ab648d08109796fc9892cea142c8c24c631c52ec8481b6f1 WatchSource:0}: Error finding container 380e8f99a62a4bf3ab648d08109796fc9892cea142c8c24c631c52ec8481b6f1: Status 404 returned error can't find the container with id 380e8f99a62a4bf3ab648d08109796fc9892cea142c8c24c631c52ec8481b6f1
Sep 16 18:45:31.749459 ip-10-0-138-127 kubenswrapper[2581]: W0916 18:45:31.749437    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod37df0dc0824bc6d3c2498649ba68632f.slice/crio-40752c933a3498141063fa2836bd1507e22d4cff5785b9fde696e638a3bbda1a WatchSource:0}: Error finding container 40752c933a3498141063fa2836bd1507e22d4cff5785b9fde696e638a3bbda1a: Status 404 returned error can't find the container with id 40752c933a3498141063fa2836bd1507e22d4cff5785b9fde696e638a3bbda1a
Sep 16 18:45:31.754113 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:45:31.754096    2581 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 16 18:45:31.780209 ip-10-0-138-127 kubenswrapper[2581]: E0916 18:45:31.780170    2581 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-138-127.ec2.internal\" not found"
Sep 16 18:45:31.864495 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:45:31.864465    2581 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 16 18:45:31.880908 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:45:31.880882    2581 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-138-127.ec2.internal"
Sep 16 18:45:31.891349 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:45:31.891323    2581 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Sep 16 18:45:31.892892 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:45:31.892877    2581 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-127.ec2.internal"
Sep 16 18:45:31.905287 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:45:31.905266    2581 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Sep 16 18:45:31.930107 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:45:31.930051    2581 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 16 18:45:32.391732 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:45:32.391411    2581 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 16 18:45:32.555515 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:45:32.555479    2581 apiserver.go:52] "Watching apiserver"
Sep 16 18:45:32.567114 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:45:32.567077    2581 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Sep 16 18:45:32.568979 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:45:32.568930    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-multus/multus-vxjzx","openshift-multus/network-metrics-daemon-c2kp2","openshift-network-diagnostics/network-check-target-bz5jk","kube-system/global-pull-secret-syncer-c78nz","kube-system/konnectivity-agent-vttdl","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-54w2k","openshift-cluster-node-tuning-operator/tuned-7t742","openshift-network-operator/iptables-alerter-fbmb7","openshift-ovn-kubernetes/ovnkube-node-csmcl","kube-system/kube-apiserver-proxy-ip-10-0-138-127.ec2.internal","openshift-dns/node-resolver-lrclz","openshift-image-registry/node-ca-mvhf8","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-127.ec2.internal","openshift-multus/multus-additional-cni-plugins-fzt7c"]
Sep 16 18:45:32.571934 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:45:32.571904    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-7t742"
Sep 16 18:45:32.574624 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:45:32.574600    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-lrclz"
Sep 16 18:45:32.574931 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:45:32.574906    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Sep 16 18:45:32.575191 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:45:32.575167    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Sep 16 18:45:32.575356 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:45:32.575338    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-98lm8\""
Sep 16 18:45:32.576677 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:45:32.576654    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-bz5jk"
Sep 16 18:45:32.576770 ip-10-0-138-127 kubenswrapper[2581]: E0916 18:45:32.576734    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-bz5jk" podUID="4775e8c4-09ba-43db-98b7-073aeb452ce8"
Sep 16 18:45:32.577006 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:45:32.576986    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Sep 16 18:45:32.577217 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:45:32.577179    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-6fx9s\""
Sep 16 18:45:32.577217 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:45:32.577213    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Sep 16 18:45:32.578983 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:45:32.578954    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-c78nz"
Sep 16 18:45:32.579084 ip-10-0-138-127 kubenswrapper[2581]: E0916 18:45:32.579040    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-c78nz" podUID="f09babc9-6295-4a0a-be9f-fa0175b0de51"
Sep 16 18:45:32.581444 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:45:32.581425    2581 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving"
Sep 16 18:45:32.583402 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:45:32.583384    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-vttdl"
Sep 16 18:45:32.586264 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:45:32.585645    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Sep 16 18:45:32.586264 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:45:32.585847    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Sep 16 18:45:32.586264 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:45:32.586086    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-zhd94\""
Sep 16 18:45:32.586264 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:45:32.586247    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-54w2k"
Sep 16 18:45:32.588676 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:45:32.588653    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Sep 16 18:45:32.588811 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:45:32.588790    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-dv4x7\""
Sep 16 18:45:32.588902 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:45:32.588879    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Sep 16 18:45:32.588967 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:45:32.588664    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-csmcl"
Sep 16 18:45:32.589053 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:45:32.588661    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Sep 16 18:45:32.591129 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:45:32.591110    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Sep 16 18:45:32.591418 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:45:32.591394    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/3f382427-dc87-4034-ae03-c457ed18d446-etc-sysctl-d\") pod \"tuned-7t742\" (UID: \"3f382427-dc87-4034-ae03-c457ed18d446\") " pod="openshift-cluster-node-tuning-operator/tuned-7t742"
Sep 16 18:45:32.591500 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:45:32.591435    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/3f382427-dc87-4034-ae03-c457ed18d446-etc-systemd\") pod \"tuned-7t742\" (UID: \"3f382427-dc87-4034-ae03-c457ed18d446\") " pod="openshift-cluster-node-tuning-operator/tuned-7t742"
Sep 16 18:45:32.591500 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:45:32.591460    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/3f382427-dc87-4034-ae03-c457ed18d446-run\") pod \"tuned-7t742\" (UID: \"3f382427-dc87-4034-ae03-c457ed18d446\") " pod="openshift-cluster-node-tuning-operator/tuned-7t742"
Sep 16 18:45:32.591500 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:45:32.591485    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/3f382427-dc87-4034-ae03-c457ed18d446-sys\") pod \"tuned-7t742\" (UID: \"3f382427-dc87-4034-ae03-c457ed18d446\") " pod="openshift-cluster-node-tuning-operator/tuned-7t742"
Sep 16 18:45:32.591665 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:45:32.591512    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/3f382427-dc87-4034-ae03-c457ed18d446-var-lib-kubelet\") pod \"tuned-7t742\" (UID: \"3f382427-dc87-4034-ae03-c457ed18d446\") " pod="openshift-cluster-node-tuning-operator/tuned-7t742"
Sep 16 18:45:32.591665 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:45:32.591538    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/f09babc9-6295-4a0a-be9f-fa0175b0de51-dbus\") pod \"global-pull-secret-syncer-c78nz\" (UID: \"f09babc9-6295-4a0a-be9f-fa0175b0de51\") " pod="kube-system/global-pull-secret-syncer-c78nz"
Sep 16 18:45:32.591665 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:45:32.591565    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/3f382427-dc87-4034-ae03-c457ed18d446-etc-sysconfig\") pod \"tuned-7t742\" (UID: \"3f382427-dc87-4034-ae03-c457ed18d446\") " pod="openshift-cluster-node-tuning-operator/tuned-7t742"
Sep 16 18:45:32.591665 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:45:32.591588    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/3f382427-dc87-4034-ae03-c457ed18d446-etc-kubernetes\") pod \"tuned-7t742\" (UID: \"3f382427-dc87-4034-ae03-c457ed18d446\") " pod="openshift-cluster-node-tuning-operator/tuned-7t742"
Sep 16 18:45:32.591665 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:45:32.591652    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/3f382427-dc87-4034-ae03-c457ed18d446-lib-modules\") pod \"tuned-7t742\" (UID: \"3f382427-dc87-4034-ae03-c457ed18d446\") " pod="openshift-cluster-node-tuning-operator/tuned-7t742"
Sep 16 18:45:32.591914 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:45:32.591681    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/3f382427-dc87-4034-ae03-c457ed18d446-tmp\") pod \"tuned-7t742\" (UID: \"3f382427-dc87-4034-ae03-c457ed18d446\") " pod="openshift-cluster-node-tuning-operator/tuned-7t742"
Sep 16 18:45:32.591914 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:45:32.591704    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/eace7050-d66d-4057-9041-62afa5df8ae7-hosts-file\") pod \"node-resolver-lrclz\" (UID: \"eace7050-d66d-4057-9041-62afa5df8ae7\") " pod="openshift-dns/node-resolver-lrclz"
Sep 16 18:45:32.591914 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:45:32.591740    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wb5w6\" (UniqueName: \"kubernetes.io/projected/eace7050-d66d-4057-9041-62afa5df8ae7-kube-api-access-wb5w6\") pod \"node-resolver-lrclz\" (UID: \"eace7050-d66d-4057-9041-62afa5df8ae7\") " pod="openshift-dns/node-resolver-lrclz"
Sep 16 18:45:32.591914 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:45:32.591769    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/3f382427-dc87-4034-ae03-c457ed18d446-etc-modprobe-d\") pod \"tuned-7t742\" (UID: \"3f382427-dc87-4034-ae03-c457ed18d446\") " pod="openshift-cluster-node-tuning-operator/tuned-7t742"
Sep 16 18:45:32.591914 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:45:32.591795    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/3f382427-dc87-4034-ae03-c457ed18d446-etc-sysctl-conf\") pod \"tuned-7t742\" (UID: \"3f382427-dc87-4034-ae03-c457ed18d446\") " pod="openshift-cluster-node-tuning-operator/tuned-7t742"
Sep 16 18:45:32.591914 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:45:32.591824    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q7kmz\" (UniqueName: \"kubernetes.io/projected/3f382427-dc87-4034-ae03-c457ed18d446-kube-api-access-q7kmz\") pod \"tuned-7t742\" (UID: \"3f382427-dc87-4034-ae03-c457ed18d446\") " pod="openshift-cluster-node-tuning-operator/tuned-7t742"
Sep 16 18:45:32.591914 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:45:32.591851    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5vmqf\" (UniqueName: \"kubernetes.io/projected/4775e8c4-09ba-43db-98b7-073aeb452ce8-kube-api-access-5vmqf\") pod \"network-check-target-bz5jk\" (UID: \"4775e8c4-09ba-43db-98b7-073aeb452ce8\") " pod="openshift-network-diagnostics/network-check-target-bz5jk"
Sep 16 18:45:32.591914 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:45:32.591880    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-g6m2q\""
Sep 16 18:45:32.591914 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:45:32.591886    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/b31b8735-005e-4387-852c-bc67df0e77cb-agent-certs\") pod \"konnectivity-agent-vttdl\" (UID: \"b31b8735-005e-4387-852c-bc67df0e77cb\") " pod="kube-system/konnectivity-agent-vttdl"
Sep 16 18:45:32.591914 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:45:32.591914    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/b31b8735-005e-4387-852c-bc67df0e77cb-konnectivity-ca\") pod \"konnectivity-agent-vttdl\" (UID: \"b31b8735-005e-4387-852c-bc67df0e77cb\") " pod="kube-system/konnectivity-agent-vttdl"
Sep 16 18:45:32.592471 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:45:32.591939    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/3f382427-dc87-4034-ae03-c457ed18d446-host\") pod \"tuned-7t742\" (UID: \"3f382427-dc87-4034-ae03-c457ed18d446\") " pod="openshift-cluster-node-tuning-operator/tuned-7t742"
Sep 16 18:45:32.592471 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:45:32.591989    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/3f382427-dc87-4034-ae03-c457ed18d446-etc-tuned\") pod \"tuned-7t742\" (UID: \"3f382427-dc87-4034-ae03-c457ed18d446\") " pod="openshift-cluster-node-tuning-operator/tuned-7t742"
Sep 16 18:45:32.592471 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:45:32.592013    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/eace7050-d66d-4057-9041-62afa5df8ae7-tmp-dir\") pod \"node-resolver-lrclz\" (UID: \"eace7050-d66d-4057-9041-62afa5df8ae7\") " pod="openshift-dns/node-resolver-lrclz"
Sep 16 18:45:32.592471 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:45:32.592042    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/f09babc9-6295-4a0a-be9f-fa0175b0de51-kubelet-config\") pod \"global-pull-secret-syncer-c78nz\" (UID: \"f09babc9-6295-4a0a-be9f-fa0175b0de51\") " pod="kube-system/global-pull-secret-syncer-c78nz"
Sep 16 18:45:32.592471 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:45:32.592068    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/f09babc9-6295-4a0a-be9f-fa0175b0de51-original-pull-secret\") pod \"global-pull-secret-syncer-c78nz\" (UID: \"f09babc9-6295-4a0a-be9f-fa0175b0de51\") " pod="kube-system/global-pull-secret-syncer-c78nz"
Sep 16 18:45:32.592471 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:45:32.592107    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Sep 16 18:45:32.592471 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:45:32.592316    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Sep 16 18:45:32.592815 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:45:32.592568    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Sep 16 18:45:32.592815 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:45:32.592595    2581 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Sep 16 18:45:32.592815 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:45:32.592674    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Sep 16 18:45:32.592963 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:45:32.592823    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Sep 16 18:45:32.593033 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:45:32.593011    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-fzt7c"
Sep 16 18:45:32.596902 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:45:32.595349    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-c2kp2"
Sep 16 18:45:32.596902 ip-10-0-138-127 kubenswrapper[2581]: E0916 18:45:32.595446    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-c2kp2" podUID="eaf5897a-76a0-4bf3-aa4e-59aa6c193ddb"
Sep 16 18:45:32.596902 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:45:32.596653    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Sep 16 18:45:32.597173 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:45:32.597150    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Sep 16 18:45:32.597310 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:45:32.597287    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-vjwhx\""
Sep 16 18:45:32.597518 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:45:32.597497    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Sep 16 18:45:32.597629 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:45:32.597541    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Sep 16 18:45:32.597695 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:45:32.597500    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Sep 16 18:45:32.598017 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:45:32.597997    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-fbmb7"
Sep 16 18:45:32.600396 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:45:32.600373    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Sep 16 18:45:32.600601 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:45:32.600582    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Sep 16 18:45:32.600601 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:45:32.600378    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Sep 16 18:45:32.601786 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:45:32.601147    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-vxjzx"
Sep 16 18:45:32.602339 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:45:32.602305    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-mdgjd\""
Sep 16 18:45:32.603451 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:45:32.603429    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Sep 16 18:45:32.603556 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:45:32.603461    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-h7jwr\""
Sep 16 18:45:32.603935 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:45:32.603918    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-mvhf8"
Sep 16 18:45:32.606001 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:45:32.605980    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Sep 16 18:45:32.606308 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:45:32.606288    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Sep 16 18:45:32.606595 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:45:32.606578    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-hssbl\""
Sep 16 18:45:32.606806 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:45:32.606790    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Sep 16 18:45:32.627911 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:45:32.627886    2581 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-nm8j4"
Sep 16 18:45:32.638099 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:45:32.637956    2581 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-nm8j4"
Sep 16 18:45:32.683339 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:45:32.683311    2581 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Sep 16 18:45:32.692945 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:45:32.692908    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/99b45db8-0fbb-445b-be31-49fba3f220ec-host\") pod \"node-ca-mvhf8\" (UID: \"99b45db8-0fbb-445b-be31-49fba3f220ec\") " pod="openshift-image-registry/node-ca-mvhf8"
Sep 16 18:45:32.693126 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:45:32.692959    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/3f382427-dc87-4034-ae03-c457ed18d446-lib-modules\") pod \"tuned-7t742\" (UID: \"3f382427-dc87-4034-ae03-c457ed18d446\") " pod="openshift-cluster-node-tuning-operator/tuned-7t742"
Sep 16 18:45:32.693126 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:45:32.692990    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wb5w6\" (UniqueName: \"kubernetes.io/projected/eace7050-d66d-4057-9041-62afa5df8ae7-kube-api-access-wb5w6\") pod \"node-resolver-lrclz\" (UID: \"eace7050-d66d-4057-9041-62afa5df8ae7\") " pod="openshift-dns/node-resolver-lrclz"
Sep 16 18:45:32.693126 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:45:32.693018    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/b31b8735-005e-4387-852c-bc67df0e77cb-konnectivity-ca\") pod \"konnectivity-agent-vttdl\" (UID: \"b31b8735-005e-4387-852c-bc67df0e77cb\") " pod="kube-system/konnectivity-agent-vttdl"
Sep 16 18:45:32.693126 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:45:32.693044    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/99e923fa-5e08-487f-ada3-1692ef20ca11-host-cni-bin\") pod \"ovnkube-node-csmcl\" (UID: \"99e923fa-5e08-487f-ada3-1692ef20ca11\") " pod="openshift-ovn-kubernetes/ovnkube-node-csmcl"
Sep 16 18:45:32.693126 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:45:32.693069    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/bf866eb9-d1c0-4cb2-ad9c-86e8b5b306d7-kubelet-dir\") pod \"aws-ebs-csi-driver-node-54w2k\" (UID: \"bf866eb9-d1c0-4cb2-ad9c-86e8b5b306d7\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-54w2k"
Sep 16 18:45:32.693126 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:45:32.693095    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/5acb5489-33eb-4c23-b35d-484caffde098-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-fzt7c\" (UID: \"5acb5489-33eb-4c23-b35d-484caffde098\") " pod="openshift-multus/multus-additional-cni-plugins-fzt7c"
Sep 16 18:45:32.693572 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:45:32.693122    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q7kmz\" (UniqueName: \"kubernetes.io/projected/3f382427-dc87-4034-ae03-c457ed18d446-kube-api-access-q7kmz\") pod \"tuned-7t742\" (UID: \"3f382427-dc87-4034-ae03-c457ed18d446\") " pod="openshift-cluster-node-tuning-operator/tuned-7t742"
Sep 16 18:45:32.693572 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:45:32.693168    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5vmqf\" (UniqueName: \"kubernetes.io/projected/4775e8c4-09ba-43db-98b7-073aeb452ce8-kube-api-access-5vmqf\") pod \"network-check-target-bz5jk\" (UID: \"4775e8c4-09ba-43db-98b7-073aeb452ce8\") " pod="openshift-network-diagnostics/network-check-target-bz5jk"
Sep 16 18:45:32.693572 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:45:32.693195    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/bf1c06c8-20b0-4816-a092-02e87cd69a7f-host-run-multus-certs\") pod \"multus-vxjzx\" (UID: \"bf1c06c8-20b0-4816-a092-02e87cd69a7f\") " pod="openshift-multus/multus-vxjzx"
Sep 16 18:45:32.693572 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:45:32.693219    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/bf866eb9-d1c0-4cb2-ad9c-86e8b5b306d7-device-dir\") pod \"aws-ebs-csi-driver-node-54w2k\" (UID: \"bf866eb9-d1c0-4cb2-ad9c-86e8b5b306d7\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-54w2k"
Sep 16 18:45:32.693572 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:45:32.693244    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/3f382427-dc87-4034-ae03-c457ed18d446-lib-modules\") pod \"tuned-7t742\" (UID: \"3f382427-dc87-4034-ae03-c457ed18d446\") " pod="openshift-cluster-node-tuning-operator/tuned-7t742"
Sep 16 18:45:32.693572 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:45:32.693273    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/5acb5489-33eb-4c23-b35d-484caffde098-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-fzt7c\" (UID: \"5acb5489-33eb-4c23-b35d-484caffde098\") " pod="openshift-multus/multus-additional-cni-plugins-fzt7c"
Sep 16 18:45:32.693572 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:45:32.693320    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/f09babc9-6295-4a0a-be9f-fa0175b0de51-original-pull-secret\") pod \"global-pull-secret-syncer-c78nz\" (UID: \"f09babc9-6295-4a0a-be9f-fa0175b0de51\") " pod="kube-system/global-pull-secret-syncer-c78nz"
Sep 16 18:45:32.693572 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:45:32.693346    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/459060f0-3e55-498a-b878-726f3e3a1060-iptables-alerter-script\") pod \"iptables-alerter-fbmb7\" (UID: \"459060f0-3e55-498a-b878-726f3e3a1060\") " pod="openshift-network-operator/iptables-alerter-fbmb7"
Sep 16 18:45:32.693572 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:45:32.693372    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/bf866eb9-d1c0-4cb2-ad9c-86e8b5b306d7-sys-fs\") pod \"aws-ebs-csi-driver-node-54w2k\" (UID: \"bf866eb9-d1c0-4cb2-ad9c-86e8b5b306d7\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-54w2k"
Sep 16 18:45:32.693572 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:45:32.693513    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7lljq\" (UniqueName: \"kubernetes.io/projected/bf866eb9-d1c0-4cb2-ad9c-86e8b5b306d7-kube-api-access-7lljq\") pod \"aws-ebs-csi-driver-node-54w2k\" (UID: \"bf866eb9-d1c0-4cb2-ad9c-86e8b5b306d7\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-54w2k"
Sep 16 18:45:32.693572 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:45:32.693551    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/5acb5489-33eb-4c23-b35d-484caffde098-tuning-conf-dir\") pod \"multus-additional-cni-plugins-fzt7c\" (UID: \"5acb5489-33eb-4c23-b35d-484caffde098\") " pod="openshift-multus/multus-additional-cni-plugins-fzt7c"
Sep 16 18:45:32.694117 ip-10-0-138-127 kubenswrapper[2581]: E0916 18:45:32.693607    2581 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 16 18:45:32.694117 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:45:32.693639    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/eaf5897a-76a0-4bf3-aa4e-59aa6c193ddb-metrics-certs\") pod \"network-metrics-daemon-c2kp2\" (UID: \"eaf5897a-76a0-4bf3-aa4e-59aa6c193ddb\") " pod="openshift-multus/network-metrics-daemon-c2kp2"
Sep 16 18:45:32.694117 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:45:32.693658    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/3f382427-dc87-4034-ae03-c457ed18d446-etc-kubernetes\") pod \"tuned-7t742\" (UID: \"3f382427-dc87-4034-ae03-c457ed18d446\") " pod="openshift-cluster-node-tuning-operator/tuned-7t742"
Sep 16 18:45:32.694117 ip-10-0-138-127 kubenswrapper[2581]: E0916 18:45:32.693740    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f09babc9-6295-4a0a-be9f-fa0175b0de51-original-pull-secret podName:f09babc9-6295-4a0a-be9f-fa0175b0de51 nodeName:}" failed. No retries permitted until 2026-09-16 18:45:33.193713986 +0000 UTC m=+3.013127901 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/f09babc9-6295-4a0a-be9f-fa0175b0de51-original-pull-secret") pod "global-pull-secret-syncer-c78nz" (UID: "f09babc9-6295-4a0a-be9f-fa0175b0de51") : object "kube-system"/"original-pull-secret" not registered
Sep 16 18:45:32.694117 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:45:32.693776    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/eace7050-d66d-4057-9041-62afa5df8ae7-hosts-file\") pod \"node-resolver-lrclz\" (UID: \"eace7050-d66d-4057-9041-62afa5df8ae7\") " pod="openshift-dns/node-resolver-lrclz"
Sep 16 18:45:32.694117 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:45:32.693784    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/b31b8735-005e-4387-852c-bc67df0e77cb-konnectivity-ca\") pod \"konnectivity-agent-vttdl\" (UID: \"b31b8735-005e-4387-852c-bc67df0e77cb\") " pod="kube-system/konnectivity-agent-vttdl"
Sep 16 18:45:32.694117 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:45:32.693831    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/5acb5489-33eb-4c23-b35d-484caffde098-cnibin\") pod \"multus-additional-cni-plugins-fzt7c\" (UID: \"5acb5489-33eb-4c23-b35d-484caffde098\") " pod="openshift-multus/multus-additional-cni-plugins-fzt7c"
Sep 16 18:45:32.694117 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:45:32.693864    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7xm4p\" (UniqueName: \"kubernetes.io/projected/5acb5489-33eb-4c23-b35d-484caffde098-kube-api-access-7xm4p\") pod \"multus-additional-cni-plugins-fzt7c\" (UID: \"5acb5489-33eb-4c23-b35d-484caffde098\") " pod="openshift-multus/multus-additional-cni-plugins-fzt7c"
Sep 16 18:45:32.694117 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:45:32.693866    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/3f382427-dc87-4034-ae03-c457ed18d446-etc-kubernetes\") pod \"tuned-7t742\" (UID: \"3f382427-dc87-4034-ae03-c457ed18d446\") " pod="openshift-cluster-node-tuning-operator/tuned-7t742"
Sep 16 18:45:32.694117 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:45:32.693903    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/b31b8735-005e-4387-852c-bc67df0e77cb-agent-certs\") pod \"konnectivity-agent-vttdl\" (UID: \"b31b8735-005e-4387-852c-bc67df0e77cb\") " pod="kube-system/konnectivity-agent-vttdl"
Sep 16 18:45:32.694117 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:45:32.693910    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/eace7050-d66d-4057-9041-62afa5df8ae7-hosts-file\") pod \"node-resolver-lrclz\" (UID: \"eace7050-d66d-4057-9041-62afa5df8ae7\") " pod="openshift-dns/node-resolver-lrclz"
Sep 16 18:45:32.694117 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:45:32.693950    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/99e923fa-5e08-487f-ada3-1692ef20ca11-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-csmcl\" (UID: \"99e923fa-5e08-487f-ada3-1692ef20ca11\") " pod="openshift-ovn-kubernetes/ovnkube-node-csmcl"
Sep 16 18:45:32.694117 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:45:32.694001    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/99e923fa-5e08-487f-ada3-1692ef20ca11-ovnkube-script-lib\") pod \"ovnkube-node-csmcl\" (UID: \"99e923fa-5e08-487f-ada3-1692ef20ca11\") " pod="openshift-ovn-kubernetes/ovnkube-node-csmcl"
Sep 16 18:45:32.694117 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:45:32.694037    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/459060f0-3e55-498a-b878-726f3e3a1060-host-slash\") pod \"iptables-alerter-fbmb7\" (UID: \"459060f0-3e55-498a-b878-726f3e3a1060\") " pod="openshift-network-operator/iptables-alerter-fbmb7"
Sep 16 18:45:32.694117 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:45:32.694063    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/bf1c06c8-20b0-4816-a092-02e87cd69a7f-host-run-k8s-cni-cncf-io\") pod \"multus-vxjzx\" (UID: \"bf1c06c8-20b0-4816-a092-02e87cd69a7f\") " pod="openshift-multus/multus-vxjzx"
Sep 16 18:45:32.694117 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:45:32.694091    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/bf866eb9-d1c0-4cb2-ad9c-86e8b5b306d7-etc-selinux\") pod \"aws-ebs-csi-driver-node-54w2k\" (UID: \"bf866eb9-d1c0-4cb2-ad9c-86e8b5b306d7\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-54w2k"
Sep 16 18:45:32.695030 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:45:32.694120    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/3f382427-dc87-4034-ae03-c457ed18d446-host\") pod \"tuned-7t742\" (UID: \"3f382427-dc87-4034-ae03-c457ed18d446\") " pod="openshift-cluster-node-tuning-operator/tuned-7t742"
Sep 16 18:45:32.695030 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:45:32.694146    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/eace7050-d66d-4057-9041-62afa5df8ae7-tmp-dir\") pod \"node-resolver-lrclz\" (UID: \"eace7050-d66d-4057-9041-62afa5df8ae7\") " pod="openshift-dns/node-resolver-lrclz"
Sep 16 18:45:32.695030 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:45:32.694170    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/99e923fa-5e08-487f-ada3-1692ef20ca11-run-systemd\") pod \"ovnkube-node-csmcl\" (UID: \"99e923fa-5e08-487f-ada3-1692ef20ca11\") " pod="openshift-ovn-kubernetes/ovnkube-node-csmcl"
Sep 16 18:45:32.695030 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:45:32.694179    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/3f382427-dc87-4034-ae03-c457ed18d446-host\") pod \"tuned-7t742\" (UID: \"3f382427-dc87-4034-ae03-c457ed18d446\") " pod="openshift-cluster-node-tuning-operator/tuned-7t742"
Sep 16 18:45:32.695030 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:45:32.694194    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/99e923fa-5e08-487f-ada3-1692ef20ca11-host-cni-netd\") pod \"ovnkube-node-csmcl\" (UID: \"99e923fa-5e08-487f-ada3-1692ef20ca11\") " pod="openshift-ovn-kubernetes/ovnkube-node-csmcl"
Sep 16 18:45:32.695030 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:45:32.694245    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/bf1c06c8-20b0-4816-a092-02e87cd69a7f-system-cni-dir\") pod \"multus-vxjzx\" (UID: \"bf1c06c8-20b0-4816-a092-02e87cd69a7f\") " pod="openshift-multus/multus-vxjzx"
Sep 16 18:45:32.695030 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:45:32.694289    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/bf1c06c8-20b0-4816-a092-02e87cd69a7f-host-run-netns\") pod \"multus-vxjzx\" (UID: \"bf1c06c8-20b0-4816-a092-02e87cd69a7f\") " pod="openshift-multus/multus-vxjzx"
Sep 16 18:45:32.695030 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:45:32.694331    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c72bq\" (UniqueName: \"kubernetes.io/projected/99b45db8-0fbb-445b-be31-49fba3f220ec-kube-api-access-c72bq\") pod \"node-ca-mvhf8\" (UID: \"99b45db8-0fbb-445b-be31-49fba3f220ec\") " pod="openshift-image-registry/node-ca-mvhf8"
Sep 16 18:45:32.695030 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:45:32.694356    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/5acb5489-33eb-4c23-b35d-484caffde098-system-cni-dir\") pod \"multus-additional-cni-plugins-fzt7c\" (UID: \"5acb5489-33eb-4c23-b35d-484caffde098\") " pod="openshift-multus/multus-additional-cni-plugins-fzt7c"
Sep 16 18:45:32.695030 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:45:32.694381    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8qzq2\" (UniqueName: \"kubernetes.io/projected/99e923fa-5e08-487f-ada3-1692ef20ca11-kube-api-access-8qzq2\") pod \"ovnkube-node-csmcl\" (UID: \"99e923fa-5e08-487f-ada3-1692ef20ca11\") " pod="openshift-ovn-kubernetes/ovnkube-node-csmcl"
Sep 16 18:45:32.695030 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:45:32.694408    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/bf866eb9-d1c0-4cb2-ad9c-86e8b5b306d7-registration-dir\") pod \"aws-ebs-csi-driver-node-54w2k\" (UID: \"bf866eb9-d1c0-4cb2-ad9c-86e8b5b306d7\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-54w2k"
Sep 16 18:45:32.695030 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:45:32.694433    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/3f382427-dc87-4034-ae03-c457ed18d446-tmp\") pod \"tuned-7t742\" (UID: \"3f382427-dc87-4034-ae03-c457ed18d446\") " pod="openshift-cluster-node-tuning-operator/tuned-7t742"
Sep 16 18:45:32.695030 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:45:32.694458    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/bf866eb9-d1c0-4cb2-ad9c-86e8b5b306d7-socket-dir\") pod \"aws-ebs-csi-driver-node-54w2k\" (UID: \"bf866eb9-d1c0-4cb2-ad9c-86e8b5b306d7\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-54w2k"
Sep 16 18:45:32.695030 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:45:32.694485    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/3f382427-dc87-4034-ae03-c457ed18d446-etc-modprobe-d\") pod \"tuned-7t742\" (UID: \"3f382427-dc87-4034-ae03-c457ed18d446\") " pod="openshift-cluster-node-tuning-operator/tuned-7t742"
Sep 16 18:45:32.695030 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:45:32.694497    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/eace7050-d66d-4057-9041-62afa5df8ae7-tmp-dir\") pod \"node-resolver-lrclz\" (UID: \"eace7050-d66d-4057-9041-62afa5df8ae7\") " pod="openshift-dns/node-resolver-lrclz"
Sep 16 18:45:32.695030 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:45:32.694508    2581 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Sep 16 18:45:32.695030 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:45:32.694510    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/3f382427-dc87-4034-ae03-c457ed18d446-etc-sysctl-conf\") pod \"tuned-7t742\" (UID: \"3f382427-dc87-4034-ae03-c457ed18d446\") " pod="openshift-cluster-node-tuning-operator/tuned-7t742"
Sep 16 18:45:32.696004 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:45:32.694569    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/99e923fa-5e08-487f-ada3-1692ef20ca11-host-run-netns\") pod \"ovnkube-node-csmcl\" (UID: \"99e923fa-5e08-487f-ada3-1692ef20ca11\") " pod="openshift-ovn-kubernetes/ovnkube-node-csmcl"
Sep 16 18:45:32.696004 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:45:32.694610    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/99e923fa-5e08-487f-ada3-1692ef20ca11-run-openvswitch\") pod \"ovnkube-node-csmcl\" (UID: \"99e923fa-5e08-487f-ada3-1692ef20ca11\") " pod="openshift-ovn-kubernetes/ovnkube-node-csmcl"
Sep 16 18:45:32.696004 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:45:32.694640    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/bf1c06c8-20b0-4816-a092-02e87cd69a7f-cnibin\") pod \"multus-vxjzx\" (UID: \"bf1c06c8-20b0-4816-a092-02e87cd69a7f\") " pod="openshift-multus/multus-vxjzx"
Sep 16 18:45:32.696004 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:45:32.694665    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/bf1c06c8-20b0-4816-a092-02e87cd69a7f-multus-conf-dir\") pod \"multus-vxjzx\" (UID: \"bf1c06c8-20b0-4816-a092-02e87cd69a7f\") " pod="openshift-multus/multus-vxjzx"
Sep 16 18:45:32.696004 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:45:32.694695    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r7qw6\" (UniqueName: \"kubernetes.io/projected/bf1c06c8-20b0-4816-a092-02e87cd69a7f-kube-api-access-r7qw6\") pod \"multus-vxjzx\" (UID: \"bf1c06c8-20b0-4816-a092-02e87cd69a7f\") " pod="openshift-multus/multus-vxjzx"
Sep 16 18:45:32.696004 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:45:32.694721    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/5acb5489-33eb-4c23-b35d-484caffde098-cni-binary-copy\") pod \"multus-additional-cni-plugins-fzt7c\" (UID: \"5acb5489-33eb-4c23-b35d-484caffde098\") " pod="openshift-multus/multus-additional-cni-plugins-fzt7c"
Sep 16 18:45:32.696004 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:45:32.694749    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/3f382427-dc87-4034-ae03-c457ed18d446-etc-tuned\") pod \"tuned-7t742\" (UID: \"3f382427-dc87-4034-ae03-c457ed18d446\") " pod="openshift-cluster-node-tuning-operator/tuned-7t742"
Sep 16 18:45:32.696004 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:45:32.694776    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/f09babc9-6295-4a0a-be9f-fa0175b0de51-kubelet-config\") pod \"global-pull-secret-syncer-c78nz\" (UID: \"f09babc9-6295-4a0a-be9f-fa0175b0de51\") " pod="kube-system/global-pull-secret-syncer-c78nz"
Sep 16 18:45:32.696004 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:45:32.694804    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/99e923fa-5e08-487f-ada3-1692ef20ca11-env-overrides\") pod \"ovnkube-node-csmcl\" (UID: \"99e923fa-5e08-487f-ada3-1692ef20ca11\") " pod="openshift-ovn-kubernetes/ovnkube-node-csmcl"
Sep 16 18:45:32.696004 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:45:32.694832    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/bf1c06c8-20b0-4816-a092-02e87cd69a7f-cni-binary-copy\") pod \"multus-vxjzx\" (UID: \"bf1c06c8-20b0-4816-a092-02e87cd69a7f\") " pod="openshift-multus/multus-vxjzx"
Sep 16 18:45:32.696004 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:45:32.694858    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/bf1c06c8-20b0-4816-a092-02e87cd69a7f-host-var-lib-cni-bin\") pod \"multus-vxjzx\" (UID: \"bf1c06c8-20b0-4816-a092-02e87cd69a7f\") " pod="openshift-multus/multus-vxjzx"
Sep 16 18:45:32.696004 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:45:32.694883    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-26lrg\" (UniqueName: \"kubernetes.io/projected/eaf5897a-76a0-4bf3-aa4e-59aa6c193ddb-kube-api-access-26lrg\") pod \"network-metrics-daemon-c2kp2\" (UID: \"eaf5897a-76a0-4bf3-aa4e-59aa6c193ddb\") " pod="openshift-multus/network-metrics-daemon-c2kp2"
Sep 16 18:45:32.696004 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:45:32.694911    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/3f382427-dc87-4034-ae03-c457ed18d446-etc-systemd\") pod \"tuned-7t742\" (UID: \"3f382427-dc87-4034-ae03-c457ed18d446\") " pod="openshift-cluster-node-tuning-operator/tuned-7t742"
Sep 16 18:45:32.696004 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:45:32.694935    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/3f382427-dc87-4034-ae03-c457ed18d446-run\") pod \"tuned-7t742\" (UID: \"3f382427-dc87-4034-ae03-c457ed18d446\") " pod="openshift-cluster-node-tuning-operator/tuned-7t742"
Sep 16 18:45:32.696004 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:45:32.694960    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/3f382427-dc87-4034-ae03-c457ed18d446-sys\") pod \"tuned-7t742\" (UID: \"3f382427-dc87-4034-ae03-c457ed18d446\") " pod="openshift-cluster-node-tuning-operator/tuned-7t742"
Sep 16 18:45:32.696004 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:45:32.694985    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/3f382427-dc87-4034-ae03-c457ed18d446-var-lib-kubelet\") pod \"tuned-7t742\" (UID: \"3f382427-dc87-4034-ae03-c457ed18d446\") " pod="openshift-cluster-node-tuning-operator/tuned-7t742"
Sep 16 18:45:32.696004 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:45:32.695011    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/99e923fa-5e08-487f-ada3-1692ef20ca11-systemd-units\") pod \"ovnkube-node-csmcl\" (UID: \"99e923fa-5e08-487f-ada3-1692ef20ca11\") " pod="openshift-ovn-kubernetes/ovnkube-node-csmcl"
Sep 16 18:45:32.697007 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:45:32.695035    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/99e923fa-5e08-487f-ada3-1692ef20ca11-node-log\") pod \"ovnkube-node-csmcl\" (UID: \"99e923fa-5e08-487f-ada3-1692ef20ca11\") " pod="openshift-ovn-kubernetes/ovnkube-node-csmcl"
Sep 16 18:45:32.697007 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:45:32.695059    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/99e923fa-5e08-487f-ada3-1692ef20ca11-log-socket\") pod \"ovnkube-node-csmcl\" (UID: \"99e923fa-5e08-487f-ada3-1692ef20ca11\") " pod="openshift-ovn-kubernetes/ovnkube-node-csmcl"
Sep 16 18:45:32.697007 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:45:32.695088    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/3f382427-dc87-4034-ae03-c457ed18d446-etc-sysconfig\") pod \"tuned-7t742\" (UID: \"3f382427-dc87-4034-ae03-c457ed18d446\") " pod="openshift-cluster-node-tuning-operator/tuned-7t742"
Sep 16 18:45:32.697007 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:45:32.695114    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/99e923fa-5e08-487f-ada3-1692ef20ca11-host-slash\") pod \"ovnkube-node-csmcl\" (UID: \"99e923fa-5e08-487f-ada3-1692ef20ca11\") " pod="openshift-ovn-kubernetes/ovnkube-node-csmcl"
Sep 16 18:45:32.697007 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:45:32.695144    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/99e923fa-5e08-487f-ada3-1692ef20ca11-etc-openvswitch\") pod \"ovnkube-node-csmcl\" (UID: \"99e923fa-5e08-487f-ada3-1692ef20ca11\") " pod="openshift-ovn-kubernetes/ovnkube-node-csmcl"
Sep 16 18:45:32.697007 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:45:32.695169    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/99e923fa-5e08-487f-ada3-1692ef20ca11-run-ovn\") pod \"ovnkube-node-csmcl\" (UID: \"99e923fa-5e08-487f-ada3-1692ef20ca11\") " pod="openshift-ovn-kubernetes/ovnkube-node-csmcl"
Sep 16 18:45:32.697007 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:45:32.695195    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/99e923fa-5e08-487f-ada3-1692ef20ca11-ovn-node-metrics-cert\") pod \"ovnkube-node-csmcl\" (UID: \"99e923fa-5e08-487f-ada3-1692ef20ca11\") " pod="openshift-ovn-kubernetes/ovnkube-node-csmcl"
Sep 16 18:45:32.697007 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:45:32.695257    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/bf1c06c8-20b0-4816-a092-02e87cd69a7f-multus-socket-dir-parent\") pod \"multus-vxjzx\" (UID: \"bf1c06c8-20b0-4816-a092-02e87cd69a7f\") " pod="openshift-multus/multus-vxjzx"
Sep 16 18:45:32.697007 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:45:32.695303    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/bf1c06c8-20b0-4816-a092-02e87cd69a7f-host-var-lib-cni-multus\") pod \"multus-vxjzx\" (UID: \"bf1c06c8-20b0-4816-a092-02e87cd69a7f\") " pod="openshift-multus/multus-vxjzx"
Sep 16 18:45:32.697007 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:45:32.695330    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/bf1c06c8-20b0-4816-a092-02e87cd69a7f-host-var-lib-kubelet\") pod \"multus-vxjzx\" (UID: \"bf1c06c8-20b0-4816-a092-02e87cd69a7f\") " pod="openshift-multus/multus-vxjzx"
Sep 16 18:45:32.697007 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:45:32.695376    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/99e923fa-5e08-487f-ada3-1692ef20ca11-host-kubelet\") pod \"ovnkube-node-csmcl\" (UID: \"99e923fa-5e08-487f-ada3-1692ef20ca11\") " pod="openshift-ovn-kubernetes/ovnkube-node-csmcl"
Sep 16 18:45:32.697007 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:45:32.695403    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/bf1c06c8-20b0-4816-a092-02e87cd69a7f-multus-cni-dir\") pod \"multus-vxjzx\" (UID: \"bf1c06c8-20b0-4816-a092-02e87cd69a7f\") " pod="openshift-multus/multus-vxjzx"
Sep 16 18:45:32.697007 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:45:32.695419    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/3f382427-dc87-4034-ae03-c457ed18d446-etc-modprobe-d\") pod \"tuned-7t742\" (UID: \"3f382427-dc87-4034-ae03-c457ed18d446\") " pod="openshift-cluster-node-tuning-operator/tuned-7t742"
Sep 16 18:45:32.697007 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:45:32.695437    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/bf1c06c8-20b0-4816-a092-02e87cd69a7f-multus-daemon-config\") pod \"multus-vxjzx\" (UID: \"bf1c06c8-20b0-4816-a092-02e87cd69a7f\") " pod="openshift-multus/multus-vxjzx"
Sep 16 18:45:32.697007 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:45:32.695465    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/99e923fa-5e08-487f-ada3-1692ef20ca11-host-run-ovn-kubernetes\") pod \"ovnkube-node-csmcl\" (UID: \"99e923fa-5e08-487f-ada3-1692ef20ca11\") " pod="openshift-ovn-kubernetes/ovnkube-node-csmcl"
Sep 16 18:45:32.697007 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:45:32.695491    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/bf1c06c8-20b0-4816-a092-02e87cd69a7f-hostroot\") pod \"multus-vxjzx\" (UID: \"bf1c06c8-20b0-4816-a092-02e87cd69a7f\") " pod="openshift-multus/multus-vxjzx"
Sep 16 18:45:32.697007 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:45:32.695520    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/bf1c06c8-20b0-4816-a092-02e87cd69a7f-etc-kubernetes\") pod \"multus-vxjzx\" (UID: \"bf1c06c8-20b0-4816-a092-02e87cd69a7f\") " pod="openshift-multus/multus-vxjzx"
Sep 16 18:45:32.698208 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:45:32.695574    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/99b45db8-0fbb-445b-be31-49fba3f220ec-serviceca\") pod \"node-ca-mvhf8\" (UID: \"99b45db8-0fbb-445b-be31-49fba3f220ec\") " pod="openshift-image-registry/node-ca-mvhf8"
Sep 16 18:45:32.698208 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:45:32.695601    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/3f382427-dc87-4034-ae03-c457ed18d446-etc-sysctl-d\") pod \"tuned-7t742\" (UID: \"3f382427-dc87-4034-ae03-c457ed18d446\") " pod="openshift-cluster-node-tuning-operator/tuned-7t742"
Sep 16 18:45:32.698208 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:45:32.695614    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/3f382427-dc87-4034-ae03-c457ed18d446-var-lib-kubelet\") pod \"tuned-7t742\" (UID: \"3f382427-dc87-4034-ae03-c457ed18d446\") " pod="openshift-cluster-node-tuning-operator/tuned-7t742"
Sep 16 18:45:32.698208 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:45:32.695618    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/3f382427-dc87-4034-ae03-c457ed18d446-etc-sysctl-conf\") pod \"tuned-7t742\" (UID: \"3f382427-dc87-4034-ae03-c457ed18d446\") " pod="openshift-cluster-node-tuning-operator/tuned-7t742"
Sep 16 18:45:32.698208 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:45:32.695636    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/f09babc9-6295-4a0a-be9f-fa0175b0de51-dbus\") pod \"global-pull-secret-syncer-c78nz\" (UID: \"f09babc9-6295-4a0a-be9f-fa0175b0de51\") " pod="kube-system/global-pull-secret-syncer-c78nz"
Sep 16 18:45:32.698208 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:45:32.695665    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/99e923fa-5e08-487f-ada3-1692ef20ca11-var-lib-openvswitch\") pod \"ovnkube-node-csmcl\" (UID: \"99e923fa-5e08-487f-ada3-1692ef20ca11\") " pod="openshift-ovn-kubernetes/ovnkube-node-csmcl"
Sep 16 18:45:32.698208 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:45:32.695683    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/f09babc9-6295-4a0a-be9f-fa0175b0de51-kubelet-config\") pod \"global-pull-secret-syncer-c78nz\" (UID: \"f09babc9-6295-4a0a-be9f-fa0175b0de51\") " pod="kube-system/global-pull-secret-syncer-c78nz"
Sep 16 18:45:32.698208 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:45:32.695699    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/99e923fa-5e08-487f-ada3-1692ef20ca11-ovnkube-config\") pod \"ovnkube-node-csmcl\" (UID: \"99e923fa-5e08-487f-ada3-1692ef20ca11\") " pod="openshift-ovn-kubernetes/ovnkube-node-csmcl"
Sep 16 18:45:32.698208 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:45:32.695743    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sgdb6\" (UniqueName: \"kubernetes.io/projected/459060f0-3e55-498a-b878-726f3e3a1060-kube-api-access-sgdb6\") pod \"iptables-alerter-fbmb7\" (UID: \"459060f0-3e55-498a-b878-726f3e3a1060\") " pod="openshift-network-operator/iptables-alerter-fbmb7"
Sep 16 18:45:32.698208 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:45:32.695870    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/3f382427-dc87-4034-ae03-c457ed18d446-etc-sysconfig\") pod \"tuned-7t742\" (UID: \"3f382427-dc87-4034-ae03-c457ed18d446\") " pod="openshift-cluster-node-tuning-operator/tuned-7t742"
Sep 16 18:45:32.698208 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:45:32.696017    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/3f382427-dc87-4034-ae03-c457ed18d446-etc-sysctl-d\") pod \"tuned-7t742\" (UID: \"3f382427-dc87-4034-ae03-c457ed18d446\") " pod="openshift-cluster-node-tuning-operator/tuned-7t742"
Sep 16 18:45:32.698208 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:45:32.696042    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/f09babc9-6295-4a0a-be9f-fa0175b0de51-dbus\") pod \"global-pull-secret-syncer-c78nz\" (UID: \"f09babc9-6295-4a0a-be9f-fa0175b0de51\") " pod="kube-system/global-pull-secret-syncer-c78nz"
Sep 16 18:45:32.698208 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:45:32.696113    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/3f382427-dc87-4034-ae03-c457ed18d446-run\") pod \"tuned-7t742\" (UID: \"3f382427-dc87-4034-ae03-c457ed18d446\") " pod="openshift-cluster-node-tuning-operator/tuned-7t742"
Sep 16 18:45:32.698208 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:45:32.696117    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/3f382427-dc87-4034-ae03-c457ed18d446-etc-systemd\") pod \"tuned-7t742\" (UID: \"3f382427-dc87-4034-ae03-c457ed18d446\") " pod="openshift-cluster-node-tuning-operator/tuned-7t742"
Sep 16 18:45:32.698208 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:45:32.696172    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/3f382427-dc87-4034-ae03-c457ed18d446-sys\") pod \"tuned-7t742\" (UID: \"3f382427-dc87-4034-ae03-c457ed18d446\") " pod="openshift-cluster-node-tuning-operator/tuned-7t742"
Sep 16 18:45:32.699120 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:45:32.699058    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/3f382427-dc87-4034-ae03-c457ed18d446-etc-tuned\") pod \"tuned-7t742\" (UID: \"3f382427-dc87-4034-ae03-c457ed18d446\") " pod="openshift-cluster-node-tuning-operator/tuned-7t742"
Sep 16 18:45:32.699288 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:45:32.699254    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/3f382427-dc87-4034-ae03-c457ed18d446-tmp\") pod \"tuned-7t742\" (UID: \"3f382427-dc87-4034-ae03-c457ed18d446\") " pod="openshift-cluster-node-tuning-operator/tuned-7t742"
Sep 16 18:45:32.699740 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:45:32.699720    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/b31b8735-005e-4387-852c-bc67df0e77cb-agent-certs\") pod \"konnectivity-agent-vttdl\" (UID: \"b31b8735-005e-4387-852c-bc67df0e77cb\") " pod="kube-system/konnectivity-agent-vttdl"
Sep 16 18:45:32.704051 ip-10-0-138-127 kubenswrapper[2581]: E0916 18:45:32.703930    2581 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 16 18:45:32.704051 ip-10-0-138-127 kubenswrapper[2581]: E0916 18:45:32.703955    2581 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 16 18:45:32.704051 ip-10-0-138-127 kubenswrapper[2581]: E0916 18:45:32.703971    2581 projected.go:196] Error preparing data for projected volume kube-api-access-5vmqf for pod openshift-network-diagnostics/network-check-target-bz5jk: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 16 18:45:32.704445 ip-10-0-138-127 kubenswrapper[2581]: E0916 18:45:32.704058    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/4775e8c4-09ba-43db-98b7-073aeb452ce8-kube-api-access-5vmqf podName:4775e8c4-09ba-43db-98b7-073aeb452ce8 nodeName:}" failed. No retries permitted until 2026-09-16 18:45:33.204038709 +0000 UTC m=+3.023452600 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-5vmqf" (UniqueName: "kubernetes.io/projected/4775e8c4-09ba-43db-98b7-073aeb452ce8-kube-api-access-5vmqf") pod "network-check-target-bz5jk" (UID: "4775e8c4-09ba-43db-98b7-073aeb452ce8") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 16 18:45:32.706695 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:45:32.706030    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wb5w6\" (UniqueName: \"kubernetes.io/projected/eace7050-d66d-4057-9041-62afa5df8ae7-kube-api-access-wb5w6\") pod \"node-resolver-lrclz\" (UID: \"eace7050-d66d-4057-9041-62afa5df8ae7\") " pod="openshift-dns/node-resolver-lrclz"
Sep 16 18:45:32.707665 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:45:32.707552    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-q7kmz\" (UniqueName: \"kubernetes.io/projected/3f382427-dc87-4034-ae03-c457ed18d446-kube-api-access-q7kmz\") pod \"tuned-7t742\" (UID: \"3f382427-dc87-4034-ae03-c457ed18d446\") " pod="openshift-cluster-node-tuning-operator/tuned-7t742"
Sep 16 18:45:32.726992 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:45:32.726912    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-127.ec2.internal" event={"ID":"3f19801b96f1829134dfd0b39bce2048","Type":"ContainerStarted","Data":"380e8f99a62a4bf3ab648d08109796fc9892cea142c8c24c631c52ec8481b6f1"}
Sep 16 18:45:32.728498 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:45:32.728443    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-138-127.ec2.internal" event={"ID":"37df0dc0824bc6d3c2498649ba68632f","Type":"ContainerStarted","Data":"40752c933a3498141063fa2836bd1507e22d4cff5785b9fde696e638a3bbda1a"}
Sep 16 18:45:32.796361 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:45:32.796314    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/99e923fa-5e08-487f-ada3-1692ef20ca11-host-run-netns\") pod \"ovnkube-node-csmcl\" (UID: \"99e923fa-5e08-487f-ada3-1692ef20ca11\") " pod="openshift-ovn-kubernetes/ovnkube-node-csmcl"
Sep 16 18:45:32.796361 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:45:32.796365    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/99e923fa-5e08-487f-ada3-1692ef20ca11-run-openvswitch\") pod \"ovnkube-node-csmcl\" (UID: \"99e923fa-5e08-487f-ada3-1692ef20ca11\") " pod="openshift-ovn-kubernetes/ovnkube-node-csmcl"
Sep 16 18:45:32.796641 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:45:32.796394    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/bf1c06c8-20b0-4816-a092-02e87cd69a7f-cnibin\") pod \"multus-vxjzx\" (UID: \"bf1c06c8-20b0-4816-a092-02e87cd69a7f\") " pod="openshift-multus/multus-vxjzx"
Sep 16 18:45:32.796641 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:45:32.796422    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/bf1c06c8-20b0-4816-a092-02e87cd69a7f-multus-conf-dir\") pod \"multus-vxjzx\" (UID: \"bf1c06c8-20b0-4816-a092-02e87cd69a7f\") " pod="openshift-multus/multus-vxjzx"
Sep 16 18:45:32.796641 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:45:32.796437    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/99e923fa-5e08-487f-ada3-1692ef20ca11-host-run-netns\") pod \"ovnkube-node-csmcl\" (UID: \"99e923fa-5e08-487f-ada3-1692ef20ca11\") " pod="openshift-ovn-kubernetes/ovnkube-node-csmcl"
Sep 16 18:45:32.796641 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:45:32.796458    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r7qw6\" (UniqueName: \"kubernetes.io/projected/bf1c06c8-20b0-4816-a092-02e87cd69a7f-kube-api-access-r7qw6\") pod \"multus-vxjzx\" (UID: \"bf1c06c8-20b0-4816-a092-02e87cd69a7f\") " pod="openshift-multus/multus-vxjzx"
Sep 16 18:45:32.796641 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:45:32.796487    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/5acb5489-33eb-4c23-b35d-484caffde098-cni-binary-copy\") pod \"multus-additional-cni-plugins-fzt7c\" (UID: \"5acb5489-33eb-4c23-b35d-484caffde098\") " pod="openshift-multus/multus-additional-cni-plugins-fzt7c"
Sep 16 18:45:32.796641 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:45:32.796513    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/99e923fa-5e08-487f-ada3-1692ef20ca11-env-overrides\") pod \"ovnkube-node-csmcl\" (UID: \"99e923fa-5e08-487f-ada3-1692ef20ca11\") " pod="openshift-ovn-kubernetes/ovnkube-node-csmcl"
Sep 16 18:45:32.796641 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:45:32.796530    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/bf1c06c8-20b0-4816-a092-02e87cd69a7f-cni-binary-copy\") pod \"multus-vxjzx\" (UID: \"bf1c06c8-20b0-4816-a092-02e87cd69a7f\") " pod="openshift-multus/multus-vxjzx"
Sep 16 18:45:32.796641 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:45:32.796536    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/99e923fa-5e08-487f-ada3-1692ef20ca11-run-openvswitch\") pod \"ovnkube-node-csmcl\" (UID: \"99e923fa-5e08-487f-ada3-1692ef20ca11\") " pod="openshift-ovn-kubernetes/ovnkube-node-csmcl"
Sep 16 18:45:32.796641 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:45:32.796545    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/bf1c06c8-20b0-4816-a092-02e87cd69a7f-host-var-lib-cni-bin\") pod \"multus-vxjzx\" (UID: \"bf1c06c8-20b0-4816-a092-02e87cd69a7f\") " pod="openshift-multus/multus-vxjzx"
Sep 16 18:45:32.796641 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:45:32.796561    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-26lrg\" (UniqueName: \"kubernetes.io/projected/eaf5897a-76a0-4bf3-aa4e-59aa6c193ddb-kube-api-access-26lrg\") pod \"network-metrics-daemon-c2kp2\" (UID: \"eaf5897a-76a0-4bf3-aa4e-59aa6c193ddb\") " pod="openshift-multus/network-metrics-daemon-c2kp2"
Sep 16 18:45:32.796641 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:45:32.796579    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/99e923fa-5e08-487f-ada3-1692ef20ca11-systemd-units\") pod \"ovnkube-node-csmcl\" (UID: \"99e923fa-5e08-487f-ada3-1692ef20ca11\") " pod="openshift-ovn-kubernetes/ovnkube-node-csmcl"
Sep 16 18:45:32.796641 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:45:32.796595    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/99e923fa-5e08-487f-ada3-1692ef20ca11-node-log\") pod \"ovnkube-node-csmcl\" (UID: \"99e923fa-5e08-487f-ada3-1692ef20ca11\") " pod="openshift-ovn-kubernetes/ovnkube-node-csmcl"
Sep 16 18:45:32.796641 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:45:32.796609    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/99e923fa-5e08-487f-ada3-1692ef20ca11-log-socket\") pod \"ovnkube-node-csmcl\" (UID: \"99e923fa-5e08-487f-ada3-1692ef20ca11\") " pod="openshift-ovn-kubernetes/ovnkube-node-csmcl"
Sep 16 18:45:32.796641 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:45:32.796624    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/99e923fa-5e08-487f-ada3-1692ef20ca11-host-slash\") pod \"ovnkube-node-csmcl\" (UID: \"99e923fa-5e08-487f-ada3-1692ef20ca11\") " pod="openshift-ovn-kubernetes/ovnkube-node-csmcl"
Sep 16 18:45:32.796641 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:45:32.796638    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/99e923fa-5e08-487f-ada3-1692ef20ca11-etc-openvswitch\") pod \"ovnkube-node-csmcl\" (UID: \"99e923fa-5e08-487f-ada3-1692ef20ca11\") " pod="openshift-ovn-kubernetes/ovnkube-node-csmcl"
Sep 16 18:45:32.797179 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:45:32.796653    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/99e923fa-5e08-487f-ada3-1692ef20ca11-run-ovn\") pod \"ovnkube-node-csmcl\" (UID: \"99e923fa-5e08-487f-ada3-1692ef20ca11\") " pod="openshift-ovn-kubernetes/ovnkube-node-csmcl"
Sep 16 18:45:32.797179 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:45:32.796669    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/99e923fa-5e08-487f-ada3-1692ef20ca11-ovn-node-metrics-cert\") pod \"ovnkube-node-csmcl\" (UID: \"99e923fa-5e08-487f-ada3-1692ef20ca11\") " pod="openshift-ovn-kubernetes/ovnkube-node-csmcl"
Sep 16 18:45:32.797179 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:45:32.796684    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/bf1c06c8-20b0-4816-a092-02e87cd69a7f-multus-socket-dir-parent\") pod \"multus-vxjzx\" (UID: \"bf1c06c8-20b0-4816-a092-02e87cd69a7f\") " pod="openshift-multus/multus-vxjzx"
Sep 16 18:45:32.797179 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:45:32.796698    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/bf1c06c8-20b0-4816-a092-02e87cd69a7f-host-var-lib-cni-multus\") pod \"multus-vxjzx\" (UID: \"bf1c06c8-20b0-4816-a092-02e87cd69a7f\") " pod="openshift-multus/multus-vxjzx"
Sep 16 18:45:32.797179 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:45:32.796714    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/bf1c06c8-20b0-4816-a092-02e87cd69a7f-host-var-lib-kubelet\") pod \"multus-vxjzx\" (UID: \"bf1c06c8-20b0-4816-a092-02e87cd69a7f\") " pod="openshift-multus/multus-vxjzx"
Sep 16 18:45:32.797179 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:45:32.796729    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/99e923fa-5e08-487f-ada3-1692ef20ca11-host-kubelet\") pod \"ovnkube-node-csmcl\" (UID: \"99e923fa-5e08-487f-ada3-1692ef20ca11\") " pod="openshift-ovn-kubernetes/ovnkube-node-csmcl"
Sep 16 18:45:32.797179 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:45:32.796743    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/bf1c06c8-20b0-4816-a092-02e87cd69a7f-multus-cni-dir\") pod \"multus-vxjzx\" (UID: \"bf1c06c8-20b0-4816-a092-02e87cd69a7f\") " pod="openshift-multus/multus-vxjzx"
Sep 16 18:45:32.797179 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:45:32.796778    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/bf1c06c8-20b0-4816-a092-02e87cd69a7f-multus-daemon-config\") pod \"multus-vxjzx\" (UID: \"bf1c06c8-20b0-4816-a092-02e87cd69a7f\") " pod="openshift-multus/multus-vxjzx"
Sep 16 18:45:32.797179 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:45:32.796795    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/99e923fa-5e08-487f-ada3-1692ef20ca11-host-run-ovn-kubernetes\") pod \"ovnkube-node-csmcl\" (UID: \"99e923fa-5e08-487f-ada3-1692ef20ca11\") " pod="openshift-ovn-kubernetes/ovnkube-node-csmcl"
Sep 16 18:45:32.797179 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:45:32.796810    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/bf1c06c8-20b0-4816-a092-02e87cd69a7f-hostroot\") pod \"multus-vxjzx\" (UID: \"bf1c06c8-20b0-4816-a092-02e87cd69a7f\") " pod="openshift-multus/multus-vxjzx"
Sep 16 18:45:32.797179 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:45:32.796824    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/bf1c06c8-20b0-4816-a092-02e87cd69a7f-etc-kubernetes\") pod \"multus-vxjzx\" (UID: \"bf1c06c8-20b0-4816-a092-02e87cd69a7f\") " pod="openshift-multus/multus-vxjzx"
Sep 16 18:45:32.797179 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:45:32.796839    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/99b45db8-0fbb-445b-be31-49fba3f220ec-serviceca\") pod \"node-ca-mvhf8\" (UID: \"99b45db8-0fbb-445b-be31-49fba3f220ec\") " pod="openshift-image-registry/node-ca-mvhf8"
Sep 16 18:45:32.797179 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:45:32.796855    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/99e923fa-5e08-487f-ada3-1692ef20ca11-var-lib-openvswitch\") pod \"ovnkube-node-csmcl\" (UID: \"99e923fa-5e08-487f-ada3-1692ef20ca11\") " pod="openshift-ovn-kubernetes/ovnkube-node-csmcl"
Sep 16 18:45:32.797179 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:45:32.796871    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/99e923fa-5e08-487f-ada3-1692ef20ca11-ovnkube-config\") pod \"ovnkube-node-csmcl\" (UID: \"99e923fa-5e08-487f-ada3-1692ef20ca11\") " pod="openshift-ovn-kubernetes/ovnkube-node-csmcl"
Sep 16 18:45:32.797179 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:45:32.796887    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sgdb6\" (UniqueName: \"kubernetes.io/projected/459060f0-3e55-498a-b878-726f3e3a1060-kube-api-access-sgdb6\") pod \"iptables-alerter-fbmb7\" (UID: \"459060f0-3e55-498a-b878-726f3e3a1060\") " pod="openshift-network-operator/iptables-alerter-fbmb7"
Sep 16 18:45:32.797179 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:45:32.796902    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/99b45db8-0fbb-445b-be31-49fba3f220ec-host\") pod \"node-ca-mvhf8\" (UID: \"99b45db8-0fbb-445b-be31-49fba3f220ec\") " pod="openshift-image-registry/node-ca-mvhf8"
Sep 16 18:45:32.797179 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:45:32.796920    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/99e923fa-5e08-487f-ada3-1692ef20ca11-host-cni-bin\") pod \"ovnkube-node-csmcl\" (UID: \"99e923fa-5e08-487f-ada3-1692ef20ca11\") " pod="openshift-ovn-kubernetes/ovnkube-node-csmcl"
Sep 16 18:45:32.797179 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:45:32.796937    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/bf866eb9-d1c0-4cb2-ad9c-86e8b5b306d7-kubelet-dir\") pod \"aws-ebs-csi-driver-node-54w2k\" (UID: \"bf866eb9-d1c0-4cb2-ad9c-86e8b5b306d7\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-54w2k"
Sep 16 18:45:32.798054 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:45:32.796963    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/5acb5489-33eb-4c23-b35d-484caffde098-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-fzt7c\" (UID: \"5acb5489-33eb-4c23-b35d-484caffde098\") " pod="openshift-multus/multus-additional-cni-plugins-fzt7c"
Sep 16 18:45:32.798054 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:45:32.796997    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/bf1c06c8-20b0-4816-a092-02e87cd69a7f-host-run-multus-certs\") pod \"multus-vxjzx\" (UID: \"bf1c06c8-20b0-4816-a092-02e87cd69a7f\") " pod="openshift-multus/multus-vxjzx"
Sep 16 18:45:32.798054 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:45:32.797021    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/bf866eb9-d1c0-4cb2-ad9c-86e8b5b306d7-device-dir\") pod \"aws-ebs-csi-driver-node-54w2k\" (UID: \"bf866eb9-d1c0-4cb2-ad9c-86e8b5b306d7\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-54w2k"
Sep 16 18:45:32.798054 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:45:32.797037    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/5acb5489-33eb-4c23-b35d-484caffde098-cni-binary-copy\") pod \"multus-additional-cni-plugins-fzt7c\" (UID: \"5acb5489-33eb-4c23-b35d-484caffde098\") " pod="openshift-multus/multus-additional-cni-plugins-fzt7c"
Sep 16 18:45:32.798054 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:45:32.797039    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/5acb5489-33eb-4c23-b35d-484caffde098-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-fzt7c\" (UID: \"5acb5489-33eb-4c23-b35d-484caffde098\") " pod="openshift-multus/multus-additional-cni-plugins-fzt7c"
Sep 16 18:45:32.798054 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:45:32.797102    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/459060f0-3e55-498a-b878-726f3e3a1060-iptables-alerter-script\") pod \"iptables-alerter-fbmb7\" (UID: \"459060f0-3e55-498a-b878-726f3e3a1060\") " pod="openshift-network-operator/iptables-alerter-fbmb7"
Sep 16 18:45:32.798054 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:45:32.797121    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/bf866eb9-d1c0-4cb2-ad9c-86e8b5b306d7-sys-fs\") pod \"aws-ebs-csi-driver-node-54w2k\" (UID: \"bf866eb9-d1c0-4cb2-ad9c-86e8b5b306d7\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-54w2k"
Sep 16 18:45:32.798054 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:45:32.797138    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7lljq\" (UniqueName: \"kubernetes.io/projected/bf866eb9-d1c0-4cb2-ad9c-86e8b5b306d7-kube-api-access-7lljq\") pod \"aws-ebs-csi-driver-node-54w2k\" (UID: \"bf866eb9-d1c0-4cb2-ad9c-86e8b5b306d7\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-54w2k"
Sep 16 18:45:32.798054 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:45:32.797155    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/5acb5489-33eb-4c23-b35d-484caffde098-tuning-conf-dir\") pod \"multus-additional-cni-plugins-fzt7c\" (UID: \"5acb5489-33eb-4c23-b35d-484caffde098\") " pod="openshift-multus/multus-additional-cni-plugins-fzt7c"
Sep 16 18:45:32.798054 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:45:32.797174    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/eaf5897a-76a0-4bf3-aa4e-59aa6c193ddb-metrics-certs\") pod \"network-metrics-daemon-c2kp2\" (UID: \"eaf5897a-76a0-4bf3-aa4e-59aa6c193ddb\") " pod="openshift-multus/network-metrics-daemon-c2kp2"
Sep 16 18:45:32.798054 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:45:32.797206    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/5acb5489-33eb-4c23-b35d-484caffde098-cnibin\") pod \"multus-additional-cni-plugins-fzt7c\" (UID: \"5acb5489-33eb-4c23-b35d-484caffde098\") " pod="openshift-multus/multus-additional-cni-plugins-fzt7c"
Sep 16 18:45:32.798054 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:45:32.797233    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7xm4p\" (UniqueName: \"kubernetes.io/projected/5acb5489-33eb-4c23-b35d-484caffde098-kube-api-access-7xm4p\") pod \"multus-additional-cni-plugins-fzt7c\" (UID: \"5acb5489-33eb-4c23-b35d-484caffde098\") " pod="openshift-multus/multus-additional-cni-plugins-fzt7c"
Sep 16 18:45:32.798054 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:45:32.797267    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/bf1c06c8-20b0-4816-a092-02e87cd69a7f-multus-conf-dir\") pod \"multus-vxjzx\" (UID: \"bf1c06c8-20b0-4816-a092-02e87cd69a7f\") " pod="openshift-multus/multus-vxjzx"
Sep 16 18:45:32.798054 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:45:32.797297    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/99e923fa-5e08-487f-ada3-1692ef20ca11-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-csmcl\" (UID: \"99e923fa-5e08-487f-ada3-1692ef20ca11\") " pod="openshift-ovn-kubernetes/ovnkube-node-csmcl"
Sep 16 18:45:32.798054 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:45:32.797327    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/99e923fa-5e08-487f-ada3-1692ef20ca11-ovnkube-script-lib\") pod \"ovnkube-node-csmcl\" (UID: \"99e923fa-5e08-487f-ada3-1692ef20ca11\") " pod="openshift-ovn-kubernetes/ovnkube-node-csmcl"
Sep 16 18:45:32.798054 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:45:32.797355    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/459060f0-3e55-498a-b878-726f3e3a1060-host-slash\") pod \"iptables-alerter-fbmb7\" (UID: \"459060f0-3e55-498a-b878-726f3e3a1060\") " pod="openshift-network-operator/iptables-alerter-fbmb7"
Sep 16 18:45:32.798942 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:45:32.797382    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/bf1c06c8-20b0-4816-a092-02e87cd69a7f-host-run-k8s-cni-cncf-io\") pod \"multus-vxjzx\" (UID: \"bf1c06c8-20b0-4816-a092-02e87cd69a7f\") " pod="openshift-multus/multus-vxjzx"
Sep 16 18:45:32.798942 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:45:32.797409    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/bf866eb9-d1c0-4cb2-ad9c-86e8b5b306d7-etc-selinux\") pod \"aws-ebs-csi-driver-node-54w2k\" (UID: \"bf866eb9-d1c0-4cb2-ad9c-86e8b5b306d7\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-54w2k"
Sep 16 18:45:32.798942 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:45:32.797441    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/99e923fa-5e08-487f-ada3-1692ef20ca11-run-systemd\") pod \"ovnkube-node-csmcl\" (UID: \"99e923fa-5e08-487f-ada3-1692ef20ca11\") " pod="openshift-ovn-kubernetes/ovnkube-node-csmcl"
Sep 16 18:45:32.798942 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:45:32.797466    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/99e923fa-5e08-487f-ada3-1692ef20ca11-host-cni-netd\") pod \"ovnkube-node-csmcl\" (UID: \"99e923fa-5e08-487f-ada3-1692ef20ca11\") " pod="openshift-ovn-kubernetes/ovnkube-node-csmcl"
Sep 16 18:45:32.798942 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:45:32.797491    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/bf1c06c8-20b0-4816-a092-02e87cd69a7f-system-cni-dir\") pod \"multus-vxjzx\" (UID: \"bf1c06c8-20b0-4816-a092-02e87cd69a7f\") " pod="openshift-multus/multus-vxjzx"
Sep 16 18:45:32.798942 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:45:32.797517    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/bf1c06c8-20b0-4816-a092-02e87cd69a7f-host-run-netns\") pod \"multus-vxjzx\" (UID: \"bf1c06c8-20b0-4816-a092-02e87cd69a7f\") " pod="openshift-multus/multus-vxjzx"
Sep 16 18:45:32.798942 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:45:32.797547    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c72bq\" (UniqueName: \"kubernetes.io/projected/99b45db8-0fbb-445b-be31-49fba3f220ec-kube-api-access-c72bq\") pod \"node-ca-mvhf8\" (UID: \"99b45db8-0fbb-445b-be31-49fba3f220ec\") " pod="openshift-image-registry/node-ca-mvhf8"
Sep 16 18:45:32.798942 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:45:32.797575    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/5acb5489-33eb-4c23-b35d-484caffde098-system-cni-dir\") pod \"multus-additional-cni-plugins-fzt7c\" (UID: \"5acb5489-33eb-4c23-b35d-484caffde098\") " pod="openshift-multus/multus-additional-cni-plugins-fzt7c"
Sep 16 18:45:32.798942 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:45:32.797576    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/459060f0-3e55-498a-b878-726f3e3a1060-host-slash\") pod \"iptables-alerter-fbmb7\" (UID: \"459060f0-3e55-498a-b878-726f3e3a1060\") " pod="openshift-network-operator/iptables-alerter-fbmb7"
Sep 16 18:45:32.798942 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:45:32.797601    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8qzq2\" (UniqueName: \"kubernetes.io/projected/99e923fa-5e08-487f-ada3-1692ef20ca11-kube-api-access-8qzq2\") pod \"ovnkube-node-csmcl\" (UID: \"99e923fa-5e08-487f-ada3-1692ef20ca11\") " pod="openshift-ovn-kubernetes/ovnkube-node-csmcl"
Sep 16 18:45:32.798942 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:45:32.797616    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/5acb5489-33eb-4c23-b35d-484caffde098-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-fzt7c\" (UID: \"5acb5489-33eb-4c23-b35d-484caffde098\") " pod="openshift-multus/multus-additional-cni-plugins-fzt7c"
Sep 16 18:45:32.798942 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:45:32.797628    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/bf866eb9-d1c0-4cb2-ad9c-86e8b5b306d7-registration-dir\") pod \"aws-ebs-csi-driver-node-54w2k\" (UID: \"bf866eb9-d1c0-4cb2-ad9c-86e8b5b306d7\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-54w2k"
Sep 16 18:45:32.798942 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:45:32.797647    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/5acb5489-33eb-4c23-b35d-484caffde098-cnibin\") pod \"multus-additional-cni-plugins-fzt7c\" (UID: \"5acb5489-33eb-4c23-b35d-484caffde098\") " pod="openshift-multus/multus-additional-cni-plugins-fzt7c"
Sep 16 18:45:32.798942 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:45:32.797685    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/bf866eb9-d1c0-4cb2-ad9c-86e8b5b306d7-registration-dir\") pod \"aws-ebs-csi-driver-node-54w2k\" (UID: \"bf866eb9-d1c0-4cb2-ad9c-86e8b5b306d7\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-54w2k"
Sep 16 18:45:32.798942 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:45:32.797701    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/bf866eb9-d1c0-4cb2-ad9c-86e8b5b306d7-sys-fs\") pod \"aws-ebs-csi-driver-node-54w2k\" (UID: \"bf866eb9-d1c0-4cb2-ad9c-86e8b5b306d7\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-54w2k"
Sep 16 18:45:32.798942 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:45:32.797722    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/bf1c06c8-20b0-4816-a092-02e87cd69a7f-host-run-k8s-cni-cncf-io\") pod \"multus-vxjzx\" (UID: \"bf1c06c8-20b0-4816-a092-02e87cd69a7f\") " pod="openshift-multus/multus-vxjzx"
Sep 16 18:45:32.798942 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:45:32.797750    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/bf866eb9-d1c0-4cb2-ad9c-86e8b5b306d7-etc-selinux\") pod \"aws-ebs-csi-driver-node-54w2k\" (UID: \"bf866eb9-d1c0-4cb2-ad9c-86e8b5b306d7\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-54w2k"
Sep 16 18:45:32.799695 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:45:32.797406    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/5acb5489-33eb-4c23-b35d-484caffde098-tuning-conf-dir\") pod \"multus-additional-cni-plugins-fzt7c\" (UID: \"5acb5489-33eb-4c23-b35d-484caffde098\") " pod="openshift-multus/multus-additional-cni-plugins-fzt7c"
Sep 16 18:45:32.799695 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:45:32.797765    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/bf1c06c8-20b0-4816-a092-02e87cd69a7f-host-run-netns\") pod \"multus-vxjzx\" (UID: \"bf1c06c8-20b0-4816-a092-02e87cd69a7f\") " pod="openshift-multus/multus-vxjzx"
Sep 16 18:45:32.799695 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:45:32.797797    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/459060f0-3e55-498a-b878-726f3e3a1060-iptables-alerter-script\") pod \"iptables-alerter-fbmb7\" (UID: \"459060f0-3e55-498a-b878-726f3e3a1060\") " pod="openshift-network-operator/iptables-alerter-fbmb7"
Sep 16 18:45:32.799695 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:45:32.797804    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/99e923fa-5e08-487f-ada3-1692ef20ca11-run-systemd\") pod \"ovnkube-node-csmcl\" (UID: \"99e923fa-5e08-487f-ada3-1692ef20ca11\") " pod="openshift-ovn-kubernetes/ovnkube-node-csmcl"
Sep 16 18:45:32.799695 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:45:32.796513    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/bf1c06c8-20b0-4816-a092-02e87cd69a7f-cnibin\") pod \"multus-vxjzx\" (UID: \"bf1c06c8-20b0-4816-a092-02e87cd69a7f\") " pod="openshift-multus/multus-vxjzx"
Sep 16 18:45:32.799695 ip-10-0-138-127 kubenswrapper[2581]: E0916 18:45:32.797973    2581 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 16 18:45:32.799695 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:45:32.797981    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/99e923fa-5e08-487f-ada3-1692ef20ca11-host-cni-netd\") pod \"ovnkube-node-csmcl\" (UID: \"99e923fa-5e08-487f-ada3-1692ef20ca11\") " pod="openshift-ovn-kubernetes/ovnkube-node-csmcl"
Sep 16 18:45:32.799695 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:45:32.798000    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/bf1c06c8-20b0-4816-a092-02e87cd69a7f-multus-socket-dir-parent\") pod \"multus-vxjzx\" (UID: \"bf1c06c8-20b0-4816-a092-02e87cd69a7f\") " pod="openshift-multus/multus-vxjzx"
Sep 16 18:45:32.799695 ip-10-0-138-127 kubenswrapper[2581]: E0916 18:45:32.798041    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/eaf5897a-76a0-4bf3-aa4e-59aa6c193ddb-metrics-certs podName:eaf5897a-76a0-4bf3-aa4e-59aa6c193ddb nodeName:}" failed. No retries permitted until 2026-09-16 18:45:33.298024227 +0000 UTC m=+3.117438114 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/eaf5897a-76a0-4bf3-aa4e-59aa6c193ddb-metrics-certs") pod "network-metrics-daemon-c2kp2" (UID: "eaf5897a-76a0-4bf3-aa4e-59aa6c193ddb") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 16 18:45:32.799695 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:45:32.798126    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/bf1c06c8-20b0-4816-a092-02e87cd69a7f-multus-cni-dir\") pod \"multus-vxjzx\" (UID: \"bf1c06c8-20b0-4816-a092-02e87cd69a7f\") " pod="openshift-multus/multus-vxjzx"
Sep 16 18:45:32.799695 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:45:32.798164    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/99e923fa-5e08-487f-ada3-1692ef20ca11-run-ovn\") pod \"ovnkube-node-csmcl\" (UID: \"99e923fa-5e08-487f-ada3-1692ef20ca11\") " pod="openshift-ovn-kubernetes/ovnkube-node-csmcl"
Sep 16 18:45:32.799695 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:45:32.798193    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/bf1c06c8-20b0-4816-a092-02e87cd69a7f-host-var-lib-cni-multus\") pod \"multus-vxjzx\" (UID: \"bf1c06c8-20b0-4816-a092-02e87cd69a7f\") " pod="openshift-multus/multus-vxjzx"
Sep 16 18:45:32.799695 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:45:32.798228    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/99e923fa-5e08-487f-ada3-1692ef20ca11-env-overrides\") pod \"ovnkube-node-csmcl\" (UID: \"99e923fa-5e08-487f-ada3-1692ef20ca11\") " pod="openshift-ovn-kubernetes/ovnkube-node-csmcl"
Sep 16 18:45:32.799695 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:45:32.798326    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/99e923fa-5e08-487f-ada3-1692ef20ca11-systemd-units\") pod \"ovnkube-node-csmcl\" (UID: \"99e923fa-5e08-487f-ada3-1692ef20ca11\") " pod="openshift-ovn-kubernetes/ovnkube-node-csmcl"
Sep 16 18:45:32.799695 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:45:32.798367    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/99e923fa-5e08-487f-ada3-1692ef20ca11-var-lib-openvswitch\") pod \"ovnkube-node-csmcl\" (UID: \"99e923fa-5e08-487f-ada3-1692ef20ca11\") " pod="openshift-ovn-kubernetes/ovnkube-node-csmcl"
Sep 16 18:45:32.799695 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:45:32.798402    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/99e923fa-5e08-487f-ada3-1692ef20ca11-node-log\") pod \"ovnkube-node-csmcl\" (UID: \"99e923fa-5e08-487f-ada3-1692ef20ca11\") " pod="openshift-ovn-kubernetes/ovnkube-node-csmcl"
Sep 16 18:45:32.799695 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:45:32.798449    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/bf1c06c8-20b0-4816-a092-02e87cd69a7f-host-var-lib-kubelet\") pod \"multus-vxjzx\" (UID: \"bf1c06c8-20b0-4816-a092-02e87cd69a7f\") " pod="openshift-multus/multus-vxjzx"
Sep 16 18:45:32.799695 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:45:32.798449    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/99b45db8-0fbb-445b-be31-49fba3f220ec-host\") pod \"node-ca-mvhf8\" (UID: \"99b45db8-0fbb-445b-be31-49fba3f220ec\") " pod="openshift-image-registry/node-ca-mvhf8"
Sep 16 18:45:32.800327 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:45:32.798485    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/99e923fa-5e08-487f-ada3-1692ef20ca11-host-kubelet\") pod \"ovnkube-node-csmcl\" (UID: \"99e923fa-5e08-487f-ada3-1692ef20ca11\") " pod="openshift-ovn-kubernetes/ovnkube-node-csmcl"
Sep 16 18:45:32.800327 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:45:32.798491    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/99e923fa-5e08-487f-ada3-1692ef20ca11-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-csmcl\" (UID: \"99e923fa-5e08-487f-ada3-1692ef20ca11\") " pod="openshift-ovn-kubernetes/ovnkube-node-csmcl"
Sep 16 18:45:32.800327 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:45:32.798493    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/99e923fa-5e08-487f-ada3-1692ef20ca11-host-cni-bin\") pod \"ovnkube-node-csmcl\" (UID: \"99e923fa-5e08-487f-ada3-1692ef20ca11\") " pod="openshift-ovn-kubernetes/ovnkube-node-csmcl"
Sep 16 18:45:32.800327 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:45:32.798515    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/bf866eb9-d1c0-4cb2-ad9c-86e8b5b306d7-kubelet-dir\") pod \"aws-ebs-csi-driver-node-54w2k\" (UID: \"bf866eb9-d1c0-4cb2-ad9c-86e8b5b306d7\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-54w2k"
Sep 16 18:45:32.800327 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:45:32.798548    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/bf1c06c8-20b0-4816-a092-02e87cd69a7f-system-cni-dir\") pod \"multus-vxjzx\" (UID: \"bf1c06c8-20b0-4816-a092-02e87cd69a7f\") " pod="openshift-multus/multus-vxjzx"
Sep 16 18:45:32.800327 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:45:32.798577    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/bf1c06c8-20b0-4816-a092-02e87cd69a7f-hostroot\") pod \"multus-vxjzx\" (UID: \"bf1c06c8-20b0-4816-a092-02e87cd69a7f\") " pod="openshift-multus/multus-vxjzx"
Sep 16 18:45:32.800327 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:45:32.798587    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/bf866eb9-d1c0-4cb2-ad9c-86e8b5b306d7-socket-dir\") pod \"aws-ebs-csi-driver-node-54w2k\" (UID: \"bf866eb9-d1c0-4cb2-ad9c-86e8b5b306d7\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-54w2k"
Sep 16 18:45:32.800327 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:45:32.798629    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/bf866eb9-d1c0-4cb2-ad9c-86e8b5b306d7-device-dir\") pod \"aws-ebs-csi-driver-node-54w2k\" (UID: \"bf866eb9-d1c0-4cb2-ad9c-86e8b5b306d7\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-54w2k"
Sep 16 18:45:32.800327 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:45:32.798570    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/bf1c06c8-20b0-4816-a092-02e87cd69a7f-host-run-multus-certs\") pod \"multus-vxjzx\" (UID: \"bf1c06c8-20b0-4816-a092-02e87cd69a7f\") " pod="openshift-multus/multus-vxjzx"
Sep 16 18:45:32.800327 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:45:32.798687    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/99e923fa-5e08-487f-ada3-1692ef20ca11-host-slash\") pod \"ovnkube-node-csmcl\" (UID: \"99e923fa-5e08-487f-ada3-1692ef20ca11\") " pod="openshift-ovn-kubernetes/ovnkube-node-csmcl"
Sep 16 18:45:32.800327 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:45:32.798730    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/99b45db8-0fbb-445b-be31-49fba3f220ec-serviceca\") pod \"node-ca-mvhf8\" (UID: \"99b45db8-0fbb-445b-be31-49fba3f220ec\") " pod="openshift-image-registry/node-ca-mvhf8"
Sep 16 18:45:32.800327 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:45:32.798738    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/bf866eb9-d1c0-4cb2-ad9c-86e8b5b306d7-socket-dir\") pod \"aws-ebs-csi-driver-node-54w2k\" (UID: \"bf866eb9-d1c0-4cb2-ad9c-86e8b5b306d7\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-54w2k"
Sep 16 18:45:32.800327 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:45:32.798780    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/99e923fa-5e08-487f-ada3-1692ef20ca11-etc-openvswitch\") pod \"ovnkube-node-csmcl\" (UID: \"99e923fa-5e08-487f-ada3-1692ef20ca11\") " pod="openshift-ovn-kubernetes/ovnkube-node-csmcl"
Sep 16 18:45:32.800327 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:45:32.798800    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/bf1c06c8-20b0-4816-a092-02e87cd69a7f-etc-kubernetes\") pod \"multus-vxjzx\" (UID: \"bf1c06c8-20b0-4816-a092-02e87cd69a7f\") " pod="openshift-multus/multus-vxjzx"
Sep 16 18:45:32.800327 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:45:32.798818    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/bf1c06c8-20b0-4816-a092-02e87cd69a7f-host-var-lib-cni-bin\") pod \"multus-vxjzx\" (UID: \"bf1c06c8-20b0-4816-a092-02e87cd69a7f\") " pod="openshift-multus/multus-vxjzx"
Sep 16 18:45:32.800327 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:45:32.798827    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/bf1c06c8-20b0-4816-a092-02e87cd69a7f-cni-binary-copy\") pod \"multus-vxjzx\" (UID: \"bf1c06c8-20b0-4816-a092-02e87cd69a7f\") " pod="openshift-multus/multus-vxjzx"
Sep 16 18:45:32.800327 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:45:32.798863    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/99e923fa-5e08-487f-ada3-1692ef20ca11-host-run-ovn-kubernetes\") pod \"ovnkube-node-csmcl\" (UID: \"99e923fa-5e08-487f-ada3-1692ef20ca11\") " pod="openshift-ovn-kubernetes/ovnkube-node-csmcl"
Sep 16 18:45:32.800327 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:45:32.798869    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/99e923fa-5e08-487f-ada3-1692ef20ca11-log-socket\") pod \"ovnkube-node-csmcl\" (UID: \"99e923fa-5e08-487f-ada3-1692ef20ca11\") " pod="openshift-ovn-kubernetes/ovnkube-node-csmcl"
Sep 16 18:45:32.801036 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:45:32.798907    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/5acb5489-33eb-4c23-b35d-484caffde098-system-cni-dir\") pod \"multus-additional-cni-plugins-fzt7c\" (UID: \"5acb5489-33eb-4c23-b35d-484caffde098\") " pod="openshift-multus/multus-additional-cni-plugins-fzt7c"
Sep 16 18:45:32.801036 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:45:32.799000    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/5acb5489-33eb-4c23-b35d-484caffde098-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-fzt7c\" (UID: \"5acb5489-33eb-4c23-b35d-484caffde098\") " pod="openshift-multus/multus-additional-cni-plugins-fzt7c"
Sep 16 18:45:32.801036 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:45:32.799317    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/99e923fa-5e08-487f-ada3-1692ef20ca11-ovnkube-config\") pod \"ovnkube-node-csmcl\" (UID: \"99e923fa-5e08-487f-ada3-1692ef20ca11\") " pod="openshift-ovn-kubernetes/ovnkube-node-csmcl"
Sep 16 18:45:32.801036 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:45:32.799365    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/bf1c06c8-20b0-4816-a092-02e87cd69a7f-multus-daemon-config\") pod \"multus-vxjzx\" (UID: \"bf1c06c8-20b0-4816-a092-02e87cd69a7f\") " pod="openshift-multus/multus-vxjzx"
Sep 16 18:45:32.801036 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:45:32.799471    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/99e923fa-5e08-487f-ada3-1692ef20ca11-ovnkube-script-lib\") pod \"ovnkube-node-csmcl\" (UID: \"99e923fa-5e08-487f-ada3-1692ef20ca11\") " pod="openshift-ovn-kubernetes/ovnkube-node-csmcl"
Sep 16 18:45:32.802170 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:45:32.802146    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/99e923fa-5e08-487f-ada3-1692ef20ca11-ovn-node-metrics-cert\") pod \"ovnkube-node-csmcl\" (UID: \"99e923fa-5e08-487f-ada3-1692ef20ca11\") " pod="openshift-ovn-kubernetes/ovnkube-node-csmcl"
Sep 16 18:45:32.812256 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:45:32.812202    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-r7qw6\" (UniqueName: \"kubernetes.io/projected/bf1c06c8-20b0-4816-a092-02e87cd69a7f-kube-api-access-r7qw6\") pod \"multus-vxjzx\" (UID: \"bf1c06c8-20b0-4816-a092-02e87cd69a7f\") " pod="openshift-multus/multus-vxjzx"
Sep 16 18:45:32.813969 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:45:32.813925    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sgdb6\" (UniqueName: \"kubernetes.io/projected/459060f0-3e55-498a-b878-726f3e3a1060-kube-api-access-sgdb6\") pod \"iptables-alerter-fbmb7\" (UID: \"459060f0-3e55-498a-b878-726f3e3a1060\") " pod="openshift-network-operator/iptables-alerter-fbmb7"
Sep 16 18:45:32.814925 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:45:32.814898    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-c72bq\" (UniqueName: \"kubernetes.io/projected/99b45db8-0fbb-445b-be31-49fba3f220ec-kube-api-access-c72bq\") pod \"node-ca-mvhf8\" (UID: \"99b45db8-0fbb-445b-be31-49fba3f220ec\") " pod="openshift-image-registry/node-ca-mvhf8"
Sep 16 18:45:32.815375 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:45:32.815352    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7xm4p\" (UniqueName: \"kubernetes.io/projected/5acb5489-33eb-4c23-b35d-484caffde098-kube-api-access-7xm4p\") pod \"multus-additional-cni-plugins-fzt7c\" (UID: \"5acb5489-33eb-4c23-b35d-484caffde098\") " pod="openshift-multus/multus-additional-cni-plugins-fzt7c"
Sep 16 18:45:32.816061 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:45:32.816035    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-26lrg\" (UniqueName: \"kubernetes.io/projected/eaf5897a-76a0-4bf3-aa4e-59aa6c193ddb-kube-api-access-26lrg\") pod \"network-metrics-daemon-c2kp2\" (UID: \"eaf5897a-76a0-4bf3-aa4e-59aa6c193ddb\") " pod="openshift-multus/network-metrics-daemon-c2kp2"
Sep 16 18:45:32.816937 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:45:32.816912    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7lljq\" (UniqueName: \"kubernetes.io/projected/bf866eb9-d1c0-4cb2-ad9c-86e8b5b306d7-kube-api-access-7lljq\") pod \"aws-ebs-csi-driver-node-54w2k\" (UID: \"bf866eb9-d1c0-4cb2-ad9c-86e8b5b306d7\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-54w2k"
Sep 16 18:45:32.817110 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:45:32.817091    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8qzq2\" (UniqueName: \"kubernetes.io/projected/99e923fa-5e08-487f-ada3-1692ef20ca11-kube-api-access-8qzq2\") pod \"ovnkube-node-csmcl\" (UID: \"99e923fa-5e08-487f-ada3-1692ef20ca11\") " pod="openshift-ovn-kubernetes/ovnkube-node-csmcl"
Sep 16 18:45:32.886382 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:45:32.886340    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-7t742"
Sep 16 18:45:32.894928 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:45:32.894896    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-lrclz"
Sep 16 18:45:32.907235 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:45:32.907155    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-vttdl"
Sep 16 18:45:32.919262 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:45:32.919236    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-54w2k"
Sep 16 18:45:32.927289 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:45:32.927260    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-csmcl"
Sep 16 18:45:32.935325 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:45:32.935300    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-fzt7c"
Sep 16 18:45:32.943249 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:45:32.943228    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-fbmb7"
Sep 16 18:45:32.953304 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:45:32.953277    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-vxjzx"
Sep 16 18:45:32.957304 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:45:32.957276    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-mvhf8"
Sep 16 18:45:33.201862 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:45:33.201759    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/f09babc9-6295-4a0a-be9f-fa0175b0de51-original-pull-secret\") pod \"global-pull-secret-syncer-c78nz\" (UID: \"f09babc9-6295-4a0a-be9f-fa0175b0de51\") " pod="kube-system/global-pull-secret-syncer-c78nz"
Sep 16 18:45:33.202044 ip-10-0-138-127 kubenswrapper[2581]: E0916 18:45:33.201949    2581 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 16 18:45:33.202044 ip-10-0-138-127 kubenswrapper[2581]: E0916 18:45:33.202037    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f09babc9-6295-4a0a-be9f-fa0175b0de51-original-pull-secret podName:f09babc9-6295-4a0a-be9f-fa0175b0de51 nodeName:}" failed. No retries permitted until 2026-09-16 18:45:34.202014603 +0000 UTC m=+4.021428509 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/f09babc9-6295-4a0a-be9f-fa0175b0de51-original-pull-secret") pod "global-pull-secret-syncer-c78nz" (UID: "f09babc9-6295-4a0a-be9f-fa0175b0de51") : object "kube-system"/"original-pull-secret" not registered
Sep 16 18:45:33.302619 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:45:33.302580    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5vmqf\" (UniqueName: \"kubernetes.io/projected/4775e8c4-09ba-43db-98b7-073aeb452ce8-kube-api-access-5vmqf\") pod \"network-check-target-bz5jk\" (UID: \"4775e8c4-09ba-43db-98b7-073aeb452ce8\") " pod="openshift-network-diagnostics/network-check-target-bz5jk"
Sep 16 18:45:33.302619 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:45:33.302633    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/eaf5897a-76a0-4bf3-aa4e-59aa6c193ddb-metrics-certs\") pod \"network-metrics-daemon-c2kp2\" (UID: \"eaf5897a-76a0-4bf3-aa4e-59aa6c193ddb\") " pod="openshift-multus/network-metrics-daemon-c2kp2"
Sep 16 18:45:33.302874 ip-10-0-138-127 kubenswrapper[2581]: E0916 18:45:33.302747    2581 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 16 18:45:33.302874 ip-10-0-138-127 kubenswrapper[2581]: E0916 18:45:33.302773    2581 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 16 18:45:33.302874 ip-10-0-138-127 kubenswrapper[2581]: E0916 18:45:33.302797    2581 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 16 18:45:33.302874 ip-10-0-138-127 kubenswrapper[2581]: E0916 18:45:33.302805    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/eaf5897a-76a0-4bf3-aa4e-59aa6c193ddb-metrics-certs podName:eaf5897a-76a0-4bf3-aa4e-59aa6c193ddb nodeName:}" failed. No retries permitted until 2026-09-16 18:45:34.302791206 +0000 UTC m=+4.122205095 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/eaf5897a-76a0-4bf3-aa4e-59aa6c193ddb-metrics-certs") pod "network-metrics-daemon-c2kp2" (UID: "eaf5897a-76a0-4bf3-aa4e-59aa6c193ddb") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 16 18:45:33.302874 ip-10-0-138-127 kubenswrapper[2581]: E0916 18:45:33.302807    2581 projected.go:196] Error preparing data for projected volume kube-api-access-5vmqf for pod openshift-network-diagnostics/network-check-target-bz5jk: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 16 18:45:33.302874 ip-10-0-138-127 kubenswrapper[2581]: E0916 18:45:33.302854    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/4775e8c4-09ba-43db-98b7-073aeb452ce8-kube-api-access-5vmqf podName:4775e8c4-09ba-43db-98b7-073aeb452ce8 nodeName:}" failed. No retries permitted until 2026-09-16 18:45:34.302842161 +0000 UTC m=+4.122256050 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-5vmqf" (UniqueName: "kubernetes.io/projected/4775e8c4-09ba-43db-98b7-073aeb452ce8-kube-api-access-5vmqf") pod "network-check-target-bz5jk" (UID: "4775e8c4-09ba-43db-98b7-073aeb452ce8") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 16 18:45:33.641307 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:45:33.641180    2581 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-09-15 18:40:33 +0000 UTC" deadline="2028-06-12 03:24:42.175742836 +0000 UTC"
Sep 16 18:45:33.641307 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:45:33.641240    2581 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="15224h39m8.534508665s"
Sep 16 18:45:33.722549 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:45:33.722514    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-c78nz"
Sep 16 18:45:33.722752 ip-10-0-138-127 kubenswrapper[2581]: E0916 18:45:33.722645    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-c78nz" podUID="f09babc9-6295-4a0a-be9f-fa0175b0de51"
Sep 16 18:45:33.817666 ip-10-0-138-127 kubenswrapper[2581]: W0916 18:45:33.817621    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod99e923fa_5e08_487f_ada3_1692ef20ca11.slice/crio-fa989806d28497d28d0c05b212cb97bfbcf425eac2bb6309e84bce9a27e249b8 WatchSource:0}: Error finding container fa989806d28497d28d0c05b212cb97bfbcf425eac2bb6309e84bce9a27e249b8: Status 404 returned error can't find the container with id fa989806d28497d28d0c05b212cb97bfbcf425eac2bb6309e84bce9a27e249b8
Sep 16 18:45:33.820667 ip-10-0-138-127 kubenswrapper[2581]: W0916 18:45:33.820639    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbf866eb9_d1c0_4cb2_ad9c_86e8b5b306d7.slice/crio-aeab529e046b0636858ec32c404ec31aba745a60a601774c8bef38d23974ca72 WatchSource:0}: Error finding container aeab529e046b0636858ec32c404ec31aba745a60a601774c8bef38d23974ca72: Status 404 returned error can't find the container with id aeab529e046b0636858ec32c404ec31aba745a60a601774c8bef38d23974ca72
Sep 16 18:45:33.826153 ip-10-0-138-127 kubenswrapper[2581]: W0916 18:45:33.826122    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb31b8735_005e_4387_852c_bc67df0e77cb.slice/crio-08c2e63b4a9dfd6c4e478ffe56959ae66af0442f5849b264d0c9dd362354b140 WatchSource:0}: Error finding container 08c2e63b4a9dfd6c4e478ffe56959ae66af0442f5849b264d0c9dd362354b140: Status 404 returned error can't find the container with id 08c2e63b4a9dfd6c4e478ffe56959ae66af0442f5849b264d0c9dd362354b140
Sep 16 18:45:33.826846 ip-10-0-138-127 kubenswrapper[2581]: W0916 18:45:33.826818    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod459060f0_3e55_498a_b878_726f3e3a1060.slice/crio-cd75a1590ba152e6c6ac3e4985ca84ca87782295be093e7264f20b04c4d2cc75 WatchSource:0}: Error finding container cd75a1590ba152e6c6ac3e4985ca84ca87782295be093e7264f20b04c4d2cc75: Status 404 returned error can't find the container with id cd75a1590ba152e6c6ac3e4985ca84ca87782295be093e7264f20b04c4d2cc75
Sep 16 18:45:33.828770 ip-10-0-138-127 kubenswrapper[2581]: W0916 18:45:33.828721    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3f382427_dc87_4034_ae03_c457ed18d446.slice/crio-5d31a4799f24c680ab813e256bb589838bc928e5208c96e5b3d73674262f2995 WatchSource:0}: Error finding container 5d31a4799f24c680ab813e256bb589838bc928e5208c96e5b3d73674262f2995: Status 404 returned error can't find the container with id 5d31a4799f24c680ab813e256bb589838bc928e5208c96e5b3d73674262f2995
Sep 16 18:45:33.829549 ip-10-0-138-127 kubenswrapper[2581]: W0916 18:45:33.829437    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podeace7050_d66d_4057_9041_62afa5df8ae7.slice/crio-98df37f3db38460e8e18f4b33a263f38ee30330946fba1d377fe63613c14360d WatchSource:0}: Error finding container 98df37f3db38460e8e18f4b33a263f38ee30330946fba1d377fe63613c14360d: Status 404 returned error can't find the container with id 98df37f3db38460e8e18f4b33a263f38ee30330946fba1d377fe63613c14360d
Sep 16 18:45:33.830094 ip-10-0-138-127 kubenswrapper[2581]: W0916 18:45:33.830067    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbf1c06c8_20b0_4816_a092_02e87cd69a7f.slice/crio-3e0dc699e80caf7833f0174f2fbddc26b3ce4c15a0d7929bc35515f769c8be99 WatchSource:0}: Error finding container 3e0dc699e80caf7833f0174f2fbddc26b3ce4c15a0d7929bc35515f769c8be99: Status 404 returned error can't find the container with id 3e0dc699e80caf7833f0174f2fbddc26b3ce4c15a0d7929bc35515f769c8be99
Sep 16 18:45:33.833155 ip-10-0-138-127 kubenswrapper[2581]: W0916 18:45:33.833123    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5acb5489_33eb_4c23_b35d_484caffde098.slice/crio-2a882ea5646e0eff25bbf80ef8e74a10cb62da8f8e5dc77144452db8e9ed5617 WatchSource:0}: Error finding container 2a882ea5646e0eff25bbf80ef8e74a10cb62da8f8e5dc77144452db8e9ed5617: Status 404 returned error can't find the container with id 2a882ea5646e0eff25bbf80ef8e74a10cb62da8f8e5dc77144452db8e9ed5617
Sep 16 18:45:34.211000 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:45:34.210884    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/f09babc9-6295-4a0a-be9f-fa0175b0de51-original-pull-secret\") pod \"global-pull-secret-syncer-c78nz\" (UID: \"f09babc9-6295-4a0a-be9f-fa0175b0de51\") " pod="kube-system/global-pull-secret-syncer-c78nz"
Sep 16 18:45:34.211166 ip-10-0-138-127 kubenswrapper[2581]: E0916 18:45:34.211061    2581 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 16 18:45:34.211166 ip-10-0-138-127 kubenswrapper[2581]: E0916 18:45:34.211132    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f09babc9-6295-4a0a-be9f-fa0175b0de51-original-pull-secret podName:f09babc9-6295-4a0a-be9f-fa0175b0de51 nodeName:}" failed. No retries permitted until 2026-09-16 18:45:36.211112884 +0000 UTC m=+6.030526779 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/f09babc9-6295-4a0a-be9f-fa0175b0de51-original-pull-secret") pod "global-pull-secret-syncer-c78nz" (UID: "f09babc9-6295-4a0a-be9f-fa0175b0de51") : object "kube-system"/"original-pull-secret" not registered
Sep 16 18:45:34.314397 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:45:34.314351    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5vmqf\" (UniqueName: \"kubernetes.io/projected/4775e8c4-09ba-43db-98b7-073aeb452ce8-kube-api-access-5vmqf\") pod \"network-check-target-bz5jk\" (UID: \"4775e8c4-09ba-43db-98b7-073aeb452ce8\") " pod="openshift-network-diagnostics/network-check-target-bz5jk"
Sep 16 18:45:34.314592 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:45:34.314463    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/eaf5897a-76a0-4bf3-aa4e-59aa6c193ddb-metrics-certs\") pod \"network-metrics-daemon-c2kp2\" (UID: \"eaf5897a-76a0-4bf3-aa4e-59aa6c193ddb\") " pod="openshift-multus/network-metrics-daemon-c2kp2"
Sep 16 18:45:34.314684 ip-10-0-138-127 kubenswrapper[2581]: E0916 18:45:34.314663    2581 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 16 18:45:34.314774 ip-10-0-138-127 kubenswrapper[2581]: E0916 18:45:34.314761    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/eaf5897a-76a0-4bf3-aa4e-59aa6c193ddb-metrics-certs podName:eaf5897a-76a0-4bf3-aa4e-59aa6c193ddb nodeName:}" failed. No retries permitted until 2026-09-16 18:45:36.314737592 +0000 UTC m=+6.134151487 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/eaf5897a-76a0-4bf3-aa4e-59aa6c193ddb-metrics-certs") pod "network-metrics-daemon-c2kp2" (UID: "eaf5897a-76a0-4bf3-aa4e-59aa6c193ddb") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 16 18:45:34.315537 ip-10-0-138-127 kubenswrapper[2581]: E0916 18:45:34.315512    2581 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 16 18:45:34.315623 ip-10-0-138-127 kubenswrapper[2581]: E0916 18:45:34.315544    2581 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 16 18:45:34.315623 ip-10-0-138-127 kubenswrapper[2581]: E0916 18:45:34.315560    2581 projected.go:196] Error preparing data for projected volume kube-api-access-5vmqf for pod openshift-network-diagnostics/network-check-target-bz5jk: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 16 18:45:34.315623 ip-10-0-138-127 kubenswrapper[2581]: E0916 18:45:34.315619    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/4775e8c4-09ba-43db-98b7-073aeb452ce8-kube-api-access-5vmqf podName:4775e8c4-09ba-43db-98b7-073aeb452ce8 nodeName:}" failed. No retries permitted until 2026-09-16 18:45:36.315599845 +0000 UTC m=+6.135013756 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-5vmqf" (UniqueName: "kubernetes.io/projected/4775e8c4-09ba-43db-98b7-073aeb452ce8-kube-api-access-5vmqf") pod "network-check-target-bz5jk" (UID: "4775e8c4-09ba-43db-98b7-073aeb452ce8") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 16 18:45:34.567101 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:45:34.567009    2581 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 16 18:45:34.641532 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:45:34.641466    2581 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-09-15 18:40:33 +0000 UTC" deadline="2028-04-04 08:03:16.271781696 +0000 UTC"
Sep 16 18:45:34.641532 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:45:34.641532    2581 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13573h17m41.630257358s"
Sep 16 18:45:34.724842 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:45:34.724804    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-bz5jk"
Sep 16 18:45:34.725054 ip-10-0-138-127 kubenswrapper[2581]: E0916 18:45:34.724954    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-bz5jk" podUID="4775e8c4-09ba-43db-98b7-073aeb452ce8"
Sep 16 18:45:34.725610 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:45:34.725426    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-c2kp2"
Sep 16 18:45:34.725610 ip-10-0-138-127 kubenswrapper[2581]: E0916 18:45:34.725542    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-c2kp2" podUID="eaf5897a-76a0-4bf3-aa4e-59aa6c193ddb"
Sep 16 18:45:34.752738 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:45:34.752696    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-fbmb7" event={"ID":"459060f0-3e55-498a-b878-726f3e3a1060","Type":"ContainerStarted","Data":"cd75a1590ba152e6c6ac3e4985ca84ca87782295be093e7264f20b04c4d2cc75"}
Sep 16 18:45:34.763842 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:45:34.763802    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-vttdl" event={"ID":"b31b8735-005e-4387-852c-bc67df0e77cb","Type":"ContainerStarted","Data":"08c2e63b4a9dfd6c4e478ffe56959ae66af0442f5849b264d0c9dd362354b140"}
Sep 16 18:45:34.792897 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:45:34.792851    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-mvhf8" event={"ID":"99b45db8-0fbb-445b-be31-49fba3f220ec","Type":"ContainerStarted","Data":"e186adec288a402a10421fafbeea48b4b92c7c3161ec65f29e4cbb4e31d76817"}
Sep 16 18:45:34.808500 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:45:34.808458    2581 generic.go:356] "Generic (PLEG): container finished" podID="3f19801b96f1829134dfd0b39bce2048" containerID="e62ea7868e6dc59728648c7525c346d5292d349e5df984a5aa8ba8e8c1a04ca8" exitCode=0
Sep 16 18:45:34.808679 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:45:34.808590    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-127.ec2.internal" event={"ID":"3f19801b96f1829134dfd0b39bce2048","Type":"ContainerDied","Data":"e62ea7868e6dc59728648c7525c346d5292d349e5df984a5aa8ba8e8c1a04ca8"}
Sep 16 18:45:34.828777 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:45:34.828728    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-138-127.ec2.internal" event={"ID":"37df0dc0824bc6d3c2498649ba68632f","Type":"ContainerStarted","Data":"ac2b351a8061af86940b17df2945c73c8867fb320ab3ade2f745203c50b90f9b"}
Sep 16 18:45:34.844795 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:45:34.844755    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-vxjzx" event={"ID":"bf1c06c8-20b0-4816-a092-02e87cd69a7f","Type":"ContainerStarted","Data":"3e0dc699e80caf7833f0174f2fbddc26b3ce4c15a0d7929bc35515f769c8be99"}
Sep 16 18:45:34.851730 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:45:34.851634    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-138-127.ec2.internal" podStartSLOduration=2.851604557 podStartE2EDuration="2.851604557s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-16 18:45:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:45:34.851252255 +0000 UTC m=+4.670666167" watchObservedRunningTime="2026-09-16 18:45:34.851604557 +0000 UTC m=+4.671018472"
Sep 16 18:45:34.861622 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:45:34.861582    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-lrclz" event={"ID":"eace7050-d66d-4057-9041-62afa5df8ae7","Type":"ContainerStarted","Data":"98df37f3db38460e8e18f4b33a263f38ee30330946fba1d377fe63613c14360d"}
Sep 16 18:45:34.867443 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:45:34.867383    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-54w2k" event={"ID":"bf866eb9-d1c0-4cb2-ad9c-86e8b5b306d7","Type":"ContainerStarted","Data":"aeab529e046b0636858ec32c404ec31aba745a60a601774c8bef38d23974ca72"}
Sep 16 18:45:34.875249 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:45:34.875210    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-csmcl" event={"ID":"99e923fa-5e08-487f-ada3-1692ef20ca11","Type":"ContainerStarted","Data":"fa989806d28497d28d0c05b212cb97bfbcf425eac2bb6309e84bce9a27e249b8"}
Sep 16 18:45:34.883388 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:45:34.883351    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-fzt7c" event={"ID":"5acb5489-33eb-4c23-b35d-484caffde098","Type":"ContainerStarted","Data":"2a882ea5646e0eff25bbf80ef8e74a10cb62da8f8e5dc77144452db8e9ed5617"}
Sep 16 18:45:34.891831 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:45:34.891745    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-7t742" event={"ID":"3f382427-dc87-4034-ae03-c457ed18d446","Type":"ContainerStarted","Data":"5d31a4799f24c680ab813e256bb589838bc928e5208c96e5b3d73674262f2995"}
Sep 16 18:45:35.722160 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:45:35.722115    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-c78nz"
Sep 16 18:45:35.722832 ip-10-0-138-127 kubenswrapper[2581]: E0916 18:45:35.722290    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-c78nz" podUID="f09babc9-6295-4a0a-be9f-fa0175b0de51"
Sep 16 18:45:35.931995 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:45:35.931920    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-127.ec2.internal" event={"ID":"3f19801b96f1829134dfd0b39bce2048","Type":"ContainerStarted","Data":"0d6a767284d61735e5ffe6c53eea7be300e190dc4b7861e3e0c19887583653ba"}
Sep 16 18:45:36.232630 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:45:36.232584    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/f09babc9-6295-4a0a-be9f-fa0175b0de51-original-pull-secret\") pod \"global-pull-secret-syncer-c78nz\" (UID: \"f09babc9-6295-4a0a-be9f-fa0175b0de51\") " pod="kube-system/global-pull-secret-syncer-c78nz"
Sep 16 18:45:36.232781 ip-10-0-138-127 kubenswrapper[2581]: E0916 18:45:36.232762    2581 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 16 18:45:36.232840 ip-10-0-138-127 kubenswrapper[2581]: E0916 18:45:36.232832    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f09babc9-6295-4a0a-be9f-fa0175b0de51-original-pull-secret podName:f09babc9-6295-4a0a-be9f-fa0175b0de51 nodeName:}" failed. No retries permitted until 2026-09-16 18:45:40.232811257 +0000 UTC m=+10.052225164 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/f09babc9-6295-4a0a-be9f-fa0175b0de51-original-pull-secret") pod "global-pull-secret-syncer-c78nz" (UID: "f09babc9-6295-4a0a-be9f-fa0175b0de51") : object "kube-system"/"original-pull-secret" not registered
Sep 16 18:45:36.334008 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:45:36.333733    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5vmqf\" (UniqueName: \"kubernetes.io/projected/4775e8c4-09ba-43db-98b7-073aeb452ce8-kube-api-access-5vmqf\") pod \"network-check-target-bz5jk\" (UID: \"4775e8c4-09ba-43db-98b7-073aeb452ce8\") " pod="openshift-network-diagnostics/network-check-target-bz5jk"
Sep 16 18:45:36.334200 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:45:36.334071    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/eaf5897a-76a0-4bf3-aa4e-59aa6c193ddb-metrics-certs\") pod \"network-metrics-daemon-c2kp2\" (UID: \"eaf5897a-76a0-4bf3-aa4e-59aa6c193ddb\") " pod="openshift-multus/network-metrics-daemon-c2kp2"
Sep 16 18:45:36.334347 ip-10-0-138-127 kubenswrapper[2581]: E0916 18:45:36.334298    2581 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 16 18:45:36.334412 ip-10-0-138-127 kubenswrapper[2581]: E0916 18:45:36.334370    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/eaf5897a-76a0-4bf3-aa4e-59aa6c193ddb-metrics-certs podName:eaf5897a-76a0-4bf3-aa4e-59aa6c193ddb nodeName:}" failed. No retries permitted until 2026-09-16 18:45:40.334349567 +0000 UTC m=+10.153763472 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/eaf5897a-76a0-4bf3-aa4e-59aa6c193ddb-metrics-certs") pod "network-metrics-daemon-c2kp2" (UID: "eaf5897a-76a0-4bf3-aa4e-59aa6c193ddb") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 16 18:45:36.334824 ip-10-0-138-127 kubenswrapper[2581]: E0916 18:45:36.334802    2581 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 16 18:45:36.334936 ip-10-0-138-127 kubenswrapper[2581]: E0916 18:45:36.334834    2581 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 16 18:45:36.334936 ip-10-0-138-127 kubenswrapper[2581]: E0916 18:45:36.334848    2581 projected.go:196] Error preparing data for projected volume kube-api-access-5vmqf for pod openshift-network-diagnostics/network-check-target-bz5jk: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 16 18:45:36.334936 ip-10-0-138-127 kubenswrapper[2581]: E0916 18:45:36.334895    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/4775e8c4-09ba-43db-98b7-073aeb452ce8-kube-api-access-5vmqf podName:4775e8c4-09ba-43db-98b7-073aeb452ce8 nodeName:}" failed. No retries permitted until 2026-09-16 18:45:40.334880217 +0000 UTC m=+10.154294108 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-5vmqf" (UniqueName: "kubernetes.io/projected/4775e8c4-09ba-43db-98b7-073aeb452ce8-kube-api-access-5vmqf") pod "network-check-target-bz5jk" (UID: "4775e8c4-09ba-43db-98b7-073aeb452ce8") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 16 18:45:36.723503 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:45:36.723398    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-bz5jk"
Sep 16 18:45:36.723503 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:45:36.723423    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-c2kp2"
Sep 16 18:45:36.724054 ip-10-0-138-127 kubenswrapper[2581]: E0916 18:45:36.723545    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-bz5jk" podUID="4775e8c4-09ba-43db-98b7-073aeb452ce8"
Sep 16 18:45:36.724054 ip-10-0-138-127 kubenswrapper[2581]: E0916 18:45:36.723677    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-c2kp2" podUID="eaf5897a-76a0-4bf3-aa4e-59aa6c193ddb"
Sep 16 18:45:37.722292 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:45:37.722247    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-c78nz"
Sep 16 18:45:37.722529 ip-10-0-138-127 kubenswrapper[2581]: E0916 18:45:37.722496    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-c78nz" podUID="f09babc9-6295-4a0a-be9f-fa0175b0de51"
Sep 16 18:45:38.722402 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:45:38.722364    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-bz5jk"
Sep 16 18:45:38.722894 ip-10-0-138-127 kubenswrapper[2581]: E0916 18:45:38.722513    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-bz5jk" podUID="4775e8c4-09ba-43db-98b7-073aeb452ce8"
Sep 16 18:45:38.722894 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:45:38.722584    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-c2kp2"
Sep 16 18:45:38.722894 ip-10-0-138-127 kubenswrapper[2581]: E0916 18:45:38.722701    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-c2kp2" podUID="eaf5897a-76a0-4bf3-aa4e-59aa6c193ddb"
Sep 16 18:45:39.722432 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:45:39.722386    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-c78nz"
Sep 16 18:45:39.723038 ip-10-0-138-127 kubenswrapper[2581]: E0916 18:45:39.722541    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-c78nz" podUID="f09babc9-6295-4a0a-be9f-fa0175b0de51"
Sep 16 18:45:40.268496 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:45:40.268450    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/f09babc9-6295-4a0a-be9f-fa0175b0de51-original-pull-secret\") pod \"global-pull-secret-syncer-c78nz\" (UID: \"f09babc9-6295-4a0a-be9f-fa0175b0de51\") " pod="kube-system/global-pull-secret-syncer-c78nz"
Sep 16 18:45:40.268687 ip-10-0-138-127 kubenswrapper[2581]: E0916 18:45:40.268634    2581 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 16 18:45:40.268743 ip-10-0-138-127 kubenswrapper[2581]: E0916 18:45:40.268708    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f09babc9-6295-4a0a-be9f-fa0175b0de51-original-pull-secret podName:f09babc9-6295-4a0a-be9f-fa0175b0de51 nodeName:}" failed. No retries permitted until 2026-09-16 18:45:48.268689517 +0000 UTC m=+18.088103414 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/f09babc9-6295-4a0a-be9f-fa0175b0de51-original-pull-secret") pod "global-pull-secret-syncer-c78nz" (UID: "f09babc9-6295-4a0a-be9f-fa0175b0de51") : object "kube-system"/"original-pull-secret" not registered
Sep 16 18:45:40.369635 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:45:40.369583    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5vmqf\" (UniqueName: \"kubernetes.io/projected/4775e8c4-09ba-43db-98b7-073aeb452ce8-kube-api-access-5vmqf\") pod \"network-check-target-bz5jk\" (UID: \"4775e8c4-09ba-43db-98b7-073aeb452ce8\") " pod="openshift-network-diagnostics/network-check-target-bz5jk"
Sep 16 18:45:40.369847 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:45:40.369658    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/eaf5897a-76a0-4bf3-aa4e-59aa6c193ddb-metrics-certs\") pod \"network-metrics-daemon-c2kp2\" (UID: \"eaf5897a-76a0-4bf3-aa4e-59aa6c193ddb\") " pod="openshift-multus/network-metrics-daemon-c2kp2"
Sep 16 18:45:40.369847 ip-10-0-138-127 kubenswrapper[2581]: E0916 18:45:40.369819    2581 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 16 18:45:40.369969 ip-10-0-138-127 kubenswrapper[2581]: E0916 18:45:40.369892    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/eaf5897a-76a0-4bf3-aa4e-59aa6c193ddb-metrics-certs podName:eaf5897a-76a0-4bf3-aa4e-59aa6c193ddb nodeName:}" failed. No retries permitted until 2026-09-16 18:45:48.369872657 +0000 UTC m=+18.189286551 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/eaf5897a-76a0-4bf3-aa4e-59aa6c193ddb-metrics-certs") pod "network-metrics-daemon-c2kp2" (UID: "eaf5897a-76a0-4bf3-aa4e-59aa6c193ddb") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 16 18:45:40.370419 ip-10-0-138-127 kubenswrapper[2581]: E0916 18:45:40.370298    2581 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 16 18:45:40.370419 ip-10-0-138-127 kubenswrapper[2581]: E0916 18:45:40.370324    2581 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 16 18:45:40.370419 ip-10-0-138-127 kubenswrapper[2581]: E0916 18:45:40.370336    2581 projected.go:196] Error preparing data for projected volume kube-api-access-5vmqf for pod openshift-network-diagnostics/network-check-target-bz5jk: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 16 18:45:40.370419 ip-10-0-138-127 kubenswrapper[2581]: E0916 18:45:40.370388    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/4775e8c4-09ba-43db-98b7-073aeb452ce8-kube-api-access-5vmqf podName:4775e8c4-09ba-43db-98b7-073aeb452ce8 nodeName:}" failed. No retries permitted until 2026-09-16 18:45:48.370371285 +0000 UTC m=+18.189785180 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-5vmqf" (UniqueName: "kubernetes.io/projected/4775e8c4-09ba-43db-98b7-073aeb452ce8-kube-api-access-5vmqf") pod "network-check-target-bz5jk" (UID: "4775e8c4-09ba-43db-98b7-073aeb452ce8") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 16 18:45:40.724088 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:45:40.723332    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-bz5jk"
Sep 16 18:45:40.724088 ip-10-0-138-127 kubenswrapper[2581]: E0916 18:45:40.723437    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-bz5jk" podUID="4775e8c4-09ba-43db-98b7-073aeb452ce8"
Sep 16 18:45:40.724088 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:45:40.723858    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-c2kp2"
Sep 16 18:45:40.724088 ip-10-0-138-127 kubenswrapper[2581]: E0916 18:45:40.723964    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-c2kp2" podUID="eaf5897a-76a0-4bf3-aa4e-59aa6c193ddb"
Sep 16 18:45:41.722175 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:45:41.722144    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-c78nz"
Sep 16 18:45:41.722388 ip-10-0-138-127 kubenswrapper[2581]: E0916 18:45:41.722308    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-c78nz" podUID="f09babc9-6295-4a0a-be9f-fa0175b0de51"
Sep 16 18:45:42.722509 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:45:42.722477    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-bz5jk"
Sep 16 18:45:42.722949 ip-10-0-138-127 kubenswrapper[2581]: E0916 18:45:42.722608    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-bz5jk" podUID="4775e8c4-09ba-43db-98b7-073aeb452ce8"
Sep 16 18:45:42.723083 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:45:42.723064    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-c2kp2"
Sep 16 18:45:42.723222 ip-10-0-138-127 kubenswrapper[2581]: E0916 18:45:42.723182    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-c2kp2" podUID="eaf5897a-76a0-4bf3-aa4e-59aa6c193ddb"
Sep 16 18:45:43.722393 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:45:43.722356    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-c78nz"
Sep 16 18:45:43.722587 ip-10-0-138-127 kubenswrapper[2581]: E0916 18:45:43.722484    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-c78nz" podUID="f09babc9-6295-4a0a-be9f-fa0175b0de51"
Sep 16 18:45:44.722830 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:45:44.722795    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-c2kp2"
Sep 16 18:45:44.723315 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:45:44.722849    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-bz5jk"
Sep 16 18:45:44.723315 ip-10-0-138-127 kubenswrapper[2581]: E0916 18:45:44.722944    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-c2kp2" podUID="eaf5897a-76a0-4bf3-aa4e-59aa6c193ddb"
Sep 16 18:45:44.723315 ip-10-0-138-127 kubenswrapper[2581]: E0916 18:45:44.723057    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-bz5jk" podUID="4775e8c4-09ba-43db-98b7-073aeb452ce8"
Sep 16 18:45:45.722086 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:45:45.721993    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-c78nz"
Sep 16 18:45:45.722276 ip-10-0-138-127 kubenswrapper[2581]: E0916 18:45:45.722128    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-c78nz" podUID="f09babc9-6295-4a0a-be9f-fa0175b0de51"
Sep 16 18:45:46.721969 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:45:46.721933    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-c2kp2"
Sep 16 18:45:46.722418 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:45:46.721986    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-bz5jk"
Sep 16 18:45:46.722418 ip-10-0-138-127 kubenswrapper[2581]: E0916 18:45:46.722105    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-c2kp2" podUID="eaf5897a-76a0-4bf3-aa4e-59aa6c193ddb"
Sep 16 18:45:46.722418 ip-10-0-138-127 kubenswrapper[2581]: E0916 18:45:46.722189    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-bz5jk" podUID="4775e8c4-09ba-43db-98b7-073aeb452ce8"
Sep 16 18:45:47.722149 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:45:47.722112    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-c78nz"
Sep 16 18:45:47.722721 ip-10-0-138-127 kubenswrapper[2581]: E0916 18:45:47.722236    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-c78nz" podUID="f09babc9-6295-4a0a-be9f-fa0175b0de51"
Sep 16 18:45:48.334811 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:45:48.334771    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/f09babc9-6295-4a0a-be9f-fa0175b0de51-original-pull-secret\") pod \"global-pull-secret-syncer-c78nz\" (UID: \"f09babc9-6295-4a0a-be9f-fa0175b0de51\") " pod="kube-system/global-pull-secret-syncer-c78nz"
Sep 16 18:45:48.335045 ip-10-0-138-127 kubenswrapper[2581]: E0916 18:45:48.334914    2581 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 16 18:45:48.335045 ip-10-0-138-127 kubenswrapper[2581]: E0916 18:45:48.334979    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f09babc9-6295-4a0a-be9f-fa0175b0de51-original-pull-secret podName:f09babc9-6295-4a0a-be9f-fa0175b0de51 nodeName:}" failed. No retries permitted until 2026-09-16 18:46:04.334961132 +0000 UTC m=+34.154375024 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/f09babc9-6295-4a0a-be9f-fa0175b0de51-original-pull-secret") pod "global-pull-secret-syncer-c78nz" (UID: "f09babc9-6295-4a0a-be9f-fa0175b0de51") : object "kube-system"/"original-pull-secret" not registered
Sep 16 18:45:48.436152 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:45:48.435599    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5vmqf\" (UniqueName: \"kubernetes.io/projected/4775e8c4-09ba-43db-98b7-073aeb452ce8-kube-api-access-5vmqf\") pod \"network-check-target-bz5jk\" (UID: \"4775e8c4-09ba-43db-98b7-073aeb452ce8\") " pod="openshift-network-diagnostics/network-check-target-bz5jk"
Sep 16 18:45:48.436152 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:45:48.435650    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/eaf5897a-76a0-4bf3-aa4e-59aa6c193ddb-metrics-certs\") pod \"network-metrics-daemon-c2kp2\" (UID: \"eaf5897a-76a0-4bf3-aa4e-59aa6c193ddb\") " pod="openshift-multus/network-metrics-daemon-c2kp2"
Sep 16 18:45:48.436152 ip-10-0-138-127 kubenswrapper[2581]: E0916 18:45:48.435781    2581 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 16 18:45:48.436152 ip-10-0-138-127 kubenswrapper[2581]: E0916 18:45:48.435790    2581 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 16 18:45:48.436152 ip-10-0-138-127 kubenswrapper[2581]: E0916 18:45:48.435816    2581 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 16 18:45:48.436152 ip-10-0-138-127 kubenswrapper[2581]: E0916 18:45:48.435828    2581 projected.go:196] Error preparing data for projected volume kube-api-access-5vmqf for pod openshift-network-diagnostics/network-check-target-bz5jk: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 16 18:45:48.436152 ip-10-0-138-127 kubenswrapper[2581]: E0916 18:45:48.435834    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/eaf5897a-76a0-4bf3-aa4e-59aa6c193ddb-metrics-certs podName:eaf5897a-76a0-4bf3-aa4e-59aa6c193ddb nodeName:}" failed. No retries permitted until 2026-09-16 18:46:04.435821109 +0000 UTC m=+34.255234996 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/eaf5897a-76a0-4bf3-aa4e-59aa6c193ddb-metrics-certs") pod "network-metrics-daemon-c2kp2" (UID: "eaf5897a-76a0-4bf3-aa4e-59aa6c193ddb") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 16 18:45:48.436152 ip-10-0-138-127 kubenswrapper[2581]: E0916 18:45:48.435867    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/4775e8c4-09ba-43db-98b7-073aeb452ce8-kube-api-access-5vmqf podName:4775e8c4-09ba-43db-98b7-073aeb452ce8 nodeName:}" failed. No retries permitted until 2026-09-16 18:46:04.4358556 +0000 UTC m=+34.255269488 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-5vmqf" (UniqueName: "kubernetes.io/projected/4775e8c4-09ba-43db-98b7-073aeb452ce8-kube-api-access-5vmqf") pod "network-check-target-bz5jk" (UID: "4775e8c4-09ba-43db-98b7-073aeb452ce8") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 16 18:45:48.721855 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:45:48.721754    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-bz5jk"
Sep 16 18:45:48.721855 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:45:48.721798    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-c2kp2"
Sep 16 18:45:48.722042 ip-10-0-138-127 kubenswrapper[2581]: E0916 18:45:48.721891    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-bz5jk" podUID="4775e8c4-09ba-43db-98b7-073aeb452ce8"
Sep 16 18:45:48.722042 ip-10-0-138-127 kubenswrapper[2581]: E0916 18:45:48.722019    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-c2kp2" podUID="eaf5897a-76a0-4bf3-aa4e-59aa6c193ddb"
Sep 16 18:45:49.722387 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:45:49.722341    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-c78nz"
Sep 16 18:45:49.722820 ip-10-0-138-127 kubenswrapper[2581]: E0916 18:45:49.722475    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-c78nz" podUID="f09babc9-6295-4a0a-be9f-fa0175b0de51"
Sep 16 18:45:50.723683 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:45:50.723637    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-bz5jk"
Sep 16 18:45:50.724157 ip-10-0-138-127 kubenswrapper[2581]: E0916 18:45:50.723776    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-bz5jk" podUID="4775e8c4-09ba-43db-98b7-073aeb452ce8"
Sep 16 18:45:50.724157 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:45:50.723834    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-c2kp2"
Sep 16 18:45:50.724157 ip-10-0-138-127 kubenswrapper[2581]: E0916 18:45:50.723953    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-c2kp2" podUID="eaf5897a-76a0-4bf3-aa4e-59aa6c193ddb"
Sep 16 18:45:51.722252 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:45:51.722211    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-c78nz"
Sep 16 18:45:51.722425 ip-10-0-138-127 kubenswrapper[2581]: E0916 18:45:51.722338    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-c78nz" podUID="f09babc9-6295-4a0a-be9f-fa0175b0de51"
Sep 16 18:45:52.724362 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:45:52.724263    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-bz5jk"
Sep 16 18:45:52.724362 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:45:52.724312    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-c2kp2"
Sep 16 18:45:52.724731 ip-10-0-138-127 kubenswrapper[2581]: E0916 18:45:52.724401    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-bz5jk" podUID="4775e8c4-09ba-43db-98b7-073aeb452ce8"
Sep 16 18:45:52.724731 ip-10-0-138-127 kubenswrapper[2581]: E0916 18:45:52.724555    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-c2kp2" podUID="eaf5897a-76a0-4bf3-aa4e-59aa6c193ddb"
Sep 16 18:45:52.972207 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:45:52.972004    2581 generic.go:356] "Generic (PLEG): container finished" podID="5acb5489-33eb-4c23-b35d-484caffde098" containerID="78f07e219efaa1dba2e2d9074dcb24c038c2953990dee6cb374143380fcbf94a" exitCode=0
Sep 16 18:45:52.972207 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:45:52.972104    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-fzt7c" event={"ID":"5acb5489-33eb-4c23-b35d-484caffde098","Type":"ContainerDied","Data":"78f07e219efaa1dba2e2d9074dcb24c038c2953990dee6cb374143380fcbf94a"}
Sep 16 18:45:52.974119 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:45:52.974087    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-7t742" event={"ID":"3f382427-dc87-4034-ae03-c457ed18d446","Type":"ContainerStarted","Data":"67177044b4542aa3deac4555c076092011b505120763350e664010433cabf695"}
Sep 16 18:45:52.975894 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:45:52.975861    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-vttdl" event={"ID":"b31b8735-005e-4387-852c-bc67df0e77cb","Type":"ContainerStarted","Data":"7d87b0365853bb2d893611650dc097c7cf001ef9ee62289e50393d2d02a75d15"}
Sep 16 18:45:52.981824 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:45:52.979769    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-mvhf8" event={"ID":"99b45db8-0fbb-445b-be31-49fba3f220ec","Type":"ContainerStarted","Data":"d1eaf717684effcb5bc197b952fcf743203954855b666280519cd0449b6b24bd"}
Sep 16 18:45:52.984916 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:45:52.984890    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-vxjzx" event={"ID":"bf1c06c8-20b0-4816-a092-02e87cd69a7f","Type":"ContainerStarted","Data":"531a9550a4409dd7b10df0c9a4ad4ea4cabd70aeb69fb76683f15b274c3c8ad3"}
Sep 16 18:45:52.986569 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:45:52.986528    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-lrclz" event={"ID":"eace7050-d66d-4057-9041-62afa5df8ae7","Type":"ContainerStarted","Data":"bbd7c1e8fa19075d61cfcff256ebc6a291df6d153eab110eb038b338369120df"}
Sep 16 18:45:52.988235 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:45:52.988184    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-54w2k" event={"ID":"bf866eb9-d1c0-4cb2-ad9c-86e8b5b306d7","Type":"ContainerStarted","Data":"26049d0e4988105f18d99b34aae84f2c97cf5bfc0b9d3acf10ab7f0d7bd827d4"}
Sep 16 18:45:52.990340 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:45:52.990320    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-csmcl" event={"ID":"99e923fa-5e08-487f-ada3-1692ef20ca11","Type":"ContainerStarted","Data":"0e332e9681d6392164b3ce2160065bf3744a4ac097c20fe7b15878fa40a346ba"}
Sep 16 18:45:52.990428 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:45:52.990391    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-csmcl" event={"ID":"99e923fa-5e08-487f-ada3-1692ef20ca11","Type":"ContainerStarted","Data":"495c48e2b613732965949e34da8e255fa084b2f47eeb446eb98c0e94f2e78390"}
Sep 16 18:45:52.994174 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:45:52.994134    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-127.ec2.internal" podStartSLOduration=20.994101945 podStartE2EDuration="20.994101945s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-16 18:45:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:45:35.95107031 +0000 UTC m=+5.770484226" watchObservedRunningTime="2026-09-16 18:45:52.994101945 +0000 UTC m=+22.813515845"
Sep 16 18:45:53.012968 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:45:53.012924    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-vxjzx" podStartSLOduration=3.145046339 podStartE2EDuration="22.012908214s" totalImagesPullingTime="18.867861875s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-16 18:45:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:45:53.012646859 +0000 UTC m=+22.832060768" watchObservedRunningTime="2026-09-16 18:45:53.012908214 +0000 UTC m=+22.832322121"
Sep 16 18:45:53.027847 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:45:53.027792    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-mvhf8" podStartSLOduration=3.213098323 podStartE2EDuration="22.027767764s" totalImagesPullingTime="18.814669441s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-16 18:45:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:45:53.027768975 +0000 UTC m=+22.847182884" watchObservedRunningTime="2026-09-16 18:45:53.027767764 +0000 UTC m=+22.847181684"
Sep 16 18:45:53.041038 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:45:53.040984    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-lrclz" podStartSLOduration=7.118314332 podStartE2EDuration="22.040968623s" totalImagesPullingTime="14.922654291s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-16 18:45:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:45:53.040648521 +0000 UTC m=+22.860062431" watchObservedRunningTime="2026-09-16 18:45:53.040968623 +0000 UTC m=+22.860382533"
Sep 16 18:45:53.056696 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:45:53.056653    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-vttdl" podStartSLOduration=7.130134641 podStartE2EDuration="22.056633922s" totalImagesPullingTime="14.926499281s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:45:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:45:53.055664352 +0000 UTC m=+22.875078262" watchObservedRunningTime="2026-09-16 18:45:53.056633922 +0000 UTC m=+22.876047832"
Sep 16 18:45:53.082331 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:45:53.082292    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-7t742" podStartSLOduration=3.273319791 podStartE2EDuration="22.082276033s" totalImagesPullingTime="18.808956242s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-16 18:45:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:45:53.081591101 +0000 UTC m=+22.901005009" watchObservedRunningTime="2026-09-16 18:45:53.082276033 +0000 UTC m=+22.901689939"
Sep 16 18:45:53.722838 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:45:53.722759    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-c78nz"
Sep 16 18:45:53.722976 ip-10-0-138-127 kubenswrapper[2581]: E0916 18:45:53.722902    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-c78nz" podUID="f09babc9-6295-4a0a-be9f-fa0175b0de51"
Sep 16 18:45:53.996793 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:45:53.996730    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-csmcl" event={"ID":"99e923fa-5e08-487f-ada3-1692ef20ca11","Type":"ContainerStarted","Data":"3db48aaa43ed33ae9591fd44ba1a27f762d1d6df7191e4c80a9a151f1186e186"}
Sep 16 18:45:53.997686 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:45:53.996822    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-csmcl" event={"ID":"99e923fa-5e08-487f-ada3-1692ef20ca11","Type":"ContainerStarted","Data":"2b1f5e72c1d62348c46b4f716be31af5da83d2eac11c05bf814f7c545b8db8eb"}
Sep 16 18:45:53.997686 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:45:53.996867    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-csmcl" event={"ID":"99e923fa-5e08-487f-ada3-1692ef20ca11","Type":"ContainerStarted","Data":"89bc76f864f07e3f55b3ad0dc175fc5aef7d2c0ce3aab94bfbed4115f8180663"}
Sep 16 18:45:53.997686 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:45:53.996924    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-csmcl" event={"ID":"99e923fa-5e08-487f-ada3-1692ef20ca11","Type":"ContainerStarted","Data":"fa420d0b1b301047878f3c2fe623e36988598b3183a2271ce240e3674735f8fc"}
Sep 16 18:45:53.998586 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:45:53.998555    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-fbmb7" event={"ID":"459060f0-3e55-498a-b878-726f3e3a1060","Type":"ContainerStarted","Data":"f0439acc76641442e59f7b5fc13ab497458ff1c0e575f4edfb0e5f4bced233d7"}
Sep 16 18:45:54.028866 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:45:54.027923    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-fbmb7" podStartSLOduration=4.217472392 podStartE2EDuration="23.027899865s" totalImagesPullingTime="18.810427473s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-16 18:45:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:45:54.027746155 +0000 UTC m=+23.847160065" watchObservedRunningTime="2026-09-16 18:45:54.027899865 +0000 UTC m=+23.847313775"
Sep 16 18:45:54.169215 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:45:54.169170    2581 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Sep 16 18:45:54.663852 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:45:54.663679    2581 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-09-16T18:45:54.16922296Z","UUID":"1ffdad4a-358e-4a81-8b48-4a5bdc476509","Handler":null,"Name":"","Endpoint":""}
Sep 16 18:45:54.665436 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:45:54.665410    2581 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Sep 16 18:45:54.665570 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:45:54.665450    2581 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Sep 16 18:45:54.723247 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:45:54.722725    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-bz5jk"
Sep 16 18:45:54.723247 ip-10-0-138-127 kubenswrapper[2581]: E0916 18:45:54.722866    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-bz5jk" podUID="4775e8c4-09ba-43db-98b7-073aeb452ce8"
Sep 16 18:45:54.723247 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:45:54.722921    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-c2kp2"
Sep 16 18:45:54.723247 ip-10-0-138-127 kubenswrapper[2581]: E0916 18:45:54.723049    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-c2kp2" podUID="eaf5897a-76a0-4bf3-aa4e-59aa6c193ddb"
Sep 16 18:45:55.006093 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:45:55.005937    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-54w2k" event={"ID":"bf866eb9-d1c0-4cb2-ad9c-86e8b5b306d7","Type":"ContainerStarted","Data":"667f7c6eacd9e8e810580693aa29a877a5f65ffa9c08810d068f1524c542a51f"}
Sep 16 18:45:55.576528 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:45:55.576493    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-vttdl"
Sep 16 18:45:55.577139 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:45:55.577104    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-vttdl"
Sep 16 18:45:55.722100 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:45:55.722009    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-c78nz"
Sep 16 18:45:55.722272 ip-10-0-138-127 kubenswrapper[2581]: E0916 18:45:55.722132    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-c78nz" podUID="f09babc9-6295-4a0a-be9f-fa0175b0de51"
Sep 16 18:45:56.010658 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:45:56.010559    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-54w2k" event={"ID":"bf866eb9-d1c0-4cb2-ad9c-86e8b5b306d7","Type":"ContainerStarted","Data":"89ea6b8ac7aa2e8154551148ac1f4ec8aaa6d95110a0436857e6ac3e926126f4"}
Sep 16 18:45:56.014869 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:45:56.014825    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-csmcl" event={"ID":"99e923fa-5e08-487f-ada3-1692ef20ca11","Type":"ContainerStarted","Data":"7e38017b54f19078e8da02565b7f0d8d1dd9a11157cd5787e2c57bcb4fb0d31e"}
Sep 16 18:45:56.015103 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:45:56.015081    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-vttdl"
Sep 16 18:45:56.015841 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:45:56.015824    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-vttdl"
Sep 16 18:45:56.036212 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:45:56.036119    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-54w2k" podStartSLOduration=3.907280534 podStartE2EDuration="25.03609947s" totalImagesPullingTime="21.128818936s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-16 18:45:31 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:45:56.035903518 +0000 UTC m=+25.855317426" watchObservedRunningTime="2026-09-16 18:45:56.03609947 +0000 UTC m=+25.855513379"
Sep 16 18:45:56.723349 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:45:56.723311    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-bz5jk"
Sep 16 18:45:56.723534 ip-10-0-138-127 kubenswrapper[2581]: E0916 18:45:56.723453    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-bz5jk" podUID="4775e8c4-09ba-43db-98b7-073aeb452ce8"
Sep 16 18:45:56.723534 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:45:56.723507    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-c2kp2"
Sep 16 18:45:56.723670 ip-10-0-138-127 kubenswrapper[2581]: E0916 18:45:56.723632    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-c2kp2" podUID="eaf5897a-76a0-4bf3-aa4e-59aa6c193ddb"
Sep 16 18:45:57.722931 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:45:57.722704    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-c78nz"
Sep 16 18:45:57.723341 ip-10-0-138-127 kubenswrapper[2581]: E0916 18:45:57.722961    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-c78nz" podUID="f09babc9-6295-4a0a-be9f-fa0175b0de51"
Sep 16 18:45:58.021667 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:45:58.021632    2581 generic.go:356] "Generic (PLEG): container finished" podID="5acb5489-33eb-4c23-b35d-484caffde098" containerID="a9c0fcc6eac1f2499064b7870cac877e64b6ed51c5a39003e0378302ce334ae0" exitCode=0
Sep 16 18:45:58.021898 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:45:58.021715    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-fzt7c" event={"ID":"5acb5489-33eb-4c23-b35d-484caffde098","Type":"ContainerDied","Data":"a9c0fcc6eac1f2499064b7870cac877e64b6ed51c5a39003e0378302ce334ae0"}
Sep 16 18:45:58.025828 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:45:58.025801    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-csmcl" event={"ID":"99e923fa-5e08-487f-ada3-1692ef20ca11","Type":"ContainerStarted","Data":"b148fd084c946ef63770fbc523c02cdb7a9390f18b2a0fe24078fcf922b063fd"}
Sep 16 18:45:58.078715 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:45:58.078665    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-csmcl" podStartSLOduration=8.205113277 podStartE2EDuration="27.078648752s" totalImagesPullingTime="18.873535475s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-16 18:45:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:45:58.075699837 +0000 UTC m=+27.895113745" watchObservedRunningTime="2026-09-16 18:45:58.078648752 +0000 UTC m=+27.898062661"
Sep 16 18:45:58.722118 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:45:58.722081    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-bz5jk"
Sep 16 18:45:58.722338 ip-10-0-138-127 kubenswrapper[2581]: E0916 18:45:58.722232    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-bz5jk" podUID="4775e8c4-09ba-43db-98b7-073aeb452ce8"
Sep 16 18:45:58.722338 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:45:58.722092    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-c2kp2"
Sep 16 18:45:58.722455 ip-10-0-138-127 kubenswrapper[2581]: E0916 18:45:58.722339    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-c2kp2" podUID="eaf5897a-76a0-4bf3-aa4e-59aa6c193ddb"
Sep 16 18:45:59.030455 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:45:59.030423    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-csmcl"
Sep 16 18:45:59.031130 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:45:59.030510    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-csmcl"
Sep 16 18:45:59.031130 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:45:59.030561    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-csmcl"
Sep 16 18:45:59.049313 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:45:59.049041    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-csmcl"
Sep 16 18:45:59.050247 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:45:59.050225    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-csmcl"
Sep 16 18:45:59.510122 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:45:59.510090    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-c78nz"]
Sep 16 18:45:59.510307 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:45:59.510245    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-c78nz"
Sep 16 18:45:59.510350 ip-10-0-138-127 kubenswrapper[2581]: E0916 18:45:59.510325    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-c78nz" podUID="f09babc9-6295-4a0a-be9f-fa0175b0de51"
Sep 16 18:45:59.513974 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:45:59.513950    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-bz5jk"]
Sep 16 18:45:59.514108 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:45:59.514051    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-bz5jk"
Sep 16 18:45:59.514158 ip-10-0-138-127 kubenswrapper[2581]: E0916 18:45:59.514126    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-bz5jk" podUID="4775e8c4-09ba-43db-98b7-073aeb452ce8"
Sep 16 18:45:59.516818 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:45:59.516794    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-c2kp2"]
Sep 16 18:45:59.516936 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:45:59.516904    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-c2kp2"
Sep 16 18:45:59.517023 ip-10-0-138-127 kubenswrapper[2581]: E0916 18:45:59.517005    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-c2kp2" podUID="eaf5897a-76a0-4bf3-aa4e-59aa6c193ddb"
Sep 16 18:46:00.034060 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:46:00.034028    2581 generic.go:356] "Generic (PLEG): container finished" podID="5acb5489-33eb-4c23-b35d-484caffde098" containerID="f9b261ecd93496319b68831523c334694ca734df5c73859f5ebecae8c839f44b" exitCode=0
Sep 16 18:46:00.034613 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:46:00.034120    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-fzt7c" event={"ID":"5acb5489-33eb-4c23-b35d-484caffde098","Type":"ContainerDied","Data":"f9b261ecd93496319b68831523c334694ca734df5c73859f5ebecae8c839f44b"}
Sep 16 18:46:00.723411 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:46:00.723373    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-c2kp2"
Sep 16 18:46:00.723583 ip-10-0-138-127 kubenswrapper[2581]: E0916 18:46:00.723505    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-c2kp2" podUID="eaf5897a-76a0-4bf3-aa4e-59aa6c193ddb"
Sep 16 18:46:00.723940 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:46:00.723914    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-bz5jk"
Sep 16 18:46:00.724038 ip-10-0-138-127 kubenswrapper[2581]: E0916 18:46:00.724016    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-bz5jk" podUID="4775e8c4-09ba-43db-98b7-073aeb452ce8"
Sep 16 18:46:01.038602 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:46:01.038565    2581 generic.go:356] "Generic (PLEG): container finished" podID="5acb5489-33eb-4c23-b35d-484caffde098" containerID="8bf91d80973e727beedf29188c08156a4e58420b3630cddb668943095e2ce6cb" exitCode=0
Sep 16 18:46:01.039022 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:46:01.038661    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-fzt7c" event={"ID":"5acb5489-33eb-4c23-b35d-484caffde098","Type":"ContainerDied","Data":"8bf91d80973e727beedf29188c08156a4e58420b3630cddb668943095e2ce6cb"}
Sep 16 18:46:01.721925 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:46:01.721891    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-c78nz"
Sep 16 18:46:01.722104 ip-10-0-138-127 kubenswrapper[2581]: E0916 18:46:01.722016    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-c78nz" podUID="f09babc9-6295-4a0a-be9f-fa0175b0de51"
Sep 16 18:46:02.722742 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:46:02.722478    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-bz5jk"
Sep 16 18:46:02.723457 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:46:02.722487    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-c2kp2"
Sep 16 18:46:02.723457 ip-10-0-138-127 kubenswrapper[2581]: E0916 18:46:02.722854    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-bz5jk" podUID="4775e8c4-09ba-43db-98b7-073aeb452ce8"
Sep 16 18:46:02.723457 ip-10-0-138-127 kubenswrapper[2581]: E0916 18:46:02.722990    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-c2kp2" podUID="eaf5897a-76a0-4bf3-aa4e-59aa6c193ddb"
Sep 16 18:46:03.722203 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:46:03.722160    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-c78nz"
Sep 16 18:46:03.722616 ip-10-0-138-127 kubenswrapper[2581]: E0916 18:46:03.722323    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-c78nz" podUID="f09babc9-6295-4a0a-be9f-fa0175b0de51"
Sep 16 18:46:04.349079 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:46:04.349043    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/f09babc9-6295-4a0a-be9f-fa0175b0de51-original-pull-secret\") pod \"global-pull-secret-syncer-c78nz\" (UID: \"f09babc9-6295-4a0a-be9f-fa0175b0de51\") " pod="kube-system/global-pull-secret-syncer-c78nz"
Sep 16 18:46:04.349547 ip-10-0-138-127 kubenswrapper[2581]: E0916 18:46:04.349223    2581 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 16 18:46:04.349547 ip-10-0-138-127 kubenswrapper[2581]: E0916 18:46:04.349298    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f09babc9-6295-4a0a-be9f-fa0175b0de51-original-pull-secret podName:f09babc9-6295-4a0a-be9f-fa0175b0de51 nodeName:}" failed. No retries permitted until 2026-09-16 18:46:36.349280599 +0000 UTC m=+66.168694486 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/f09babc9-6295-4a0a-be9f-fa0175b0de51-original-pull-secret") pod "global-pull-secret-syncer-c78nz" (UID: "f09babc9-6295-4a0a-be9f-fa0175b0de51") : object "kube-system"/"original-pull-secret" not registered
Sep 16 18:46:04.449808 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:46:04.449767    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5vmqf\" (UniqueName: \"kubernetes.io/projected/4775e8c4-09ba-43db-98b7-073aeb452ce8-kube-api-access-5vmqf\") pod \"network-check-target-bz5jk\" (UID: \"4775e8c4-09ba-43db-98b7-073aeb452ce8\") " pod="openshift-network-diagnostics/network-check-target-bz5jk"
Sep 16 18:46:04.450090 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:46:04.449837    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/eaf5897a-76a0-4bf3-aa4e-59aa6c193ddb-metrics-certs\") pod \"network-metrics-daemon-c2kp2\" (UID: \"eaf5897a-76a0-4bf3-aa4e-59aa6c193ddb\") " pod="openshift-multus/network-metrics-daemon-c2kp2"
Sep 16 18:46:04.450090 ip-10-0-138-127 kubenswrapper[2581]: E0916 18:46:04.449965    2581 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 16 18:46:04.450090 ip-10-0-138-127 kubenswrapper[2581]: E0916 18:46:04.450032    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/eaf5897a-76a0-4bf3-aa4e-59aa6c193ddb-metrics-certs podName:eaf5897a-76a0-4bf3-aa4e-59aa6c193ddb nodeName:}" failed. No retries permitted until 2026-09-16 18:46:36.450011846 +0000 UTC m=+66.269425733 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/eaf5897a-76a0-4bf3-aa4e-59aa6c193ddb-metrics-certs") pod "network-metrics-daemon-c2kp2" (UID: "eaf5897a-76a0-4bf3-aa4e-59aa6c193ddb") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 16 18:46:04.450564 ip-10-0-138-127 kubenswrapper[2581]: E0916 18:46:04.450523    2581 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 16 18:46:04.450564 ip-10-0-138-127 kubenswrapper[2581]: E0916 18:46:04.450549    2581 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 16 18:46:04.450564 ip-10-0-138-127 kubenswrapper[2581]: E0916 18:46:04.450562    2581 projected.go:196] Error preparing data for projected volume kube-api-access-5vmqf for pod openshift-network-diagnostics/network-check-target-bz5jk: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 16 18:46:04.450772 ip-10-0-138-127 kubenswrapper[2581]: E0916 18:46:04.450606    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/4775e8c4-09ba-43db-98b7-073aeb452ce8-kube-api-access-5vmqf podName:4775e8c4-09ba-43db-98b7-073aeb452ce8 nodeName:}" failed. No retries permitted until 2026-09-16 18:46:36.450591651 +0000 UTC m=+66.270005574 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-5vmqf" (UniqueName: "kubernetes.io/projected/4775e8c4-09ba-43db-98b7-073aeb452ce8-kube-api-access-5vmqf") pod "network-check-target-bz5jk" (UID: "4775e8c4-09ba-43db-98b7-073aeb452ce8") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 16 18:46:04.723248 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:46:04.722665    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-bz5jk"
Sep 16 18:46:04.723248 ip-10-0-138-127 kubenswrapper[2581]: E0916 18:46:04.722796    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-bz5jk" podUID="4775e8c4-09ba-43db-98b7-073aeb452ce8"
Sep 16 18:46:04.723248 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:46:04.722812    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-c2kp2"
Sep 16 18:46:04.723248 ip-10-0-138-127 kubenswrapper[2581]: E0916 18:46:04.722919    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-c2kp2" podUID="eaf5897a-76a0-4bf3-aa4e-59aa6c193ddb"
Sep 16 18:46:05.722125 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:46:05.722102    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-c78nz"
Sep 16 18:46:05.722475 ip-10-0-138-127 kubenswrapper[2581]: E0916 18:46:05.722212    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-c78nz" podUID="f09babc9-6295-4a0a-be9f-fa0175b0de51"
Sep 16 18:46:06.053743 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:46:06.053705    2581 generic.go:356] "Generic (PLEG): container finished" podID="5acb5489-33eb-4c23-b35d-484caffde098" containerID="7e9587d00555f4e160e6760ab9b2489e63de4e6ecbccad854cdf14df49929d41" exitCode=0
Sep 16 18:46:06.053932 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:46:06.053755    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-fzt7c" event={"ID":"5acb5489-33eb-4c23-b35d-484caffde098","Type":"ContainerDied","Data":"7e9587d00555f4e160e6760ab9b2489e63de4e6ecbccad854cdf14df49929d41"}
Sep 16 18:46:06.448721 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:46:06.448639    2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-127.ec2.internal" event="NodeReady"
Sep 16 18:46:06.448859 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:46:06.448760    2581 kubelet_node_status.go:489] "Fast updating node status as it just became ready"
Sep 16 18:46:06.497871 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:46:06.497828    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-649b4fc7ff-zmk84"]
Sep 16 18:46:06.513999 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:46:06.513968    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-649b4fc7ff-zmk84"
Sep 16 18:46:06.515580 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:46:06.515554    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-qsxrs"]
Sep 16 18:46:06.516622 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:46:06.516593    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\""
Sep 16 18:46:06.516710 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:46:06.516641    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\""
Sep 16 18:46:06.516710 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:46:06.516688    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-5jk86\""
Sep 16 18:46:06.516913 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:46:06.516894    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\""
Sep 16 18:46:06.522215 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:46:06.522176    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\""
Sep 16 18:46:06.530006 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:46:06.529986    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-z69gl"]
Sep 16 18:46:06.530124 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:46:06.530110    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-qsxrs"
Sep 16 18:46:06.533517 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:46:06.533494    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Sep 16 18:46:06.533622 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:46:06.533501    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-nqnnl\""
Sep 16 18:46:06.534254 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:46:06.534235    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Sep 16 18:46:06.552058 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:46:06.552028    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-649b4fc7ff-zmk84"]
Sep 16 18:46:06.552058 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:46:06.552059    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-z69gl"]
Sep 16 18:46:06.552317 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:46:06.552071    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-qsxrs"]
Sep 16 18:46:06.552317 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:46:06.552165    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-z69gl"
Sep 16 18:46:06.555167 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:46:06.555147    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-wrlvp\""
Sep 16 18:46:06.555379 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:46:06.555367    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Sep 16 18:46:06.555618 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:46:06.555596    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Sep 16 18:46:06.556211 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:46:06.556178    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Sep 16 18:46:06.665452 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:46:06.665418    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zkkrz\" (UniqueName: \"kubernetes.io/projected/60f08a67-da8a-40c3-a0b9-93d7668c7da1-kube-api-access-zkkrz\") pod \"ingress-canary-z69gl\" (UID: \"60f08a67-da8a-40c3-a0b9-93d7668c7da1\") " pod="openshift-ingress-canary/ingress-canary-z69gl"
Sep 16 18:46:06.665642 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:46:06.665462    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/57ce0be5-62d9-473f-a19f-4253cb333fe2-ca-trust-extracted\") pod \"image-registry-649b4fc7ff-zmk84\" (UID: \"57ce0be5-62d9-473f-a19f-4253cb333fe2\") " pod="openshift-image-registry/image-registry-649b4fc7ff-zmk84"
Sep 16 18:46:06.665642 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:46:06.665483    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6xnhc\" (UniqueName: \"kubernetes.io/projected/57ce0be5-62d9-473f-a19f-4253cb333fe2-kube-api-access-6xnhc\") pod \"image-registry-649b4fc7ff-zmk84\" (UID: \"57ce0be5-62d9-473f-a19f-4253cb333fe2\") " pod="openshift-image-registry/image-registry-649b4fc7ff-zmk84"
Sep 16 18:46:06.665642 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:46:06.665508    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tml2c\" (UniqueName: \"kubernetes.io/projected/61521525-706f-428c-80f6-2cdfcb30fdc4-kube-api-access-tml2c\") pod \"dns-default-qsxrs\" (UID: \"61521525-706f-428c-80f6-2cdfcb30fdc4\") " pod="openshift-dns/dns-default-qsxrs"
Sep 16 18:46:06.665642 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:46:06.665556    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/57ce0be5-62d9-473f-a19f-4253cb333fe2-installation-pull-secrets\") pod \"image-registry-649b4fc7ff-zmk84\" (UID: \"57ce0be5-62d9-473f-a19f-4253cb333fe2\") " pod="openshift-image-registry/image-registry-649b4fc7ff-zmk84"
Sep 16 18:46:06.665642 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:46:06.665611    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/60f08a67-da8a-40c3-a0b9-93d7668c7da1-cert\") pod \"ingress-canary-z69gl\" (UID: \"60f08a67-da8a-40c3-a0b9-93d7668c7da1\") " pod="openshift-ingress-canary/ingress-canary-z69gl"
Sep 16 18:46:06.665894 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:46:06.665700    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/57ce0be5-62d9-473f-a19f-4253cb333fe2-image-registry-private-configuration\") pod \"image-registry-649b4fc7ff-zmk84\" (UID: \"57ce0be5-62d9-473f-a19f-4253cb333fe2\") " pod="openshift-image-registry/image-registry-649b4fc7ff-zmk84"
Sep 16 18:46:06.665894 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:46:06.665734    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/57ce0be5-62d9-473f-a19f-4253cb333fe2-trusted-ca\") pod \"image-registry-649b4fc7ff-zmk84\" (UID: \"57ce0be5-62d9-473f-a19f-4253cb333fe2\") " pod="openshift-image-registry/image-registry-649b4fc7ff-zmk84"
Sep 16 18:46:06.665894 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:46:06.665791    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/57ce0be5-62d9-473f-a19f-4253cb333fe2-registry-certificates\") pod \"image-registry-649b4fc7ff-zmk84\" (UID: \"57ce0be5-62d9-473f-a19f-4253cb333fe2\") " pod="openshift-image-registry/image-registry-649b4fc7ff-zmk84"
Sep 16 18:46:06.665894 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:46:06.665833    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/61521525-706f-428c-80f6-2cdfcb30fdc4-metrics-tls\") pod \"dns-default-qsxrs\" (UID: \"61521525-706f-428c-80f6-2cdfcb30fdc4\") " pod="openshift-dns/dns-default-qsxrs"
Sep 16 18:46:06.665894 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:46:06.665883    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/57ce0be5-62d9-473f-a19f-4253cb333fe2-registry-tls\") pod \"image-registry-649b4fc7ff-zmk84\" (UID: \"57ce0be5-62d9-473f-a19f-4253cb333fe2\") " pod="openshift-image-registry/image-registry-649b4fc7ff-zmk84"
Sep 16 18:46:06.666098 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:46:06.665905    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/61521525-706f-428c-80f6-2cdfcb30fdc4-config-volume\") pod \"dns-default-qsxrs\" (UID: \"61521525-706f-428c-80f6-2cdfcb30fdc4\") " pod="openshift-dns/dns-default-qsxrs"
Sep 16 18:46:06.666098 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:46:06.665936    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/61521525-706f-428c-80f6-2cdfcb30fdc4-tmp-dir\") pod \"dns-default-qsxrs\" (UID: \"61521525-706f-428c-80f6-2cdfcb30fdc4\") " pod="openshift-dns/dns-default-qsxrs"
Sep 16 18:46:06.666098 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:46:06.665995    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/57ce0be5-62d9-473f-a19f-4253cb333fe2-bound-sa-token\") pod \"image-registry-649b4fc7ff-zmk84\" (UID: \"57ce0be5-62d9-473f-a19f-4253cb333fe2\") " pod="openshift-image-registry/image-registry-649b4fc7ff-zmk84"
Sep 16 18:46:06.722917 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:46:06.722832    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-bz5jk"
Sep 16 18:46:06.723345 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:46:06.723056    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-c2kp2"
Sep 16 18:46:06.725998 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:46:06.725956    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Sep 16 18:46:06.726132 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:46:06.725956    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Sep 16 18:46:06.726132 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:46:06.726060    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Sep 16 18:46:06.726274 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:46:06.726148    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-4ccjj\""
Sep 16 18:46:06.726274 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:46:06.726219    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-pl7qs\""
Sep 16 18:46:06.766360 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:46:06.766324    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/57ce0be5-62d9-473f-a19f-4253cb333fe2-image-registry-private-configuration\") pod \"image-registry-649b4fc7ff-zmk84\" (UID: \"57ce0be5-62d9-473f-a19f-4253cb333fe2\") " pod="openshift-image-registry/image-registry-649b4fc7ff-zmk84"
Sep 16 18:46:06.766360 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:46:06.766362    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/57ce0be5-62d9-473f-a19f-4253cb333fe2-trusted-ca\") pod \"image-registry-649b4fc7ff-zmk84\" (UID: \"57ce0be5-62d9-473f-a19f-4253cb333fe2\") " pod="openshift-image-registry/image-registry-649b4fc7ff-zmk84"
Sep 16 18:46:06.766580 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:46:06.766407    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/57ce0be5-62d9-473f-a19f-4253cb333fe2-registry-certificates\") pod \"image-registry-649b4fc7ff-zmk84\" (UID: \"57ce0be5-62d9-473f-a19f-4253cb333fe2\") " pod="openshift-image-registry/image-registry-649b4fc7ff-zmk84"
Sep 16 18:46:06.766580 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:46:06.766434    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/61521525-706f-428c-80f6-2cdfcb30fdc4-metrics-tls\") pod \"dns-default-qsxrs\" (UID: \"61521525-706f-428c-80f6-2cdfcb30fdc4\") " pod="openshift-dns/dns-default-qsxrs"
Sep 16 18:46:06.766580 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:46:06.766461    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/57ce0be5-62d9-473f-a19f-4253cb333fe2-registry-tls\") pod \"image-registry-649b4fc7ff-zmk84\" (UID: \"57ce0be5-62d9-473f-a19f-4253cb333fe2\") " pod="openshift-image-registry/image-registry-649b4fc7ff-zmk84"
Sep 16 18:46:06.766580 ip-10-0-138-127 kubenswrapper[2581]: E0916 18:46:06.766533    2581 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 16 18:46:06.766580 ip-10-0-138-127 kubenswrapper[2581]: E0916 18:46:06.766546    2581 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-649b4fc7ff-zmk84: secret "image-registry-tls" not found
Sep 16 18:46:06.766580 ip-10-0-138-127 kubenswrapper[2581]: E0916 18:46:06.766556    2581 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 16 18:46:06.766854 ip-10-0-138-127 kubenswrapper[2581]: E0916 18:46:06.766608    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/57ce0be5-62d9-473f-a19f-4253cb333fe2-registry-tls podName:57ce0be5-62d9-473f-a19f-4253cb333fe2 nodeName:}" failed. No retries permitted until 2026-09-16 18:46:07.266589622 +0000 UTC m=+37.086003526 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/57ce0be5-62d9-473f-a19f-4253cb333fe2-registry-tls") pod "image-registry-649b4fc7ff-zmk84" (UID: "57ce0be5-62d9-473f-a19f-4253cb333fe2") : secret "image-registry-tls" not found
Sep 16 18:46:06.766854 ip-10-0-138-127 kubenswrapper[2581]: E0916 18:46:06.766626    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/61521525-706f-428c-80f6-2cdfcb30fdc4-metrics-tls podName:61521525-706f-428c-80f6-2cdfcb30fdc4 nodeName:}" failed. No retries permitted until 2026-09-16 18:46:07.266617196 +0000 UTC m=+37.086031086 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/61521525-706f-428c-80f6-2cdfcb30fdc4-metrics-tls") pod "dns-default-qsxrs" (UID: "61521525-706f-428c-80f6-2cdfcb30fdc4") : secret "dns-default-metrics-tls" not found
Sep 16 18:46:06.766854 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:46:06.766684    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/61521525-706f-428c-80f6-2cdfcb30fdc4-config-volume\") pod \"dns-default-qsxrs\" (UID: \"61521525-706f-428c-80f6-2cdfcb30fdc4\") " pod="openshift-dns/dns-default-qsxrs"
Sep 16 18:46:06.766854 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:46:06.766719    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/61521525-706f-428c-80f6-2cdfcb30fdc4-tmp-dir\") pod \"dns-default-qsxrs\" (UID: \"61521525-706f-428c-80f6-2cdfcb30fdc4\") " pod="openshift-dns/dns-default-qsxrs"
Sep 16 18:46:06.766854 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:46:06.766764    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/57ce0be5-62d9-473f-a19f-4253cb333fe2-bound-sa-token\") pod \"image-registry-649b4fc7ff-zmk84\" (UID: \"57ce0be5-62d9-473f-a19f-4253cb333fe2\") " pod="openshift-image-registry/image-registry-649b4fc7ff-zmk84"
Sep 16 18:46:06.766854 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:46:06.766810    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zkkrz\" (UniqueName: \"kubernetes.io/projected/60f08a67-da8a-40c3-a0b9-93d7668c7da1-kube-api-access-zkkrz\") pod \"ingress-canary-z69gl\" (UID: \"60f08a67-da8a-40c3-a0b9-93d7668c7da1\") " pod="openshift-ingress-canary/ingress-canary-z69gl"
Sep 16 18:46:06.766854 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:46:06.766852    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/57ce0be5-62d9-473f-a19f-4253cb333fe2-ca-trust-extracted\") pod \"image-registry-649b4fc7ff-zmk84\" (UID: \"57ce0be5-62d9-473f-a19f-4253cb333fe2\") " pod="openshift-image-registry/image-registry-649b4fc7ff-zmk84"
Sep 16 18:46:06.767229 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:46:06.766875    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6xnhc\" (UniqueName: \"kubernetes.io/projected/57ce0be5-62d9-473f-a19f-4253cb333fe2-kube-api-access-6xnhc\") pod \"image-registry-649b4fc7ff-zmk84\" (UID: \"57ce0be5-62d9-473f-a19f-4253cb333fe2\") " pod="openshift-image-registry/image-registry-649b4fc7ff-zmk84"
Sep 16 18:46:06.767229 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:46:06.766902    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tml2c\" (UniqueName: \"kubernetes.io/projected/61521525-706f-428c-80f6-2cdfcb30fdc4-kube-api-access-tml2c\") pod \"dns-default-qsxrs\" (UID: \"61521525-706f-428c-80f6-2cdfcb30fdc4\") " pod="openshift-dns/dns-default-qsxrs"
Sep 16 18:46:06.767229 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:46:06.766933    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/57ce0be5-62d9-473f-a19f-4253cb333fe2-installation-pull-secrets\") pod \"image-registry-649b4fc7ff-zmk84\" (UID: \"57ce0be5-62d9-473f-a19f-4253cb333fe2\") " pod="openshift-image-registry/image-registry-649b4fc7ff-zmk84"
Sep 16 18:46:06.767229 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:46:06.766956    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/60f08a67-da8a-40c3-a0b9-93d7668c7da1-cert\") pod \"ingress-canary-z69gl\" (UID: \"60f08a67-da8a-40c3-a0b9-93d7668c7da1\") " pod="openshift-ingress-canary/ingress-canary-z69gl"
Sep 16 18:46:06.767438 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:46:06.767246    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/57ce0be5-62d9-473f-a19f-4253cb333fe2-ca-trust-extracted\") pod \"image-registry-649b4fc7ff-zmk84\" (UID: \"57ce0be5-62d9-473f-a19f-4253cb333fe2\") " pod="openshift-image-registry/image-registry-649b4fc7ff-zmk84"
Sep 16 18:46:06.767438 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:46:06.767272    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/57ce0be5-62d9-473f-a19f-4253cb333fe2-registry-certificates\") pod \"image-registry-649b4fc7ff-zmk84\" (UID: \"57ce0be5-62d9-473f-a19f-4253cb333fe2\") " pod="openshift-image-registry/image-registry-649b4fc7ff-zmk84"
Sep 16 18:46:06.767438 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:46:06.767305    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/61521525-706f-428c-80f6-2cdfcb30fdc4-config-volume\") pod \"dns-default-qsxrs\" (UID: \"61521525-706f-428c-80f6-2cdfcb30fdc4\") " pod="openshift-dns/dns-default-qsxrs"
Sep 16 18:46:06.767551 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:46:06.767511    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/57ce0be5-62d9-473f-a19f-4253cb333fe2-trusted-ca\") pod \"image-registry-649b4fc7ff-zmk84\" (UID: \"57ce0be5-62d9-473f-a19f-4253cb333fe2\") " pod="openshift-image-registry/image-registry-649b4fc7ff-zmk84"
Sep 16 18:46:06.767740 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:46:06.767722    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/61521525-706f-428c-80f6-2cdfcb30fdc4-tmp-dir\") pod \"dns-default-qsxrs\" (UID: \"61521525-706f-428c-80f6-2cdfcb30fdc4\") " pod="openshift-dns/dns-default-qsxrs"
Sep 16 18:46:06.771231 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:46:06.771104    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/57ce0be5-62d9-473f-a19f-4253cb333fe2-installation-pull-secrets\") pod \"image-registry-649b4fc7ff-zmk84\" (UID: \"57ce0be5-62d9-473f-a19f-4253cb333fe2\") " pod="openshift-image-registry/image-registry-649b4fc7ff-zmk84"
Sep 16 18:46:06.771325 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:46:06.771129    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/60f08a67-da8a-40c3-a0b9-93d7668c7da1-cert\") pod \"ingress-canary-z69gl\" (UID: \"60f08a67-da8a-40c3-a0b9-93d7668c7da1\") " pod="openshift-ingress-canary/ingress-canary-z69gl"
Sep 16 18:46:06.771325 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:46:06.771159    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/57ce0be5-62d9-473f-a19f-4253cb333fe2-image-registry-private-configuration\") pod \"image-registry-649b4fc7ff-zmk84\" (UID: \"57ce0be5-62d9-473f-a19f-4253cb333fe2\") " pod="openshift-image-registry/image-registry-649b4fc7ff-zmk84"
Sep 16 18:46:06.777111 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:46:06.777088    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/57ce0be5-62d9-473f-a19f-4253cb333fe2-bound-sa-token\") pod \"image-registry-649b4fc7ff-zmk84\" (UID: \"57ce0be5-62d9-473f-a19f-4253cb333fe2\") " pod="openshift-image-registry/image-registry-649b4fc7ff-zmk84"
Sep 16 18:46:06.777259 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:46:06.777221    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zkkrz\" (UniqueName: \"kubernetes.io/projected/60f08a67-da8a-40c3-a0b9-93d7668c7da1-kube-api-access-zkkrz\") pod \"ingress-canary-z69gl\" (UID: \"60f08a67-da8a-40c3-a0b9-93d7668c7da1\") " pod="openshift-ingress-canary/ingress-canary-z69gl"
Sep 16 18:46:06.777580 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:46:06.777564    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6xnhc\" (UniqueName: \"kubernetes.io/projected/57ce0be5-62d9-473f-a19f-4253cb333fe2-kube-api-access-6xnhc\") pod \"image-registry-649b4fc7ff-zmk84\" (UID: \"57ce0be5-62d9-473f-a19f-4253cb333fe2\") " pod="openshift-image-registry/image-registry-649b4fc7ff-zmk84"
Sep 16 18:46:06.778306 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:46:06.778286    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tml2c\" (UniqueName: \"kubernetes.io/projected/61521525-706f-428c-80f6-2cdfcb30fdc4-kube-api-access-tml2c\") pod \"dns-default-qsxrs\" (UID: \"61521525-706f-428c-80f6-2cdfcb30fdc4\") " pod="openshift-dns/dns-default-qsxrs"
Sep 16 18:46:06.860702 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:46:06.860654    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-z69gl"
Sep 16 18:46:07.059258 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:46:07.059227    2581 generic.go:356] "Generic (PLEG): container finished" podID="5acb5489-33eb-4c23-b35d-484caffde098" containerID="2901397f6ac1d550a07bb9082367474e12ad0b0cbe1ae5ef191d96c0bca41d4a" exitCode=0
Sep 16 18:46:07.059430 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:46:07.059274    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-fzt7c" event={"ID":"5acb5489-33eb-4c23-b35d-484caffde098","Type":"ContainerDied","Data":"2901397f6ac1d550a07bb9082367474e12ad0b0cbe1ae5ef191d96c0bca41d4a"}
Sep 16 18:46:07.201799 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:46:07.201767    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-z69gl"]
Sep 16 18:46:07.204914 ip-10-0-138-127 kubenswrapper[2581]: W0916 18:46:07.204890    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod60f08a67_da8a_40c3_a0b9_93d7668c7da1.slice/crio-818d403dc5585d9c096d8ebfcd32ba14efb254006800f36ccc2ff0207f1fa942 WatchSource:0}: Error finding container 818d403dc5585d9c096d8ebfcd32ba14efb254006800f36ccc2ff0207f1fa942: Status 404 returned error can't find the container with id 818d403dc5585d9c096d8ebfcd32ba14efb254006800f36ccc2ff0207f1fa942
Sep 16 18:46:07.271418 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:46:07.271382    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/61521525-706f-428c-80f6-2cdfcb30fdc4-metrics-tls\") pod \"dns-default-qsxrs\" (UID: \"61521525-706f-428c-80f6-2cdfcb30fdc4\") " pod="openshift-dns/dns-default-qsxrs"
Sep 16 18:46:07.271418 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:46:07.271421    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/57ce0be5-62d9-473f-a19f-4253cb333fe2-registry-tls\") pod \"image-registry-649b4fc7ff-zmk84\" (UID: \"57ce0be5-62d9-473f-a19f-4253cb333fe2\") " pod="openshift-image-registry/image-registry-649b4fc7ff-zmk84"
Sep 16 18:46:07.271621 ip-10-0-138-127 kubenswrapper[2581]: E0916 18:46:07.271520    2581 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 16 18:46:07.271621 ip-10-0-138-127 kubenswrapper[2581]: E0916 18:46:07.271587    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/61521525-706f-428c-80f6-2cdfcb30fdc4-metrics-tls podName:61521525-706f-428c-80f6-2cdfcb30fdc4 nodeName:}" failed. No retries permitted until 2026-09-16 18:46:08.271572105 +0000 UTC m=+38.090985991 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/61521525-706f-428c-80f6-2cdfcb30fdc4-metrics-tls") pod "dns-default-qsxrs" (UID: "61521525-706f-428c-80f6-2cdfcb30fdc4") : secret "dns-default-metrics-tls" not found
Sep 16 18:46:07.271621 ip-10-0-138-127 kubenswrapper[2581]: E0916 18:46:07.271525    2581 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 16 18:46:07.271756 ip-10-0-138-127 kubenswrapper[2581]: E0916 18:46:07.271631    2581 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-649b4fc7ff-zmk84: secret "image-registry-tls" not found
Sep 16 18:46:07.271756 ip-10-0-138-127 kubenswrapper[2581]: E0916 18:46:07.271683    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/57ce0be5-62d9-473f-a19f-4253cb333fe2-registry-tls podName:57ce0be5-62d9-473f-a19f-4253cb333fe2 nodeName:}" failed. No retries permitted until 2026-09-16 18:46:08.27167188 +0000 UTC m=+38.091085770 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/57ce0be5-62d9-473f-a19f-4253cb333fe2-registry-tls") pod "image-registry-649b4fc7ff-zmk84" (UID: "57ce0be5-62d9-473f-a19f-4253cb333fe2") : secret "image-registry-tls" not found
Sep 16 18:46:07.722429 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:46:07.722387    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-c78nz"
Sep 16 18:46:07.725459 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:46:07.725429    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Sep 16 18:46:08.063376 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:46:08.063339    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-z69gl" event={"ID":"60f08a67-da8a-40c3-a0b9-93d7668c7da1","Type":"ContainerStarted","Data":"818d403dc5585d9c096d8ebfcd32ba14efb254006800f36ccc2ff0207f1fa942"}
Sep 16 18:46:08.067647 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:46:08.067604    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-fzt7c" event={"ID":"5acb5489-33eb-4c23-b35d-484caffde098","Type":"ContainerStarted","Data":"921655d32f9fab7db969743500cfc23bba7d95378648cd4ab76c6c2e6d304cf4"}
Sep 16 18:46:08.098220 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:46:08.098018    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-fzt7c" podStartSLOduration=7.277561629 podStartE2EDuration="37.097993909s" totalImagesPullingTime="29.82043228s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-16 18:45:31 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:46:08.097039784 +0000 UTC m=+37.916453694" watchObservedRunningTime="2026-09-16 18:46:08.097993909 +0000 UTC m=+37.917407817"
Sep 16 18:46:08.280901 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:46:08.280863    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/61521525-706f-428c-80f6-2cdfcb30fdc4-metrics-tls\") pod \"dns-default-qsxrs\" (UID: \"61521525-706f-428c-80f6-2cdfcb30fdc4\") " pod="openshift-dns/dns-default-qsxrs"
Sep 16 18:46:08.281083 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:46:08.280909    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/57ce0be5-62d9-473f-a19f-4253cb333fe2-registry-tls\") pod \"image-registry-649b4fc7ff-zmk84\" (UID: \"57ce0be5-62d9-473f-a19f-4253cb333fe2\") " pod="openshift-image-registry/image-registry-649b4fc7ff-zmk84"
Sep 16 18:46:08.281083 ip-10-0-138-127 kubenswrapper[2581]: E0916 18:46:08.281056    2581 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 16 18:46:08.281232 ip-10-0-138-127 kubenswrapper[2581]: E0916 18:46:08.281147    2581 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 16 18:46:08.281232 ip-10-0-138-127 kubenswrapper[2581]: E0916 18:46:08.281164    2581 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-649b4fc7ff-zmk84: secret "image-registry-tls" not found
Sep 16 18:46:08.281232 ip-10-0-138-127 kubenswrapper[2581]: E0916 18:46:08.281211    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/61521525-706f-428c-80f6-2cdfcb30fdc4-metrics-tls podName:61521525-706f-428c-80f6-2cdfcb30fdc4 nodeName:}" failed. No retries permitted until 2026-09-16 18:46:10.281168369 +0000 UTC m=+40.100582258 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/61521525-706f-428c-80f6-2cdfcb30fdc4-metrics-tls") pod "dns-default-qsxrs" (UID: "61521525-706f-428c-80f6-2cdfcb30fdc4") : secret "dns-default-metrics-tls" not found
Sep 16 18:46:08.281356 ip-10-0-138-127 kubenswrapper[2581]: E0916 18:46:08.281246    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/57ce0be5-62d9-473f-a19f-4253cb333fe2-registry-tls podName:57ce0be5-62d9-473f-a19f-4253cb333fe2 nodeName:}" failed. No retries permitted until 2026-09-16 18:46:10.281232485 +0000 UTC m=+40.100646376 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/57ce0be5-62d9-473f-a19f-4253cb333fe2-registry-tls") pod "image-registry-649b4fc7ff-zmk84" (UID: "57ce0be5-62d9-473f-a19f-4253cb333fe2") : secret "image-registry-tls" not found
Sep 16 18:46:10.074689 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:46:10.074609    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-z69gl" event={"ID":"60f08a67-da8a-40c3-a0b9-93d7668c7da1","Type":"ContainerStarted","Data":"692c1777edcf7bf32604f7e88756c1023b512dd46dd8ecfb3206dafac2095e8f"}
Sep 16 18:46:10.091981 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:46:10.091930    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-z69gl" podStartSLOduration=1.5136376729999999 podStartE2EDuration="4.091912625s" totalImagesPullingTime="2.578274952s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:46:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:46:10.091300678 +0000 UTC m=+39.910714584" watchObservedRunningTime="2026-09-16 18:46:10.091912625 +0000 UTC m=+39.911326567"
Sep 16 18:46:10.298971 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:46:10.298776    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/61521525-706f-428c-80f6-2cdfcb30fdc4-metrics-tls\") pod \"dns-default-qsxrs\" (UID: \"61521525-706f-428c-80f6-2cdfcb30fdc4\") " pod="openshift-dns/dns-default-qsxrs"
Sep 16 18:46:10.298971 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:46:10.298975    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/57ce0be5-62d9-473f-a19f-4253cb333fe2-registry-tls\") pod \"image-registry-649b4fc7ff-zmk84\" (UID: \"57ce0be5-62d9-473f-a19f-4253cb333fe2\") " pod="openshift-image-registry/image-registry-649b4fc7ff-zmk84"
Sep 16 18:46:10.299255 ip-10-0-138-127 kubenswrapper[2581]: E0916 18:46:10.298929    2581 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 16 18:46:10.299255 ip-10-0-138-127 kubenswrapper[2581]: E0916 18:46:10.299055    2581 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 16 18:46:10.299255 ip-10-0-138-127 kubenswrapper[2581]: E0916 18:46:10.299066    2581 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-649b4fc7ff-zmk84: secret "image-registry-tls" not found
Sep 16 18:46:10.299255 ip-10-0-138-127 kubenswrapper[2581]: E0916 18:46:10.299110    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/61521525-706f-428c-80f6-2cdfcb30fdc4-metrics-tls podName:61521525-706f-428c-80f6-2cdfcb30fdc4 nodeName:}" failed. No retries permitted until 2026-09-16 18:46:14.299089608 +0000 UTC m=+44.118503516 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/61521525-706f-428c-80f6-2cdfcb30fdc4-metrics-tls") pod "dns-default-qsxrs" (UID: "61521525-706f-428c-80f6-2cdfcb30fdc4") : secret "dns-default-metrics-tls" not found
Sep 16 18:46:10.299255 ip-10-0-138-127 kubenswrapper[2581]: E0916 18:46:10.299129    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/57ce0be5-62d9-473f-a19f-4253cb333fe2-registry-tls podName:57ce0be5-62d9-473f-a19f-4253cb333fe2 nodeName:}" failed. No retries permitted until 2026-09-16 18:46:14.29912112 +0000 UTC m=+44.118535013 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/57ce0be5-62d9-473f-a19f-4253cb333fe2-registry-tls") pod "image-registry-649b4fc7ff-zmk84" (UID: "57ce0be5-62d9-473f-a19f-4253cb333fe2") : secret "image-registry-tls" not found
Sep 16 18:46:14.331974 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:46:14.331930    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/61521525-706f-428c-80f6-2cdfcb30fdc4-metrics-tls\") pod \"dns-default-qsxrs\" (UID: \"61521525-706f-428c-80f6-2cdfcb30fdc4\") " pod="openshift-dns/dns-default-qsxrs"
Sep 16 18:46:14.331974 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:46:14.331976    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/57ce0be5-62d9-473f-a19f-4253cb333fe2-registry-tls\") pod \"image-registry-649b4fc7ff-zmk84\" (UID: \"57ce0be5-62d9-473f-a19f-4253cb333fe2\") " pod="openshift-image-registry/image-registry-649b4fc7ff-zmk84"
Sep 16 18:46:14.332436 ip-10-0-138-127 kubenswrapper[2581]: E0916 18:46:14.332105    2581 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 16 18:46:14.332436 ip-10-0-138-127 kubenswrapper[2581]: E0916 18:46:14.332103    2581 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 16 18:46:14.332436 ip-10-0-138-127 kubenswrapper[2581]: E0916 18:46:14.332119    2581 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-649b4fc7ff-zmk84: secret "image-registry-tls" not found
Sep 16 18:46:14.332436 ip-10-0-138-127 kubenswrapper[2581]: E0916 18:46:14.332187    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/61521525-706f-428c-80f6-2cdfcb30fdc4-metrics-tls podName:61521525-706f-428c-80f6-2cdfcb30fdc4 nodeName:}" failed. No retries permitted until 2026-09-16 18:46:22.332165089 +0000 UTC m=+52.151578977 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/61521525-706f-428c-80f6-2cdfcb30fdc4-metrics-tls") pod "dns-default-qsxrs" (UID: "61521525-706f-428c-80f6-2cdfcb30fdc4") : secret "dns-default-metrics-tls" not found
Sep 16 18:46:14.332436 ip-10-0-138-127 kubenswrapper[2581]: E0916 18:46:14.332228    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/57ce0be5-62d9-473f-a19f-4253cb333fe2-registry-tls podName:57ce0be5-62d9-473f-a19f-4253cb333fe2 nodeName:}" failed. No retries permitted until 2026-09-16 18:46:22.332218651 +0000 UTC m=+52.151632539 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/57ce0be5-62d9-473f-a19f-4253cb333fe2-registry-tls") pod "image-registry-649b4fc7ff-zmk84" (UID: "57ce0be5-62d9-473f-a19f-4253cb333fe2") : secret "image-registry-tls" not found
Sep 16 18:46:22.394525 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:46:22.394467    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/61521525-706f-428c-80f6-2cdfcb30fdc4-metrics-tls\") pod \"dns-default-qsxrs\" (UID: \"61521525-706f-428c-80f6-2cdfcb30fdc4\") " pod="openshift-dns/dns-default-qsxrs"
Sep 16 18:46:22.394525 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:46:22.394530    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/57ce0be5-62d9-473f-a19f-4253cb333fe2-registry-tls\") pod \"image-registry-649b4fc7ff-zmk84\" (UID: \"57ce0be5-62d9-473f-a19f-4253cb333fe2\") " pod="openshift-image-registry/image-registry-649b4fc7ff-zmk84"
Sep 16 18:46:22.394988 ip-10-0-138-127 kubenswrapper[2581]: E0916 18:46:22.394622    2581 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 16 18:46:22.394988 ip-10-0-138-127 kubenswrapper[2581]: E0916 18:46:22.394633    2581 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 16 18:46:22.394988 ip-10-0-138-127 kubenswrapper[2581]: E0916 18:46:22.394645    2581 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-649b4fc7ff-zmk84: secret "image-registry-tls" not found
Sep 16 18:46:22.394988 ip-10-0-138-127 kubenswrapper[2581]: E0916 18:46:22.394689    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/61521525-706f-428c-80f6-2cdfcb30fdc4-metrics-tls podName:61521525-706f-428c-80f6-2cdfcb30fdc4 nodeName:}" failed. No retries permitted until 2026-09-16 18:46:38.394671006 +0000 UTC m=+68.214084893 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/61521525-706f-428c-80f6-2cdfcb30fdc4-metrics-tls") pod "dns-default-qsxrs" (UID: "61521525-706f-428c-80f6-2cdfcb30fdc4") : secret "dns-default-metrics-tls" not found
Sep 16 18:46:22.394988 ip-10-0-138-127 kubenswrapper[2581]: E0916 18:46:22.394707    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/57ce0be5-62d9-473f-a19f-4253cb333fe2-registry-tls podName:57ce0be5-62d9-473f-a19f-4253cb333fe2 nodeName:}" failed. No retries permitted until 2026-09-16 18:46:38.394694594 +0000 UTC m=+68.214108480 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/57ce0be5-62d9-473f-a19f-4253cb333fe2-registry-tls") pod "image-registry-649b4fc7ff-zmk84" (UID: "57ce0be5-62d9-473f-a19f-4253cb333fe2") : secret "image-registry-tls" not found
Sep 16 18:46:31.049833 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:46:31.049799    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-csmcl"
Sep 16 18:46:36.398483 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:46:36.398425    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/f09babc9-6295-4a0a-be9f-fa0175b0de51-original-pull-secret\") pod \"global-pull-secret-syncer-c78nz\" (UID: \"f09babc9-6295-4a0a-be9f-fa0175b0de51\") " pod="kube-system/global-pull-secret-syncer-c78nz"
Sep 16 18:46:36.402916 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:46:36.402898    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Sep 16 18:46:36.411841 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:46:36.411806    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/f09babc9-6295-4a0a-be9f-fa0175b0de51-original-pull-secret\") pod \"global-pull-secret-syncer-c78nz\" (UID: \"f09babc9-6295-4a0a-be9f-fa0175b0de51\") " pod="kube-system/global-pull-secret-syncer-c78nz"
Sep 16 18:46:36.499516 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:46:36.499460    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5vmqf\" (UniqueName: \"kubernetes.io/projected/4775e8c4-09ba-43db-98b7-073aeb452ce8-kube-api-access-5vmqf\") pod \"network-check-target-bz5jk\" (UID: \"4775e8c4-09ba-43db-98b7-073aeb452ce8\") " pod="openshift-network-diagnostics/network-check-target-bz5jk"
Sep 16 18:46:36.499516 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:46:36.499517    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/eaf5897a-76a0-4bf3-aa4e-59aa6c193ddb-metrics-certs\") pod \"network-metrics-daemon-c2kp2\" (UID: \"eaf5897a-76a0-4bf3-aa4e-59aa6c193ddb\") " pod="openshift-multus/network-metrics-daemon-c2kp2"
Sep 16 18:46:36.504321 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:46:36.504301    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Sep 16 18:46:36.504633 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:46:36.504618    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Sep 16 18:46:36.510517 ip-10-0-138-127 kubenswrapper[2581]: E0916 18:46:36.510495    2581 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
Sep 16 18:46:36.510593 ip-10-0-138-127 kubenswrapper[2581]: E0916 18:46:36.510576    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/eaf5897a-76a0-4bf3-aa4e-59aa6c193ddb-metrics-certs podName:eaf5897a-76a0-4bf3-aa4e-59aa6c193ddb nodeName:}" failed. No retries permitted until 2026-09-16 18:47:40.510552687 +0000 UTC m=+130.329966591 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/eaf5897a-76a0-4bf3-aa4e-59aa6c193ddb-metrics-certs") pod "network-metrics-daemon-c2kp2" (UID: "eaf5897a-76a0-4bf3-aa4e-59aa6c193ddb") : secret "metrics-daemon-secret" not found
Sep 16 18:46:36.513112 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:46:36.513095    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Sep 16 18:46:36.523654 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:46:36.523634    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5vmqf\" (UniqueName: \"kubernetes.io/projected/4775e8c4-09ba-43db-98b7-073aeb452ce8-kube-api-access-5vmqf\") pod \"network-check-target-bz5jk\" (UID: \"4775e8c4-09ba-43db-98b7-073aeb452ce8\") " pod="openshift-network-diagnostics/network-check-target-bz5jk"
Sep 16 18:46:36.531465 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:46:36.531446    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-c78nz"
Sep 16 18:46:36.735411 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:46:36.735331    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-4ccjj\""
Sep 16 18:46:36.743211 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:46:36.743178    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-bz5jk"
Sep 16 18:46:36.807764 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:46:36.807704    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-c78nz"]
Sep 16 18:46:36.812695 ip-10-0-138-127 kubenswrapper[2581]: W0916 18:46:36.812664    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf09babc9_6295_4a0a_be9f_fa0175b0de51.slice/crio-c3d5c62a4a0412440e4717197b58900934051eb80be1856231832a4d0815ff36 WatchSource:0}: Error finding container c3d5c62a4a0412440e4717197b58900934051eb80be1856231832a4d0815ff36: Status 404 returned error can't find the container with id c3d5c62a4a0412440e4717197b58900934051eb80be1856231832a4d0815ff36
Sep 16 18:46:37.013388 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:46:37.013316    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-bz5jk"]
Sep 16 18:46:37.016016 ip-10-0-138-127 kubenswrapper[2581]: W0916 18:46:37.015990    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4775e8c4_09ba_43db_98b7_073aeb452ce8.slice/crio-ff3ba4d164fa937406bed9595b8a8f334a50c6c12a85f5363c29ee881f28cb0b WatchSource:0}: Error finding container ff3ba4d164fa937406bed9595b8a8f334a50c6c12a85f5363c29ee881f28cb0b: Status 404 returned error can't find the container with id ff3ba4d164fa937406bed9595b8a8f334a50c6c12a85f5363c29ee881f28cb0b
Sep 16 18:46:37.141544 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:46:37.141508    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-bz5jk" event={"ID":"4775e8c4-09ba-43db-98b7-073aeb452ce8","Type":"ContainerStarted","Data":"ff3ba4d164fa937406bed9595b8a8f334a50c6c12a85f5363c29ee881f28cb0b"}
Sep 16 18:46:37.142674 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:46:37.142651    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-c78nz" event={"ID":"f09babc9-6295-4a0a-be9f-fa0175b0de51","Type":"ContainerStarted","Data":"c3d5c62a4a0412440e4717197b58900934051eb80be1856231832a4d0815ff36"}
Sep 16 18:46:38.414041 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:46:38.414000    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/61521525-706f-428c-80f6-2cdfcb30fdc4-metrics-tls\") pod \"dns-default-qsxrs\" (UID: \"61521525-706f-428c-80f6-2cdfcb30fdc4\") " pod="openshift-dns/dns-default-qsxrs"
Sep 16 18:46:38.414505 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:46:38.414048    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/57ce0be5-62d9-473f-a19f-4253cb333fe2-registry-tls\") pod \"image-registry-649b4fc7ff-zmk84\" (UID: \"57ce0be5-62d9-473f-a19f-4253cb333fe2\") " pod="openshift-image-registry/image-registry-649b4fc7ff-zmk84"
Sep 16 18:46:38.414505 ip-10-0-138-127 kubenswrapper[2581]: E0916 18:46:38.414211    2581 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 16 18:46:38.414505 ip-10-0-138-127 kubenswrapper[2581]: E0916 18:46:38.414223    2581 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 16 18:46:38.414505 ip-10-0-138-127 kubenswrapper[2581]: E0916 18:46:38.414243    2581 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-649b4fc7ff-zmk84: secret "image-registry-tls" not found
Sep 16 18:46:38.414505 ip-10-0-138-127 kubenswrapper[2581]: E0916 18:46:38.414291    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/61521525-706f-428c-80f6-2cdfcb30fdc4-metrics-tls podName:61521525-706f-428c-80f6-2cdfcb30fdc4 nodeName:}" failed. No retries permitted until 2026-09-16 18:47:10.414273215 +0000 UTC m=+100.233687104 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/61521525-706f-428c-80f6-2cdfcb30fdc4-metrics-tls") pod "dns-default-qsxrs" (UID: "61521525-706f-428c-80f6-2cdfcb30fdc4") : secret "dns-default-metrics-tls" not found
Sep 16 18:46:38.414505 ip-10-0-138-127 kubenswrapper[2581]: E0916 18:46:38.414308    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/57ce0be5-62d9-473f-a19f-4253cb333fe2-registry-tls podName:57ce0be5-62d9-473f-a19f-4253cb333fe2 nodeName:}" failed. No retries permitted until 2026-09-16 18:47:10.414302082 +0000 UTC m=+100.233715969 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/57ce0be5-62d9-473f-a19f-4253cb333fe2-registry-tls") pod "image-registry-649b4fc7ff-zmk84" (UID: "57ce0be5-62d9-473f-a19f-4253cb333fe2") : secret "image-registry-tls" not found
Sep 16 18:46:40.153330 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:46:40.153280    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-bz5jk" event={"ID":"4775e8c4-09ba-43db-98b7-073aeb452ce8","Type":"ContainerStarted","Data":"4f080a84cc119e73f843efc1946e83d088d11c32a712e3125d80e920f4974d92"}
Sep 16 18:46:40.153815 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:46:40.153391    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-bz5jk"
Sep 16 18:46:40.172809 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:46:40.172289    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-bz5jk" podStartSLOduration=66.261296112 podStartE2EDuration="1m9.17226584s" totalImagesPullingTime="2.910969728s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:45:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:46:40.171818768 +0000 UTC m=+69.991232678" watchObservedRunningTime="2026-09-16 18:46:40.17226584 +0000 UTC m=+69.991679751"
Sep 16 18:46:42.159876 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:46:42.159790    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-c78nz" event={"ID":"f09babc9-6295-4a0a-be9f-fa0175b0de51","Type":"ContainerStarted","Data":"36b2ca8ec92e90e2805ea0d669b48968f72f3a513d4daaa68588eb4cab11c72e"}
Sep 16 18:46:42.176755 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:46:42.176700    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-c78nz" podStartSLOduration=66.183194314 podStartE2EDuration="1m11.176683585s" totalImagesPullingTime="4.993489271s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-16 18:45:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:46:42.176054267 +0000 UTC m=+71.995468176" watchObservedRunningTime="2026-09-16 18:46:42.176683585 +0000 UTC m=+71.996097493"
Sep 16 18:47:10.458930 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:47:10.458878    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/61521525-706f-428c-80f6-2cdfcb30fdc4-metrics-tls\") pod \"dns-default-qsxrs\" (UID: \"61521525-706f-428c-80f6-2cdfcb30fdc4\") " pod="openshift-dns/dns-default-qsxrs"
Sep 16 18:47:10.458930 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:47:10.458931    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/57ce0be5-62d9-473f-a19f-4253cb333fe2-registry-tls\") pod \"image-registry-649b4fc7ff-zmk84\" (UID: \"57ce0be5-62d9-473f-a19f-4253cb333fe2\") " pod="openshift-image-registry/image-registry-649b4fc7ff-zmk84"
Sep 16 18:47:10.459440 ip-10-0-138-127 kubenswrapper[2581]: E0916 18:47:10.459027    2581 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 16 18:47:10.459440 ip-10-0-138-127 kubenswrapper[2581]: E0916 18:47:10.459027    2581 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 16 18:47:10.459440 ip-10-0-138-127 kubenswrapper[2581]: E0916 18:47:10.459038    2581 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-649b4fc7ff-zmk84: secret "image-registry-tls" not found
Sep 16 18:47:10.459440 ip-10-0-138-127 kubenswrapper[2581]: E0916 18:47:10.459106    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/57ce0be5-62d9-473f-a19f-4253cb333fe2-registry-tls podName:57ce0be5-62d9-473f-a19f-4253cb333fe2 nodeName:}" failed. No retries permitted until 2026-09-16 18:48:14.459091958 +0000 UTC m=+164.278505850 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/57ce0be5-62d9-473f-a19f-4253cb333fe2-registry-tls") pod "image-registry-649b4fc7ff-zmk84" (UID: "57ce0be5-62d9-473f-a19f-4253cb333fe2") : secret "image-registry-tls" not found
Sep 16 18:47:10.459440 ip-10-0-138-127 kubenswrapper[2581]: E0916 18:47:10.459117    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/61521525-706f-428c-80f6-2cdfcb30fdc4-metrics-tls podName:61521525-706f-428c-80f6-2cdfcb30fdc4 nodeName:}" failed. No retries permitted until 2026-09-16 18:48:14.459111868 +0000 UTC m=+164.278525755 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/61521525-706f-428c-80f6-2cdfcb30fdc4-metrics-tls") pod "dns-default-qsxrs" (UID: "61521525-706f-428c-80f6-2cdfcb30fdc4") : secret "dns-default-metrics-tls" not found
Sep 16 18:47:11.158952 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:47:11.158921    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-bz5jk"
Sep 16 18:47:40.577483 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:47:40.577421    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/eaf5897a-76a0-4bf3-aa4e-59aa6c193ddb-metrics-certs\") pod \"network-metrics-daemon-c2kp2\" (UID: \"eaf5897a-76a0-4bf3-aa4e-59aa6c193ddb\") " pod="openshift-multus/network-metrics-daemon-c2kp2"
Sep 16 18:47:40.577979 ip-10-0-138-127 kubenswrapper[2581]: E0916 18:47:40.577577    2581 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
Sep 16 18:47:40.577979 ip-10-0-138-127 kubenswrapper[2581]: E0916 18:47:40.577656    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/eaf5897a-76a0-4bf3-aa4e-59aa6c193ddb-metrics-certs podName:eaf5897a-76a0-4bf3-aa4e-59aa6c193ddb nodeName:}" failed. No retries permitted until 2026-09-16 18:49:42.577639213 +0000 UTC m=+252.397053100 (durationBeforeRetry 2m2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/eaf5897a-76a0-4bf3-aa4e-59aa6c193ddb-metrics-certs") pod "network-metrics-daemon-c2kp2" (UID: "eaf5897a-76a0-4bf3-aa4e-59aa6c193ddb") : secret "metrics-daemon-secret" not found
Sep 16 18:47:41.748470 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:47:41.748436    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-c2dhr"]
Sep 16 18:47:41.751526 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:47:41.751507    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-c2dhr"
Sep 16 18:47:41.759818 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:47:41.759797    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Sep 16 18:47:41.760252 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:47:41.760235    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Sep 16 18:47:41.762098 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:47:41.762084    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Sep 16 18:47:41.764879 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:47:41.764865    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-kb6xm\""
Sep 16 18:47:41.766908 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:47:41.766895    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Sep 16 18:47:41.776029 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:47:41.776007    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-c2dhr"]
Sep 16 18:47:41.885606 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:47:41.885572    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/6e068f1b-74ad-44e1-aee0-5307a9eae059-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-c2dhr\" (UID: \"6e068f1b-74ad-44e1-aee0-5307a9eae059\") " pod="openshift-insights/insights-runtime-extractor-c2dhr"
Sep 16 18:47:41.885782 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:47:41.885626    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/6e068f1b-74ad-44e1-aee0-5307a9eae059-data-volume\") pod \"insights-runtime-extractor-c2dhr\" (UID: \"6e068f1b-74ad-44e1-aee0-5307a9eae059\") " pod="openshift-insights/insights-runtime-extractor-c2dhr"
Sep 16 18:47:41.885782 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:47:41.885652    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/6e068f1b-74ad-44e1-aee0-5307a9eae059-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-c2dhr\" (UID: \"6e068f1b-74ad-44e1-aee0-5307a9eae059\") " pod="openshift-insights/insights-runtime-extractor-c2dhr"
Sep 16 18:47:41.885782 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:47:41.885715    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/6e068f1b-74ad-44e1-aee0-5307a9eae059-crio-socket\") pod \"insights-runtime-extractor-c2dhr\" (UID: \"6e068f1b-74ad-44e1-aee0-5307a9eae059\") " pod="openshift-insights/insights-runtime-extractor-c2dhr"
Sep 16 18:47:41.885782 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:47:41.885757    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-87trb\" (UniqueName: \"kubernetes.io/projected/6e068f1b-74ad-44e1-aee0-5307a9eae059-kube-api-access-87trb\") pod \"insights-runtime-extractor-c2dhr\" (UID: \"6e068f1b-74ad-44e1-aee0-5307a9eae059\") " pod="openshift-insights/insights-runtime-extractor-c2dhr"
Sep 16 18:47:41.986243 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:47:41.986173    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/6e068f1b-74ad-44e1-aee0-5307a9eae059-data-volume\") pod \"insights-runtime-extractor-c2dhr\" (UID: \"6e068f1b-74ad-44e1-aee0-5307a9eae059\") " pod="openshift-insights/insights-runtime-extractor-c2dhr"
Sep 16 18:47:41.986362 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:47:41.986255    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/6e068f1b-74ad-44e1-aee0-5307a9eae059-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-c2dhr\" (UID: \"6e068f1b-74ad-44e1-aee0-5307a9eae059\") " pod="openshift-insights/insights-runtime-extractor-c2dhr"
Sep 16 18:47:41.986362 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:47:41.986288    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/6e068f1b-74ad-44e1-aee0-5307a9eae059-crio-socket\") pod \"insights-runtime-extractor-c2dhr\" (UID: \"6e068f1b-74ad-44e1-aee0-5307a9eae059\") " pod="openshift-insights/insights-runtime-extractor-c2dhr"
Sep 16 18:47:41.986362 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:47:41.986328    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-87trb\" (UniqueName: \"kubernetes.io/projected/6e068f1b-74ad-44e1-aee0-5307a9eae059-kube-api-access-87trb\") pod \"insights-runtime-extractor-c2dhr\" (UID: \"6e068f1b-74ad-44e1-aee0-5307a9eae059\") " pod="openshift-insights/insights-runtime-extractor-c2dhr"
Sep 16 18:47:41.986502 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:47:41.986386    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/6e068f1b-74ad-44e1-aee0-5307a9eae059-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-c2dhr\" (UID: \"6e068f1b-74ad-44e1-aee0-5307a9eae059\") " pod="openshift-insights/insights-runtime-extractor-c2dhr"
Sep 16 18:47:41.986502 ip-10-0-138-127 kubenswrapper[2581]: E0916 18:47:41.986430    2581 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 16 18:47:41.986502 ip-10-0-138-127 kubenswrapper[2581]: E0916 18:47:41.986478    2581 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 16 18:47:41.986624 ip-10-0-138-127 kubenswrapper[2581]: E0916 18:47:41.986523    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6e068f1b-74ad-44e1-aee0-5307a9eae059-insights-runtime-extractor-tls podName:6e068f1b-74ad-44e1-aee0-5307a9eae059 nodeName:}" failed. No retries permitted until 2026-09-16 18:47:42.486502336 +0000 UTC m=+132.305916222 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/6e068f1b-74ad-44e1-aee0-5307a9eae059-insights-runtime-extractor-tls") pod "insights-runtime-extractor-c2dhr" (UID: "6e068f1b-74ad-44e1-aee0-5307a9eae059") : secret "insights-runtime-extractor-tls" not found
Sep 16 18:47:41.986624 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:47:41.986433    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/6e068f1b-74ad-44e1-aee0-5307a9eae059-crio-socket\") pod \"insights-runtime-extractor-c2dhr\" (UID: \"6e068f1b-74ad-44e1-aee0-5307a9eae059\") " pod="openshift-insights/insights-runtime-extractor-c2dhr"
Sep 16 18:47:41.986624 ip-10-0-138-127 kubenswrapper[2581]: E0916 18:47:41.986548    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/6e068f1b-74ad-44e1-aee0-5307a9eae059-kube-rbac-proxy-cm podName:6e068f1b-74ad-44e1-aee0-5307a9eae059 nodeName:}" failed. No retries permitted until 2026-09-16 18:47:42.486531941 +0000 UTC m=+132.305945848 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/6e068f1b-74ad-44e1-aee0-5307a9eae059-kube-rbac-proxy-cm") pod "insights-runtime-extractor-c2dhr" (UID: "6e068f1b-74ad-44e1-aee0-5307a9eae059") : configmap "kube-rbac-proxy" not found
Sep 16 18:47:41.986737 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:47:41.986643    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/6e068f1b-74ad-44e1-aee0-5307a9eae059-data-volume\") pod \"insights-runtime-extractor-c2dhr\" (UID: \"6e068f1b-74ad-44e1-aee0-5307a9eae059\") " pod="openshift-insights/insights-runtime-extractor-c2dhr"
Sep 16 18:47:41.999227 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:47:41.999139    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-87trb\" (UniqueName: \"kubernetes.io/projected/6e068f1b-74ad-44e1-aee0-5307a9eae059-kube-api-access-87trb\") pod \"insights-runtime-extractor-c2dhr\" (UID: \"6e068f1b-74ad-44e1-aee0-5307a9eae059\") " pod="openshift-insights/insights-runtime-extractor-c2dhr"
Sep 16 18:47:42.490272 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:47:42.490230    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/6e068f1b-74ad-44e1-aee0-5307a9eae059-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-c2dhr\" (UID: \"6e068f1b-74ad-44e1-aee0-5307a9eae059\") " pod="openshift-insights/insights-runtime-extractor-c2dhr"
Sep 16 18:47:42.490490 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:47:42.490308    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/6e068f1b-74ad-44e1-aee0-5307a9eae059-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-c2dhr\" (UID: \"6e068f1b-74ad-44e1-aee0-5307a9eae059\") " pod="openshift-insights/insights-runtime-extractor-c2dhr"
Sep 16 18:47:42.490490 ip-10-0-138-127 kubenswrapper[2581]: E0916 18:47:42.490382    2581 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 16 18:47:42.490490 ip-10-0-138-127 kubenswrapper[2581]: E0916 18:47:42.490393    2581 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 16 18:47:42.490490 ip-10-0-138-127 kubenswrapper[2581]: E0916 18:47:42.490450    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/6e068f1b-74ad-44e1-aee0-5307a9eae059-kube-rbac-proxy-cm podName:6e068f1b-74ad-44e1-aee0-5307a9eae059 nodeName:}" failed. No retries permitted until 2026-09-16 18:47:43.490433675 +0000 UTC m=+133.309847569 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/6e068f1b-74ad-44e1-aee0-5307a9eae059-kube-rbac-proxy-cm") pod "insights-runtime-extractor-c2dhr" (UID: "6e068f1b-74ad-44e1-aee0-5307a9eae059") : configmap "kube-rbac-proxy" not found
Sep 16 18:47:42.490490 ip-10-0-138-127 kubenswrapper[2581]: E0916 18:47:42.490474    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6e068f1b-74ad-44e1-aee0-5307a9eae059-insights-runtime-extractor-tls podName:6e068f1b-74ad-44e1-aee0-5307a9eae059 nodeName:}" failed. No retries permitted until 2026-09-16 18:47:43.490456847 +0000 UTC m=+133.309870749 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/6e068f1b-74ad-44e1-aee0-5307a9eae059-insights-runtime-extractor-tls") pod "insights-runtime-extractor-c2dhr" (UID: "6e068f1b-74ad-44e1-aee0-5307a9eae059") : secret "insights-runtime-extractor-tls" not found
Sep 16 18:47:43.414150 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:47:43.414123    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-lrclz_eace7050-d66d-4057-9041-62afa5df8ae7/dns-node-resolver/0.log"
Sep 16 18:47:43.500814 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:47:43.500786    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/6e068f1b-74ad-44e1-aee0-5307a9eae059-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-c2dhr\" (UID: \"6e068f1b-74ad-44e1-aee0-5307a9eae059\") " pod="openshift-insights/insights-runtime-extractor-c2dhr"
Sep 16 18:47:43.500981 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:47:43.500835    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/6e068f1b-74ad-44e1-aee0-5307a9eae059-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-c2dhr\" (UID: \"6e068f1b-74ad-44e1-aee0-5307a9eae059\") " pod="openshift-insights/insights-runtime-extractor-c2dhr"
Sep 16 18:47:43.500981 ip-10-0-138-127 kubenswrapper[2581]: E0916 18:47:43.500936    2581 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 16 18:47:43.500981 ip-10-0-138-127 kubenswrapper[2581]: E0916 18:47:43.500948    2581 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 16 18:47:43.501131 ip-10-0-138-127 kubenswrapper[2581]: E0916 18:47:43.501011    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6e068f1b-74ad-44e1-aee0-5307a9eae059-insights-runtime-extractor-tls podName:6e068f1b-74ad-44e1-aee0-5307a9eae059 nodeName:}" failed. No retries permitted until 2026-09-16 18:47:45.500997264 +0000 UTC m=+135.320411150 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/6e068f1b-74ad-44e1-aee0-5307a9eae059-insights-runtime-extractor-tls") pod "insights-runtime-extractor-c2dhr" (UID: "6e068f1b-74ad-44e1-aee0-5307a9eae059") : secret "insights-runtime-extractor-tls" not found
Sep 16 18:47:43.501131 ip-10-0-138-127 kubenswrapper[2581]: E0916 18:47:43.501023    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/6e068f1b-74ad-44e1-aee0-5307a9eae059-kube-rbac-proxy-cm podName:6e068f1b-74ad-44e1-aee0-5307a9eae059 nodeName:}" failed. No retries permitted until 2026-09-16 18:47:45.501016651 +0000 UTC m=+135.320430538 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/6e068f1b-74ad-44e1-aee0-5307a9eae059-kube-rbac-proxy-cm") pod "insights-runtime-extractor-c2dhr" (UID: "6e068f1b-74ad-44e1-aee0-5307a9eae059") : configmap "kube-rbac-proxy" not found
Sep 16 18:47:44.614219 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:47:44.614173    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-mvhf8_99b45db8-0fbb-445b-be31-49fba3f220ec/node-ca/0.log"
Sep 16 18:47:45.413103 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:47:45.413077    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-z69gl_60f08a67-da8a-40c3-a0b9-93d7668c7da1/serve-healthcheck-canary/0.log"
Sep 16 18:47:45.514972 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:47:45.514938    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/6e068f1b-74ad-44e1-aee0-5307a9eae059-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-c2dhr\" (UID: \"6e068f1b-74ad-44e1-aee0-5307a9eae059\") " pod="openshift-insights/insights-runtime-extractor-c2dhr"
Sep 16 18:47:45.515099 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:47:45.515009    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/6e068f1b-74ad-44e1-aee0-5307a9eae059-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-c2dhr\" (UID: \"6e068f1b-74ad-44e1-aee0-5307a9eae059\") " pod="openshift-insights/insights-runtime-extractor-c2dhr"
Sep 16 18:47:45.515160 ip-10-0-138-127 kubenswrapper[2581]: E0916 18:47:45.515106    2581 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 16 18:47:45.515160 ip-10-0-138-127 kubenswrapper[2581]: E0916 18:47:45.515118    2581 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 16 18:47:45.515284 ip-10-0-138-127 kubenswrapper[2581]: E0916 18:47:45.515165    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/6e068f1b-74ad-44e1-aee0-5307a9eae059-kube-rbac-proxy-cm podName:6e068f1b-74ad-44e1-aee0-5307a9eae059 nodeName:}" failed. No retries permitted until 2026-09-16 18:47:49.515150406 +0000 UTC m=+139.334564313 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/6e068f1b-74ad-44e1-aee0-5307a9eae059-kube-rbac-proxy-cm") pod "insights-runtime-extractor-c2dhr" (UID: "6e068f1b-74ad-44e1-aee0-5307a9eae059") : configmap "kube-rbac-proxy" not found
Sep 16 18:47:45.515284 ip-10-0-138-127 kubenswrapper[2581]: E0916 18:47:45.515216    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6e068f1b-74ad-44e1-aee0-5307a9eae059-insights-runtime-extractor-tls podName:6e068f1b-74ad-44e1-aee0-5307a9eae059 nodeName:}" failed. No retries permitted until 2026-09-16 18:47:49.515172139 +0000 UTC m=+139.334586043 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/6e068f1b-74ad-44e1-aee0-5307a9eae059-insights-runtime-extractor-tls") pod "insights-runtime-extractor-c2dhr" (UID: "6e068f1b-74ad-44e1-aee0-5307a9eae059") : secret "insights-runtime-extractor-tls" not found
Sep 16 18:47:49.546405 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:47:49.546360    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/6e068f1b-74ad-44e1-aee0-5307a9eae059-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-c2dhr\" (UID: \"6e068f1b-74ad-44e1-aee0-5307a9eae059\") " pod="openshift-insights/insights-runtime-extractor-c2dhr"
Sep 16 18:47:49.546841 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:47:49.546441    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/6e068f1b-74ad-44e1-aee0-5307a9eae059-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-c2dhr\" (UID: \"6e068f1b-74ad-44e1-aee0-5307a9eae059\") " pod="openshift-insights/insights-runtime-extractor-c2dhr"
Sep 16 18:47:49.546841 ip-10-0-138-127 kubenswrapper[2581]: E0916 18:47:49.546511    2581 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 16 18:47:49.546841 ip-10-0-138-127 kubenswrapper[2581]: E0916 18:47:49.546511    2581 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 16 18:47:49.546841 ip-10-0-138-127 kubenswrapper[2581]: E0916 18:47:49.546574    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/6e068f1b-74ad-44e1-aee0-5307a9eae059-kube-rbac-proxy-cm podName:6e068f1b-74ad-44e1-aee0-5307a9eae059 nodeName:}" failed. No retries permitted until 2026-09-16 18:47:57.546560197 +0000 UTC m=+147.365974087 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/6e068f1b-74ad-44e1-aee0-5307a9eae059-kube-rbac-proxy-cm") pod "insights-runtime-extractor-c2dhr" (UID: "6e068f1b-74ad-44e1-aee0-5307a9eae059") : configmap "kube-rbac-proxy" not found
Sep 16 18:47:49.546841 ip-10-0-138-127 kubenswrapper[2581]: E0916 18:47:49.546587    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6e068f1b-74ad-44e1-aee0-5307a9eae059-insights-runtime-extractor-tls podName:6e068f1b-74ad-44e1-aee0-5307a9eae059 nodeName:}" failed. No retries permitted until 2026-09-16 18:47:57.546581946 +0000 UTC m=+147.365995833 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/6e068f1b-74ad-44e1-aee0-5307a9eae059-insights-runtime-extractor-tls") pod "insights-runtime-extractor-c2dhr" (UID: "6e068f1b-74ad-44e1-aee0-5307a9eae059") : secret "insights-runtime-extractor-tls" not found
Sep 16 18:47:50.985808 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:47:50.985773    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-dd6d5"]
Sep 16 18:47:50.988477 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:47:50.988461    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-dd6d5"
Sep 16 18:47:50.992162 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:47:50.992140    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"serving-cert\""
Sep 16 18:47:50.992330 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:47:50.992146    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"openshift-service-ca.crt\""
Sep 16 18:47:50.992330 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:47:50.992269    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"config\""
Sep 16 18:47:50.992895 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:47:50.992876    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-storage-version-migrator-operator-dockercfg-ps5xc\""
Sep 16 18:47:50.992943 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:47:50.992893    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-root-ca.crt\""
Sep 16 18:47:51.001808 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:47:51.001787    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-dd6d5"]
Sep 16 18:47:51.157802 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:47:51.157769    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/f3b6e193-3e4d-4dea-9909-b9eb2040ce3d-serving-cert\") pod \"kube-storage-version-migrator-operator-6c8f84f897-dd6d5\" (UID: \"f3b6e193-3e4d-4dea-9909-b9eb2040ce3d\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-dd6d5"
Sep 16 18:47:51.157988 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:47:51.157854    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/f3b6e193-3e4d-4dea-9909-b9eb2040ce3d-config\") pod \"kube-storage-version-migrator-operator-6c8f84f897-dd6d5\" (UID: \"f3b6e193-3e4d-4dea-9909-b9eb2040ce3d\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-dd6d5"
Sep 16 18:47:51.157988 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:47:51.157896    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9sz4d\" (UniqueName: \"kubernetes.io/projected/f3b6e193-3e4d-4dea-9909-b9eb2040ce3d-kube-api-access-9sz4d\") pod \"kube-storage-version-migrator-operator-6c8f84f897-dd6d5\" (UID: \"f3b6e193-3e4d-4dea-9909-b9eb2040ce3d\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-dd6d5"
Sep 16 18:47:51.258789 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:47:51.258715    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/f3b6e193-3e4d-4dea-9909-b9eb2040ce3d-serving-cert\") pod \"kube-storage-version-migrator-operator-6c8f84f897-dd6d5\" (UID: \"f3b6e193-3e4d-4dea-9909-b9eb2040ce3d\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-dd6d5"
Sep 16 18:47:51.258789 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:47:51.258771    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/f3b6e193-3e4d-4dea-9909-b9eb2040ce3d-config\") pod \"kube-storage-version-migrator-operator-6c8f84f897-dd6d5\" (UID: \"f3b6e193-3e4d-4dea-9909-b9eb2040ce3d\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-dd6d5"
Sep 16 18:47:51.258956 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:47:51.258801    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9sz4d\" (UniqueName: \"kubernetes.io/projected/f3b6e193-3e4d-4dea-9909-b9eb2040ce3d-kube-api-access-9sz4d\") pod \"kube-storage-version-migrator-operator-6c8f84f897-dd6d5\" (UID: \"f3b6e193-3e4d-4dea-9909-b9eb2040ce3d\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-dd6d5"
Sep 16 18:47:51.259388 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:47:51.259366    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/f3b6e193-3e4d-4dea-9909-b9eb2040ce3d-config\") pod \"kube-storage-version-migrator-operator-6c8f84f897-dd6d5\" (UID: \"f3b6e193-3e4d-4dea-9909-b9eb2040ce3d\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-dd6d5"
Sep 16 18:47:51.261312 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:47:51.261291    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/f3b6e193-3e4d-4dea-9909-b9eb2040ce3d-serving-cert\") pod \"kube-storage-version-migrator-operator-6c8f84f897-dd6d5\" (UID: \"f3b6e193-3e4d-4dea-9909-b9eb2040ce3d\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-dd6d5"
Sep 16 18:47:51.268427 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:47:51.268406    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9sz4d\" (UniqueName: \"kubernetes.io/projected/f3b6e193-3e4d-4dea-9909-b9eb2040ce3d-kube-api-access-9sz4d\") pod \"kube-storage-version-migrator-operator-6c8f84f897-dd6d5\" (UID: \"f3b6e193-3e4d-4dea-9909-b9eb2040ce3d\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-dd6d5"
Sep 16 18:47:51.297329 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:47:51.297278    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-dd6d5"
Sep 16 18:47:51.572180 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:47:51.572151    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-dd6d5"]
Sep 16 18:47:51.575639 ip-10-0-138-127 kubenswrapper[2581]: W0916 18:47:51.575608    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf3b6e193_3e4d_4dea_9909_b9eb2040ce3d.slice/crio-680baa73f5c640966f0e2fa7478eb9977d0dba305b494c98b066bd8fe428841f WatchSource:0}: Error finding container 680baa73f5c640966f0e2fa7478eb9977d0dba305b494c98b066bd8fe428841f: Status 404 returned error can't find the container with id 680baa73f5c640966f0e2fa7478eb9977d0dba305b494c98b066bd8fe428841f
Sep 16 18:47:52.332259 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:47:52.332211    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-dd6d5" event={"ID":"f3b6e193-3e4d-4dea-9909-b9eb2040ce3d","Type":"ContainerStarted","Data":"680baa73f5c640966f0e2fa7478eb9977d0dba305b494c98b066bd8fe428841f"}
Sep 16 18:47:54.339751 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:47:54.339710    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-dd6d5" event={"ID":"f3b6e193-3e4d-4dea-9909-b9eb2040ce3d","Type":"ContainerStarted","Data":"4d8f4dcd8a0abb2aa1de72362dab755aa5cdf005ccc972c9377eb483230bd4fc"}
Sep 16 18:47:57.611384 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:47:57.611345    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/6e068f1b-74ad-44e1-aee0-5307a9eae059-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-c2dhr\" (UID: \"6e068f1b-74ad-44e1-aee0-5307a9eae059\") " pod="openshift-insights/insights-runtime-extractor-c2dhr"
Sep 16 18:47:57.611772 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:47:57.611400    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/6e068f1b-74ad-44e1-aee0-5307a9eae059-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-c2dhr\" (UID: \"6e068f1b-74ad-44e1-aee0-5307a9eae059\") " pod="openshift-insights/insights-runtime-extractor-c2dhr"
Sep 16 18:47:57.611772 ip-10-0-138-127 kubenswrapper[2581]: E0916 18:47:57.611503    2581 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 16 18:47:57.611772 ip-10-0-138-127 kubenswrapper[2581]: E0916 18:47:57.611562    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6e068f1b-74ad-44e1-aee0-5307a9eae059-insights-runtime-extractor-tls podName:6e068f1b-74ad-44e1-aee0-5307a9eae059 nodeName:}" failed. No retries permitted until 2026-09-16 18:48:13.611547683 +0000 UTC m=+163.430961573 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/6e068f1b-74ad-44e1-aee0-5307a9eae059-insights-runtime-extractor-tls") pod "insights-runtime-extractor-c2dhr" (UID: "6e068f1b-74ad-44e1-aee0-5307a9eae059") : secret "insights-runtime-extractor-tls" not found
Sep 16 18:47:57.611891 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:47:57.611843    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/6e068f1b-74ad-44e1-aee0-5307a9eae059-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-c2dhr\" (UID: \"6e068f1b-74ad-44e1-aee0-5307a9eae059\") " pod="openshift-insights/insights-runtime-extractor-c2dhr"
Sep 16 18:48:09.536403 ip-10-0-138-127 kubenswrapper[2581]: E0916 18:48:09.536353    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[registry-tls], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-image-registry/image-registry-649b4fc7ff-zmk84" podUID="57ce0be5-62d9-473f-a19f-4253cb333fe2"
Sep 16 18:48:09.541469 ip-10-0-138-127 kubenswrapper[2581]: E0916 18:48:09.541439    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[metrics-tls], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-dns/dns-default-qsxrs" podUID="61521525-706f-428c-80f6-2cdfcb30fdc4"
Sep 16 18:48:09.737087 ip-10-0-138-127 kubenswrapper[2581]: E0916 18:48:09.737026    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[metrics-certs], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-multus/network-metrics-daemon-c2kp2" podUID="eaf5897a-76a0-4bf3-aa4e-59aa6c193ddb"
Sep 16 18:48:10.377810 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:48:10.377775    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-649b4fc7ff-zmk84"
Sep 16 18:48:10.377810 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:48:10.377802    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-qsxrs"
Sep 16 18:48:13.634961 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:48:13.634911    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/6e068f1b-74ad-44e1-aee0-5307a9eae059-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-c2dhr\" (UID: \"6e068f1b-74ad-44e1-aee0-5307a9eae059\") " pod="openshift-insights/insights-runtime-extractor-c2dhr"
Sep 16 18:48:13.637439 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:48:13.637407    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/6e068f1b-74ad-44e1-aee0-5307a9eae059-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-c2dhr\" (UID: \"6e068f1b-74ad-44e1-aee0-5307a9eae059\") " pod="openshift-insights/insights-runtime-extractor-c2dhr"
Sep 16 18:48:13.859837 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:48:13.859774    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-c2dhr"
Sep 16 18:48:14.139021 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:48:14.138960    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-dd6d5" podStartSLOduration=22.170068276 podStartE2EDuration="24.138942509s" totalImagesPullingTime="1.968874233s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:47:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:47:54.36665994 +0000 UTC m=+144.186073848" watchObservedRunningTime="2026-09-16 18:48:14.138942509 +0000 UTC m=+163.958356441"
Sep 16 18:48:14.139659 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:48:14.139633    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-c2dhr"]
Sep 16 18:48:14.143228 ip-10-0-138-127 kubenswrapper[2581]: W0916 18:48:14.143189    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6e068f1b_74ad_44e1_aee0_5307a9eae059.slice/crio-209862a4b3947daf9137d5296fc50ba98a0b19c3de912b902d9a3c541f0538b0 WatchSource:0}: Error finding container 209862a4b3947daf9137d5296fc50ba98a0b19c3de912b902d9a3c541f0538b0: Status 404 returned error can't find the container with id 209862a4b3947daf9137d5296fc50ba98a0b19c3de912b902d9a3c541f0538b0
Sep 16 18:48:14.389951 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:48:14.389916    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-c2dhr" event={"ID":"6e068f1b-74ad-44e1-aee0-5307a9eae059","Type":"ContainerStarted","Data":"37f4e28b8bcd296c03e3b9cfc3e8b39d6b1fbb671862169b7f4b2834d6f67935"}
Sep 16 18:48:14.390111 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:48:14.389974    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-c2dhr" event={"ID":"6e068f1b-74ad-44e1-aee0-5307a9eae059","Type":"ContainerStarted","Data":"209862a4b3947daf9137d5296fc50ba98a0b19c3de912b902d9a3c541f0538b0"}
Sep 16 18:48:14.542880 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:48:14.542850    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/61521525-706f-428c-80f6-2cdfcb30fdc4-metrics-tls\") pod \"dns-default-qsxrs\" (UID: \"61521525-706f-428c-80f6-2cdfcb30fdc4\") " pod="openshift-dns/dns-default-qsxrs"
Sep 16 18:48:14.543056 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:48:14.542893    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/57ce0be5-62d9-473f-a19f-4253cb333fe2-registry-tls\") pod \"image-registry-649b4fc7ff-zmk84\" (UID: \"57ce0be5-62d9-473f-a19f-4253cb333fe2\") " pod="openshift-image-registry/image-registry-649b4fc7ff-zmk84"
Sep 16 18:48:14.545468 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:48:14.545442    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/61521525-706f-428c-80f6-2cdfcb30fdc4-metrics-tls\") pod \"dns-default-qsxrs\" (UID: \"61521525-706f-428c-80f6-2cdfcb30fdc4\") " pod="openshift-dns/dns-default-qsxrs"
Sep 16 18:48:14.545468 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:48:14.545453    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/57ce0be5-62d9-473f-a19f-4253cb333fe2-registry-tls\") pod \"image-registry-649b4fc7ff-zmk84\" (UID: \"57ce0be5-62d9-473f-a19f-4253cb333fe2\") " pod="openshift-image-registry/image-registry-649b4fc7ff-zmk84"
Sep 16 18:48:14.581416 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:48:14.581382    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-nqnnl\""
Sep 16 18:48:14.581416 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:48:14.581382    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-5jk86\""
Sep 16 18:48:14.589449 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:48:14.589420    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-qsxrs"
Sep 16 18:48:14.589567 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:48:14.589522    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-649b4fc7ff-zmk84"
Sep 16 18:48:14.915923 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:48:14.915894    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-qsxrs"]
Sep 16 18:48:14.919047 ip-10-0-138-127 kubenswrapper[2581]: W0916 18:48:14.919019    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod61521525_706f_428c_80f6_2cdfcb30fdc4.slice/crio-2ec7d8715c74448d0c1b12adfb6039fbf4df7bbaa96989513ce468ad2a4b03aa WatchSource:0}: Error finding container 2ec7d8715c74448d0c1b12adfb6039fbf4df7bbaa96989513ce468ad2a4b03aa: Status 404 returned error can't find the container with id 2ec7d8715c74448d0c1b12adfb6039fbf4df7bbaa96989513ce468ad2a4b03aa
Sep 16 18:48:14.926646 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:48:14.926624    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-649b4fc7ff-zmk84"]
Sep 16 18:48:14.927901 ip-10-0-138-127 kubenswrapper[2581]: W0916 18:48:14.927881    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod57ce0be5_62d9_473f_a19f_4253cb333fe2.slice/crio-ad7297e7b4e6c92640415cf383179c4f21686b987ff0d1208aed684a8a11579c WatchSource:0}: Error finding container ad7297e7b4e6c92640415cf383179c4f21686b987ff0d1208aed684a8a11579c: Status 404 returned error can't find the container with id ad7297e7b4e6c92640415cf383179c4f21686b987ff0d1208aed684a8a11579c
Sep 16 18:48:15.396628 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:48:15.396587    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-649b4fc7ff-zmk84" event={"ID":"57ce0be5-62d9-473f-a19f-4253cb333fe2","Type":"ContainerStarted","Data":"4bb6224a269aee88f1547d6fce9f13932ef1f10341445cc66c6c87ee3eceec6c"}
Sep 16 18:48:15.396831 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:48:15.396693    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-649b4fc7ff-zmk84" event={"ID":"57ce0be5-62d9-473f-a19f-4253cb333fe2","Type":"ContainerStarted","Data":"ad7297e7b4e6c92640415cf383179c4f21686b987ff0d1208aed684a8a11579c"}
Sep 16 18:48:15.396831 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:48:15.396756    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-649b4fc7ff-zmk84"
Sep 16 18:48:15.398974 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:48:15.398942    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-c2dhr" event={"ID":"6e068f1b-74ad-44e1-aee0-5307a9eae059","Type":"ContainerStarted","Data":"d1e018c2aa9d3a17a012020b9b73558932b23c8979b710fce1923c45905e9d23"}
Sep 16 18:48:15.400365 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:48:15.400336    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-qsxrs" event={"ID":"61521525-706f-428c-80f6-2cdfcb30fdc4","Type":"ContainerStarted","Data":"2ec7d8715c74448d0c1b12adfb6039fbf4df7bbaa96989513ce468ad2a4b03aa"}
Sep 16 18:48:15.684338 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:48:15.684227    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/image-registry-649b4fc7ff-zmk84" podStartSLOduration=164.684189132 podStartE2EDuration="2m44.684189132s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:45:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:48:15.68401919 +0000 UTC m=+165.503433100" watchObservedRunningTime="2026-09-16 18:48:15.684189132 +0000 UTC m=+165.503603039"
Sep 16 18:48:15.816279 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:48:15.816227    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-c8fd974bd-zcgx2"]
Sep 16 18:48:15.819921 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:48:15.819889    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-c8fd974bd-zcgx2"
Sep 16 18:48:15.821953 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:48:15.821920    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"open-cluster-management-image-pull-credentials\""
Sep 16 18:48:15.822086 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:48:15.822014    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"kube-root-ca.crt\""
Sep 16 18:48:15.822892 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:48:15.822870    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"openshift-service-ca.crt\""
Sep 16 18:48:15.826514 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:48:15.826496    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"work-manager-hub-kubeconfig\""
Sep 16 18:48:15.838666 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:48:15.838639    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-c8fd974bd-zcgx2"]
Sep 16 18:48:15.889616 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:48:15.889577    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator/migrator-8b4d697c5-7s7hs"]
Sep 16 18:48:15.893089 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:48:15.893034    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-8b4d697c5-7s7hs"
Sep 16 18:48:15.895212 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:48:15.895170    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-storage-version-migrator-sa-dockercfg-hmmvp\""
Sep 16 18:48:15.895739 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:48:15.895718    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"openshift-service-ca.crt\""
Sep 16 18:48:15.895840 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:48:15.895831    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-root-ca.crt\""
Sep 16 18:48:15.910967 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:48:15.910938    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-8b4d697c5-7s7hs"]
Sep 16 18:48:15.953637 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:48:15.953554    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/0f94c58b-0100-4cf7-bb65-912e6edc15c4-tmp\") pod \"klusterlet-addon-workmgr-c8fd974bd-zcgx2\" (UID: \"0f94c58b-0100-4cf7-bb65-912e6edc15c4\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-c8fd974bd-zcgx2"
Sep 16 18:48:15.953637 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:48:15.953601    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tmm4r\" (UniqueName: \"kubernetes.io/projected/39b80a07-c65f-4e50-88e9-a5269e8c6d66-kube-api-access-tmm4r\") pod \"migrator-8b4d697c5-7s7hs\" (UID: \"39b80a07-c65f-4e50-88e9-a5269e8c6d66\") " pod="openshift-kube-storage-version-migrator/migrator-8b4d697c5-7s7hs"
Sep 16 18:48:15.953637 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:48:15.953628    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/0f94c58b-0100-4cf7-bb65-912e6edc15c4-klusterlet-config\") pod \"klusterlet-addon-workmgr-c8fd974bd-zcgx2\" (UID: \"0f94c58b-0100-4cf7-bb65-912e6edc15c4\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-c8fd974bd-zcgx2"
Sep 16 18:48:15.954169 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:48:15.953649    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zbtx8\" (UniqueName: \"kubernetes.io/projected/0f94c58b-0100-4cf7-bb65-912e6edc15c4-kube-api-access-zbtx8\") pod \"klusterlet-addon-workmgr-c8fd974bd-zcgx2\" (UID: \"0f94c58b-0100-4cf7-bb65-912e6edc15c4\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-c8fd974bd-zcgx2"
Sep 16 18:48:15.981925 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:48:15.981893    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/downloads-76f99584f4-67c8w"]
Sep 16 18:48:15.984928 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:48:15.984908    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-76f99584f4-67c8w"
Sep 16 18:48:15.995514 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:48:15.995482    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"default-dockercfg-l2hmb\""
Sep 16 18:48:15.995691 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:48:15.995665    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\""
Sep 16 18:48:15.995835 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:48:15.995758    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\""
Sep 16 18:48:16.013770 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:48:16.013738    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-76f99584f4-67c8w"]
Sep 16 18:48:16.054546 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:48:16.054480    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2fchh\" (UniqueName: \"kubernetes.io/projected/ddcab39a-92a7-4106-927a-8160c999373c-kube-api-access-2fchh\") pod \"downloads-76f99584f4-67c8w\" (UID: \"ddcab39a-92a7-4106-927a-8160c999373c\") " pod="openshift-console/downloads-76f99584f4-67c8w"
Sep 16 18:48:16.054740 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:48:16.054589    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/0f94c58b-0100-4cf7-bb65-912e6edc15c4-tmp\") pod \"klusterlet-addon-workmgr-c8fd974bd-zcgx2\" (UID: \"0f94c58b-0100-4cf7-bb65-912e6edc15c4\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-c8fd974bd-zcgx2"
Sep 16 18:48:16.054740 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:48:16.054654    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tmm4r\" (UniqueName: \"kubernetes.io/projected/39b80a07-c65f-4e50-88e9-a5269e8c6d66-kube-api-access-tmm4r\") pod \"migrator-8b4d697c5-7s7hs\" (UID: \"39b80a07-c65f-4e50-88e9-a5269e8c6d66\") " pod="openshift-kube-storage-version-migrator/migrator-8b4d697c5-7s7hs"
Sep 16 18:48:16.054740 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:48:16.054682    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/0f94c58b-0100-4cf7-bb65-912e6edc15c4-klusterlet-config\") pod \"klusterlet-addon-workmgr-c8fd974bd-zcgx2\" (UID: \"0f94c58b-0100-4cf7-bb65-912e6edc15c4\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-c8fd974bd-zcgx2"
Sep 16 18:48:16.054740 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:48:16.054704    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zbtx8\" (UniqueName: \"kubernetes.io/projected/0f94c58b-0100-4cf7-bb65-912e6edc15c4-kube-api-access-zbtx8\") pod \"klusterlet-addon-workmgr-c8fd974bd-zcgx2\" (UID: \"0f94c58b-0100-4cf7-bb65-912e6edc15c4\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-c8fd974bd-zcgx2"
Sep 16 18:48:16.055068 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:48:16.055022    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/0f94c58b-0100-4cf7-bb65-912e6edc15c4-tmp\") pod \"klusterlet-addon-workmgr-c8fd974bd-zcgx2\" (UID: \"0f94c58b-0100-4cf7-bb65-912e6edc15c4\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-c8fd974bd-zcgx2"
Sep 16 18:48:16.057940 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:48:16.057919    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/0f94c58b-0100-4cf7-bb65-912e6edc15c4-klusterlet-config\") pod \"klusterlet-addon-workmgr-c8fd974bd-zcgx2\" (UID: \"0f94c58b-0100-4cf7-bb65-912e6edc15c4\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-c8fd974bd-zcgx2"
Sep 16 18:48:16.073587 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:48:16.073562    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tmm4r\" (UniqueName: \"kubernetes.io/projected/39b80a07-c65f-4e50-88e9-a5269e8c6d66-kube-api-access-tmm4r\") pod \"migrator-8b4d697c5-7s7hs\" (UID: \"39b80a07-c65f-4e50-88e9-a5269e8c6d66\") " pod="openshift-kube-storage-version-migrator/migrator-8b4d697c5-7s7hs"
Sep 16 18:48:16.076412 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:48:16.076384    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zbtx8\" (UniqueName: \"kubernetes.io/projected/0f94c58b-0100-4cf7-bb65-912e6edc15c4-kube-api-access-zbtx8\") pod \"klusterlet-addon-workmgr-c8fd974bd-zcgx2\" (UID: \"0f94c58b-0100-4cf7-bb65-912e6edc15c4\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-c8fd974bd-zcgx2"
Sep 16 18:48:16.132124 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:48:16.132078    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-c8fd974bd-zcgx2"
Sep 16 18:48:16.155941 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:48:16.155908    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2fchh\" (UniqueName: \"kubernetes.io/projected/ddcab39a-92a7-4106-927a-8160c999373c-kube-api-access-2fchh\") pod \"downloads-76f99584f4-67c8w\" (UID: \"ddcab39a-92a7-4106-927a-8160c999373c\") " pod="openshift-console/downloads-76f99584f4-67c8w"
Sep 16 18:48:16.166385 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:48:16.166360    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2fchh\" (UniqueName: \"kubernetes.io/projected/ddcab39a-92a7-4106-927a-8160c999373c-kube-api-access-2fchh\") pod \"downloads-76f99584f4-67c8w\" (UID: \"ddcab39a-92a7-4106-927a-8160c999373c\") " pod="openshift-console/downloads-76f99584f4-67c8w"
Sep 16 18:48:16.205658 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:48:16.205581    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-8b4d697c5-7s7hs"
Sep 16 18:48:16.295420 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:48:16.295386    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-76f99584f4-67c8w"
Sep 16 18:48:16.974493 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:48:16.974416    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-76f99584f4-67c8w"]
Sep 16 18:48:16.977771 ip-10-0-138-127 kubenswrapper[2581]: W0916 18:48:16.977739    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podddcab39a_92a7_4106_927a_8160c999373c.slice/crio-0f88ec79302971a514b7332757e594d4cc4cb536c97edb25d905a2f3f3990501 WatchSource:0}: Error finding container 0f88ec79302971a514b7332757e594d4cc4cb536c97edb25d905a2f3f3990501: Status 404 returned error can't find the container with id 0f88ec79302971a514b7332757e594d4cc4cb536c97edb25d905a2f3f3990501
Sep 16 18:48:16.985497 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:48:16.984290    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-c8fd974bd-zcgx2"]
Sep 16 18:48:16.991519 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:48:16.991498    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-8b4d697c5-7s7hs"]
Sep 16 18:48:16.993927 ip-10-0-138-127 kubenswrapper[2581]: W0916 18:48:16.993902    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod39b80a07_c65f_4e50_88e9_a5269e8c6d66.slice/crio-bb7fe0b15e07b96351ca352ae6ae6b85f0843f227366aa09d6fbf9843ce1a682 WatchSource:0}: Error finding container bb7fe0b15e07b96351ca352ae6ae6b85f0843f227366aa09d6fbf9843ce1a682: Status 404 returned error can't find the container with id bb7fe0b15e07b96351ca352ae6ae6b85f0843f227366aa09d6fbf9843ce1a682
Sep 16 18:48:17.410489 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:48:17.410435    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-c2dhr" event={"ID":"6e068f1b-74ad-44e1-aee0-5307a9eae059","Type":"ContainerStarted","Data":"dee7388e5a33dfda4e0be0b1637d0d171540e857be28a746087d70f3d51d8ca5"}
Sep 16 18:48:17.412763 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:48:17.412732    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-qsxrs" event={"ID":"61521525-706f-428c-80f6-2cdfcb30fdc4","Type":"ContainerStarted","Data":"5cbd2579478d6a06c8768ead438b3c38c261ee40376155a7081f10df9405bea4"}
Sep 16 18:48:17.412922 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:48:17.412788    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-qsxrs"
Sep 16 18:48:17.412922 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:48:17.412804    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-qsxrs" event={"ID":"61521525-706f-428c-80f6-2cdfcb30fdc4","Type":"ContainerStarted","Data":"fccc06e84851584f39edabe95a259c6c41696e981d243eeb0ea595343751cf22"}
Sep 16 18:48:17.414180 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:48:17.414121    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-8b4d697c5-7s7hs" event={"ID":"39b80a07-c65f-4e50-88e9-a5269e8c6d66","Type":"ContainerStarted","Data":"bb7fe0b15e07b96351ca352ae6ae6b85f0843f227366aa09d6fbf9843ce1a682"}
Sep 16 18:48:17.415538 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:48:17.415510    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-76f99584f4-67c8w" event={"ID":"ddcab39a-92a7-4106-927a-8160c999373c","Type":"ContainerStarted","Data":"0f88ec79302971a514b7332757e594d4cc4cb536c97edb25d905a2f3f3990501"}
Sep 16 18:48:17.416827 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:48:17.416800    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-c8fd974bd-zcgx2" event={"ID":"0f94c58b-0100-4cf7-bb65-912e6edc15c4","Type":"ContainerStarted","Data":"a49fd52ba5fd7e4a67ef4b95793c41e7033cc7223395ff16b410ffce0ee63398"}
Sep 16 18:48:17.439117 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:48:17.439056    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-c2dhr" podStartSLOduration=34.019831385 podStartE2EDuration="36.439041472s" totalImagesPullingTime="2.419210087s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-16 18:47:41 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:48:17.43866729 +0000 UTC m=+167.258081200" watchObservedRunningTime="2026-09-16 18:48:17.439041472 +0000 UTC m=+167.258455380"
Sep 16 18:48:17.466074 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:48:17.466014    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-qsxrs" podStartSLOduration=129.688636162 podStartE2EDuration="2m11.465995672s" totalImagesPullingTime="1.77735951s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:46:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:48:17.464810988 +0000 UTC m=+167.284224898" watchObservedRunningTime="2026-09-16 18:48:17.465995672 +0000 UTC m=+167.285409581"
Sep 16 18:48:18.423094 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:48:18.423051    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-8b4d697c5-7s7hs" event={"ID":"39b80a07-c65f-4e50-88e9-a5269e8c6d66","Type":"ContainerStarted","Data":"5c4570f36ff8ffdee2a5f8812ed59440e1d2e16406593503778726601919b5ee"}
Sep 16 18:48:18.423527 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:48:18.423116    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-8b4d697c5-7s7hs" event={"ID":"39b80a07-c65f-4e50-88e9-a5269e8c6d66","Type":"ContainerStarted","Data":"2a56772066b7966587a61f18b9b4c5d009c6e3231ac83ba9d31f0b44c8d19691"}
Sep 16 18:48:18.449452 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:48:18.449388    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator/migrator-8b4d697c5-7s7hs" podStartSLOduration=2.23056373 podStartE2EDuration="3.449365967s" totalImagesPullingTime="1.218802237s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:48:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:48:18.449084946 +0000 UTC m=+168.268498859" watchObservedRunningTime="2026-09-16 18:48:18.449365967 +0000 UTC m=+168.268779877"
Sep 16 18:48:19.743235 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:48:19.743205    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-6456555d6b-r87j6"]
Sep 16 18:48:19.746494 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:48:19.746463    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6456555d6b-r87j6"
Sep 16 18:48:19.751084 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:48:19.751058    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\""
Sep 16 18:48:19.751258 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:48:19.751108    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-6gpst\""
Sep 16 18:48:19.751258 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:48:19.751169    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\""
Sep 16 18:48:19.751258 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:48:19.751108    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\""
Sep 16 18:48:19.751258 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:48:19.751108    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\""
Sep 16 18:48:19.751258 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:48:19.751219    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\""
Sep 16 18:48:19.761150 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:48:19.761125    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-6456555d6b-r87j6"]
Sep 16 18:48:19.895587 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:48:19.895547    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/d9df7594-85dc-4836-a7e7-7777f474e948-console-oauth-config\") pod \"console-6456555d6b-r87j6\" (UID: \"d9df7594-85dc-4836-a7e7-7777f474e948\") " pod="openshift-console/console-6456555d6b-r87j6"
Sep 16 18:48:19.895587 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:48:19.895588    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/d9df7594-85dc-4836-a7e7-7777f474e948-oauth-serving-cert\") pod \"console-6456555d6b-r87j6\" (UID: \"d9df7594-85dc-4836-a7e7-7777f474e948\") " pod="openshift-console/console-6456555d6b-r87j6"
Sep 16 18:48:19.895851 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:48:19.895616    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/d9df7594-85dc-4836-a7e7-7777f474e948-console-serving-cert\") pod \"console-6456555d6b-r87j6\" (UID: \"d9df7594-85dc-4836-a7e7-7777f474e948\") " pod="openshift-console/console-6456555d6b-r87j6"
Sep 16 18:48:19.895851 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:48:19.895710    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/d9df7594-85dc-4836-a7e7-7777f474e948-service-ca\") pod \"console-6456555d6b-r87j6\" (UID: \"d9df7594-85dc-4836-a7e7-7777f474e948\") " pod="openshift-console/console-6456555d6b-r87j6"
Sep 16 18:48:19.895851 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:48:19.895776    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c5fhh\" (UniqueName: \"kubernetes.io/projected/d9df7594-85dc-4836-a7e7-7777f474e948-kube-api-access-c5fhh\") pod \"console-6456555d6b-r87j6\" (UID: \"d9df7594-85dc-4836-a7e7-7777f474e948\") " pod="openshift-console/console-6456555d6b-r87j6"
Sep 16 18:48:19.895851 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:48:19.895820    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/d9df7594-85dc-4836-a7e7-7777f474e948-console-config\") pod \"console-6456555d6b-r87j6\" (UID: \"d9df7594-85dc-4836-a7e7-7777f474e948\") " pod="openshift-console/console-6456555d6b-r87j6"
Sep 16 18:48:19.996980 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:48:19.996901    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/d9df7594-85dc-4836-a7e7-7777f474e948-console-oauth-config\") pod \"console-6456555d6b-r87j6\" (UID: \"d9df7594-85dc-4836-a7e7-7777f474e948\") " pod="openshift-console/console-6456555d6b-r87j6"
Sep 16 18:48:19.996980 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:48:19.996938    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/d9df7594-85dc-4836-a7e7-7777f474e948-oauth-serving-cert\") pod \"console-6456555d6b-r87j6\" (UID: \"d9df7594-85dc-4836-a7e7-7777f474e948\") " pod="openshift-console/console-6456555d6b-r87j6"
Sep 16 18:48:19.996980 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:48:19.996962    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/d9df7594-85dc-4836-a7e7-7777f474e948-console-serving-cert\") pod \"console-6456555d6b-r87j6\" (UID: \"d9df7594-85dc-4836-a7e7-7777f474e948\") " pod="openshift-console/console-6456555d6b-r87j6"
Sep 16 18:48:19.997286 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:48:19.997024    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/d9df7594-85dc-4836-a7e7-7777f474e948-service-ca\") pod \"console-6456555d6b-r87j6\" (UID: \"d9df7594-85dc-4836-a7e7-7777f474e948\") " pod="openshift-console/console-6456555d6b-r87j6"
Sep 16 18:48:19.997286 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:48:19.997053    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c5fhh\" (UniqueName: \"kubernetes.io/projected/d9df7594-85dc-4836-a7e7-7777f474e948-kube-api-access-c5fhh\") pod \"console-6456555d6b-r87j6\" (UID: \"d9df7594-85dc-4836-a7e7-7777f474e948\") " pod="openshift-console/console-6456555d6b-r87j6"
Sep 16 18:48:19.997286 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:48:19.997077    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/d9df7594-85dc-4836-a7e7-7777f474e948-console-config\") pod \"console-6456555d6b-r87j6\" (UID: \"d9df7594-85dc-4836-a7e7-7777f474e948\") " pod="openshift-console/console-6456555d6b-r87j6"
Sep 16 18:48:19.997837 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:48:19.997805    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/d9df7594-85dc-4836-a7e7-7777f474e948-oauth-serving-cert\") pod \"console-6456555d6b-r87j6\" (UID: \"d9df7594-85dc-4836-a7e7-7777f474e948\") " pod="openshift-console/console-6456555d6b-r87j6"
Sep 16 18:48:19.997945 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:48:19.997836    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/d9df7594-85dc-4836-a7e7-7777f474e948-console-config\") pod \"console-6456555d6b-r87j6\" (UID: \"d9df7594-85dc-4836-a7e7-7777f474e948\") " pod="openshift-console/console-6456555d6b-r87j6"
Sep 16 18:48:19.998106 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:48:19.998087    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/d9df7594-85dc-4836-a7e7-7777f474e948-service-ca\") pod \"console-6456555d6b-r87j6\" (UID: \"d9df7594-85dc-4836-a7e7-7777f474e948\") " pod="openshift-console/console-6456555d6b-r87j6"
Sep 16 18:48:19.999958 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:48:19.999935    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/d9df7594-85dc-4836-a7e7-7777f474e948-console-oauth-config\") pod \"console-6456555d6b-r87j6\" (UID: \"d9df7594-85dc-4836-a7e7-7777f474e948\") " pod="openshift-console/console-6456555d6b-r87j6"
Sep 16 18:48:20.000101 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:48:20.000074    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/d9df7594-85dc-4836-a7e7-7777f474e948-console-serving-cert\") pod \"console-6456555d6b-r87j6\" (UID: \"d9df7594-85dc-4836-a7e7-7777f474e948\") " pod="openshift-console/console-6456555d6b-r87j6"
Sep 16 18:48:20.009241 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:48:20.009212    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-c5fhh\" (UniqueName: \"kubernetes.io/projected/d9df7594-85dc-4836-a7e7-7777f474e948-kube-api-access-c5fhh\") pod \"console-6456555d6b-r87j6\" (UID: \"d9df7594-85dc-4836-a7e7-7777f474e948\") " pod="openshift-console/console-6456555d6b-r87j6"
Sep 16 18:48:20.056375 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:48:20.056332    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6456555d6b-r87j6"
Sep 16 18:48:20.344993 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:48:20.344962    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-6456555d6b-r87j6"]
Sep 16 18:48:20.349030 ip-10-0-138-127 kubenswrapper[2581]: W0916 18:48:20.348998    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd9df7594_85dc_4836_a7e7_7777f474e948.slice/crio-60398e677427b1bc93f7e47cc61f975879b2da199b726e77e4dece879de483fd WatchSource:0}: Error finding container 60398e677427b1bc93f7e47cc61f975879b2da199b726e77e4dece879de483fd: Status 404 returned error can't find the container with id 60398e677427b1bc93f7e47cc61f975879b2da199b726e77e4dece879de483fd
Sep 16 18:48:20.433170 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:48:20.433127    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6456555d6b-r87j6" event={"ID":"d9df7594-85dc-4836-a7e7-7777f474e948","Type":"ContainerStarted","Data":"60398e677427b1bc93f7e47cc61f975879b2da199b726e77e4dece879de483fd"}
Sep 16 18:48:21.722205 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:48:21.722153    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-c2kp2"
Sep 16 18:48:25.456163 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:48:25.456119    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6456555d6b-r87j6" event={"ID":"d9df7594-85dc-4836-a7e7-7777f474e948","Type":"ContainerStarted","Data":"038c1b0a8f4e9961d82562646484e27ec3b5e65aae0f9d32e4c8da29be3caca4"}
Sep 16 18:48:25.459085 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:48:25.459053    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-c8fd974bd-zcgx2" event={"ID":"0f94c58b-0100-4cf7-bb65-912e6edc15c4","Type":"ContainerStarted","Data":"c5b1fe9a9605294fb1bf7b4690badc356d882e9555f0aabb69081f77133b0d8d"}
Sep 16 18:48:25.459504 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:48:25.459486    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-c8fd974bd-zcgx2"
Sep 16 18:48:25.460816 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:48:25.460780    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-c8fd974bd-zcgx2" podUID="0f94c58b-0100-4cf7-bb65-912e6edc15c4" containerName="acm-agent" probeResult="failure" output="Get \"http://10.132.0.11:8000/readyz\": dial tcp 10.132.0.11:8000: connect: connection refused"
Sep 16 18:48:25.481581 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:48:25.481519    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-6456555d6b-r87j6" podStartSLOduration=1.479048828 podStartE2EDuration="6.48149779s" totalImagesPullingTime="5.002448962s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:48:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:48:25.48026634 +0000 UTC m=+175.299680249" watchObservedRunningTime="2026-09-16 18:48:25.48149779 +0000 UTC m=+175.300911691"
Sep 16 18:48:25.500837 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:48:25.500788    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-c8fd974bd-zcgx2" podStartSLOduration=2.122579383 podStartE2EDuration="10.500769004s" totalImagesPullingTime="8.378189621s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:48:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:48:25.499442264 +0000 UTC m=+175.318856172" watchObservedRunningTime="2026-09-16 18:48:25.500769004 +0000 UTC m=+175.320182912"
Sep 16 18:48:26.443752 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:48:26.443715    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/openshift-state-metrics-67dcf8c979-4hdsw"]
Sep 16 18:48:26.447577 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:48:26.447551    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-4hdsw"
Sep 16 18:48:26.452929 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:48:26.451780    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-dockercfg-g4zsf\""
Sep 16 18:48:26.452929 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:48:26.452082    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\""
Sep 16 18:48:26.452929 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:48:26.452314    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
Sep 16 18:48:26.452929 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:48:26.452518    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-tls\""
Sep 16 18:48:26.452929 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:48:26.452746    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-kube-rbac-proxy-config\""
Sep 16 18:48:26.453291 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:48:26.453002    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
Sep 16 18:48:26.463349 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:48:26.463325    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-c8fd974bd-zcgx2"
Sep 16 18:48:26.480715 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:48:26.480674    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-67dcf8c979-4hdsw"]
Sep 16 18:48:26.481701 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:48:26.481677    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-hsp8v"]
Sep 16 18:48:26.485370 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:48:26.485350    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-hsp8v"
Sep 16 18:48:26.488074 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:48:26.488055    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-zt92r\""
Sep 16 18:48:26.488227 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:48:26.488183    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\""
Sep 16 18:48:26.488858 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:48:26.488827    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\""
Sep 16 18:48:26.493938 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:48:26.493904    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\""
Sep 16 18:48:26.555160 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:48:26.555124    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/b5732a85-64c1-4cf8-b80a-c06f8d0a4b3b-node-exporter-textfile\") pod \"node-exporter-hsp8v\" (UID: \"b5732a85-64c1-4cf8-b80a-c06f8d0a4b3b\") " pod="openshift-monitoring/node-exporter-hsp8v"
Sep 16 18:48:26.555357 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:48:26.555206    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/b5732a85-64c1-4cf8-b80a-c06f8d0a4b3b-sys\") pod \"node-exporter-hsp8v\" (UID: \"b5732a85-64c1-4cf8-b80a-c06f8d0a4b3b\") " pod="openshift-monitoring/node-exporter-hsp8v"
Sep 16 18:48:26.555357 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:48:26.555276    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/dc53a5ab-d957-42f4-9eb7-a3c38c3ca302-openshift-state-metrics-tls\") pod \"openshift-state-metrics-67dcf8c979-4hdsw\" (UID: \"dc53a5ab-d957-42f4-9eb7-a3c38c3ca302\") " pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-4hdsw"
Sep 16 18:48:26.555470 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:48:26.555381    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/b5732a85-64c1-4cf8-b80a-c06f8d0a4b3b-node-exporter-accelerators-collector-config\") pod \"node-exporter-hsp8v\" (UID: \"b5732a85-64c1-4cf8-b80a-c06f8d0a4b3b\") " pod="openshift-monitoring/node-exporter-hsp8v"
Sep 16 18:48:26.555624 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:48:26.555598    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/dc53a5ab-d957-42f4-9eb7-a3c38c3ca302-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-67dcf8c979-4hdsw\" (UID: \"dc53a5ab-d957-42f4-9eb7-a3c38c3ca302\") " pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-4hdsw"
Sep 16 18:48:26.555752 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:48:26.555636    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/dc53a5ab-d957-42f4-9eb7-a3c38c3ca302-metrics-client-ca\") pod \"openshift-state-metrics-67dcf8c979-4hdsw\" (UID: \"dc53a5ab-d957-42f4-9eb7-a3c38c3ca302\") " pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-4hdsw"
Sep 16 18:48:26.555752 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:48:26.555669    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/b5732a85-64c1-4cf8-b80a-c06f8d0a4b3b-node-exporter-tls\") pod \"node-exporter-hsp8v\" (UID: \"b5732a85-64c1-4cf8-b80a-c06f8d0a4b3b\") " pod="openshift-monitoring/node-exporter-hsp8v"
Sep 16 18:48:26.555752 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:48:26.555701    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pkg2c\" (UniqueName: \"kubernetes.io/projected/dc53a5ab-d957-42f4-9eb7-a3c38c3ca302-kube-api-access-pkg2c\") pod \"openshift-state-metrics-67dcf8c979-4hdsw\" (UID: \"dc53a5ab-d957-42f4-9eb7-a3c38c3ca302\") " pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-4hdsw"
Sep 16 18:48:26.555752 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:48:26.555725    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/b5732a85-64c1-4cf8-b80a-c06f8d0a4b3b-root\") pod \"node-exporter-hsp8v\" (UID: \"b5732a85-64c1-4cf8-b80a-c06f8d0a4b3b\") " pod="openshift-monitoring/node-exporter-hsp8v"
Sep 16 18:48:26.555972 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:48:26.555802    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p8npw\" (UniqueName: \"kubernetes.io/projected/b5732a85-64c1-4cf8-b80a-c06f8d0a4b3b-kube-api-access-p8npw\") pod \"node-exporter-hsp8v\" (UID: \"b5732a85-64c1-4cf8-b80a-c06f8d0a4b3b\") " pod="openshift-monitoring/node-exporter-hsp8v"
Sep 16 18:48:26.555972 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:48:26.555961    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/b5732a85-64c1-4cf8-b80a-c06f8d0a4b3b-node-exporter-wtmp\") pod \"node-exporter-hsp8v\" (UID: \"b5732a85-64c1-4cf8-b80a-c06f8d0a4b3b\") " pod="openshift-monitoring/node-exporter-hsp8v"
Sep 16 18:48:26.556078 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:48:26.555994    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/b5732a85-64c1-4cf8-b80a-c06f8d0a4b3b-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-hsp8v\" (UID: \"b5732a85-64c1-4cf8-b80a-c06f8d0a4b3b\") " pod="openshift-monitoring/node-exporter-hsp8v"
Sep 16 18:48:26.556134 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:48:26.556092    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/b5732a85-64c1-4cf8-b80a-c06f8d0a4b3b-metrics-client-ca\") pod \"node-exporter-hsp8v\" (UID: \"b5732a85-64c1-4cf8-b80a-c06f8d0a4b3b\") " pod="openshift-monitoring/node-exporter-hsp8v"
Sep 16 18:48:26.657467 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:48:26.657427    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/b5732a85-64c1-4cf8-b80a-c06f8d0a4b3b-node-exporter-textfile\") pod \"node-exporter-hsp8v\" (UID: \"b5732a85-64c1-4cf8-b80a-c06f8d0a4b3b\") " pod="openshift-monitoring/node-exporter-hsp8v"
Sep 16 18:48:26.657773 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:48:26.657488    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/b5732a85-64c1-4cf8-b80a-c06f8d0a4b3b-sys\") pod \"node-exporter-hsp8v\" (UID: \"b5732a85-64c1-4cf8-b80a-c06f8d0a4b3b\") " pod="openshift-monitoring/node-exporter-hsp8v"
Sep 16 18:48:26.657773 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:48:26.657517    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/dc53a5ab-d957-42f4-9eb7-a3c38c3ca302-openshift-state-metrics-tls\") pod \"openshift-state-metrics-67dcf8c979-4hdsw\" (UID: \"dc53a5ab-d957-42f4-9eb7-a3c38c3ca302\") " pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-4hdsw"
Sep 16 18:48:26.657773 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:48:26.657541    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/b5732a85-64c1-4cf8-b80a-c06f8d0a4b3b-node-exporter-accelerators-collector-config\") pod \"node-exporter-hsp8v\" (UID: \"b5732a85-64c1-4cf8-b80a-c06f8d0a4b3b\") " pod="openshift-monitoring/node-exporter-hsp8v"
Sep 16 18:48:26.657773 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:48:26.657583    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/dc53a5ab-d957-42f4-9eb7-a3c38c3ca302-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-67dcf8c979-4hdsw\" (UID: \"dc53a5ab-d957-42f4-9eb7-a3c38c3ca302\") " pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-4hdsw"
Sep 16 18:48:26.657773 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:48:26.657600    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/dc53a5ab-d957-42f4-9eb7-a3c38c3ca302-metrics-client-ca\") pod \"openshift-state-metrics-67dcf8c979-4hdsw\" (UID: \"dc53a5ab-d957-42f4-9eb7-a3c38c3ca302\") " pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-4hdsw"
Sep 16 18:48:26.657773 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:48:26.657620    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/b5732a85-64c1-4cf8-b80a-c06f8d0a4b3b-node-exporter-tls\") pod \"node-exporter-hsp8v\" (UID: \"b5732a85-64c1-4cf8-b80a-c06f8d0a4b3b\") " pod="openshift-monitoring/node-exporter-hsp8v"
Sep 16 18:48:26.657773 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:48:26.657646    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pkg2c\" (UniqueName: \"kubernetes.io/projected/dc53a5ab-d957-42f4-9eb7-a3c38c3ca302-kube-api-access-pkg2c\") pod \"openshift-state-metrics-67dcf8c979-4hdsw\" (UID: \"dc53a5ab-d957-42f4-9eb7-a3c38c3ca302\") " pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-4hdsw"
Sep 16 18:48:26.657773 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:48:26.657667    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/b5732a85-64c1-4cf8-b80a-c06f8d0a4b3b-root\") pod \"node-exporter-hsp8v\" (UID: \"b5732a85-64c1-4cf8-b80a-c06f8d0a4b3b\") " pod="openshift-monitoring/node-exporter-hsp8v"
Sep 16 18:48:26.657773 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:48:26.657687    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p8npw\" (UniqueName: \"kubernetes.io/projected/b5732a85-64c1-4cf8-b80a-c06f8d0a4b3b-kube-api-access-p8npw\") pod \"node-exporter-hsp8v\" (UID: \"b5732a85-64c1-4cf8-b80a-c06f8d0a4b3b\") " pod="openshift-monitoring/node-exporter-hsp8v"
Sep 16 18:48:26.657773 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:48:26.657704    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/b5732a85-64c1-4cf8-b80a-c06f8d0a4b3b-node-exporter-wtmp\") pod \"node-exporter-hsp8v\" (UID: \"b5732a85-64c1-4cf8-b80a-c06f8d0a4b3b\") " pod="openshift-monitoring/node-exporter-hsp8v"
Sep 16 18:48:26.657773 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:48:26.657727    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/b5732a85-64c1-4cf8-b80a-c06f8d0a4b3b-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-hsp8v\" (UID: \"b5732a85-64c1-4cf8-b80a-c06f8d0a4b3b\") " pod="openshift-monitoring/node-exporter-hsp8v"
Sep 16 18:48:26.657773 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:48:26.657776    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/b5732a85-64c1-4cf8-b80a-c06f8d0a4b3b-metrics-client-ca\") pod \"node-exporter-hsp8v\" (UID: \"b5732a85-64c1-4cf8-b80a-c06f8d0a4b3b\") " pod="openshift-monitoring/node-exporter-hsp8v"
Sep 16 18:48:26.658383 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:48:26.658240    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/b5732a85-64c1-4cf8-b80a-c06f8d0a4b3b-sys\") pod \"node-exporter-hsp8v\" (UID: \"b5732a85-64c1-4cf8-b80a-c06f8d0a4b3b\") " pod="openshift-monitoring/node-exporter-hsp8v"
Sep 16 18:48:26.658383 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:48:26.658338    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/b5732a85-64c1-4cf8-b80a-c06f8d0a4b3b-node-exporter-wtmp\") pod \"node-exporter-hsp8v\" (UID: \"b5732a85-64c1-4cf8-b80a-c06f8d0a4b3b\") " pod="openshift-monitoring/node-exporter-hsp8v"
Sep 16 18:48:26.658492 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:48:26.658387    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/b5732a85-64c1-4cf8-b80a-c06f8d0a4b3b-node-exporter-textfile\") pod \"node-exporter-hsp8v\" (UID: \"b5732a85-64c1-4cf8-b80a-c06f8d0a4b3b\") " pod="openshift-monitoring/node-exporter-hsp8v"
Sep 16 18:48:26.658492 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:48:26.658450    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/b5732a85-64c1-4cf8-b80a-c06f8d0a4b3b-root\") pod \"node-exporter-hsp8v\" (UID: \"b5732a85-64c1-4cf8-b80a-c06f8d0a4b3b\") " pod="openshift-monitoring/node-exporter-hsp8v"
Sep 16 18:48:26.658601 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:48:26.658497    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/b5732a85-64c1-4cf8-b80a-c06f8d0a4b3b-metrics-client-ca\") pod \"node-exporter-hsp8v\" (UID: \"b5732a85-64c1-4cf8-b80a-c06f8d0a4b3b\") " pod="openshift-monitoring/node-exporter-hsp8v"
Sep 16 18:48:26.658763 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:48:26.658706    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/b5732a85-64c1-4cf8-b80a-c06f8d0a4b3b-node-exporter-accelerators-collector-config\") pod \"node-exporter-hsp8v\" (UID: \"b5732a85-64c1-4cf8-b80a-c06f8d0a4b3b\") " pod="openshift-monitoring/node-exporter-hsp8v"
Sep 16 18:48:26.659613 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:48:26.659585    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/dc53a5ab-d957-42f4-9eb7-a3c38c3ca302-metrics-client-ca\") pod \"openshift-state-metrics-67dcf8c979-4hdsw\" (UID: \"dc53a5ab-d957-42f4-9eb7-a3c38c3ca302\") " pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-4hdsw"
Sep 16 18:48:26.660933 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:48:26.660914    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/b5732a85-64c1-4cf8-b80a-c06f8d0a4b3b-node-exporter-tls\") pod \"node-exporter-hsp8v\" (UID: \"b5732a85-64c1-4cf8-b80a-c06f8d0a4b3b\") " pod="openshift-monitoring/node-exporter-hsp8v"
Sep 16 18:48:26.661230 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:48:26.661184    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/dc53a5ab-d957-42f4-9eb7-a3c38c3ca302-openshift-state-metrics-tls\") pod \"openshift-state-metrics-67dcf8c979-4hdsw\" (UID: \"dc53a5ab-d957-42f4-9eb7-a3c38c3ca302\") " pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-4hdsw"
Sep 16 18:48:26.661886 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:48:26.661866    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/b5732a85-64c1-4cf8-b80a-c06f8d0a4b3b-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-hsp8v\" (UID: \"b5732a85-64c1-4cf8-b80a-c06f8d0a4b3b\") " pod="openshift-monitoring/node-exporter-hsp8v"
Sep 16 18:48:26.662338 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:48:26.662314    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/dc53a5ab-d957-42f4-9eb7-a3c38c3ca302-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-67dcf8c979-4hdsw\" (UID: \"dc53a5ab-d957-42f4-9eb7-a3c38c3ca302\") " pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-4hdsw"
Sep 16 18:48:26.673863 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:48:26.673835    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pkg2c\" (UniqueName: \"kubernetes.io/projected/dc53a5ab-d957-42f4-9eb7-a3c38c3ca302-kube-api-access-pkg2c\") pod \"openshift-state-metrics-67dcf8c979-4hdsw\" (UID: \"dc53a5ab-d957-42f4-9eb7-a3c38c3ca302\") " pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-4hdsw"
Sep 16 18:48:26.673967 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:48:26.673833    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-p8npw\" (UniqueName: \"kubernetes.io/projected/b5732a85-64c1-4cf8-b80a-c06f8d0a4b3b-kube-api-access-p8npw\") pod \"node-exporter-hsp8v\" (UID: \"b5732a85-64c1-4cf8-b80a-c06f8d0a4b3b\") " pod="openshift-monitoring/node-exporter-hsp8v"
Sep 16 18:48:26.761886 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:48:26.761808    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-4hdsw"
Sep 16 18:48:26.797534 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:48:26.797499    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-hsp8v"
Sep 16 18:48:26.810955 ip-10-0-138-127 kubenswrapper[2581]: W0916 18:48:26.810916    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb5732a85_64c1_4cf8_b80a_c06f8d0a4b3b.slice/crio-25c597cd938cbddcaffc9c58ecbef78b6244b47a251d2c2c5fd802fac49dafe5 WatchSource:0}: Error finding container 25c597cd938cbddcaffc9c58ecbef78b6244b47a251d2c2c5fd802fac49dafe5: Status 404 returned error can't find the container with id 25c597cd938cbddcaffc9c58ecbef78b6244b47a251d2c2c5fd802fac49dafe5
Sep 16 18:48:27.049576 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:48:27.049538    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-67dcf8c979-4hdsw"]
Sep 16 18:48:27.053401 ip-10-0-138-127 kubenswrapper[2581]: W0916 18:48:27.053373    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddc53a5ab_d957_42f4_9eb7_a3c38c3ca302.slice/crio-f2d5810b301a94a5bf8368a509dc0e61d45ff3a9171b1c2c6e8d1a5ea13842c5 WatchSource:0}: Error finding container f2d5810b301a94a5bf8368a509dc0e61d45ff3a9171b1c2c6e8d1a5ea13842c5: Status 404 returned error can't find the container with id f2d5810b301a94a5bf8368a509dc0e61d45ff3a9171b1c2c6e8d1a5ea13842c5
Sep 16 18:48:27.389977 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:48:27.389895    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Sep 16 18:48:27.393947 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:48:27.393921    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Sep 16 18:48:27.396468 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:48:27.396442    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\""
Sep 16 18:48:27.397382 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:48:27.396728    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\""
Sep 16 18:48:27.397382 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:48:27.396963    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\""
Sep 16 18:48:27.397382 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:48:27.397042    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-bm4pd\""
Sep 16 18:48:27.397382 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:48:27.397068    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\""
Sep 16 18:48:27.397382 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:48:27.397221    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\""
Sep 16 18:48:27.397382 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:48:27.397307    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\""
Sep 16 18:48:27.397382 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:48:27.397325    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\""
Sep 16 18:48:27.397769 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:48:27.397746    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\""
Sep 16 18:48:27.400486 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:48:27.400229    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\""
Sep 16 18:48:27.421396 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:48:27.421364    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Sep 16 18:48:27.425345 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:48:27.425319    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-qsxrs"
Sep 16 18:48:27.468165 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:48:27.468125    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-hsp8v" event={"ID":"b5732a85-64c1-4cf8-b80a-c06f8d0a4b3b","Type":"ContainerStarted","Data":"25c597cd938cbddcaffc9c58ecbef78b6244b47a251d2c2c5fd802fac49dafe5"}
Sep 16 18:48:27.470474 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:48:27.470446    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-4hdsw" event={"ID":"dc53a5ab-d957-42f4-9eb7-a3c38c3ca302","Type":"ContainerStarted","Data":"08cee2091513baa68a9924d50c298d97d266ac86b8c5c0ac5ceec35b33eb80d1"}
Sep 16 18:48:27.470610 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:48:27.470510    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-4hdsw" event={"ID":"dc53a5ab-d957-42f4-9eb7-a3c38c3ca302","Type":"ContainerStarted","Data":"0638eb300ddb3eba08fc1e6c031dfd6cbc7012b18338ca56dae94afa834b796a"}
Sep 16 18:48:27.470610 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:48:27.470541    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-4hdsw" event={"ID":"dc53a5ab-d957-42f4-9eb7-a3c38c3ca302","Type":"ContainerStarted","Data":"f2d5810b301a94a5bf8368a509dc0e61d45ff3a9171b1c2c6e8d1a5ea13842c5"}
Sep 16 18:48:27.567382 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:48:27.567341    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/89fc85f4-ec94-4459-8311-f3dbf4681bf8-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"89fc85f4-ec94-4459-8311-f3dbf4681bf8\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 18:48:27.567543 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:48:27.567402    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/89fc85f4-ec94-4459-8311-f3dbf4681bf8-web-config\") pod \"alertmanager-main-0\" (UID: \"89fc85f4-ec94-4459-8311-f3dbf4681bf8\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 18:48:27.567543 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:48:27.567432    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/89fc85f4-ec94-4459-8311-f3dbf4681bf8-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"89fc85f4-ec94-4459-8311-f3dbf4681bf8\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 18:48:27.567644 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:48:27.567549    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/89fc85f4-ec94-4459-8311-f3dbf4681bf8-tls-assets\") pod \"alertmanager-main-0\" (UID: \"89fc85f4-ec94-4459-8311-f3dbf4681bf8\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 18:48:27.567644 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:48:27.567601    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/89fc85f4-ec94-4459-8311-f3dbf4681bf8-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"89fc85f4-ec94-4459-8311-f3dbf4681bf8\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 18:48:27.567644 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:48:27.567623    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/89fc85f4-ec94-4459-8311-f3dbf4681bf8-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"89fc85f4-ec94-4459-8311-f3dbf4681bf8\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 18:48:27.567775 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:48:27.567695    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/89fc85f4-ec94-4459-8311-f3dbf4681bf8-config-out\") pod \"alertmanager-main-0\" (UID: \"89fc85f4-ec94-4459-8311-f3dbf4681bf8\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 18:48:27.567775 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:48:27.567753    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/89fc85f4-ec94-4459-8311-f3dbf4681bf8-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"89fc85f4-ec94-4459-8311-f3dbf4681bf8\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 18:48:27.567875 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:48:27.567779    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-krrrv\" (UniqueName: \"kubernetes.io/projected/89fc85f4-ec94-4459-8311-f3dbf4681bf8-kube-api-access-krrrv\") pod \"alertmanager-main-0\" (UID: \"89fc85f4-ec94-4459-8311-f3dbf4681bf8\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 18:48:27.567875 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:48:27.567852    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/89fc85f4-ec94-4459-8311-f3dbf4681bf8-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"89fc85f4-ec94-4459-8311-f3dbf4681bf8\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 18:48:27.567947 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:48:27.567894    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/89fc85f4-ec94-4459-8311-f3dbf4681bf8-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"89fc85f4-ec94-4459-8311-f3dbf4681bf8\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 18:48:27.567947 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:48:27.567923    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/89fc85f4-ec94-4459-8311-f3dbf4681bf8-config-volume\") pod \"alertmanager-main-0\" (UID: \"89fc85f4-ec94-4459-8311-f3dbf4681bf8\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 18:48:27.568001 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:48:27.567953    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/89fc85f4-ec94-4459-8311-f3dbf4681bf8-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"89fc85f4-ec94-4459-8311-f3dbf4681bf8\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 18:48:27.669415 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:48:27.669320    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/89fc85f4-ec94-4459-8311-f3dbf4681bf8-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"89fc85f4-ec94-4459-8311-f3dbf4681bf8\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 18:48:27.669415 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:48:27.669370    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/89fc85f4-ec94-4459-8311-f3dbf4681bf8-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"89fc85f4-ec94-4459-8311-f3dbf4681bf8\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 18:48:27.669415 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:48:27.669404    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/89fc85f4-ec94-4459-8311-f3dbf4681bf8-web-config\") pod \"alertmanager-main-0\" (UID: \"89fc85f4-ec94-4459-8311-f3dbf4681bf8\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 18:48:27.669689 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:48:27.669426    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/89fc85f4-ec94-4459-8311-f3dbf4681bf8-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"89fc85f4-ec94-4459-8311-f3dbf4681bf8\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 18:48:27.669689 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:48:27.669489    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/89fc85f4-ec94-4459-8311-f3dbf4681bf8-tls-assets\") pod \"alertmanager-main-0\" (UID: \"89fc85f4-ec94-4459-8311-f3dbf4681bf8\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 18:48:27.669689 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:48:27.669524    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/89fc85f4-ec94-4459-8311-f3dbf4681bf8-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"89fc85f4-ec94-4459-8311-f3dbf4681bf8\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 18:48:27.669689 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:48:27.669549    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/89fc85f4-ec94-4459-8311-f3dbf4681bf8-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"89fc85f4-ec94-4459-8311-f3dbf4681bf8\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 18:48:27.669689 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:48:27.669605    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/89fc85f4-ec94-4459-8311-f3dbf4681bf8-config-out\") pod \"alertmanager-main-0\" (UID: \"89fc85f4-ec94-4459-8311-f3dbf4681bf8\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 18:48:27.669689 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:48:27.669680    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/89fc85f4-ec94-4459-8311-f3dbf4681bf8-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"89fc85f4-ec94-4459-8311-f3dbf4681bf8\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 18:48:27.669989 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:48:27.669706    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-krrrv\" (UniqueName: \"kubernetes.io/projected/89fc85f4-ec94-4459-8311-f3dbf4681bf8-kube-api-access-krrrv\") pod \"alertmanager-main-0\" (UID: \"89fc85f4-ec94-4459-8311-f3dbf4681bf8\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 18:48:27.669989 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:48:27.669738    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/89fc85f4-ec94-4459-8311-f3dbf4681bf8-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"89fc85f4-ec94-4459-8311-f3dbf4681bf8\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 18:48:27.669989 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:48:27.669767    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/89fc85f4-ec94-4459-8311-f3dbf4681bf8-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"89fc85f4-ec94-4459-8311-f3dbf4681bf8\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 18:48:27.669989 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:48:27.669801    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/89fc85f4-ec94-4459-8311-f3dbf4681bf8-config-volume\") pod \"alertmanager-main-0\" (UID: \"89fc85f4-ec94-4459-8311-f3dbf4681bf8\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 18:48:27.670922 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:48:27.670398    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/89fc85f4-ec94-4459-8311-f3dbf4681bf8-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"89fc85f4-ec94-4459-8311-f3dbf4681bf8\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 18:48:27.671411 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:48:27.671281    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/89fc85f4-ec94-4459-8311-f3dbf4681bf8-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"89fc85f4-ec94-4459-8311-f3dbf4681bf8\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 18:48:27.675021 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:48:27.674400    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/89fc85f4-ec94-4459-8311-f3dbf4681bf8-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"89fc85f4-ec94-4459-8311-f3dbf4681bf8\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 18:48:27.675021 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:48:27.674800    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/89fc85f4-ec94-4459-8311-f3dbf4681bf8-tls-assets\") pod \"alertmanager-main-0\" (UID: \"89fc85f4-ec94-4459-8311-f3dbf4681bf8\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 18:48:27.675021 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:48:27.674825    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/89fc85f4-ec94-4459-8311-f3dbf4681bf8-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"89fc85f4-ec94-4459-8311-f3dbf4681bf8\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 18:48:27.677524 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:48:27.677499    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/89fc85f4-ec94-4459-8311-f3dbf4681bf8-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"89fc85f4-ec94-4459-8311-f3dbf4681bf8\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 18:48:27.677630 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:48:27.677533    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/89fc85f4-ec94-4459-8311-f3dbf4681bf8-config-out\") pod \"alertmanager-main-0\" (UID: \"89fc85f4-ec94-4459-8311-f3dbf4681bf8\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 18:48:27.677801 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:48:27.677779    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/89fc85f4-ec94-4459-8311-f3dbf4681bf8-web-config\") pod \"alertmanager-main-0\" (UID: \"89fc85f4-ec94-4459-8311-f3dbf4681bf8\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 18:48:27.678095 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:48:27.678066    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/89fc85f4-ec94-4459-8311-f3dbf4681bf8-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"89fc85f4-ec94-4459-8311-f3dbf4681bf8\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 18:48:27.678292 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:48:27.678218    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/89fc85f4-ec94-4459-8311-f3dbf4681bf8-config-volume\") pod \"alertmanager-main-0\" (UID: \"89fc85f4-ec94-4459-8311-f3dbf4681bf8\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 18:48:27.679305 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:48:27.679276    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/89fc85f4-ec94-4459-8311-f3dbf4681bf8-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"89fc85f4-ec94-4459-8311-f3dbf4681bf8\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 18:48:27.681019 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:48:27.680983    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/89fc85f4-ec94-4459-8311-f3dbf4681bf8-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"89fc85f4-ec94-4459-8311-f3dbf4681bf8\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 18:48:27.690459 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:48:27.690434    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-krrrv\" (UniqueName: \"kubernetes.io/projected/89fc85f4-ec94-4459-8311-f3dbf4681bf8-kube-api-access-krrrv\") pod \"alertmanager-main-0\" (UID: \"89fc85f4-ec94-4459-8311-f3dbf4681bf8\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 18:48:27.706582 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:48:27.706549    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Sep 16 18:48:28.007892 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:48:28.007780    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Sep 16 18:48:28.011688 ip-10-0-138-127 kubenswrapper[2581]: W0916 18:48:28.011658    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod89fc85f4_ec94_4459_8311_f3dbf4681bf8.slice/crio-cc87715509bcacd6541405775c70754578b08db2ca4e69c4c946d7a31ca9c1c9 WatchSource:0}: Error finding container cc87715509bcacd6541405775c70754578b08db2ca4e69c4c946d7a31ca9c1c9: Status 404 returned error can't find the container with id cc87715509bcacd6541405775c70754578b08db2ca4e69c4c946d7a31ca9c1c9
Sep 16 18:48:28.475676 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:48:28.475641    2581 generic.go:356] "Generic (PLEG): container finished" podID="b5732a85-64c1-4cf8-b80a-c06f8d0a4b3b" containerID="5da7938d2660bb1d5a0d70a6805db3cb0bc6db7e02d44edad27031adf3d05c82" exitCode=0
Sep 16 18:48:28.476146 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:48:28.475730    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-hsp8v" event={"ID":"b5732a85-64c1-4cf8-b80a-c06f8d0a4b3b","Type":"ContainerDied","Data":"5da7938d2660bb1d5a0d70a6805db3cb0bc6db7e02d44edad27031adf3d05c82"}
Sep 16 18:48:28.477210 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:48:28.477165    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"89fc85f4-ec94-4459-8311-f3dbf4681bf8","Type":"ContainerStarted","Data":"cc87715509bcacd6541405775c70754578b08db2ca4e69c4c946d7a31ca9c1c9"}
Sep 16 18:48:30.057017 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:48:30.056975    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-6456555d6b-r87j6"
Sep 16 18:48:30.057482 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:48:30.057051    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-6456555d6b-r87j6"
Sep 16 18:48:30.058967 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:48:30.058942    2581 patch_prober.go:28] interesting pod/console-6456555d6b-r87j6 container/console namespace/openshift-console: Startup probe status=failure output="Get \"https://10.132.0.14:8443/health\": dial tcp 10.132.0.14:8443: connect: connection refused" start-of-body=
Sep 16 18:48:30.059101 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:48:30.058994    2581 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-console/console-6456555d6b-r87j6" podUID="d9df7594-85dc-4836-a7e7-7777f474e948" containerName="console" probeResult="failure" output="Get \"https://10.132.0.14:8443/health\": dial tcp 10.132.0.14:8443: connect: connection refused"
Sep 16 18:48:30.103967 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:48:30.103929    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-56fb5c8694-qcn8h"]
Sep 16 18:48:30.107415 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:48:30.107388    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-56fb5c8694-qcn8h"
Sep 16 18:48:30.116102 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:48:30.116080    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\""
Sep 16 18:48:30.122283 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:48:30.122247    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-56fb5c8694-qcn8h"]
Sep 16 18:48:30.193328 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:48:30.193291    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/5bb743b0-66c4-44c0-bd47-b99cbaa2f24e-console-config\") pod \"console-56fb5c8694-qcn8h\" (UID: \"5bb743b0-66c4-44c0-bd47-b99cbaa2f24e\") " pod="openshift-console/console-56fb5c8694-qcn8h"
Sep 16 18:48:30.193517 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:48:30.193446    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5bb743b0-66c4-44c0-bd47-b99cbaa2f24e-trusted-ca-bundle\") pod \"console-56fb5c8694-qcn8h\" (UID: \"5bb743b0-66c4-44c0-bd47-b99cbaa2f24e\") " pod="openshift-console/console-56fb5c8694-qcn8h"
Sep 16 18:48:30.193517 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:48:30.193493    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/5bb743b0-66c4-44c0-bd47-b99cbaa2f24e-oauth-serving-cert\") pod \"console-56fb5c8694-qcn8h\" (UID: \"5bb743b0-66c4-44c0-bd47-b99cbaa2f24e\") " pod="openshift-console/console-56fb5c8694-qcn8h"
Sep 16 18:48:30.193517 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:48:30.193514    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hwqjl\" (UniqueName: \"kubernetes.io/projected/5bb743b0-66c4-44c0-bd47-b99cbaa2f24e-kube-api-access-hwqjl\") pod \"console-56fb5c8694-qcn8h\" (UID: \"5bb743b0-66c4-44c0-bd47-b99cbaa2f24e\") " pod="openshift-console/console-56fb5c8694-qcn8h"
Sep 16 18:48:30.193690 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:48:30.193547    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/5bb743b0-66c4-44c0-bd47-b99cbaa2f24e-service-ca\") pod \"console-56fb5c8694-qcn8h\" (UID: \"5bb743b0-66c4-44c0-bd47-b99cbaa2f24e\") " pod="openshift-console/console-56fb5c8694-qcn8h"
Sep 16 18:48:30.193690 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:48:30.193584    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/5bb743b0-66c4-44c0-bd47-b99cbaa2f24e-console-oauth-config\") pod \"console-56fb5c8694-qcn8h\" (UID: \"5bb743b0-66c4-44c0-bd47-b99cbaa2f24e\") " pod="openshift-console/console-56fb5c8694-qcn8h"
Sep 16 18:48:30.193690 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:48:30.193623    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/5bb743b0-66c4-44c0-bd47-b99cbaa2f24e-console-serving-cert\") pod \"console-56fb5c8694-qcn8h\" (UID: \"5bb743b0-66c4-44c0-bd47-b99cbaa2f24e\") " pod="openshift-console/console-56fb5c8694-qcn8h"
Sep 16 18:48:30.294637 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:48:30.294600    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/5bb743b0-66c4-44c0-bd47-b99cbaa2f24e-console-serving-cert\") pod \"console-56fb5c8694-qcn8h\" (UID: \"5bb743b0-66c4-44c0-bd47-b99cbaa2f24e\") " pod="openshift-console/console-56fb5c8694-qcn8h"
Sep 16 18:48:30.294825 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:48:30.294678    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/5bb743b0-66c4-44c0-bd47-b99cbaa2f24e-console-config\") pod \"console-56fb5c8694-qcn8h\" (UID: \"5bb743b0-66c4-44c0-bd47-b99cbaa2f24e\") " pod="openshift-console/console-56fb5c8694-qcn8h"
Sep 16 18:48:30.294825 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:48:30.294718    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5bb743b0-66c4-44c0-bd47-b99cbaa2f24e-trusted-ca-bundle\") pod \"console-56fb5c8694-qcn8h\" (UID: \"5bb743b0-66c4-44c0-bd47-b99cbaa2f24e\") " pod="openshift-console/console-56fb5c8694-qcn8h"
Sep 16 18:48:30.294825 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:48:30.294764    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/5bb743b0-66c4-44c0-bd47-b99cbaa2f24e-oauth-serving-cert\") pod \"console-56fb5c8694-qcn8h\" (UID: \"5bb743b0-66c4-44c0-bd47-b99cbaa2f24e\") " pod="openshift-console/console-56fb5c8694-qcn8h"
Sep 16 18:48:30.294825 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:48:30.294784    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hwqjl\" (UniqueName: \"kubernetes.io/projected/5bb743b0-66c4-44c0-bd47-b99cbaa2f24e-kube-api-access-hwqjl\") pod \"console-56fb5c8694-qcn8h\" (UID: \"5bb743b0-66c4-44c0-bd47-b99cbaa2f24e\") " pod="openshift-console/console-56fb5c8694-qcn8h"
Sep 16 18:48:30.294825 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:48:30.294811    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/5bb743b0-66c4-44c0-bd47-b99cbaa2f24e-service-ca\") pod \"console-56fb5c8694-qcn8h\" (UID: \"5bb743b0-66c4-44c0-bd47-b99cbaa2f24e\") " pod="openshift-console/console-56fb5c8694-qcn8h"
Sep 16 18:48:30.295084 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:48:30.294853    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/5bb743b0-66c4-44c0-bd47-b99cbaa2f24e-console-oauth-config\") pod \"console-56fb5c8694-qcn8h\" (UID: \"5bb743b0-66c4-44c0-bd47-b99cbaa2f24e\") " pod="openshift-console/console-56fb5c8694-qcn8h"
Sep 16 18:48:30.296013 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:48:30.295978    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/5bb743b0-66c4-44c0-bd47-b99cbaa2f24e-service-ca\") pod \"console-56fb5c8694-qcn8h\" (UID: \"5bb743b0-66c4-44c0-bd47-b99cbaa2f24e\") " pod="openshift-console/console-56fb5c8694-qcn8h"
Sep 16 18:48:30.296013 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:48:30.296000    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/5bb743b0-66c4-44c0-bd47-b99cbaa2f24e-console-config\") pod \"console-56fb5c8694-qcn8h\" (UID: \"5bb743b0-66c4-44c0-bd47-b99cbaa2f24e\") " pod="openshift-console/console-56fb5c8694-qcn8h"
Sep 16 18:48:30.296209 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:48:30.296059    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/5bb743b0-66c4-44c0-bd47-b99cbaa2f24e-oauth-serving-cert\") pod \"console-56fb5c8694-qcn8h\" (UID: \"5bb743b0-66c4-44c0-bd47-b99cbaa2f24e\") " pod="openshift-console/console-56fb5c8694-qcn8h"
Sep 16 18:48:30.296209 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:48:30.296155    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5bb743b0-66c4-44c0-bd47-b99cbaa2f24e-trusted-ca-bundle\") pod \"console-56fb5c8694-qcn8h\" (UID: \"5bb743b0-66c4-44c0-bd47-b99cbaa2f24e\") " pod="openshift-console/console-56fb5c8694-qcn8h"
Sep 16 18:48:30.297657 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:48:30.297633    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/5bb743b0-66c4-44c0-bd47-b99cbaa2f24e-console-oauth-config\") pod \"console-56fb5c8694-qcn8h\" (UID: \"5bb743b0-66c4-44c0-bd47-b99cbaa2f24e\") " pod="openshift-console/console-56fb5c8694-qcn8h"
Sep 16 18:48:30.297839 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:48:30.297819    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/5bb743b0-66c4-44c0-bd47-b99cbaa2f24e-console-serving-cert\") pod \"console-56fb5c8694-qcn8h\" (UID: \"5bb743b0-66c4-44c0-bd47-b99cbaa2f24e\") " pod="openshift-console/console-56fb5c8694-qcn8h"
Sep 16 18:48:30.304357 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:48:30.304332    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hwqjl\" (UniqueName: \"kubernetes.io/projected/5bb743b0-66c4-44c0-bd47-b99cbaa2f24e-kube-api-access-hwqjl\") pod \"console-56fb5c8694-qcn8h\" (UID: \"5bb743b0-66c4-44c0-bd47-b99cbaa2f24e\") " pod="openshift-console/console-56fb5c8694-qcn8h"
Sep 16 18:48:30.418334 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:48:30.418248    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-56fb5c8694-qcn8h"
Sep 16 18:48:30.854912 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:48:30.854875    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/metrics-server-5c9d8b5484-7hpps"]
Sep 16 18:48:30.858765 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:48:30.858743    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-5c9d8b5484-7hpps"
Sep 16 18:48:30.862140 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:48:30.862108    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-client-certs\""
Sep 16 18:48:30.862607 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:48:30.862548    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\""
Sep 16 18:48:30.862607 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:48:30.862584    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-dockercfg-fz7sn\""
Sep 16 18:48:30.862607 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:48:30.862584    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-tls\""
Sep 16 18:48:30.862820 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:48:30.862587    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-1ta7kf26ncg0o\""
Sep 16 18:48:30.862880 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:48:30.862852    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-server-audit-profiles\""
Sep 16 18:48:30.868228 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:48:30.868205    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-5c9d8b5484-7hpps"]
Sep 16 18:48:30.985385 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:48:30.985348    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/monitoring-plugin-5d85778b9d-h492n"]
Sep 16 18:48:30.988561 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:48:30.988527    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-5d85778b9d-h492n"
Sep 16 18:48:30.990883 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:48:30.990858    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-dockercfg-cp2r4\""
Sep 16 18:48:30.991439 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:48:30.991239    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-cert\""
Sep 16 18:48:30.998912 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:48:30.998887    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-5d85778b9d-h492n"]
Sep 16 18:48:31.002391 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:48:31.002370    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qwfb9\" (UniqueName: \"kubernetes.io/projected/df30013a-07d7-491e-b50a-4c208c6ce24d-kube-api-access-qwfb9\") pod \"metrics-server-5c9d8b5484-7hpps\" (UID: \"df30013a-07d7-491e-b50a-4c208c6ce24d\") " pod="openshift-monitoring/metrics-server-5c9d8b5484-7hpps"
Sep 16 18:48:31.002488 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:48:31.002409    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/df30013a-07d7-491e-b50a-4c208c6ce24d-secret-metrics-server-client-certs\") pod \"metrics-server-5c9d8b5484-7hpps\" (UID: \"df30013a-07d7-491e-b50a-4c208c6ce24d\") " pod="openshift-monitoring/metrics-server-5c9d8b5484-7hpps"
Sep 16 18:48:31.002543 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:48:31.002529    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/df30013a-07d7-491e-b50a-4c208c6ce24d-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-5c9d8b5484-7hpps\" (UID: \"df30013a-07d7-491e-b50a-4c208c6ce24d\") " pod="openshift-monitoring/metrics-server-5c9d8b5484-7hpps"
Sep 16 18:48:31.002580 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:48:31.002563    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/df30013a-07d7-491e-b50a-4c208c6ce24d-audit-log\") pod \"metrics-server-5c9d8b5484-7hpps\" (UID: \"df30013a-07d7-491e-b50a-4c208c6ce24d\") " pod="openshift-monitoring/metrics-server-5c9d8b5484-7hpps"
Sep 16 18:48:31.002620 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:48:31.002587    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/df30013a-07d7-491e-b50a-4c208c6ce24d-metrics-server-audit-profiles\") pod \"metrics-server-5c9d8b5484-7hpps\" (UID: \"df30013a-07d7-491e-b50a-4c208c6ce24d\") " pod="openshift-monitoring/metrics-server-5c9d8b5484-7hpps"
Sep 16 18:48:31.002700 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:48:31.002673    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/df30013a-07d7-491e-b50a-4c208c6ce24d-client-ca-bundle\") pod \"metrics-server-5c9d8b5484-7hpps\" (UID: \"df30013a-07d7-491e-b50a-4c208c6ce24d\") " pod="openshift-monitoring/metrics-server-5c9d8b5484-7hpps"
Sep 16 18:48:31.002798 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:48:31.002713    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/df30013a-07d7-491e-b50a-4c208c6ce24d-secret-metrics-server-tls\") pod \"metrics-server-5c9d8b5484-7hpps\" (UID: \"df30013a-07d7-491e-b50a-4c208c6ce24d\") " pod="openshift-monitoring/metrics-server-5c9d8b5484-7hpps"
Sep 16 18:48:31.104212 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:48:31.104148    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/df30013a-07d7-491e-b50a-4c208c6ce24d-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-5c9d8b5484-7hpps\" (UID: \"df30013a-07d7-491e-b50a-4c208c6ce24d\") " pod="openshift-monitoring/metrics-server-5c9d8b5484-7hpps"
Sep 16 18:48:31.104719 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:48:31.104229    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/df30013a-07d7-491e-b50a-4c208c6ce24d-audit-log\") pod \"metrics-server-5c9d8b5484-7hpps\" (UID: \"df30013a-07d7-491e-b50a-4c208c6ce24d\") " pod="openshift-monitoring/metrics-server-5c9d8b5484-7hpps"
Sep 16 18:48:31.104719 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:48:31.104267    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/df30013a-07d7-491e-b50a-4c208c6ce24d-metrics-server-audit-profiles\") pod \"metrics-server-5c9d8b5484-7hpps\" (UID: \"df30013a-07d7-491e-b50a-4c208c6ce24d\") " pod="openshift-monitoring/metrics-server-5c9d8b5484-7hpps"
Sep 16 18:48:31.104719 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:48:31.104292    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/511bd723-e9cb-4806-b954-bf03f6b23d9d-monitoring-plugin-cert\") pod \"monitoring-plugin-5d85778b9d-h492n\" (UID: \"511bd723-e9cb-4806-b954-bf03f6b23d9d\") " pod="openshift-monitoring/monitoring-plugin-5d85778b9d-h492n"
Sep 16 18:48:31.104719 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:48:31.104323    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/df30013a-07d7-491e-b50a-4c208c6ce24d-client-ca-bundle\") pod \"metrics-server-5c9d8b5484-7hpps\" (UID: \"df30013a-07d7-491e-b50a-4c208c6ce24d\") " pod="openshift-monitoring/metrics-server-5c9d8b5484-7hpps"
Sep 16 18:48:31.104719 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:48:31.104355    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/df30013a-07d7-491e-b50a-4c208c6ce24d-secret-metrics-server-tls\") pod \"metrics-server-5c9d8b5484-7hpps\" (UID: \"df30013a-07d7-491e-b50a-4c208c6ce24d\") " pod="openshift-monitoring/metrics-server-5c9d8b5484-7hpps"
Sep 16 18:48:31.104719 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:48:31.104391    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qwfb9\" (UniqueName: \"kubernetes.io/projected/df30013a-07d7-491e-b50a-4c208c6ce24d-kube-api-access-qwfb9\") pod \"metrics-server-5c9d8b5484-7hpps\" (UID: \"df30013a-07d7-491e-b50a-4c208c6ce24d\") " pod="openshift-monitoring/metrics-server-5c9d8b5484-7hpps"
Sep 16 18:48:31.104719 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:48:31.104463    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/df30013a-07d7-491e-b50a-4c208c6ce24d-secret-metrics-server-client-certs\") pod \"metrics-server-5c9d8b5484-7hpps\" (UID: \"df30013a-07d7-491e-b50a-4c208c6ce24d\") " pod="openshift-monitoring/metrics-server-5c9d8b5484-7hpps"
Sep 16 18:48:31.104719 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:48:31.104689    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pz2tc\" (UniqueName: \"kubernetes.io/projected/511bd723-e9cb-4806-b954-bf03f6b23d9d-kube-api-access-pz2tc\") pod \"monitoring-plugin-5d85778b9d-h492n\" (UID: \"511bd723-e9cb-4806-b954-bf03f6b23d9d\") " pod="openshift-monitoring/monitoring-plugin-5d85778b9d-h492n"
Sep 16 18:48:31.104719 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:48:31.104719    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/df30013a-07d7-491e-b50a-4c208c6ce24d-audit-log\") pod \"metrics-server-5c9d8b5484-7hpps\" (UID: \"df30013a-07d7-491e-b50a-4c208c6ce24d\") " pod="openshift-monitoring/metrics-server-5c9d8b5484-7hpps"
Sep 16 18:48:31.105800 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:48:31.105724    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/df30013a-07d7-491e-b50a-4c208c6ce24d-metrics-server-audit-profiles\") pod \"metrics-server-5c9d8b5484-7hpps\" (UID: \"df30013a-07d7-491e-b50a-4c208c6ce24d\") " pod="openshift-monitoring/metrics-server-5c9d8b5484-7hpps"
Sep 16 18:48:31.106098 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:48:31.106065    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/df30013a-07d7-491e-b50a-4c208c6ce24d-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-5c9d8b5484-7hpps\" (UID: \"df30013a-07d7-491e-b50a-4c208c6ce24d\") " pod="openshift-monitoring/metrics-server-5c9d8b5484-7hpps"
Sep 16 18:48:31.107600 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:48:31.107577    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/df30013a-07d7-491e-b50a-4c208c6ce24d-secret-metrics-server-client-certs\") pod \"metrics-server-5c9d8b5484-7hpps\" (UID: \"df30013a-07d7-491e-b50a-4c208c6ce24d\") " pod="openshift-monitoring/metrics-server-5c9d8b5484-7hpps"
Sep 16 18:48:31.107893 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:48:31.107876    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/df30013a-07d7-491e-b50a-4c208c6ce24d-client-ca-bundle\") pod \"metrics-server-5c9d8b5484-7hpps\" (UID: \"df30013a-07d7-491e-b50a-4c208c6ce24d\") " pod="openshift-monitoring/metrics-server-5c9d8b5484-7hpps"
Sep 16 18:48:31.108020 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:48:31.107996    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/df30013a-07d7-491e-b50a-4c208c6ce24d-secret-metrics-server-tls\") pod \"metrics-server-5c9d8b5484-7hpps\" (UID: \"df30013a-07d7-491e-b50a-4c208c6ce24d\") " pod="openshift-monitoring/metrics-server-5c9d8b5484-7hpps"
Sep 16 18:48:31.113636 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:48:31.113608    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qwfb9\" (UniqueName: \"kubernetes.io/projected/df30013a-07d7-491e-b50a-4c208c6ce24d-kube-api-access-qwfb9\") pod \"metrics-server-5c9d8b5484-7hpps\" (UID: \"df30013a-07d7-491e-b50a-4c208c6ce24d\") " pod="openshift-monitoring/metrics-server-5c9d8b5484-7hpps"
Sep 16 18:48:31.170465 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:48:31.170427    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-5c9d8b5484-7hpps"
Sep 16 18:48:31.206085 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:48:31.206040    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/511bd723-e9cb-4806-b954-bf03f6b23d9d-monitoring-plugin-cert\") pod \"monitoring-plugin-5d85778b9d-h492n\" (UID: \"511bd723-e9cb-4806-b954-bf03f6b23d9d\") " pod="openshift-monitoring/monitoring-plugin-5d85778b9d-h492n"
Sep 16 18:48:31.206288 ip-10-0-138-127 kubenswrapper[2581]: E0916 18:48:31.206169    2581 secret.go:189] Couldn't get secret openshift-monitoring/monitoring-plugin-cert: secret "monitoring-plugin-cert" not found
Sep 16 18:48:31.206288 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:48:31.206170    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pz2tc\" (UniqueName: \"kubernetes.io/projected/511bd723-e9cb-4806-b954-bf03f6b23d9d-kube-api-access-pz2tc\") pod \"monitoring-plugin-5d85778b9d-h492n\" (UID: \"511bd723-e9cb-4806-b954-bf03f6b23d9d\") " pod="openshift-monitoring/monitoring-plugin-5d85778b9d-h492n"
Sep 16 18:48:31.206288 ip-10-0-138-127 kubenswrapper[2581]: E0916 18:48:31.206256    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/511bd723-e9cb-4806-b954-bf03f6b23d9d-monitoring-plugin-cert podName:511bd723-e9cb-4806-b954-bf03f6b23d9d nodeName:}" failed. No retries permitted until 2026-09-16 18:48:31.70623554 +0000 UTC m=+181.525649447 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "monitoring-plugin-cert" (UniqueName: "kubernetes.io/secret/511bd723-e9cb-4806-b954-bf03f6b23d9d-monitoring-plugin-cert") pod "monitoring-plugin-5d85778b9d-h492n" (UID: "511bd723-e9cb-4806-b954-bf03f6b23d9d") : secret "monitoring-plugin-cert" not found
Sep 16 18:48:31.222409 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:48:31.222376    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pz2tc\" (UniqueName: \"kubernetes.io/projected/511bd723-e9cb-4806-b954-bf03f6b23d9d-kube-api-access-pz2tc\") pod \"monitoring-plugin-5d85778b9d-h492n\" (UID: \"511bd723-e9cb-4806-b954-bf03f6b23d9d\") " pod="openshift-monitoring/monitoring-plugin-5d85778b9d-h492n"
Sep 16 18:48:31.643776 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:48:31.643740    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/telemeter-client-6f6cf8bc4f-phh6w"]
Sep 16 18:48:31.647701 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:48:31.647674    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-6f6cf8bc4f-phh6w"
Sep 16 18:48:31.653276 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:48:31.653253    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"federate-client-certs\""
Sep 16 18:48:31.653655 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:48:31.653614    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-kube-rbac-proxy-config\""
Sep 16 18:48:31.653761 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:48:31.653740    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-dockercfg-db7jd\""
Sep 16 18:48:31.654283 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:48:31.654259    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client\""
Sep 16 18:48:31.654481 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:48:31.654264    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-client-serving-certs-ca-bundle\""
Sep 16 18:48:31.654481 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:48:31.654379    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-tls\""
Sep 16 18:48:31.658693 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:48:31.658672    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-trusted-ca-bundle-8i12ta5c71j38\""
Sep 16 18:48:31.672824 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:48:31.672796    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-6f6cf8bc4f-phh6w"]
Sep 16 18:48:31.711982 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:48:31.711941    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/511bd723-e9cb-4806-b954-bf03f6b23d9d-monitoring-plugin-cert\") pod \"monitoring-plugin-5d85778b9d-h492n\" (UID: \"511bd723-e9cb-4806-b954-bf03f6b23d9d\") " pod="openshift-monitoring/monitoring-plugin-5d85778b9d-h492n"
Sep 16 18:48:31.714898 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:48:31.714869    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/511bd723-e9cb-4806-b954-bf03f6b23d9d-monitoring-plugin-cert\") pod \"monitoring-plugin-5d85778b9d-h492n\" (UID: \"511bd723-e9cb-4806-b954-bf03f6b23d9d\") " pod="openshift-monitoring/monitoring-plugin-5d85778b9d-h492n"
Sep 16 18:48:31.813287 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:48:31.813250    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/2f040fb2-a2fb-432b-ae79-d71cb4c69847-metrics-client-ca\") pod \"telemeter-client-6f6cf8bc4f-phh6w\" (UID: \"2f040fb2-a2fb-432b-ae79-d71cb4c69847\") " pod="openshift-monitoring/telemeter-client-6f6cf8bc4f-phh6w"
Sep 16 18:48:31.813485 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:48:31.813315    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2f040fb2-a2fb-432b-ae79-d71cb4c69847-telemeter-trusted-ca-bundle\") pod \"telemeter-client-6f6cf8bc4f-phh6w\" (UID: \"2f040fb2-a2fb-432b-ae79-d71cb4c69847\") " pod="openshift-monitoring/telemeter-client-6f6cf8bc4f-phh6w"
Sep 16 18:48:31.813485 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:48:31.813368    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/2f040fb2-a2fb-432b-ae79-d71cb4c69847-secret-telemeter-client\") pod \"telemeter-client-6f6cf8bc4f-phh6w\" (UID: \"2f040fb2-a2fb-432b-ae79-d71cb4c69847\") " pod="openshift-monitoring/telemeter-client-6f6cf8bc4f-phh6w"
Sep 16 18:48:31.813485 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:48:31.813393    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/2f040fb2-a2fb-432b-ae79-d71cb4c69847-telemeter-client-tls\") pod \"telemeter-client-6f6cf8bc4f-phh6w\" (UID: \"2f040fb2-a2fb-432b-ae79-d71cb4c69847\") " pod="openshift-monitoring/telemeter-client-6f6cf8bc4f-phh6w"
Sep 16 18:48:31.813485 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:48:31.813442    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/2f040fb2-a2fb-432b-ae79-d71cb4c69847-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-6f6cf8bc4f-phh6w\" (UID: \"2f040fb2-a2fb-432b-ae79-d71cb4c69847\") " pod="openshift-monitoring/telemeter-client-6f6cf8bc4f-phh6w"
Sep 16 18:48:31.813707 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:48:31.813501    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lwjkb\" (UniqueName: \"kubernetes.io/projected/2f040fb2-a2fb-432b-ae79-d71cb4c69847-kube-api-access-lwjkb\") pod \"telemeter-client-6f6cf8bc4f-phh6w\" (UID: \"2f040fb2-a2fb-432b-ae79-d71cb4c69847\") " pod="openshift-monitoring/telemeter-client-6f6cf8bc4f-phh6w"
Sep 16 18:48:31.813707 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:48:31.813566    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/2f040fb2-a2fb-432b-ae79-d71cb4c69847-federate-client-tls\") pod \"telemeter-client-6f6cf8bc4f-phh6w\" (UID: \"2f040fb2-a2fb-432b-ae79-d71cb4c69847\") " pod="openshift-monitoring/telemeter-client-6f6cf8bc4f-phh6w"
Sep 16 18:48:31.813707 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:48:31.813619    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2f040fb2-a2fb-432b-ae79-d71cb4c69847-serving-certs-ca-bundle\") pod \"telemeter-client-6f6cf8bc4f-phh6w\" (UID: \"2f040fb2-a2fb-432b-ae79-d71cb4c69847\") " pod="openshift-monitoring/telemeter-client-6f6cf8bc4f-phh6w"
Sep 16 18:48:31.900509 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:48:31.900405    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-5d85778b9d-h492n"
Sep 16 18:48:31.914872 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:48:31.914837    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2f040fb2-a2fb-432b-ae79-d71cb4c69847-serving-certs-ca-bundle\") pod \"telemeter-client-6f6cf8bc4f-phh6w\" (UID: \"2f040fb2-a2fb-432b-ae79-d71cb4c69847\") " pod="openshift-monitoring/telemeter-client-6f6cf8bc4f-phh6w"
Sep 16 18:48:31.914996 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:48:31.914885    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/2f040fb2-a2fb-432b-ae79-d71cb4c69847-metrics-client-ca\") pod \"telemeter-client-6f6cf8bc4f-phh6w\" (UID: \"2f040fb2-a2fb-432b-ae79-d71cb4c69847\") " pod="openshift-monitoring/telemeter-client-6f6cf8bc4f-phh6w"
Sep 16 18:48:31.914996 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:48:31.914936    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2f040fb2-a2fb-432b-ae79-d71cb4c69847-telemeter-trusted-ca-bundle\") pod \"telemeter-client-6f6cf8bc4f-phh6w\" (UID: \"2f040fb2-a2fb-432b-ae79-d71cb4c69847\") " pod="openshift-monitoring/telemeter-client-6f6cf8bc4f-phh6w"
Sep 16 18:48:31.914996 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:48:31.914974    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/2f040fb2-a2fb-432b-ae79-d71cb4c69847-secret-telemeter-client\") pod \"telemeter-client-6f6cf8bc4f-phh6w\" (UID: \"2f040fb2-a2fb-432b-ae79-d71cb4c69847\") " pod="openshift-monitoring/telemeter-client-6f6cf8bc4f-phh6w"
Sep 16 18:48:31.915156 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:48:31.915004    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/2f040fb2-a2fb-432b-ae79-d71cb4c69847-telemeter-client-tls\") pod \"telemeter-client-6f6cf8bc4f-phh6w\" (UID: \"2f040fb2-a2fb-432b-ae79-d71cb4c69847\") " pod="openshift-monitoring/telemeter-client-6f6cf8bc4f-phh6w"
Sep 16 18:48:31.915156 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:48:31.915046    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/2f040fb2-a2fb-432b-ae79-d71cb4c69847-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-6f6cf8bc4f-phh6w\" (UID: \"2f040fb2-a2fb-432b-ae79-d71cb4c69847\") " pod="openshift-monitoring/telemeter-client-6f6cf8bc4f-phh6w"
Sep 16 18:48:31.915156 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:48:31.915080    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lwjkb\" (UniqueName: \"kubernetes.io/projected/2f040fb2-a2fb-432b-ae79-d71cb4c69847-kube-api-access-lwjkb\") pod \"telemeter-client-6f6cf8bc4f-phh6w\" (UID: \"2f040fb2-a2fb-432b-ae79-d71cb4c69847\") " pod="openshift-monitoring/telemeter-client-6f6cf8bc4f-phh6w"
Sep 16 18:48:31.915156 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:48:31.915120    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/2f040fb2-a2fb-432b-ae79-d71cb4c69847-federate-client-tls\") pod \"telemeter-client-6f6cf8bc4f-phh6w\" (UID: \"2f040fb2-a2fb-432b-ae79-d71cb4c69847\") " pod="openshift-monitoring/telemeter-client-6f6cf8bc4f-phh6w"
Sep 16 18:48:31.915727 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:48:31.915698    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2f040fb2-a2fb-432b-ae79-d71cb4c69847-serving-certs-ca-bundle\") pod \"telemeter-client-6f6cf8bc4f-phh6w\" (UID: \"2f040fb2-a2fb-432b-ae79-d71cb4c69847\") " pod="openshift-monitoring/telemeter-client-6f6cf8bc4f-phh6w"
Sep 16 18:48:31.915914 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:48:31.915864    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2f040fb2-a2fb-432b-ae79-d71cb4c69847-telemeter-trusted-ca-bundle\") pod \"telemeter-client-6f6cf8bc4f-phh6w\" (UID: \"2f040fb2-a2fb-432b-ae79-d71cb4c69847\") " pod="openshift-monitoring/telemeter-client-6f6cf8bc4f-phh6w"
Sep 16 18:48:31.916189 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:48:31.916075    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/2f040fb2-a2fb-432b-ae79-d71cb4c69847-metrics-client-ca\") pod \"telemeter-client-6f6cf8bc4f-phh6w\" (UID: \"2f040fb2-a2fb-432b-ae79-d71cb4c69847\") " pod="openshift-monitoring/telemeter-client-6f6cf8bc4f-phh6w"
Sep 16 18:48:31.918320 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:48:31.918296    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/2f040fb2-a2fb-432b-ae79-d71cb4c69847-federate-client-tls\") pod \"telemeter-client-6f6cf8bc4f-phh6w\" (UID: \"2f040fb2-a2fb-432b-ae79-d71cb4c69847\") " pod="openshift-monitoring/telemeter-client-6f6cf8bc4f-phh6w"
Sep 16 18:48:31.918499 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:48:31.918469    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/2f040fb2-a2fb-432b-ae79-d71cb4c69847-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-6f6cf8bc4f-phh6w\" (UID: \"2f040fb2-a2fb-432b-ae79-d71cb4c69847\") " pod="openshift-monitoring/telemeter-client-6f6cf8bc4f-phh6w"
Sep 16 18:48:31.918600 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:48:31.918531    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/2f040fb2-a2fb-432b-ae79-d71cb4c69847-telemeter-client-tls\") pod \"telemeter-client-6f6cf8bc4f-phh6w\" (UID: \"2f040fb2-a2fb-432b-ae79-d71cb4c69847\") " pod="openshift-monitoring/telemeter-client-6f6cf8bc4f-phh6w"
Sep 16 18:48:31.918651 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:48:31.918623    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/2f040fb2-a2fb-432b-ae79-d71cb4c69847-secret-telemeter-client\") pod \"telemeter-client-6f6cf8bc4f-phh6w\" (UID: \"2f040fb2-a2fb-432b-ae79-d71cb4c69847\") " pod="openshift-monitoring/telemeter-client-6f6cf8bc4f-phh6w"
Sep 16 18:48:31.924967 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:48:31.924940    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lwjkb\" (UniqueName: \"kubernetes.io/projected/2f040fb2-a2fb-432b-ae79-d71cb4c69847-kube-api-access-lwjkb\") pod \"telemeter-client-6f6cf8bc4f-phh6w\" (UID: \"2f040fb2-a2fb-432b-ae79-d71cb4c69847\") " pod="openshift-monitoring/telemeter-client-6f6cf8bc4f-phh6w"
Sep 16 18:48:31.960389 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:48:31.960351    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-6f6cf8bc4f-phh6w"
Sep 16 18:48:32.681653 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:48:32.681611    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 16 18:48:32.685796 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:48:32.685766    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:48:32.695567 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:48:32.695540    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\""
Sep 16 18:48:32.695732 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:48:32.695655    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\""
Sep 16 18:48:32.695882 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:48:32.695864    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\""
Sep 16 18:48:32.695994 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:48:32.695889    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-828mq\""
Sep 16 18:48:32.695994 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:48:32.695954    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\""
Sep 16 18:48:32.696623 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:48:32.696604    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\""
Sep 16 18:48:32.696788 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:48:32.696724    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-6ijsm9j35gbld\""
Sep 16 18:48:32.696861 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:48:32.696832    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\""
Sep 16 18:48:32.706294 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:48:32.706039    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\""
Sep 16 18:48:32.706294 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:48:32.706060    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\""
Sep 16 18:48:32.706294 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:48:32.706071    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\""
Sep 16 18:48:32.706294 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:48:32.706104    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\""
Sep 16 18:48:32.706294 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:48:32.706131    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\""
Sep 16 18:48:32.706294 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:48:32.706140    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\""
Sep 16 18:48:32.706623 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:48:32.706392    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\""
Sep 16 18:48:32.706623 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:48:32.706427    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\""
Sep 16 18:48:32.730878 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:48:32.730846    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 16 18:48:32.823047 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:48:32.823005    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/33280f86-4497-4343-829b-4781cbfd5e5f-web-config\") pod \"prometheus-k8s-0\" (UID: \"33280f86-4497-4343-829b-4781cbfd5e5f\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:48:32.823273 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:48:32.823060    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/33280f86-4497-4343-829b-4781cbfd5e5f-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"33280f86-4497-4343-829b-4781cbfd5e5f\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:48:32.823273 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:48:32.823157    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/33280f86-4497-4343-829b-4781cbfd5e5f-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"33280f86-4497-4343-829b-4781cbfd5e5f\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:48:32.823273 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:48:32.823220    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/33280f86-4497-4343-829b-4781cbfd5e5f-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"33280f86-4497-4343-829b-4781cbfd5e5f\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:48:32.823273 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:48:32.823247    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/33280f86-4497-4343-829b-4781cbfd5e5f-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"33280f86-4497-4343-829b-4781cbfd5e5f\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:48:32.823431 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:48:32.823274    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/33280f86-4497-4343-829b-4781cbfd5e5f-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"33280f86-4497-4343-829b-4781cbfd5e5f\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:48:32.823431 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:48:32.823386    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/33280f86-4497-4343-829b-4781cbfd5e5f-config\") pod \"prometheus-k8s-0\" (UID: \"33280f86-4497-4343-829b-4781cbfd5e5f\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:48:32.823431 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:48:32.823416    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/33280f86-4497-4343-829b-4781cbfd5e5f-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"33280f86-4497-4343-829b-4781cbfd5e5f\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:48:32.823559 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:48:32.823436    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/33280f86-4497-4343-829b-4781cbfd5e5f-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"33280f86-4497-4343-829b-4781cbfd5e5f\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:48:32.823559 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:48:32.823469    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/33280f86-4497-4343-829b-4781cbfd5e5f-config-out\") pod \"prometheus-k8s-0\" (UID: \"33280f86-4497-4343-829b-4781cbfd5e5f\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:48:32.823559 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:48:32.823490    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/33280f86-4497-4343-829b-4781cbfd5e5f-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"33280f86-4497-4343-829b-4781cbfd5e5f\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:48:32.823559 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:48:32.823545    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/33280f86-4497-4343-829b-4781cbfd5e5f-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"33280f86-4497-4343-829b-4781cbfd5e5f\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:48:32.823715 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:48:32.823585    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/33280f86-4497-4343-829b-4781cbfd5e5f-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"33280f86-4497-4343-829b-4781cbfd5e5f\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:48:32.823715 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:48:32.823613    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/33280f86-4497-4343-829b-4781cbfd5e5f-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"33280f86-4497-4343-829b-4781cbfd5e5f\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:48:32.823715 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:48:32.823650    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/33280f86-4497-4343-829b-4781cbfd5e5f-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"33280f86-4497-4343-829b-4781cbfd5e5f\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:48:32.823715 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:48:32.823679    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/33280f86-4497-4343-829b-4781cbfd5e5f-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"33280f86-4497-4343-829b-4781cbfd5e5f\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:48:32.823877 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:48:32.823725    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/33280f86-4497-4343-829b-4781cbfd5e5f-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"33280f86-4497-4343-829b-4781cbfd5e5f\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:48:32.823877 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:48:32.823765    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/33280f86-4497-4343-829b-4781cbfd5e5f-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"33280f86-4497-4343-829b-4781cbfd5e5f\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:48:32.823877 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:48:32.823800    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9nk74\" (UniqueName: \"kubernetes.io/projected/33280f86-4497-4343-829b-4781cbfd5e5f-kube-api-access-9nk74\") pod \"prometheus-k8s-0\" (UID: \"33280f86-4497-4343-829b-4781cbfd5e5f\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:48:32.823877 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:48:32.823824    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/33280f86-4497-4343-829b-4781cbfd5e5f-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"33280f86-4497-4343-829b-4781cbfd5e5f\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:48:32.924913 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:48:32.924872    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/33280f86-4497-4343-829b-4781cbfd5e5f-web-config\") pod \"prometheus-k8s-0\" (UID: \"33280f86-4497-4343-829b-4781cbfd5e5f\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:48:32.925113 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:48:32.924939    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/33280f86-4497-4343-829b-4781cbfd5e5f-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"33280f86-4497-4343-829b-4781cbfd5e5f\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:48:32.925113 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:48:32.924988    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/33280f86-4497-4343-829b-4781cbfd5e5f-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"33280f86-4497-4343-829b-4781cbfd5e5f\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:48:32.925113 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:48:32.925016    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/33280f86-4497-4343-829b-4781cbfd5e5f-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"33280f86-4497-4343-829b-4781cbfd5e5f\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:48:32.925294 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:48:32.925260    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/33280f86-4497-4343-829b-4781cbfd5e5f-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"33280f86-4497-4343-829b-4781cbfd5e5f\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:48:32.925352 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:48:32.925297    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/33280f86-4497-4343-829b-4781cbfd5e5f-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"33280f86-4497-4343-829b-4781cbfd5e5f\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:48:32.925396 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:48:32.925373    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/33280f86-4497-4343-829b-4781cbfd5e5f-config\") pod \"prometheus-k8s-0\" (UID: \"33280f86-4497-4343-829b-4781cbfd5e5f\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:48:32.925448 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:48:32.925401    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/33280f86-4497-4343-829b-4781cbfd5e5f-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"33280f86-4497-4343-829b-4781cbfd5e5f\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:48:32.925448 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:48:32.925424    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/33280f86-4497-4343-829b-4781cbfd5e5f-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"33280f86-4497-4343-829b-4781cbfd5e5f\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:48:32.925547 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:48:32.925464    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/33280f86-4497-4343-829b-4781cbfd5e5f-config-out\") pod \"prometheus-k8s-0\" (UID: \"33280f86-4497-4343-829b-4781cbfd5e5f\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:48:32.925547 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:48:32.925489    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/33280f86-4497-4343-829b-4781cbfd5e5f-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"33280f86-4497-4343-829b-4781cbfd5e5f\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:48:32.925547 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:48:32.925514    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/33280f86-4497-4343-829b-4781cbfd5e5f-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"33280f86-4497-4343-829b-4781cbfd5e5f\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:48:32.925547 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:48:32.925539    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/33280f86-4497-4343-829b-4781cbfd5e5f-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"33280f86-4497-4343-829b-4781cbfd5e5f\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:48:32.925732 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:48:32.925565    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/33280f86-4497-4343-829b-4781cbfd5e5f-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"33280f86-4497-4343-829b-4781cbfd5e5f\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:48:32.925732 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:48:32.925595    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/33280f86-4497-4343-829b-4781cbfd5e5f-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"33280f86-4497-4343-829b-4781cbfd5e5f\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:48:32.925732 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:48:32.925623    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/33280f86-4497-4343-829b-4781cbfd5e5f-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"33280f86-4497-4343-829b-4781cbfd5e5f\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:48:32.925732 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:48:32.925640    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/33280f86-4497-4343-829b-4781cbfd5e5f-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"33280f86-4497-4343-829b-4781cbfd5e5f\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:48:32.925732 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:48:32.925660    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/33280f86-4497-4343-829b-4781cbfd5e5f-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"33280f86-4497-4343-829b-4781cbfd5e5f\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:48:32.925732 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:48:32.925687    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/33280f86-4497-4343-829b-4781cbfd5e5f-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"33280f86-4497-4343-829b-4781cbfd5e5f\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:48:32.925732 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:48:32.925692    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/33280f86-4497-4343-829b-4781cbfd5e5f-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"33280f86-4497-4343-829b-4781cbfd5e5f\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:48:32.925732 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:48:32.925718    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9nk74\" (UniqueName: \"kubernetes.io/projected/33280f86-4497-4343-829b-4781cbfd5e5f-kube-api-access-9nk74\") pod \"prometheus-k8s-0\" (UID: \"33280f86-4497-4343-829b-4781cbfd5e5f\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:48:32.926042 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:48:32.925736    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/33280f86-4497-4343-829b-4781cbfd5e5f-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"33280f86-4497-4343-829b-4781cbfd5e5f\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:48:32.926042 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:48:32.925744    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/33280f86-4497-4343-829b-4781cbfd5e5f-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"33280f86-4497-4343-829b-4781cbfd5e5f\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:48:32.926260 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:48:32.926227    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/33280f86-4497-4343-829b-4781cbfd5e5f-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"33280f86-4497-4343-829b-4781cbfd5e5f\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:48:32.928142 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:48:32.927232    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/33280f86-4497-4343-829b-4781cbfd5e5f-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"33280f86-4497-4343-829b-4781cbfd5e5f\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:48:32.928142 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:48:32.927849    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/33280f86-4497-4343-829b-4781cbfd5e5f-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"33280f86-4497-4343-829b-4781cbfd5e5f\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:48:32.928950 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:48:32.928435    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/33280f86-4497-4343-829b-4781cbfd5e5f-web-config\") pod \"prometheus-k8s-0\" (UID: \"33280f86-4497-4343-829b-4781cbfd5e5f\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:48:32.928950 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:48:32.928608    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/33280f86-4497-4343-829b-4781cbfd5e5f-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"33280f86-4497-4343-829b-4781cbfd5e5f\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:48:32.928950 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:48:32.928817    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/33280f86-4497-4343-829b-4781cbfd5e5f-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"33280f86-4497-4343-829b-4781cbfd5e5f\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:48:32.929151 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:48:32.929016    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/33280f86-4497-4343-829b-4781cbfd5e5f-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"33280f86-4497-4343-829b-4781cbfd5e5f\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:48:32.929272 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:48:32.929254    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/33280f86-4497-4343-829b-4781cbfd5e5f-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"33280f86-4497-4343-829b-4781cbfd5e5f\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:48:32.929352 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:48:32.929329    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/33280f86-4497-4343-829b-4781cbfd5e5f-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"33280f86-4497-4343-829b-4781cbfd5e5f\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:48:32.929861 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:48:32.929785    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/33280f86-4497-4343-829b-4781cbfd5e5f-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"33280f86-4497-4343-829b-4781cbfd5e5f\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:48:32.930747 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:48:32.930720    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/33280f86-4497-4343-829b-4781cbfd5e5f-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"33280f86-4497-4343-829b-4781cbfd5e5f\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:48:32.931105 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:48:32.931069    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/33280f86-4497-4343-829b-4781cbfd5e5f-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"33280f86-4497-4343-829b-4781cbfd5e5f\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:48:32.931497 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:48:32.931477    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/33280f86-4497-4343-829b-4781cbfd5e5f-config-out\") pod \"prometheus-k8s-0\" (UID: \"33280f86-4497-4343-829b-4781cbfd5e5f\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:48:32.932090 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:48:32.932015    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/33280f86-4497-4343-829b-4781cbfd5e5f-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"33280f86-4497-4343-829b-4781cbfd5e5f\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:48:32.932474 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:48:32.932424    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/33280f86-4497-4343-829b-4781cbfd5e5f-config\") pod \"prometheus-k8s-0\" (UID: \"33280f86-4497-4343-829b-4781cbfd5e5f\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:48:32.932667 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:48:32.932646    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/33280f86-4497-4343-829b-4781cbfd5e5f-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"33280f86-4497-4343-829b-4781cbfd5e5f\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:48:32.938109 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:48:32.938085    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9nk74\" (UniqueName: \"kubernetes.io/projected/33280f86-4497-4343-829b-4781cbfd5e5f-kube-api-access-9nk74\") pod \"prometheus-k8s-0\" (UID: \"33280f86-4497-4343-829b-4781cbfd5e5f\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:48:32.997620 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:48:32.997582    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:48:34.595254 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:48:34.595216    2581 patch_prober.go:28] interesting pod/image-registry-649b4fc7ff-zmk84 container/registry namespace/openshift-image-registry: Liveness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body={"errors":[{"code":"UNAVAILABLE","message":"service unavailable","detail":"health check failed: please see /debug/health"}]}
Sep 16 18:48:34.595655 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:48:34.595281    2581 prober.go:120] "Probe failed" probeType="Liveness" pod="openshift-image-registry/image-registry-649b4fc7ff-zmk84" podUID="57ce0be5-62d9-473f-a19f-4253cb333fe2" containerName="registry" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 16 18:48:35.716555 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:48:35.716525    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-5d85778b9d-h492n"]
Sep 16 18:48:35.719948 ip-10-0-138-127 kubenswrapper[2581]: W0916 18:48:35.719849    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod511bd723_e9cb_4806_b954_bf03f6b23d9d.slice/crio-dbbc8f6a6eea6da03077bad019d178723b58c378faa4c665d439c47d27fe5a8a WatchSource:0}: Error finding container dbbc8f6a6eea6da03077bad019d178723b58c378faa4c665d439c47d27fe5a8a: Status 404 returned error can't find the container with id dbbc8f6a6eea6da03077bad019d178723b58c378faa4c665d439c47d27fe5a8a
Sep 16 18:48:35.722730 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:48:35.722637    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-5c9d8b5484-7hpps"]
Sep 16 18:48:35.723855 ip-10-0-138-127 kubenswrapper[2581]: W0916 18:48:35.723829    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddf30013a_07d7_491e_b50a_4c208c6ce24d.slice/crio-2906622005f134ca25d104fc52888fdb43cf02b225754f6491c51fedda18a53c WatchSource:0}: Error finding container 2906622005f134ca25d104fc52888fdb43cf02b225754f6491c51fedda18a53c: Status 404 returned error can't find the container with id 2906622005f134ca25d104fc52888fdb43cf02b225754f6491c51fedda18a53c
Sep 16 18:48:35.724386 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:48:35.724369    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-56fb5c8694-qcn8h"]
Sep 16 18:48:35.725839 ip-10-0-138-127 kubenswrapper[2581]: W0916 18:48:35.725739    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5bb743b0_66c4_44c0_bd47_b99cbaa2f24e.slice/crio-48c777457f2a10ae515b0243e9f10380bc556b748caaffb867212d11e3afbd67 WatchSource:0}: Error finding container 48c777457f2a10ae515b0243e9f10380bc556b748caaffb867212d11e3afbd67: Status 404 returned error can't find the container with id 48c777457f2a10ae515b0243e9f10380bc556b748caaffb867212d11e3afbd67
Sep 16 18:48:35.733698 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:48:35.733672    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 16 18:48:35.737852 ip-10-0-138-127 kubenswrapper[2581]: W0916 18:48:35.737823    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod33280f86_4497_4343_829b_4781cbfd5e5f.slice/crio-b4b25c7a54e34f336cf21f069c208a6bf0973b30aca905dc2a5dced1936c5302 WatchSource:0}: Error finding container b4b25c7a54e34f336cf21f069c208a6bf0973b30aca905dc2a5dced1936c5302: Status 404 returned error can't find the container with id b4b25c7a54e34f336cf21f069c208a6bf0973b30aca905dc2a5dced1936c5302
Sep 16 18:48:35.760128 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:48:35.759022    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-6f6cf8bc4f-phh6w"]
Sep 16 18:48:35.768594 ip-10-0-138-127 kubenswrapper[2581]: W0916 18:48:35.768536    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2f040fb2_a2fb_432b_ae79_d71cb4c69847.slice/crio-df2f4e775f6352b15bef93a8bc79abfcff1eee6ca09a3178129b285ff2f740b9 WatchSource:0}: Error finding container df2f4e775f6352b15bef93a8bc79abfcff1eee6ca09a3178129b285ff2f740b9: Status 404 returned error can't find the container with id df2f4e775f6352b15bef93a8bc79abfcff1eee6ca09a3178129b285ff2f740b9
Sep 16 18:48:36.410766 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:48:36.410512    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-649b4fc7ff-zmk84"
Sep 16 18:48:36.547478 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:48:36.547436    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-4hdsw" event={"ID":"dc53a5ab-d957-42f4-9eb7-a3c38c3ca302","Type":"ContainerStarted","Data":"19955a65c4c7781061add66dc64e6cdd4cd7cacfcc115a203b050f8f3b92a2e8"}
Sep 16 18:48:36.550160 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:48:36.550119    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-5d85778b9d-h492n" event={"ID":"511bd723-e9cb-4806-b954-bf03f6b23d9d","Type":"ContainerStarted","Data":"dbbc8f6a6eea6da03077bad019d178723b58c378faa4c665d439c47d27fe5a8a"}
Sep 16 18:48:36.553163 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:48:36.553128    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-76f99584f4-67c8w" event={"ID":"ddcab39a-92a7-4106-927a-8160c999373c","Type":"ContainerStarted","Data":"7d9f7ef0a0fb7b3fb81f934a11da8ef8ef2d7a95c545fd275d7f62ed8b498f5a"}
Sep 16 18:48:36.553392 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:48:36.553372    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/downloads-76f99584f4-67c8w"
Sep 16 18:48:36.556419 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:48:36.556390    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-6f6cf8bc4f-phh6w" event={"ID":"2f040fb2-a2fb-432b-ae79-d71cb4c69847","Type":"ContainerStarted","Data":"df2f4e775f6352b15bef93a8bc79abfcff1eee6ca09a3178129b285ff2f740b9"}
Sep 16 18:48:36.559237 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:48:36.559047    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-5c9d8b5484-7hpps" event={"ID":"df30013a-07d7-491e-b50a-4c208c6ce24d","Type":"ContainerStarted","Data":"2906622005f134ca25d104fc52888fdb43cf02b225754f6491c51fedda18a53c"}
Sep 16 18:48:36.561688 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:48:36.561664    2581 generic.go:356] "Generic (PLEG): container finished" podID="89fc85f4-ec94-4459-8311-f3dbf4681bf8" containerID="533a951e6971354aecdc7e706995006228615b0302070d4796da3b7ee87b2e85" exitCode=0
Sep 16 18:48:36.562814 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:48:36.561800    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"89fc85f4-ec94-4459-8311-f3dbf4681bf8","Type":"ContainerDied","Data":"533a951e6971354aecdc7e706995006228615b0302070d4796da3b7ee87b2e85"}
Sep 16 18:48:36.571283 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:48:36.570730    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-4hdsw" podStartSLOduration=2.365142888 podStartE2EDuration="10.570708775s" totalImagesPullingTime="8.205565887s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:48:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:48:36.569400121 +0000 UTC m=+186.388814032" watchObservedRunningTime="2026-09-16 18:48:36.570708775 +0000 UTC m=+186.390122685"
Sep 16 18:48:36.573783 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:48:36.573713    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-hsp8v" event={"ID":"b5732a85-64c1-4cf8-b80a-c06f8d0a4b3b","Type":"ContainerStarted","Data":"8f05fd82fcbff56b8ee7d89a876f009d0030c6300c6c82c88b894d5c9313479c"}
Sep 16 18:48:36.574113 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:48:36.573959    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-hsp8v" event={"ID":"b5732a85-64c1-4cf8-b80a-c06f8d0a4b3b","Type":"ContainerStarted","Data":"6e9421760fa40a3eee86d0bbc61c83f7539a239f589f619271c1af1a051a9548"}
Sep 16 18:48:36.575532 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:48:36.575378    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/downloads-76f99584f4-67c8w"
Sep 16 18:48:36.576638 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:48:36.576414    2581 generic.go:356] "Generic (PLEG): container finished" podID="33280f86-4497-4343-829b-4781cbfd5e5f" containerID="b247a13bc56273cb77128be0df26ac1bd41f05879d206a7acaeb33fd20a62d0b" exitCode=0
Sep 16 18:48:36.576638 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:48:36.576501    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"33280f86-4497-4343-829b-4781cbfd5e5f","Type":"ContainerDied","Data":"b247a13bc56273cb77128be0df26ac1bd41f05879d206a7acaeb33fd20a62d0b"}
Sep 16 18:48:36.576638 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:48:36.576559    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"33280f86-4497-4343-829b-4781cbfd5e5f","Type":"ContainerStarted","Data":"b4b25c7a54e34f336cf21f069c208a6bf0973b30aca905dc2a5dced1936c5302"}
Sep 16 18:48:36.586901 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:48:36.584022    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-56fb5c8694-qcn8h" event={"ID":"5bb743b0-66c4-44c0-bd47-b99cbaa2f24e","Type":"ContainerStarted","Data":"e0eec99ce99a62ea9fdcb3f7f621bd18dbef0a192d71f4641a8c57a8e27ef691"}
Sep 16 18:48:36.586901 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:48:36.585031    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-56fb5c8694-qcn8h" event={"ID":"5bb743b0-66c4-44c0-bd47-b99cbaa2f24e","Type":"ContainerStarted","Data":"48c777457f2a10ae515b0243e9f10380bc556b748caaffb867212d11e3afbd67"}
Sep 16 18:48:36.632470 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:48:36.630711    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/downloads-76f99584f4-67c8w" podStartSLOduration=3.104092955 podStartE2EDuration="21.630689058s" totalImagesPullingTime="18.526596103s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:48:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:48:36.629461592 +0000 UTC m=+186.448875514" watchObservedRunningTime="2026-09-16 18:48:36.630689058 +0000 UTC m=+186.450102968"
Sep 16 18:48:36.745647 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:48:36.745558    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-56fb5c8694-qcn8h" podStartSLOduration=6.74553236 podStartE2EDuration="6.74553236s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:48:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:48:36.720448204 +0000 UTC m=+186.539862113" watchObservedRunningTime="2026-09-16 18:48:36.74553236 +0000 UTC m=+186.564946276"
Sep 16 18:48:36.748119 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:48:36.746755    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-hsp8v" podStartSLOduration=9.722443545 podStartE2EDuration="10.746738566s" totalImagesPullingTime="940.018492ms" totalInitContainerRuntime="84.276529ms" isStatefulPod=true podCreationTimestamp="2026-09-16 18:48:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:48:36.743866743 +0000 UTC m=+186.563280649" watchObservedRunningTime="2026-09-16 18:48:36.746738566 +0000 UTC m=+186.566152482"
Sep 16 18:48:38.001222 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:48:38.001046    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-649b4fc7ff-zmk84"]
Sep 16 18:48:38.908044 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:48:38.907991    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-6456555d6b-r87j6"]
Sep 16 18:48:40.419105 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:48:40.419034    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-56fb5c8694-qcn8h"
Sep 16 18:48:40.419105 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:48:40.419128    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-56fb5c8694-qcn8h"
Sep 16 18:48:40.425799 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:48:40.425580    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-56fb5c8694-qcn8h"
Sep 16 18:48:40.609627 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:48:40.609596    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-56fb5c8694-qcn8h"
Sep 16 18:48:43.628921 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:48:43.628812    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-5d85778b9d-h492n" event={"ID":"511bd723-e9cb-4806-b954-bf03f6b23d9d","Type":"ContainerStarted","Data":"f8f9f0b9aa64eff51022ef6492643140896ff70b9fbdca1d9bd9998dd51bac6a"}
Sep 16 18:48:43.629441 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:48:43.628972    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/monitoring-plugin-5d85778b9d-h492n"
Sep 16 18:48:43.632575 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:48:43.632541    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-6f6cf8bc4f-phh6w" event={"ID":"2f040fb2-a2fb-432b-ae79-d71cb4c69847","Type":"ContainerStarted","Data":"e25fff7e6caf96b42e715c08902dd07a28b635242ee281735671b5c18f97b785"}
Sep 16 18:48:43.632735 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:48:43.632607    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-6f6cf8bc4f-phh6w" event={"ID":"2f040fb2-a2fb-432b-ae79-d71cb4c69847","Type":"ContainerStarted","Data":"f5ebef0b84602edd857c92aa3aeaa5b77a5d25a853bfeea127be41f7ae1706e6"}
Sep 16 18:48:43.632735 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:48:43.632635    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-6f6cf8bc4f-phh6w" event={"ID":"2f040fb2-a2fb-432b-ae79-d71cb4c69847","Type":"ContainerStarted","Data":"7e01767db1bd4e8d02574ea81e8c8d196b19d4c6bb4747fa15becd4e652b1748"}
Sep 16 18:48:43.634509 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:48:43.634475    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-5c9d8b5484-7hpps" event={"ID":"df30013a-07d7-491e-b50a-4c208c6ce24d","Type":"ContainerStarted","Data":"df3fa4ca80376cb244a79478ff3aea585a438df53d54e70afdac7b7c4618d64f"}
Sep 16 18:48:43.635562 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:48:43.635541    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/monitoring-plugin-5d85778b9d-h492n"
Sep 16 18:48:43.638543 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:48:43.638515    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"89fc85f4-ec94-4459-8311-f3dbf4681bf8","Type":"ContainerStarted","Data":"8d7b8e7f9e6b1851ac6b5e6beb3beadd022252cd070b853e77209f3d532e0025"}
Sep 16 18:48:43.638652 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:48:43.638589    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"89fc85f4-ec94-4459-8311-f3dbf4681bf8","Type":"ContainerStarted","Data":"45a58d97808bbe7579fb16590a57b26a5743ac74ecfe45ada969fc1685ed05d7"}
Sep 16 18:48:43.638652 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:48:43.638634    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"89fc85f4-ec94-4459-8311-f3dbf4681bf8","Type":"ContainerStarted","Data":"b95ace2ac8e0f37bc18c57982ae4b718ba41df9190333b9656adde5cdfbac64c"}
Sep 16 18:48:43.638768 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:48:43.638675    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"89fc85f4-ec94-4459-8311-f3dbf4681bf8","Type":"ContainerStarted","Data":"bef7df271946c5737d62c61b1cf3e85e1e9e4fd5970c843ad20c77bd5c32a57a"}
Sep 16 18:48:43.638768 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:48:43.638716    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"89fc85f4-ec94-4459-8311-f3dbf4681bf8","Type":"ContainerStarted","Data":"f4663becc1918a7e6924c74d9a68748b56439cd09776d57e9ab09ca5fff6bf04"}
Sep 16 18:48:43.641292 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:48:43.641267    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"33280f86-4497-4343-829b-4781cbfd5e5f","Type":"ContainerStarted","Data":"fe8880e1d60a0122daff9a84c8c263336a34080fa9d654bae40ab0456d777281"}
Sep 16 18:48:43.641391 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:48:43.641336    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"33280f86-4497-4343-829b-4781cbfd5e5f","Type":"ContainerStarted","Data":"83ac526d7a86604d68fe8d92bf00597bc299f57b92d72f39a7fb68e890df4232"}
Sep 16 18:48:43.646445 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:48:43.646395    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/monitoring-plugin-5d85778b9d-h492n" podStartSLOduration=6.75259725 podStartE2EDuration="13.6463797s" totalImagesPullingTime="6.89378245s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:48:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:48:43.646222816 +0000 UTC m=+193.465636716" watchObservedRunningTime="2026-09-16 18:48:43.6463797 +0000 UTC m=+193.465793611"
Sep 16 18:48:43.675354 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:48:43.675293    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/telemeter-client-6f6cf8bc4f-phh6w" podStartSLOduration=5.807721115 podStartE2EDuration="12.675275076s" totalImagesPullingTime="6.867553961s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:48:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:48:43.672980689 +0000 UTC m=+193.492394601" watchObservedRunningTime="2026-09-16 18:48:43.675275076 +0000 UTC m=+193.494688985"
Sep 16 18:48:43.705272 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:48:43.705207    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/metrics-server-5c9d8b5484-7hpps" podStartSLOduration=6.791188026 podStartE2EDuration="13.705172662s" totalImagesPullingTime="6.913984636s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:48:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:48:43.703150284 +0000 UTC m=+193.522564194" watchObservedRunningTime="2026-09-16 18:48:43.705172662 +0000 UTC m=+193.524586570"
Sep 16 18:48:46.659522 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:48:46.659488    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"89fc85f4-ec94-4459-8311-f3dbf4681bf8","Type":"ContainerStarted","Data":"1a89a9e7a2375bf34bb12157b14aa09a24a45dab3add18d098c5dcb6c5c0a79a"}
Sep 16 18:48:46.662750 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:48:46.662720    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"33280f86-4497-4343-829b-4781cbfd5e5f","Type":"ContainerStarted","Data":"3f3dd1883e45a82010062071dbb5a547b0398022c662e0974bb6035db93c527b"}
Sep 16 18:48:46.662868 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:48:46.662793    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"33280f86-4497-4343-829b-4781cbfd5e5f","Type":"ContainerStarted","Data":"631c5ce2aac8760ec03de8d2609386e5bd65a6ab1a2ff3a684b3cc1c0d53f5a5"}
Sep 16 18:48:46.662868 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:48:46.662834    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"33280f86-4497-4343-829b-4781cbfd5e5f","Type":"ContainerStarted","Data":"314e4f5337c0ea1e4e5e6b86e35ae4ee43a7775322585a4488852c09df93923f"}
Sep 16 18:48:46.662936 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:48:46.662881    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"33280f86-4497-4343-829b-4781cbfd5e5f","Type":"ContainerStarted","Data":"f3007c04a6c7666ea52a294ed272da999f867f7ef844acc86315cb2a8cf1bec2"}
Sep 16 18:48:46.690911 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:48:46.690830    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/alertmanager-main-0" podStartSLOduration=3.281301767 podStartE2EDuration="19.690807735s" totalImagesPullingTime="16.380184661s" totalInitContainerRuntime="29.321307ms" isStatefulPod=false podCreationTimestamp="2026-09-16 18:48:27 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:48:46.689652218 +0000 UTC m=+196.509066127" watchObservedRunningTime="2026-09-16 18:48:46.690807735 +0000 UTC m=+196.510221643"
Sep 16 18:48:46.723994 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:48:46.723931    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-k8s-0" podStartSLOduration=5.235809228 podStartE2EDuration="14.723913314s" totalImagesPullingTime="9.461557366s" totalInitContainerRuntime="26.54672ms" isStatefulPod=false podCreationTimestamp="2026-09-16 18:48:32 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:48:46.72209152 +0000 UTC m=+196.541505429" watchObservedRunningTime="2026-09-16 18:48:46.723913314 +0000 UTC m=+196.543327222"
Sep 16 18:48:47.998373 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:48:47.998322    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:48:51.170727 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:48:51.170689    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/metrics-server-5c9d8b5484-7hpps"
Sep 16 18:48:51.171129 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:48:51.170765    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/metrics-server-5c9d8b5484-7hpps"
Sep 16 18:48:55.162293 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:48:55.162258    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-56fb5c8694-qcn8h"]
Sep 16 18:49:03.033564 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:49:03.033491    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-image-registry/image-registry-649b4fc7ff-zmk84" podUID="57ce0be5-62d9-473f-a19f-4253cb333fe2" containerName="registry" containerID="cri-o://4bb6224a269aee88f1547d6fce9f13932ef1f10341445cc66c6c87ee3eceec6c" gracePeriod=30
Sep 16 18:49:03.314779 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:49:03.314747    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-649b4fc7ff-zmk84"
Sep 16 18:49:03.458461 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:49:03.458411    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/57ce0be5-62d9-473f-a19f-4253cb333fe2-image-registry-private-configuration\") pod \"57ce0be5-62d9-473f-a19f-4253cb333fe2\" (UID: \"57ce0be5-62d9-473f-a19f-4253cb333fe2\") "
Sep 16 18:49:03.458686 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:49:03.458477    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6xnhc\" (UniqueName: \"kubernetes.io/projected/57ce0be5-62d9-473f-a19f-4253cb333fe2-kube-api-access-6xnhc\") pod \"57ce0be5-62d9-473f-a19f-4253cb333fe2\" (UID: \"57ce0be5-62d9-473f-a19f-4253cb333fe2\") "
Sep 16 18:49:03.458686 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:49:03.458510    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/57ce0be5-62d9-473f-a19f-4253cb333fe2-ca-trust-extracted\") pod \"57ce0be5-62d9-473f-a19f-4253cb333fe2\" (UID: \"57ce0be5-62d9-473f-a19f-4253cb333fe2\") "
Sep 16 18:49:03.458686 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:49:03.458556    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/57ce0be5-62d9-473f-a19f-4253cb333fe2-trusted-ca\") pod \"57ce0be5-62d9-473f-a19f-4253cb333fe2\" (UID: \"57ce0be5-62d9-473f-a19f-4253cb333fe2\") "
Sep 16 18:49:03.458686 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:49:03.458609    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/57ce0be5-62d9-473f-a19f-4253cb333fe2-registry-tls\") pod \"57ce0be5-62d9-473f-a19f-4253cb333fe2\" (UID: \"57ce0be5-62d9-473f-a19f-4253cb333fe2\") "
Sep 16 18:49:03.458686 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:49:03.458644    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/57ce0be5-62d9-473f-a19f-4253cb333fe2-installation-pull-secrets\") pod \"57ce0be5-62d9-473f-a19f-4253cb333fe2\" (UID: \"57ce0be5-62d9-473f-a19f-4253cb333fe2\") "
Sep 16 18:49:03.458928 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:49:03.458690    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/57ce0be5-62d9-473f-a19f-4253cb333fe2-registry-certificates\") pod \"57ce0be5-62d9-473f-a19f-4253cb333fe2\" (UID: \"57ce0be5-62d9-473f-a19f-4253cb333fe2\") "
Sep 16 18:49:03.458928 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:49:03.458744    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/57ce0be5-62d9-473f-a19f-4253cb333fe2-bound-sa-token\") pod \"57ce0be5-62d9-473f-a19f-4253cb333fe2\" (UID: \"57ce0be5-62d9-473f-a19f-4253cb333fe2\") "
Sep 16 18:49:03.459134 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:49:03.459101    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/57ce0be5-62d9-473f-a19f-4253cb333fe2-trusted-ca" (OuterVolumeSpecName: "trusted-ca") pod "57ce0be5-62d9-473f-a19f-4253cb333fe2" (UID: "57ce0be5-62d9-473f-a19f-4253cb333fe2"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 18:49:03.459134 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:49:03.459120    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/57ce0be5-62d9-473f-a19f-4253cb333fe2-registry-certificates" (OuterVolumeSpecName: "registry-certificates") pod "57ce0be5-62d9-473f-a19f-4253cb333fe2" (UID: "57ce0be5-62d9-473f-a19f-4253cb333fe2"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 18:49:03.461629 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:49:03.461601    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/57ce0be5-62d9-473f-a19f-4253cb333fe2-kube-api-access-6xnhc" (OuterVolumeSpecName: "kube-api-access-6xnhc") pod "57ce0be5-62d9-473f-a19f-4253cb333fe2" (UID: "57ce0be5-62d9-473f-a19f-4253cb333fe2"). InnerVolumeSpecName "kube-api-access-6xnhc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 18:49:03.461755 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:49:03.461603    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/57ce0be5-62d9-473f-a19f-4253cb333fe2-installation-pull-secrets" (OuterVolumeSpecName: "installation-pull-secrets") pod "57ce0be5-62d9-473f-a19f-4253cb333fe2" (UID: "57ce0be5-62d9-473f-a19f-4253cb333fe2"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 18:49:03.461755 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:49:03.461709    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/57ce0be5-62d9-473f-a19f-4253cb333fe2-image-registry-private-configuration" (OuterVolumeSpecName: "image-registry-private-configuration") pod "57ce0be5-62d9-473f-a19f-4253cb333fe2" (UID: "57ce0be5-62d9-473f-a19f-4253cb333fe2"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 18:49:03.461755 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:49:03.461737    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/57ce0be5-62d9-473f-a19f-4253cb333fe2-bound-sa-token" (OuterVolumeSpecName: "bound-sa-token") pod "57ce0be5-62d9-473f-a19f-4253cb333fe2" (UID: "57ce0be5-62d9-473f-a19f-4253cb333fe2"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 18:49:03.461917 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:49:03.461750    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/57ce0be5-62d9-473f-a19f-4253cb333fe2-registry-tls" (OuterVolumeSpecName: "registry-tls") pod "57ce0be5-62d9-473f-a19f-4253cb333fe2" (UID: "57ce0be5-62d9-473f-a19f-4253cb333fe2"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 18:49:03.467442 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:49:03.467416    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/57ce0be5-62d9-473f-a19f-4253cb333fe2-ca-trust-extracted" (OuterVolumeSpecName: "ca-trust-extracted") pod "57ce0be5-62d9-473f-a19f-4253cb333fe2" (UID: "57ce0be5-62d9-473f-a19f-4253cb333fe2"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 18:49:03.559528 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:49:03.559436    2581 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/57ce0be5-62d9-473f-a19f-4253cb333fe2-ca-trust-extracted\") on node \"ip-10-0-138-127.ec2.internal\" DevicePath \"\""
Sep 16 18:49:03.559528 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:49:03.559469    2581 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/57ce0be5-62d9-473f-a19f-4253cb333fe2-trusted-ca\") on node \"ip-10-0-138-127.ec2.internal\" DevicePath \"\""
Sep 16 18:49:03.559528 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:49:03.559478    2581 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/57ce0be5-62d9-473f-a19f-4253cb333fe2-registry-tls\") on node \"ip-10-0-138-127.ec2.internal\" DevicePath \"\""
Sep 16 18:49:03.559528 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:49:03.559486    2581 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/57ce0be5-62d9-473f-a19f-4253cb333fe2-installation-pull-secrets\") on node \"ip-10-0-138-127.ec2.internal\" DevicePath \"\""
Sep 16 18:49:03.559528 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:49:03.559495    2581 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/57ce0be5-62d9-473f-a19f-4253cb333fe2-registry-certificates\") on node \"ip-10-0-138-127.ec2.internal\" DevicePath \"\""
Sep 16 18:49:03.559528 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:49:03.559504    2581 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/57ce0be5-62d9-473f-a19f-4253cb333fe2-bound-sa-token\") on node \"ip-10-0-138-127.ec2.internal\" DevicePath \"\""
Sep 16 18:49:03.559528 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:49:03.559513    2581 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/57ce0be5-62d9-473f-a19f-4253cb333fe2-image-registry-private-configuration\") on node \"ip-10-0-138-127.ec2.internal\" DevicePath \"\""
Sep 16 18:49:03.559528 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:49:03.559521    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6xnhc\" (UniqueName: \"kubernetes.io/projected/57ce0be5-62d9-473f-a19f-4253cb333fe2-kube-api-access-6xnhc\") on node \"ip-10-0-138-127.ec2.internal\" DevicePath \"\""
Sep 16 18:49:03.733360 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:49:03.733323    2581 generic.go:356] "Generic (PLEG): container finished" podID="57ce0be5-62d9-473f-a19f-4253cb333fe2" containerID="4bb6224a269aee88f1547d6fce9f13932ef1f10341445cc66c6c87ee3eceec6c" exitCode=0
Sep 16 18:49:03.733531 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:49:03.733371    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-649b4fc7ff-zmk84"
Sep 16 18:49:03.733531 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:49:03.733401    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-649b4fc7ff-zmk84" event={"ID":"57ce0be5-62d9-473f-a19f-4253cb333fe2","Type":"ContainerDied","Data":"4bb6224a269aee88f1547d6fce9f13932ef1f10341445cc66c6c87ee3eceec6c"}
Sep 16 18:49:03.733531 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:49:03.733443    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-649b4fc7ff-zmk84" event={"ID":"57ce0be5-62d9-473f-a19f-4253cb333fe2","Type":"ContainerDied","Data":"ad7297e7b4e6c92640415cf383179c4f21686b987ff0d1208aed684a8a11579c"}
Sep 16 18:49:03.733531 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:49:03.733464    2581 scope.go:117] "RemoveContainer" containerID="4bb6224a269aee88f1547d6fce9f13932ef1f10341445cc66c6c87ee3eceec6c"
Sep 16 18:49:03.742755 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:49:03.742734    2581 scope.go:117] "RemoveContainer" containerID="4bb6224a269aee88f1547d6fce9f13932ef1f10341445cc66c6c87ee3eceec6c"
Sep 16 18:49:03.743066 ip-10-0-138-127 kubenswrapper[2581]: E0916 18:49:03.743044    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4bb6224a269aee88f1547d6fce9f13932ef1f10341445cc66c6c87ee3eceec6c\": container with ID starting with 4bb6224a269aee88f1547d6fce9f13932ef1f10341445cc66c6c87ee3eceec6c not found: ID does not exist" containerID="4bb6224a269aee88f1547d6fce9f13932ef1f10341445cc66c6c87ee3eceec6c"
Sep 16 18:49:03.743130 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:49:03.743075    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4bb6224a269aee88f1547d6fce9f13932ef1f10341445cc66c6c87ee3eceec6c"} err="failed to get container status \"4bb6224a269aee88f1547d6fce9f13932ef1f10341445cc66c6c87ee3eceec6c\": rpc error: code = NotFound desc = could not find container \"4bb6224a269aee88f1547d6fce9f13932ef1f10341445cc66c6c87ee3eceec6c\": container with ID starting with 4bb6224a269aee88f1547d6fce9f13932ef1f10341445cc66c6c87ee3eceec6c not found: ID does not exist"
Sep 16 18:49:03.756461 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:49:03.756427    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-649b4fc7ff-zmk84"]
Sep 16 18:49:03.760457 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:49:03.760424    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-649b4fc7ff-zmk84"]
Sep 16 18:49:03.938753 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:49:03.938648    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-6456555d6b-r87j6" podUID="d9df7594-85dc-4836-a7e7-7777f474e948" containerName="console" containerID="cri-o://038c1b0a8f4e9961d82562646484e27ec3b5e65aae0f9d32e4c8da29be3caca4" gracePeriod=15
Sep 16 18:49:04.205114 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:49:04.205038    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6456555d6b-r87j6"
Sep 16 18:49:04.367284 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:49:04.367246    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/d9df7594-85dc-4836-a7e7-7777f474e948-oauth-serving-cert\") pod \"d9df7594-85dc-4836-a7e7-7777f474e948\" (UID: \"d9df7594-85dc-4836-a7e7-7777f474e948\") "
Sep 16 18:49:04.367479 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:49:04.367292    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/d9df7594-85dc-4836-a7e7-7777f474e948-service-ca\") pod \"d9df7594-85dc-4836-a7e7-7777f474e948\" (UID: \"d9df7594-85dc-4836-a7e7-7777f474e948\") "
Sep 16 18:49:04.367479 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:49:04.367316    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/d9df7594-85dc-4836-a7e7-7777f474e948-console-oauth-config\") pod \"d9df7594-85dc-4836-a7e7-7777f474e948\" (UID: \"d9df7594-85dc-4836-a7e7-7777f474e948\") "
Sep 16 18:49:04.367479 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:49:04.367341    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/d9df7594-85dc-4836-a7e7-7777f474e948-console-serving-cert\") pod \"d9df7594-85dc-4836-a7e7-7777f474e948\" (UID: \"d9df7594-85dc-4836-a7e7-7777f474e948\") "
Sep 16 18:49:04.367479 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:49:04.367405    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/d9df7594-85dc-4836-a7e7-7777f474e948-console-config\") pod \"d9df7594-85dc-4836-a7e7-7777f474e948\" (UID: \"d9df7594-85dc-4836-a7e7-7777f474e948\") "
Sep 16 18:49:04.367479 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:49:04.367431    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-c5fhh\" (UniqueName: \"kubernetes.io/projected/d9df7594-85dc-4836-a7e7-7777f474e948-kube-api-access-c5fhh\") pod \"d9df7594-85dc-4836-a7e7-7777f474e948\" (UID: \"d9df7594-85dc-4836-a7e7-7777f474e948\") "
Sep 16 18:49:04.367780 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:49:04.367733    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d9df7594-85dc-4836-a7e7-7777f474e948-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "d9df7594-85dc-4836-a7e7-7777f474e948" (UID: "d9df7594-85dc-4836-a7e7-7777f474e948"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 18:49:04.367780 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:49:04.367747    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d9df7594-85dc-4836-a7e7-7777f474e948-service-ca" (OuterVolumeSpecName: "service-ca") pod "d9df7594-85dc-4836-a7e7-7777f474e948" (UID: "d9df7594-85dc-4836-a7e7-7777f474e948"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 18:49:04.367853 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:49:04.367838    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d9df7594-85dc-4836-a7e7-7777f474e948-console-config" (OuterVolumeSpecName: "console-config") pod "d9df7594-85dc-4836-a7e7-7777f474e948" (UID: "d9df7594-85dc-4836-a7e7-7777f474e948"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 18:49:04.369839 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:49:04.369809    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d9df7594-85dc-4836-a7e7-7777f474e948-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "d9df7594-85dc-4836-a7e7-7777f474e948" (UID: "d9df7594-85dc-4836-a7e7-7777f474e948"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 18:49:04.369940 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:49:04.369851    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d9df7594-85dc-4836-a7e7-7777f474e948-kube-api-access-c5fhh" (OuterVolumeSpecName: "kube-api-access-c5fhh") pod "d9df7594-85dc-4836-a7e7-7777f474e948" (UID: "d9df7594-85dc-4836-a7e7-7777f474e948"). InnerVolumeSpecName "kube-api-access-c5fhh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 18:49:04.369940 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:49:04.369876    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d9df7594-85dc-4836-a7e7-7777f474e948-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "d9df7594-85dc-4836-a7e7-7777f474e948" (UID: "d9df7594-85dc-4836-a7e7-7777f474e948"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 18:49:04.468636 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:49:04.468545    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c5fhh\" (UniqueName: \"kubernetes.io/projected/d9df7594-85dc-4836-a7e7-7777f474e948-kube-api-access-c5fhh\") on node \"ip-10-0-138-127.ec2.internal\" DevicePath \"\""
Sep 16 18:49:04.468636 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:49:04.468577    2581 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/d9df7594-85dc-4836-a7e7-7777f474e948-oauth-serving-cert\") on node \"ip-10-0-138-127.ec2.internal\" DevicePath \"\""
Sep 16 18:49:04.468636 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:49:04.468587    2581 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/d9df7594-85dc-4836-a7e7-7777f474e948-service-ca\") on node \"ip-10-0-138-127.ec2.internal\" DevicePath \"\""
Sep 16 18:49:04.468636 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:49:04.468597    2581 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/d9df7594-85dc-4836-a7e7-7777f474e948-console-oauth-config\") on node \"ip-10-0-138-127.ec2.internal\" DevicePath \"\""
Sep 16 18:49:04.468636 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:49:04.468606    2581 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/d9df7594-85dc-4836-a7e7-7777f474e948-console-serving-cert\") on node \"ip-10-0-138-127.ec2.internal\" DevicePath \"\""
Sep 16 18:49:04.468636 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:49:04.468614    2581 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/d9df7594-85dc-4836-a7e7-7777f474e948-console-config\") on node \"ip-10-0-138-127.ec2.internal\" DevicePath \"\""
Sep 16 18:49:04.727912 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:49:04.727834    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="57ce0be5-62d9-473f-a19f-4253cb333fe2" path="/var/lib/kubelet/pods/57ce0be5-62d9-473f-a19f-4253cb333fe2/volumes"
Sep 16 18:49:04.739871 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:49:04.739845    2581 generic.go:356] "Generic (PLEG): container finished" podID="d9df7594-85dc-4836-a7e7-7777f474e948" containerID="038c1b0a8f4e9961d82562646484e27ec3b5e65aae0f9d32e4c8da29be3caca4" exitCode=0
Sep 16 18:49:04.740012 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:49:04.739896    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6456555d6b-r87j6"
Sep 16 18:49:04.740012 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:49:04.739910    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6456555d6b-r87j6" event={"ID":"d9df7594-85dc-4836-a7e7-7777f474e948","Type":"ContainerDied","Data":"038c1b0a8f4e9961d82562646484e27ec3b5e65aae0f9d32e4c8da29be3caca4"}
Sep 16 18:49:04.740012 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:49:04.739948    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6456555d6b-r87j6" event={"ID":"d9df7594-85dc-4836-a7e7-7777f474e948","Type":"ContainerDied","Data":"60398e677427b1bc93f7e47cc61f975879b2da199b726e77e4dece879de483fd"}
Sep 16 18:49:04.740012 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:49:04.739967    2581 scope.go:117] "RemoveContainer" containerID="038c1b0a8f4e9961d82562646484e27ec3b5e65aae0f9d32e4c8da29be3caca4"
Sep 16 18:49:04.748187 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:49:04.748171    2581 scope.go:117] "RemoveContainer" containerID="038c1b0a8f4e9961d82562646484e27ec3b5e65aae0f9d32e4c8da29be3caca4"
Sep 16 18:49:04.748499 ip-10-0-138-127 kubenswrapper[2581]: E0916 18:49:04.748479    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"038c1b0a8f4e9961d82562646484e27ec3b5e65aae0f9d32e4c8da29be3caca4\": container with ID starting with 038c1b0a8f4e9961d82562646484e27ec3b5e65aae0f9d32e4c8da29be3caca4 not found: ID does not exist" containerID="038c1b0a8f4e9961d82562646484e27ec3b5e65aae0f9d32e4c8da29be3caca4"
Sep 16 18:49:04.748552 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:49:04.748509    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"038c1b0a8f4e9961d82562646484e27ec3b5e65aae0f9d32e4c8da29be3caca4"} err="failed to get container status \"038c1b0a8f4e9961d82562646484e27ec3b5e65aae0f9d32e4c8da29be3caca4\": rpc error: code = NotFound desc = could not find container \"038c1b0a8f4e9961d82562646484e27ec3b5e65aae0f9d32e4c8da29be3caca4\": container with ID starting with 038c1b0a8f4e9961d82562646484e27ec3b5e65aae0f9d32e4c8da29be3caca4 not found: ID does not exist"
Sep 16 18:49:04.756401 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:49:04.756373    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-6456555d6b-r87j6"]
Sep 16 18:49:04.758164 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:49:04.758143    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-6456555d6b-r87j6"]
Sep 16 18:49:06.727002 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:49:06.726957    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d9df7594-85dc-4836-a7e7-7777f474e948" path="/var/lib/kubelet/pods/d9df7594-85dc-4836-a7e7-7777f474e948/volumes"
Sep 16 18:49:09.761096 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:49:09.761011    2581 generic.go:356] "Generic (PLEG): container finished" podID="f3b6e193-3e4d-4dea-9909-b9eb2040ce3d" containerID="4d8f4dcd8a0abb2aa1de72362dab755aa5cdf005ccc972c9377eb483230bd4fc" exitCode=0
Sep 16 18:49:09.761096 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:49:09.761058    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-dd6d5" event={"ID":"f3b6e193-3e4d-4dea-9909-b9eb2040ce3d","Type":"ContainerDied","Data":"4d8f4dcd8a0abb2aa1de72362dab755aa5cdf005ccc972c9377eb483230bd4fc"}
Sep 16 18:49:09.761511 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:49:09.761415    2581 scope.go:117] "RemoveContainer" containerID="4d8f4dcd8a0abb2aa1de72362dab755aa5cdf005ccc972c9377eb483230bd4fc"
Sep 16 18:49:10.766827 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:49:10.766794    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-dd6d5" event={"ID":"f3b6e193-3e4d-4dea-9909-b9eb2040ce3d","Type":"ContainerStarted","Data":"9cc697889e8b292d8113dbf8981a8adebcb50bd0d1e5583f96778a4837f7a904"}
Sep 16 18:49:11.175867 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:49:11.175838    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/metrics-server-5c9d8b5484-7hpps"
Sep 16 18:49:11.180176 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:49:11.180145    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/metrics-server-5c9d8b5484-7hpps"
Sep 16 18:49:20.181815 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:49:20.181743    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-56fb5c8694-qcn8h" podUID="5bb743b0-66c4-44c0-bd47-b99cbaa2f24e" containerName="console" containerID="cri-o://e0eec99ce99a62ea9fdcb3f7f621bd18dbef0a192d71f4641a8c57a8e27ef691" gracePeriod=15
Sep 16 18:49:20.471139 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:49:20.471104    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-56fb5c8694-qcn8h"
Sep 16 18:49:20.517919 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:49:20.517889    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/5bb743b0-66c4-44c0-bd47-b99cbaa2f24e-console-serving-cert\") pod \"5bb743b0-66c4-44c0-bd47-b99cbaa2f24e\" (UID: \"5bb743b0-66c4-44c0-bd47-b99cbaa2f24e\") "
Sep 16 18:49:20.518116 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:49:20.517930    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/5bb743b0-66c4-44c0-bd47-b99cbaa2f24e-console-oauth-config\") pod \"5bb743b0-66c4-44c0-bd47-b99cbaa2f24e\" (UID: \"5bb743b0-66c4-44c0-bd47-b99cbaa2f24e\") "
Sep 16 18:49:20.518116 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:49:20.517976    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/5bb743b0-66c4-44c0-bd47-b99cbaa2f24e-console-config\") pod \"5bb743b0-66c4-44c0-bd47-b99cbaa2f24e\" (UID: \"5bb743b0-66c4-44c0-bd47-b99cbaa2f24e\") "
Sep 16 18:49:20.518116 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:49:20.517991    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/5bb743b0-66c4-44c0-bd47-b99cbaa2f24e-service-ca\") pod \"5bb743b0-66c4-44c0-bd47-b99cbaa2f24e\" (UID: \"5bb743b0-66c4-44c0-bd47-b99cbaa2f24e\") "
Sep 16 18:49:20.518116 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:49:20.518028    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hwqjl\" (UniqueName: \"kubernetes.io/projected/5bb743b0-66c4-44c0-bd47-b99cbaa2f24e-kube-api-access-hwqjl\") pod \"5bb743b0-66c4-44c0-bd47-b99cbaa2f24e\" (UID: \"5bb743b0-66c4-44c0-bd47-b99cbaa2f24e\") "
Sep 16 18:49:20.518116 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:49:20.518054    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5bb743b0-66c4-44c0-bd47-b99cbaa2f24e-trusted-ca-bundle\") pod \"5bb743b0-66c4-44c0-bd47-b99cbaa2f24e\" (UID: \"5bb743b0-66c4-44c0-bd47-b99cbaa2f24e\") "
Sep 16 18:49:20.518431 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:49:20.518405    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/5bb743b0-66c4-44c0-bd47-b99cbaa2f24e-oauth-serving-cert\") pod \"5bb743b0-66c4-44c0-bd47-b99cbaa2f24e\" (UID: \"5bb743b0-66c4-44c0-bd47-b99cbaa2f24e\") "
Sep 16 18:49:20.518515 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:49:20.518478    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5bb743b0-66c4-44c0-bd47-b99cbaa2f24e-service-ca" (OuterVolumeSpecName: "service-ca") pod "5bb743b0-66c4-44c0-bd47-b99cbaa2f24e" (UID: "5bb743b0-66c4-44c0-bd47-b99cbaa2f24e"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 18:49:20.518515 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:49:20.518504    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5bb743b0-66c4-44c0-bd47-b99cbaa2f24e-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "5bb743b0-66c4-44c0-bd47-b99cbaa2f24e" (UID: "5bb743b0-66c4-44c0-bd47-b99cbaa2f24e"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 18:49:20.518607 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:49:20.518497    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5bb743b0-66c4-44c0-bd47-b99cbaa2f24e-console-config" (OuterVolumeSpecName: "console-config") pod "5bb743b0-66c4-44c0-bd47-b99cbaa2f24e" (UID: "5bb743b0-66c4-44c0-bd47-b99cbaa2f24e"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 18:49:20.518771 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:49:20.518747    2581 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/5bb743b0-66c4-44c0-bd47-b99cbaa2f24e-console-config\") on node \"ip-10-0-138-127.ec2.internal\" DevicePath \"\""
Sep 16 18:49:20.518771 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:49:20.518775    2581 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/5bb743b0-66c4-44c0-bd47-b99cbaa2f24e-service-ca\") on node \"ip-10-0-138-127.ec2.internal\" DevicePath \"\""
Sep 16 18:49:20.518972 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:49:20.518789    2581 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5bb743b0-66c4-44c0-bd47-b99cbaa2f24e-trusted-ca-bundle\") on node \"ip-10-0-138-127.ec2.internal\" DevicePath \"\""
Sep 16 18:49:20.519186 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:49:20.519038    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5bb743b0-66c4-44c0-bd47-b99cbaa2f24e-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "5bb743b0-66c4-44c0-bd47-b99cbaa2f24e" (UID: "5bb743b0-66c4-44c0-bd47-b99cbaa2f24e"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 18:49:20.520460 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:49:20.520435    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5bb743b0-66c4-44c0-bd47-b99cbaa2f24e-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "5bb743b0-66c4-44c0-bd47-b99cbaa2f24e" (UID: "5bb743b0-66c4-44c0-bd47-b99cbaa2f24e"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 18:49:20.520601 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:49:20.520582    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5bb743b0-66c4-44c0-bd47-b99cbaa2f24e-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "5bb743b0-66c4-44c0-bd47-b99cbaa2f24e" (UID: "5bb743b0-66c4-44c0-bd47-b99cbaa2f24e"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 18:49:20.520870 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:49:20.520843    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5bb743b0-66c4-44c0-bd47-b99cbaa2f24e-kube-api-access-hwqjl" (OuterVolumeSpecName: "kube-api-access-hwqjl") pod "5bb743b0-66c4-44c0-bd47-b99cbaa2f24e" (UID: "5bb743b0-66c4-44c0-bd47-b99cbaa2f24e"). InnerVolumeSpecName "kube-api-access-hwqjl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 18:49:20.619910 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:49:20.619873    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hwqjl\" (UniqueName: \"kubernetes.io/projected/5bb743b0-66c4-44c0-bd47-b99cbaa2f24e-kube-api-access-hwqjl\") on node \"ip-10-0-138-127.ec2.internal\" DevicePath \"\""
Sep 16 18:49:20.619910 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:49:20.619904    2581 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/5bb743b0-66c4-44c0-bd47-b99cbaa2f24e-oauth-serving-cert\") on node \"ip-10-0-138-127.ec2.internal\" DevicePath \"\""
Sep 16 18:49:20.619910 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:49:20.619913    2581 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/5bb743b0-66c4-44c0-bd47-b99cbaa2f24e-console-serving-cert\") on node \"ip-10-0-138-127.ec2.internal\" DevicePath \"\""
Sep 16 18:49:20.619910 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:49:20.619925    2581 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/5bb743b0-66c4-44c0-bd47-b99cbaa2f24e-console-oauth-config\") on node \"ip-10-0-138-127.ec2.internal\" DevicePath \"\""
Sep 16 18:49:20.815172 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:49:20.815134    2581 generic.go:356] "Generic (PLEG): container finished" podID="5bb743b0-66c4-44c0-bd47-b99cbaa2f24e" containerID="e0eec99ce99a62ea9fdcb3f7f621bd18dbef0a192d71f4641a8c57a8e27ef691" exitCode=0
Sep 16 18:49:20.815351 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:49:20.815180    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-56fb5c8694-qcn8h"
Sep 16 18:49:20.815351 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:49:20.815221    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-56fb5c8694-qcn8h" event={"ID":"5bb743b0-66c4-44c0-bd47-b99cbaa2f24e","Type":"ContainerDied","Data":"e0eec99ce99a62ea9fdcb3f7f621bd18dbef0a192d71f4641a8c57a8e27ef691"}
Sep 16 18:49:20.815351 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:49:20.815276    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-56fb5c8694-qcn8h" event={"ID":"5bb743b0-66c4-44c0-bd47-b99cbaa2f24e","Type":"ContainerDied","Data":"48c777457f2a10ae515b0243e9f10380bc556b748caaffb867212d11e3afbd67"}
Sep 16 18:49:20.815351 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:49:20.815303    2581 scope.go:117] "RemoveContainer" containerID="e0eec99ce99a62ea9fdcb3f7f621bd18dbef0a192d71f4641a8c57a8e27ef691"
Sep 16 18:49:20.823653 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:49:20.823632    2581 scope.go:117] "RemoveContainer" containerID="e0eec99ce99a62ea9fdcb3f7f621bd18dbef0a192d71f4641a8c57a8e27ef691"
Sep 16 18:49:20.823956 ip-10-0-138-127 kubenswrapper[2581]: E0916 18:49:20.823938    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e0eec99ce99a62ea9fdcb3f7f621bd18dbef0a192d71f4641a8c57a8e27ef691\": container with ID starting with e0eec99ce99a62ea9fdcb3f7f621bd18dbef0a192d71f4641a8c57a8e27ef691 not found: ID does not exist" containerID="e0eec99ce99a62ea9fdcb3f7f621bd18dbef0a192d71f4641a8c57a8e27ef691"
Sep 16 18:49:20.824012 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:49:20.823964    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e0eec99ce99a62ea9fdcb3f7f621bd18dbef0a192d71f4641a8c57a8e27ef691"} err="failed to get container status \"e0eec99ce99a62ea9fdcb3f7f621bd18dbef0a192d71f4641a8c57a8e27ef691\": rpc error: code = NotFound desc = could not find container \"e0eec99ce99a62ea9fdcb3f7f621bd18dbef0a192d71f4641a8c57a8e27ef691\": container with ID starting with e0eec99ce99a62ea9fdcb3f7f621bd18dbef0a192d71f4641a8c57a8e27ef691 not found: ID does not exist"
Sep 16 18:49:20.834936 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:49:20.834908    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-56fb5c8694-qcn8h"]
Sep 16 18:49:20.836545 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:49:20.836524    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-56fb5c8694-qcn8h"]
Sep 16 18:49:22.727613 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:49:22.727577    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5bb743b0-66c4-44c0-bd47-b99cbaa2f24e" path="/var/lib/kubelet/pods/5bb743b0-66c4-44c0-bd47-b99cbaa2f24e/volumes"
Sep 16 18:49:32.998098 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:49:32.998059    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:49:33.017863 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:49:33.017839    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:49:33.884430 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:49:33.884402    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:49:42.623008 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:49:42.622972    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/eaf5897a-76a0-4bf3-aa4e-59aa6c193ddb-metrics-certs\") pod \"network-metrics-daemon-c2kp2\" (UID: \"eaf5897a-76a0-4bf3-aa4e-59aa6c193ddb\") " pod="openshift-multus/network-metrics-daemon-c2kp2"
Sep 16 18:49:42.625475 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:49:42.625452    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/eaf5897a-76a0-4bf3-aa4e-59aa6c193ddb-metrics-certs\") pod \"network-metrics-daemon-c2kp2\" (UID: \"eaf5897a-76a0-4bf3-aa4e-59aa6c193ddb\") " pod="openshift-multus/network-metrics-daemon-c2kp2"
Sep 16 18:49:42.725604 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:49:42.725576    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-pl7qs\""
Sep 16 18:49:42.733600 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:49:42.733579    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-c2kp2"
Sep 16 18:49:43.007922 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:49:43.007849    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-c2kp2"]
Sep 16 18:49:43.010793 ip-10-0-138-127 kubenswrapper[2581]: W0916 18:49:43.010765    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podeaf5897a_76a0_4bf3_aa4e_59aa6c193ddb.slice/crio-f5ed2efd2cc28eee3dbd154124fede7d941e3af9f41329533131052b20d5d7bd WatchSource:0}: Error finding container f5ed2efd2cc28eee3dbd154124fede7d941e3af9f41329533131052b20d5d7bd: Status 404 returned error can't find the container with id f5ed2efd2cc28eee3dbd154124fede7d941e3af9f41329533131052b20d5d7bd
Sep 16 18:49:43.907716 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:49:43.907681    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-c2kp2" event={"ID":"eaf5897a-76a0-4bf3-aa4e-59aa6c193ddb","Type":"ContainerStarted","Data":"f5ed2efd2cc28eee3dbd154124fede7d941e3af9f41329533131052b20d5d7bd"}
Sep 16 18:49:44.913783 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:49:44.913744    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-c2kp2" event={"ID":"eaf5897a-76a0-4bf3-aa4e-59aa6c193ddb","Type":"ContainerStarted","Data":"c4f0ca2a9f9229673bce428d79feec04109213f8240218129e09736cfaf4b928"}
Sep 16 18:49:44.914253 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:49:44.913794    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-c2kp2" event={"ID":"eaf5897a-76a0-4bf3-aa4e-59aa6c193ddb","Type":"ContainerStarted","Data":"19d2dfa6aee3a15ff3ff3c8d20c7b25ebf4855566be63c6c1d49e6a8b812ffe7"}
Sep 16 18:49:44.946685 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:49:44.946632    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-c2kp2" podStartSLOduration=253.013047525 podStartE2EDuration="4m13.946614801s" totalImagesPullingTime="933.567276ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:45:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:49:44.945984059 +0000 UTC m=+254.765397968" watchObservedRunningTime="2026-09-16 18:49:44.946614801 +0000 UTC m=+254.766028691"
Sep 16 18:49:46.867547 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:49:46.867512    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Sep 16 18:49:46.868079 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:49:46.867865    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="89fc85f4-ec94-4459-8311-f3dbf4681bf8" containerName="alertmanager" containerID="cri-o://f4663becc1918a7e6924c74d9a68748b56439cd09776d57e9ab09ca5fff6bf04" gracePeriod=120
Sep 16 18:49:46.868159 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:49:46.868078    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="89fc85f4-ec94-4459-8311-f3dbf4681bf8" containerName="kube-rbac-proxy" containerID="cri-o://45a58d97808bbe7579fb16590a57b26a5743ac74ecfe45ada969fc1685ed05d7" gracePeriod=120
Sep 16 18:49:46.868159 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:49:46.868100    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="89fc85f4-ec94-4459-8311-f3dbf4681bf8" containerName="config-reloader" containerID="cri-o://bef7df271946c5737d62c61b1cf3e85e1e9e4fd5970c843ad20c77bd5c32a57a" gracePeriod=120
Sep 16 18:49:46.868159 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:49:46.868118    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="89fc85f4-ec94-4459-8311-f3dbf4681bf8" containerName="prom-label-proxy" containerID="cri-o://1a89a9e7a2375bf34bb12157b14aa09a24a45dab3add18d098c5dcb6c5c0a79a" gracePeriod=120
Sep 16 18:49:46.868359 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:49:46.868110    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="89fc85f4-ec94-4459-8311-f3dbf4681bf8" containerName="kube-rbac-proxy-web" containerID="cri-o://b95ace2ac8e0f37bc18c57982ae4b718ba41df9190333b9656adde5cdfbac64c" gracePeriod=120
Sep 16 18:49:46.868359 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:49:46.868248    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="89fc85f4-ec94-4459-8311-f3dbf4681bf8" containerName="kube-rbac-proxy-metric" containerID="cri-o://8d7b8e7f9e6b1851ac6b5e6beb3beadd022252cd070b853e77209f3d532e0025" gracePeriod=120
Sep 16 18:49:47.928752 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:49:47.928711    2581 generic.go:356] "Generic (PLEG): container finished" podID="89fc85f4-ec94-4459-8311-f3dbf4681bf8" containerID="1a89a9e7a2375bf34bb12157b14aa09a24a45dab3add18d098c5dcb6c5c0a79a" exitCode=0
Sep 16 18:49:47.928752 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:49:47.928735    2581 generic.go:356] "Generic (PLEG): container finished" podID="89fc85f4-ec94-4459-8311-f3dbf4681bf8" containerID="45a58d97808bbe7579fb16590a57b26a5743ac74ecfe45ada969fc1685ed05d7" exitCode=0
Sep 16 18:49:47.928752 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:49:47.928741    2581 generic.go:356] "Generic (PLEG): container finished" podID="89fc85f4-ec94-4459-8311-f3dbf4681bf8" containerID="bef7df271946c5737d62c61b1cf3e85e1e9e4fd5970c843ad20c77bd5c32a57a" exitCode=0
Sep 16 18:49:47.928752 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:49:47.928746    2581 generic.go:356] "Generic (PLEG): container finished" podID="89fc85f4-ec94-4459-8311-f3dbf4681bf8" containerID="f4663becc1918a7e6924c74d9a68748b56439cd09776d57e9ab09ca5fff6bf04" exitCode=0
Sep 16 18:49:47.929239 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:49:47.928786    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"89fc85f4-ec94-4459-8311-f3dbf4681bf8","Type":"ContainerDied","Data":"1a89a9e7a2375bf34bb12157b14aa09a24a45dab3add18d098c5dcb6c5c0a79a"}
Sep 16 18:49:47.929239 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:49:47.928850    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"89fc85f4-ec94-4459-8311-f3dbf4681bf8","Type":"ContainerDied","Data":"45a58d97808bbe7579fb16590a57b26a5743ac74ecfe45ada969fc1685ed05d7"}
Sep 16 18:49:47.929239 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:49:47.928878    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"89fc85f4-ec94-4459-8311-f3dbf4681bf8","Type":"ContainerDied","Data":"bef7df271946c5737d62c61b1cf3e85e1e9e4fd5970c843ad20c77bd5c32a57a"}
Sep 16 18:49:47.929239 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:49:47.928901    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"89fc85f4-ec94-4459-8311-f3dbf4681bf8","Type":"ContainerDied","Data":"f4663becc1918a7e6924c74d9a68748b56439cd09776d57e9ab09ca5fff6bf04"}
Sep 16 18:49:48.131119 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:49:48.131093    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Sep 16 18:49:48.268370 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:49:48.268258    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/89fc85f4-ec94-4459-8311-f3dbf4681bf8-metrics-client-ca\") pod \"89fc85f4-ec94-4459-8311-f3dbf4681bf8\" (UID: \"89fc85f4-ec94-4459-8311-f3dbf4681bf8\") "
Sep 16 18:49:48.268370 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:49:48.268310    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/89fc85f4-ec94-4459-8311-f3dbf4681bf8-secret-alertmanager-kube-rbac-proxy-web\") pod \"89fc85f4-ec94-4459-8311-f3dbf4681bf8\" (UID: \"89fc85f4-ec94-4459-8311-f3dbf4681bf8\") "
Sep 16 18:49:48.268370 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:49:48.268352    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/89fc85f4-ec94-4459-8311-f3dbf4681bf8-web-config\") pod \"89fc85f4-ec94-4459-8311-f3dbf4681bf8\" (UID: \"89fc85f4-ec94-4459-8311-f3dbf4681bf8\") "
Sep 16 18:49:48.268370 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:49:48.268369    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/89fc85f4-ec94-4459-8311-f3dbf4681bf8-secret-alertmanager-kube-rbac-proxy\") pod \"89fc85f4-ec94-4459-8311-f3dbf4681bf8\" (UID: \"89fc85f4-ec94-4459-8311-f3dbf4681bf8\") "
Sep 16 18:49:48.268707 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:49:48.268491    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/89fc85f4-ec94-4459-8311-f3dbf4681bf8-config-out\") pod \"89fc85f4-ec94-4459-8311-f3dbf4681bf8\" (UID: \"89fc85f4-ec94-4459-8311-f3dbf4681bf8\") "
Sep 16 18:49:48.268707 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:49:48.268531    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-krrrv\" (UniqueName: \"kubernetes.io/projected/89fc85f4-ec94-4459-8311-f3dbf4681bf8-kube-api-access-krrrv\") pod \"89fc85f4-ec94-4459-8311-f3dbf4681bf8\" (UID: \"89fc85f4-ec94-4459-8311-f3dbf4681bf8\") "
Sep 16 18:49:48.268707 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:49:48.268553    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/89fc85f4-ec94-4459-8311-f3dbf4681bf8-tls-assets\") pod \"89fc85f4-ec94-4459-8311-f3dbf4681bf8\" (UID: \"89fc85f4-ec94-4459-8311-f3dbf4681bf8\") "
Sep 16 18:49:48.268707 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:49:48.268587    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/89fc85f4-ec94-4459-8311-f3dbf4681bf8-alertmanager-trusted-ca-bundle\") pod \"89fc85f4-ec94-4459-8311-f3dbf4681bf8\" (UID: \"89fc85f4-ec94-4459-8311-f3dbf4681bf8\") "
Sep 16 18:49:48.268707 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:49:48.268617    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/89fc85f4-ec94-4459-8311-f3dbf4681bf8-config-volume\") pod \"89fc85f4-ec94-4459-8311-f3dbf4681bf8\" (UID: \"89fc85f4-ec94-4459-8311-f3dbf4681bf8\") "
Sep 16 18:49:48.268707 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:49:48.268641    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/89fc85f4-ec94-4459-8311-f3dbf4681bf8-secret-alertmanager-main-tls\") pod \"89fc85f4-ec94-4459-8311-f3dbf4681bf8\" (UID: \"89fc85f4-ec94-4459-8311-f3dbf4681bf8\") "
Sep 16 18:49:48.268707 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:49:48.268673    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/89fc85f4-ec94-4459-8311-f3dbf4681bf8-cluster-tls-config\") pod \"89fc85f4-ec94-4459-8311-f3dbf4681bf8\" (UID: \"89fc85f4-ec94-4459-8311-f3dbf4681bf8\") "
Sep 16 18:49:48.268707 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:49:48.268702    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/89fc85f4-ec94-4459-8311-f3dbf4681bf8-secret-alertmanager-kube-rbac-proxy-metric\") pod \"89fc85f4-ec94-4459-8311-f3dbf4681bf8\" (UID: \"89fc85f4-ec94-4459-8311-f3dbf4681bf8\") "
Sep 16 18:49:48.269098 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:49:48.268732    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/89fc85f4-ec94-4459-8311-f3dbf4681bf8-metrics-client-ca" (OuterVolumeSpecName: "metrics-client-ca") pod "89fc85f4-ec94-4459-8311-f3dbf4681bf8" (UID: "89fc85f4-ec94-4459-8311-f3dbf4681bf8"). InnerVolumeSpecName "metrics-client-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 18:49:48.269098 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:49:48.268761    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/89fc85f4-ec94-4459-8311-f3dbf4681bf8-alertmanager-main-db\") pod \"89fc85f4-ec94-4459-8311-f3dbf4681bf8\" (UID: \"89fc85f4-ec94-4459-8311-f3dbf4681bf8\") "
Sep 16 18:49:48.269098 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:49:48.269082    2581 reconciler_common.go:299] "Volume detached for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/89fc85f4-ec94-4459-8311-f3dbf4681bf8-metrics-client-ca\") on node \"ip-10-0-138-127.ec2.internal\" DevicePath \"\""
Sep 16 18:49:48.269506 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:49:48.269404    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/89fc85f4-ec94-4459-8311-f3dbf4681bf8-alertmanager-main-db" (OuterVolumeSpecName: "alertmanager-main-db") pod "89fc85f4-ec94-4459-8311-f3dbf4681bf8" (UID: "89fc85f4-ec94-4459-8311-f3dbf4681bf8"). InnerVolumeSpecName "alertmanager-main-db". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 18:49:48.271064 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:49:48.271031    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/89fc85f4-ec94-4459-8311-f3dbf4681bf8-alertmanager-trusted-ca-bundle" (OuterVolumeSpecName: "alertmanager-trusted-ca-bundle") pod "89fc85f4-ec94-4459-8311-f3dbf4681bf8" (UID: "89fc85f4-ec94-4459-8311-f3dbf4681bf8"). InnerVolumeSpecName "alertmanager-trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 18:49:48.271620 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:49:48.271592    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/89fc85f4-ec94-4459-8311-f3dbf4681bf8-secret-alertmanager-kube-rbac-proxy-web" (OuterVolumeSpecName: "secret-alertmanager-kube-rbac-proxy-web") pod "89fc85f4-ec94-4459-8311-f3dbf4681bf8" (UID: "89fc85f4-ec94-4459-8311-f3dbf4681bf8"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy-web". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 18:49:48.272276 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:49:48.272250    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/89fc85f4-ec94-4459-8311-f3dbf4681bf8-secret-alertmanager-kube-rbac-proxy-metric" (OuterVolumeSpecName: "secret-alertmanager-kube-rbac-proxy-metric") pod "89fc85f4-ec94-4459-8311-f3dbf4681bf8" (UID: "89fc85f4-ec94-4459-8311-f3dbf4681bf8"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy-metric". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 18:49:48.272380 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:49:48.272329    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/89fc85f4-ec94-4459-8311-f3dbf4681bf8-secret-alertmanager-kube-rbac-proxy" (OuterVolumeSpecName: "secret-alertmanager-kube-rbac-proxy") pod "89fc85f4-ec94-4459-8311-f3dbf4681bf8" (UID: "89fc85f4-ec94-4459-8311-f3dbf4681bf8"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 18:49:48.272610 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:49:48.272574    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/89fc85f4-ec94-4459-8311-f3dbf4681bf8-config-volume" (OuterVolumeSpecName: "config-volume") pod "89fc85f4-ec94-4459-8311-f3dbf4681bf8" (UID: "89fc85f4-ec94-4459-8311-f3dbf4681bf8"). InnerVolumeSpecName "config-volume". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 18:49:48.272725 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:49:48.272694    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/89fc85f4-ec94-4459-8311-f3dbf4681bf8-kube-api-access-krrrv" (OuterVolumeSpecName: "kube-api-access-krrrv") pod "89fc85f4-ec94-4459-8311-f3dbf4681bf8" (UID: "89fc85f4-ec94-4459-8311-f3dbf4681bf8"). InnerVolumeSpecName "kube-api-access-krrrv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 18:49:48.272849 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:49:48.272832    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/89fc85f4-ec94-4459-8311-f3dbf4681bf8-config-out" (OuterVolumeSpecName: "config-out") pod "89fc85f4-ec94-4459-8311-f3dbf4681bf8" (UID: "89fc85f4-ec94-4459-8311-f3dbf4681bf8"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 18:49:48.273582 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:49:48.273555    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/89fc85f4-ec94-4459-8311-f3dbf4681bf8-secret-alertmanager-main-tls" (OuterVolumeSpecName: "secret-alertmanager-main-tls") pod "89fc85f4-ec94-4459-8311-f3dbf4681bf8" (UID: "89fc85f4-ec94-4459-8311-f3dbf4681bf8"). InnerVolumeSpecName "secret-alertmanager-main-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 18:49:48.273900 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:49:48.273867    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/89fc85f4-ec94-4459-8311-f3dbf4681bf8-tls-assets" (OuterVolumeSpecName: "tls-assets") pod "89fc85f4-ec94-4459-8311-f3dbf4681bf8" (UID: "89fc85f4-ec94-4459-8311-f3dbf4681bf8"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 18:49:48.276344 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:49:48.276322    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/89fc85f4-ec94-4459-8311-f3dbf4681bf8-cluster-tls-config" (OuterVolumeSpecName: "cluster-tls-config") pod "89fc85f4-ec94-4459-8311-f3dbf4681bf8" (UID: "89fc85f4-ec94-4459-8311-f3dbf4681bf8"). InnerVolumeSpecName "cluster-tls-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 18:49:48.284270 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:49:48.284248    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/89fc85f4-ec94-4459-8311-f3dbf4681bf8-web-config" (OuterVolumeSpecName: "web-config") pod "89fc85f4-ec94-4459-8311-f3dbf4681bf8" (UID: "89fc85f4-ec94-4459-8311-f3dbf4681bf8"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 18:49:48.369940 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:49:48.369908    2581 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/89fc85f4-ec94-4459-8311-f3dbf4681bf8-web-config\") on node \"ip-10-0-138-127.ec2.internal\" DevicePath \"\""
Sep 16 18:49:48.369940 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:49:48.369936    2581 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/89fc85f4-ec94-4459-8311-f3dbf4681bf8-secret-alertmanager-kube-rbac-proxy\") on node \"ip-10-0-138-127.ec2.internal\" DevicePath \"\""
Sep 16 18:49:48.369940 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:49:48.369947    2581 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/89fc85f4-ec94-4459-8311-f3dbf4681bf8-config-out\") on node \"ip-10-0-138-127.ec2.internal\" DevicePath \"\""
Sep 16 18:49:48.370143 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:49:48.369955    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-krrrv\" (UniqueName: \"kubernetes.io/projected/89fc85f4-ec94-4459-8311-f3dbf4681bf8-kube-api-access-krrrv\") on node \"ip-10-0-138-127.ec2.internal\" DevicePath \"\""
Sep 16 18:49:48.370143 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:49:48.369962    2581 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/89fc85f4-ec94-4459-8311-f3dbf4681bf8-tls-assets\") on node \"ip-10-0-138-127.ec2.internal\" DevicePath \"\""
Sep 16 18:49:48.370143 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:49:48.369971    2581 reconciler_common.go:299] "Volume detached for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/89fc85f4-ec94-4459-8311-f3dbf4681bf8-alertmanager-trusted-ca-bundle\") on node \"ip-10-0-138-127.ec2.internal\" DevicePath \"\""
Sep 16 18:49:48.370143 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:49:48.369980    2581 reconciler_common.go:299] "Volume detached for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/89fc85f4-ec94-4459-8311-f3dbf4681bf8-config-volume\") on node \"ip-10-0-138-127.ec2.internal\" DevicePath \"\""
Sep 16 18:49:48.370143 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:49:48.369989    2581 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/89fc85f4-ec94-4459-8311-f3dbf4681bf8-secret-alertmanager-main-tls\") on node \"ip-10-0-138-127.ec2.internal\" DevicePath \"\""
Sep 16 18:49:48.370143 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:49:48.369996    2581 reconciler_common.go:299] "Volume detached for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/89fc85f4-ec94-4459-8311-f3dbf4681bf8-cluster-tls-config\") on node \"ip-10-0-138-127.ec2.internal\" DevicePath \"\""
Sep 16 18:49:48.370143 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:49:48.370008    2581 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/89fc85f4-ec94-4459-8311-f3dbf4681bf8-secret-alertmanager-kube-rbac-proxy-metric\") on node \"ip-10-0-138-127.ec2.internal\" DevicePath \"\""
Sep 16 18:49:48.370143 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:49:48.370016    2581 reconciler_common.go:299] "Volume detached for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/89fc85f4-ec94-4459-8311-f3dbf4681bf8-alertmanager-main-db\") on node \"ip-10-0-138-127.ec2.internal\" DevicePath \"\""
Sep 16 18:49:48.370143 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:49:48.370024    2581 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/89fc85f4-ec94-4459-8311-f3dbf4681bf8-secret-alertmanager-kube-rbac-proxy-web\") on node \"ip-10-0-138-127.ec2.internal\" DevicePath \"\""
Sep 16 18:49:48.935791 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:49:48.935756    2581 generic.go:356] "Generic (PLEG): container finished" podID="89fc85f4-ec94-4459-8311-f3dbf4681bf8" containerID="8d7b8e7f9e6b1851ac6b5e6beb3beadd022252cd070b853e77209f3d532e0025" exitCode=0
Sep 16 18:49:48.935791 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:49:48.935782    2581 generic.go:356] "Generic (PLEG): container finished" podID="89fc85f4-ec94-4459-8311-f3dbf4681bf8" containerID="b95ace2ac8e0f37bc18c57982ae4b718ba41df9190333b9656adde5cdfbac64c" exitCode=0
Sep 16 18:49:48.936236 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:49:48.935813    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"89fc85f4-ec94-4459-8311-f3dbf4681bf8","Type":"ContainerDied","Data":"8d7b8e7f9e6b1851ac6b5e6beb3beadd022252cd070b853e77209f3d532e0025"}
Sep 16 18:49:48.936236 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:49:48.935841    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Sep 16 18:49:48.936236 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:49:48.935872    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"89fc85f4-ec94-4459-8311-f3dbf4681bf8","Type":"ContainerDied","Data":"b95ace2ac8e0f37bc18c57982ae4b718ba41df9190333b9656adde5cdfbac64c"}
Sep 16 18:49:48.936236 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:49:48.935906    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"89fc85f4-ec94-4459-8311-f3dbf4681bf8","Type":"ContainerDied","Data":"cc87715509bcacd6541405775c70754578b08db2ca4e69c4c946d7a31ca9c1c9"}
Sep 16 18:49:48.936236 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:49:48.935937    2581 scope.go:117] "RemoveContainer" containerID="1a89a9e7a2375bf34bb12157b14aa09a24a45dab3add18d098c5dcb6c5c0a79a"
Sep 16 18:49:48.947352 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:49:48.947327    2581 scope.go:117] "RemoveContainer" containerID="8d7b8e7f9e6b1851ac6b5e6beb3beadd022252cd070b853e77209f3d532e0025"
Sep 16 18:49:48.958279 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:49:48.958256    2581 scope.go:117] "RemoveContainer" containerID="45a58d97808bbe7579fb16590a57b26a5743ac74ecfe45ada969fc1685ed05d7"
Sep 16 18:49:48.961019 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:49:48.960997    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Sep 16 18:49:48.964533 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:49:48.964509    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Sep 16 18:49:48.967233 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:49:48.967216    2581 scope.go:117] "RemoveContainer" containerID="b95ace2ac8e0f37bc18c57982ae4b718ba41df9190333b9656adde5cdfbac64c"
Sep 16 18:49:48.974764 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:49:48.974747    2581 scope.go:117] "RemoveContainer" containerID="bef7df271946c5737d62c61b1cf3e85e1e9e4fd5970c843ad20c77bd5c32a57a"
Sep 16 18:49:48.982302 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:49:48.982278    2581 scope.go:117] "RemoveContainer" containerID="f4663becc1918a7e6924c74d9a68748b56439cd09776d57e9ab09ca5fff6bf04"
Sep 16 18:49:48.994032 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:49:48.993875    2581 scope.go:117] "RemoveContainer" containerID="533a951e6971354aecdc7e706995006228615b0302070d4796da3b7ee87b2e85"
Sep 16 18:49:48.997399 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:49:48.997376    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Sep 16 18:49:48.997711 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:49:48.997698    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="89fc85f4-ec94-4459-8311-f3dbf4681bf8" containerName="prom-label-proxy"
Sep 16 18:49:48.997757 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:49:48.997715    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="89fc85f4-ec94-4459-8311-f3dbf4681bf8" containerName="prom-label-proxy"
Sep 16 18:49:48.997757 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:49:48.997732    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="89fc85f4-ec94-4459-8311-f3dbf4681bf8" containerName="alertmanager"
Sep 16 18:49:48.997757 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:49:48.997738    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="89fc85f4-ec94-4459-8311-f3dbf4681bf8" containerName="alertmanager"
Sep 16 18:49:48.997757 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:49:48.997748    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d9df7594-85dc-4836-a7e7-7777f474e948" containerName="console"
Sep 16 18:49:48.997757 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:49:48.997753    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="d9df7594-85dc-4836-a7e7-7777f474e948" containerName="console"
Sep 16 18:49:48.997933 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:49:48.997761    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="89fc85f4-ec94-4459-8311-f3dbf4681bf8" containerName="config-reloader"
Sep 16 18:49:48.997933 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:49:48.997766    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="89fc85f4-ec94-4459-8311-f3dbf4681bf8" containerName="config-reloader"
Sep 16 18:49:48.997933 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:49:48.997774    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="89fc85f4-ec94-4459-8311-f3dbf4681bf8" containerName="kube-rbac-proxy-metric"
Sep 16 18:49:48.997933 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:49:48.997779    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="89fc85f4-ec94-4459-8311-f3dbf4681bf8" containerName="kube-rbac-proxy-metric"
Sep 16 18:49:48.997933 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:49:48.997789    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="89fc85f4-ec94-4459-8311-f3dbf4681bf8" containerName="init-config-reloader"
Sep 16 18:49:48.997933 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:49:48.997795    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="89fc85f4-ec94-4459-8311-f3dbf4681bf8" containerName="init-config-reloader"
Sep 16 18:49:48.997933 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:49:48.997806    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="89fc85f4-ec94-4459-8311-f3dbf4681bf8" containerName="kube-rbac-proxy"
Sep 16 18:49:48.997933 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:49:48.997811    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="89fc85f4-ec94-4459-8311-f3dbf4681bf8" containerName="kube-rbac-proxy"
Sep 16 18:49:48.997933 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:49:48.997822    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="89fc85f4-ec94-4459-8311-f3dbf4681bf8" containerName="kube-rbac-proxy-web"
Sep 16 18:49:48.997933 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:49:48.997827    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="89fc85f4-ec94-4459-8311-f3dbf4681bf8" containerName="kube-rbac-proxy-web"
Sep 16 18:49:48.997933 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:49:48.997834    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="57ce0be5-62d9-473f-a19f-4253cb333fe2" containerName="registry"
Sep 16 18:49:48.997933 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:49:48.997838    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="57ce0be5-62d9-473f-a19f-4253cb333fe2" containerName="registry"
Sep 16 18:49:48.997933 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:49:48.997844    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5bb743b0-66c4-44c0-bd47-b99cbaa2f24e" containerName="console"
Sep 16 18:49:48.997933 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:49:48.997849    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="5bb743b0-66c4-44c0-bd47-b99cbaa2f24e" containerName="console"
Sep 16 18:49:48.997933 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:49:48.997894    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="89fc85f4-ec94-4459-8311-f3dbf4681bf8" containerName="prom-label-proxy"
Sep 16 18:49:48.997933 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:49:48.997903    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="57ce0be5-62d9-473f-a19f-4253cb333fe2" containerName="registry"
Sep 16 18:49:48.997933 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:49:48.997911    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="89fc85f4-ec94-4459-8311-f3dbf4681bf8" containerName="kube-rbac-proxy-metric"
Sep 16 18:49:48.997933 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:49:48.997917    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="89fc85f4-ec94-4459-8311-f3dbf4681bf8" containerName="kube-rbac-proxy-web"
Sep 16 18:49:48.997933 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:49:48.997923    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="5bb743b0-66c4-44c0-bd47-b99cbaa2f24e" containerName="console"
Sep 16 18:49:48.997933 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:49:48.997929    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="89fc85f4-ec94-4459-8311-f3dbf4681bf8" containerName="alertmanager"
Sep 16 18:49:48.997933 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:49:48.997934    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="89fc85f4-ec94-4459-8311-f3dbf4681bf8" containerName="kube-rbac-proxy"
Sep 16 18:49:48.997933 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:49:48.997939    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="d9df7594-85dc-4836-a7e7-7777f474e948" containerName="console"
Sep 16 18:49:48.998617 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:49:48.997945    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="89fc85f4-ec94-4459-8311-f3dbf4681bf8" containerName="config-reloader"
Sep 16 18:49:49.002003 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:49:49.001984    2581 scope.go:117] "RemoveContainer" containerID="1a89a9e7a2375bf34bb12157b14aa09a24a45dab3add18d098c5dcb6c5c0a79a"
Sep 16 18:49:49.002367 ip-10-0-138-127 kubenswrapper[2581]: E0916 18:49:49.002275    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1a89a9e7a2375bf34bb12157b14aa09a24a45dab3add18d098c5dcb6c5c0a79a\": container with ID starting with 1a89a9e7a2375bf34bb12157b14aa09a24a45dab3add18d098c5dcb6c5c0a79a not found: ID does not exist" containerID="1a89a9e7a2375bf34bb12157b14aa09a24a45dab3add18d098c5dcb6c5c0a79a"
Sep 16 18:49:49.002367 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:49:49.002302    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1a89a9e7a2375bf34bb12157b14aa09a24a45dab3add18d098c5dcb6c5c0a79a"} err="failed to get container status \"1a89a9e7a2375bf34bb12157b14aa09a24a45dab3add18d098c5dcb6c5c0a79a\": rpc error: code = NotFound desc = could not find container \"1a89a9e7a2375bf34bb12157b14aa09a24a45dab3add18d098c5dcb6c5c0a79a\": container with ID starting with 1a89a9e7a2375bf34bb12157b14aa09a24a45dab3add18d098c5dcb6c5c0a79a not found: ID does not exist"
Sep 16 18:49:49.002367 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:49:49.002321    2581 scope.go:117] "RemoveContainer" containerID="8d7b8e7f9e6b1851ac6b5e6beb3beadd022252cd070b853e77209f3d532e0025"
Sep 16 18:49:49.002563 ip-10-0-138-127 kubenswrapper[2581]: E0916 18:49:49.002517    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8d7b8e7f9e6b1851ac6b5e6beb3beadd022252cd070b853e77209f3d532e0025\": container with ID starting with 8d7b8e7f9e6b1851ac6b5e6beb3beadd022252cd070b853e77209f3d532e0025 not found: ID does not exist" containerID="8d7b8e7f9e6b1851ac6b5e6beb3beadd022252cd070b853e77209f3d532e0025"
Sep 16 18:49:49.002563 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:49:49.002535    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8d7b8e7f9e6b1851ac6b5e6beb3beadd022252cd070b853e77209f3d532e0025"} err="failed to get container status \"8d7b8e7f9e6b1851ac6b5e6beb3beadd022252cd070b853e77209f3d532e0025\": rpc error: code = NotFound desc = could not find container \"8d7b8e7f9e6b1851ac6b5e6beb3beadd022252cd070b853e77209f3d532e0025\": container with ID starting with 8d7b8e7f9e6b1851ac6b5e6beb3beadd022252cd070b853e77209f3d532e0025 not found: ID does not exist"
Sep 16 18:49:49.002563 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:49:49.002548    2581 scope.go:117] "RemoveContainer" containerID="45a58d97808bbe7579fb16590a57b26a5743ac74ecfe45ada969fc1685ed05d7"
Sep 16 18:49:49.002759 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:49:49.002740    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Sep 16 18:49:49.002846 ip-10-0-138-127 kubenswrapper[2581]: E0916 18:49:49.002826    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"45a58d97808bbe7579fb16590a57b26a5743ac74ecfe45ada969fc1685ed05d7\": container with ID starting with 45a58d97808bbe7579fb16590a57b26a5743ac74ecfe45ada969fc1685ed05d7 not found: ID does not exist" containerID="45a58d97808bbe7579fb16590a57b26a5743ac74ecfe45ada969fc1685ed05d7"
Sep 16 18:49:49.002898 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:49:49.002852    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"45a58d97808bbe7579fb16590a57b26a5743ac74ecfe45ada969fc1685ed05d7"} err="failed to get container status \"45a58d97808bbe7579fb16590a57b26a5743ac74ecfe45ada969fc1685ed05d7\": rpc error: code = NotFound desc = could not find container \"45a58d97808bbe7579fb16590a57b26a5743ac74ecfe45ada969fc1685ed05d7\": container with ID starting with 45a58d97808bbe7579fb16590a57b26a5743ac74ecfe45ada969fc1685ed05d7 not found: ID does not exist"
Sep 16 18:49:49.002898 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:49:49.002870    2581 scope.go:117] "RemoveContainer" containerID="b95ace2ac8e0f37bc18c57982ae4b718ba41df9190333b9656adde5cdfbac64c"
Sep 16 18:49:49.003358 ip-10-0-138-127 kubenswrapper[2581]: E0916 18:49:49.003332    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b95ace2ac8e0f37bc18c57982ae4b718ba41df9190333b9656adde5cdfbac64c\": container with ID starting with b95ace2ac8e0f37bc18c57982ae4b718ba41df9190333b9656adde5cdfbac64c not found: ID does not exist" containerID="b95ace2ac8e0f37bc18c57982ae4b718ba41df9190333b9656adde5cdfbac64c"
Sep 16 18:49:49.003471 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:49:49.003365    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b95ace2ac8e0f37bc18c57982ae4b718ba41df9190333b9656adde5cdfbac64c"} err="failed to get container status \"b95ace2ac8e0f37bc18c57982ae4b718ba41df9190333b9656adde5cdfbac64c\": rpc error: code = NotFound desc = could not find container \"b95ace2ac8e0f37bc18c57982ae4b718ba41df9190333b9656adde5cdfbac64c\": container with ID starting with b95ace2ac8e0f37bc18c57982ae4b718ba41df9190333b9656adde5cdfbac64c not found: ID does not exist"
Sep 16 18:49:49.003471 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:49:49.003380    2581 scope.go:117] "RemoveContainer" containerID="bef7df271946c5737d62c61b1cf3e85e1e9e4fd5970c843ad20c77bd5c32a57a"
Sep 16 18:49:49.003664 ip-10-0-138-127 kubenswrapper[2581]: E0916 18:49:49.003647    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bef7df271946c5737d62c61b1cf3e85e1e9e4fd5970c843ad20c77bd5c32a57a\": container with ID starting with bef7df271946c5737d62c61b1cf3e85e1e9e4fd5970c843ad20c77bd5c32a57a not found: ID does not exist" containerID="bef7df271946c5737d62c61b1cf3e85e1e9e4fd5970c843ad20c77bd5c32a57a"
Sep 16 18:49:49.003714 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:49:49.003672    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bef7df271946c5737d62c61b1cf3e85e1e9e4fd5970c843ad20c77bd5c32a57a"} err="failed to get container status \"bef7df271946c5737d62c61b1cf3e85e1e9e4fd5970c843ad20c77bd5c32a57a\": rpc error: code = NotFound desc = could not find container \"bef7df271946c5737d62c61b1cf3e85e1e9e4fd5970c843ad20c77bd5c32a57a\": container with ID starting with bef7df271946c5737d62c61b1cf3e85e1e9e4fd5970c843ad20c77bd5c32a57a not found: ID does not exist"
Sep 16 18:49:49.003714 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:49:49.003690    2581 scope.go:117] "RemoveContainer" containerID="f4663becc1918a7e6924c74d9a68748b56439cd09776d57e9ab09ca5fff6bf04"
Sep 16 18:49:49.003976 ip-10-0-138-127 kubenswrapper[2581]: E0916 18:49:49.003952    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f4663becc1918a7e6924c74d9a68748b56439cd09776d57e9ab09ca5fff6bf04\": container with ID starting with f4663becc1918a7e6924c74d9a68748b56439cd09776d57e9ab09ca5fff6bf04 not found: ID does not exist" containerID="f4663becc1918a7e6924c74d9a68748b56439cd09776d57e9ab09ca5fff6bf04"
Sep 16 18:49:49.004058 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:49:49.003984    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f4663becc1918a7e6924c74d9a68748b56439cd09776d57e9ab09ca5fff6bf04"} err="failed to get container status \"f4663becc1918a7e6924c74d9a68748b56439cd09776d57e9ab09ca5fff6bf04\": rpc error: code = NotFound desc = could not find container \"f4663becc1918a7e6924c74d9a68748b56439cd09776d57e9ab09ca5fff6bf04\": container with ID starting with f4663becc1918a7e6924c74d9a68748b56439cd09776d57e9ab09ca5fff6bf04 not found: ID does not exist"
Sep 16 18:49:49.004058 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:49:49.004000    2581 scope.go:117] "RemoveContainer" containerID="533a951e6971354aecdc7e706995006228615b0302070d4796da3b7ee87b2e85"
Sep 16 18:49:49.004301 ip-10-0-138-127 kubenswrapper[2581]: E0916 18:49:49.004284    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"533a951e6971354aecdc7e706995006228615b0302070d4796da3b7ee87b2e85\": container with ID starting with 533a951e6971354aecdc7e706995006228615b0302070d4796da3b7ee87b2e85 not found: ID does not exist" containerID="533a951e6971354aecdc7e706995006228615b0302070d4796da3b7ee87b2e85"
Sep 16 18:49:49.004374 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:49:49.004309    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"533a951e6971354aecdc7e706995006228615b0302070d4796da3b7ee87b2e85"} err="failed to get container status \"533a951e6971354aecdc7e706995006228615b0302070d4796da3b7ee87b2e85\": rpc error: code = NotFound desc = could not find container \"533a951e6971354aecdc7e706995006228615b0302070d4796da3b7ee87b2e85\": container with ID starting with 533a951e6971354aecdc7e706995006228615b0302070d4796da3b7ee87b2e85 not found: ID does not exist"
Sep 16 18:49:49.004374 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:49:49.004328    2581 scope.go:117] "RemoveContainer" containerID="1a89a9e7a2375bf34bb12157b14aa09a24a45dab3add18d098c5dcb6c5c0a79a"
Sep 16 18:49:49.004614 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:49:49.004598    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1a89a9e7a2375bf34bb12157b14aa09a24a45dab3add18d098c5dcb6c5c0a79a"} err="failed to get container status \"1a89a9e7a2375bf34bb12157b14aa09a24a45dab3add18d098c5dcb6c5c0a79a\": rpc error: code = NotFound desc = could not find container \"1a89a9e7a2375bf34bb12157b14aa09a24a45dab3add18d098c5dcb6c5c0a79a\": container with ID starting with 1a89a9e7a2375bf34bb12157b14aa09a24a45dab3add18d098c5dcb6c5c0a79a not found: ID does not exist"
Sep 16 18:49:49.004684 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:49:49.004615    2581 scope.go:117] "RemoveContainer" containerID="8d7b8e7f9e6b1851ac6b5e6beb3beadd022252cd070b853e77209f3d532e0025"
Sep 16 18:49:49.004922 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:49:49.004899    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8d7b8e7f9e6b1851ac6b5e6beb3beadd022252cd070b853e77209f3d532e0025"} err="failed to get container status \"8d7b8e7f9e6b1851ac6b5e6beb3beadd022252cd070b853e77209f3d532e0025\": rpc error: code = NotFound desc = could not find container \"8d7b8e7f9e6b1851ac6b5e6beb3beadd022252cd070b853e77209f3d532e0025\": container with ID starting with 8d7b8e7f9e6b1851ac6b5e6beb3beadd022252cd070b853e77209f3d532e0025 not found: ID does not exist"
Sep 16 18:49:49.004997 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:49:49.004923    2581 scope.go:117] "RemoveContainer" containerID="45a58d97808bbe7579fb16590a57b26a5743ac74ecfe45ada969fc1685ed05d7"
Sep 16 18:49:49.005115 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:49:49.005098    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\""
Sep 16 18:49:49.005179 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:49:49.005128    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\""
Sep 16 18:49:49.005179 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:49:49.005141    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\""
Sep 16 18:49:49.005179 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:49:49.005156    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-bm4pd\""
Sep 16 18:49:49.005354 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:49:49.005098    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\""
Sep 16 18:49:49.005354 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:49:49.005100    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\""
Sep 16 18:49:49.005354 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:49:49.005279    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"45a58d97808bbe7579fb16590a57b26a5743ac74ecfe45ada969fc1685ed05d7"} err="failed to get container status \"45a58d97808bbe7579fb16590a57b26a5743ac74ecfe45ada969fc1685ed05d7\": rpc error: code = NotFound desc = could not find container \"45a58d97808bbe7579fb16590a57b26a5743ac74ecfe45ada969fc1685ed05d7\": container with ID starting with 45a58d97808bbe7579fb16590a57b26a5743ac74ecfe45ada969fc1685ed05d7 not found: ID does not exist"
Sep 16 18:49:49.005354 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:49:49.005297    2581 scope.go:117] "RemoveContainer" containerID="b95ace2ac8e0f37bc18c57982ae4b718ba41df9190333b9656adde5cdfbac64c"
Sep 16 18:49:49.005354 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:49:49.005298    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\""
Sep 16 18:49:49.005354 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:49:49.005098    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\""
Sep 16 18:49:49.005354 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:49:49.005350    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\""
Sep 16 18:49:49.005677 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:49:49.005508    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b95ace2ac8e0f37bc18c57982ae4b718ba41df9190333b9656adde5cdfbac64c"} err="failed to get container status \"b95ace2ac8e0f37bc18c57982ae4b718ba41df9190333b9656adde5cdfbac64c\": rpc error: code = NotFound desc = could not find container \"b95ace2ac8e0f37bc18c57982ae4b718ba41df9190333b9656adde5cdfbac64c\": container with ID starting with b95ace2ac8e0f37bc18c57982ae4b718ba41df9190333b9656adde5cdfbac64c not found: ID does not exist"
Sep 16 18:49:49.005677 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:49:49.005523    2581 scope.go:117] "RemoveContainer" containerID="bef7df271946c5737d62c61b1cf3e85e1e9e4fd5970c843ad20c77bd5c32a57a"
Sep 16 18:49:49.005759 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:49:49.005743    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bef7df271946c5737d62c61b1cf3e85e1e9e4fd5970c843ad20c77bd5c32a57a"} err="failed to get container status \"bef7df271946c5737d62c61b1cf3e85e1e9e4fd5970c843ad20c77bd5c32a57a\": rpc error: code = NotFound desc = could not find container \"bef7df271946c5737d62c61b1cf3e85e1e9e4fd5970c843ad20c77bd5c32a57a\": container with ID starting with bef7df271946c5737d62c61b1cf3e85e1e9e4fd5970c843ad20c77bd5c32a57a not found: ID does not exist"
Sep 16 18:49:49.005808 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:49:49.005763    2581 scope.go:117] "RemoveContainer" containerID="f4663becc1918a7e6924c74d9a68748b56439cd09776d57e9ab09ca5fff6bf04"
Sep 16 18:49:49.006048 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:49:49.006030    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f4663becc1918a7e6924c74d9a68748b56439cd09776d57e9ab09ca5fff6bf04"} err="failed to get container status \"f4663becc1918a7e6924c74d9a68748b56439cd09776d57e9ab09ca5fff6bf04\": rpc error: code = NotFound desc = could not find container \"f4663becc1918a7e6924c74d9a68748b56439cd09776d57e9ab09ca5fff6bf04\": container with ID starting with f4663becc1918a7e6924c74d9a68748b56439cd09776d57e9ab09ca5fff6bf04 not found: ID does not exist"
Sep 16 18:49:49.006048 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:49:49.006046    2581 scope.go:117] "RemoveContainer" containerID="533a951e6971354aecdc7e706995006228615b0302070d4796da3b7ee87b2e85"
Sep 16 18:49:49.006326 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:49:49.006307    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"533a951e6971354aecdc7e706995006228615b0302070d4796da3b7ee87b2e85"} err="failed to get container status \"533a951e6971354aecdc7e706995006228615b0302070d4796da3b7ee87b2e85\": rpc error: code = NotFound desc = could not find container \"533a951e6971354aecdc7e706995006228615b0302070d4796da3b7ee87b2e85\": container with ID starting with 533a951e6971354aecdc7e706995006228615b0302070d4796da3b7ee87b2e85 not found: ID does not exist"
Sep 16 18:49:49.012116 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:49:49.012095    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\""
Sep 16 18:49:49.020891 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:49:49.020862    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Sep 16 18:49:49.176950 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:49:49.176916    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/f3f31704-4b1f-4604-aac3-6cb89d0f642e-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"f3f31704-4b1f-4604-aac3-6cb89d0f642e\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 18:49:49.177125 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:49:49.176967    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s4x5r\" (UniqueName: \"kubernetes.io/projected/f3f31704-4b1f-4604-aac3-6cb89d0f642e-kube-api-access-s4x5r\") pod \"alertmanager-main-0\" (UID: \"f3f31704-4b1f-4604-aac3-6cb89d0f642e\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 18:49:49.177125 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:49:49.176998    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/f3f31704-4b1f-4604-aac3-6cb89d0f642e-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"f3f31704-4b1f-4604-aac3-6cb89d0f642e\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 18:49:49.177125 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:49:49.177070    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/f3f31704-4b1f-4604-aac3-6cb89d0f642e-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"f3f31704-4b1f-4604-aac3-6cb89d0f642e\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 18:49:49.177125 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:49:49.177104    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/f3f31704-4b1f-4604-aac3-6cb89d0f642e-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"f3f31704-4b1f-4604-aac3-6cb89d0f642e\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 18:49:49.177390 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:49:49.177133    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/f3f31704-4b1f-4604-aac3-6cb89d0f642e-web-config\") pod \"alertmanager-main-0\" (UID: \"f3f31704-4b1f-4604-aac3-6cb89d0f642e\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 18:49:49.177390 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:49:49.177265    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/f3f31704-4b1f-4604-aac3-6cb89d0f642e-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"f3f31704-4b1f-4604-aac3-6cb89d0f642e\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 18:49:49.177390 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:49:49.177301    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/f3f31704-4b1f-4604-aac3-6cb89d0f642e-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"f3f31704-4b1f-4604-aac3-6cb89d0f642e\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 18:49:49.177390 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:49:49.177333    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/f3f31704-4b1f-4604-aac3-6cb89d0f642e-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"f3f31704-4b1f-4604-aac3-6cb89d0f642e\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 18:49:49.177390 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:49:49.177369    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/f3f31704-4b1f-4604-aac3-6cb89d0f642e-config-volume\") pod \"alertmanager-main-0\" (UID: \"f3f31704-4b1f-4604-aac3-6cb89d0f642e\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 18:49:49.177613 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:49:49.177390    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/f3f31704-4b1f-4604-aac3-6cb89d0f642e-tls-assets\") pod \"alertmanager-main-0\" (UID: \"f3f31704-4b1f-4604-aac3-6cb89d0f642e\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 18:49:49.177613 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:49:49.177416    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f3f31704-4b1f-4604-aac3-6cb89d0f642e-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"f3f31704-4b1f-4604-aac3-6cb89d0f642e\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 18:49:49.177613 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:49:49.177491    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/f3f31704-4b1f-4604-aac3-6cb89d0f642e-config-out\") pod \"alertmanager-main-0\" (UID: \"f3f31704-4b1f-4604-aac3-6cb89d0f642e\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 18:49:49.278653 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:49:49.278618    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/f3f31704-4b1f-4604-aac3-6cb89d0f642e-config-volume\") pod \"alertmanager-main-0\" (UID: \"f3f31704-4b1f-4604-aac3-6cb89d0f642e\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 18:49:49.278653 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:49:49.278653    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/f3f31704-4b1f-4604-aac3-6cb89d0f642e-tls-assets\") pod \"alertmanager-main-0\" (UID: \"f3f31704-4b1f-4604-aac3-6cb89d0f642e\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 18:49:49.278837 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:49:49.278673    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f3f31704-4b1f-4604-aac3-6cb89d0f642e-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"f3f31704-4b1f-4604-aac3-6cb89d0f642e\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 18:49:49.278906 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:49:49.278887    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/f3f31704-4b1f-4604-aac3-6cb89d0f642e-config-out\") pod \"alertmanager-main-0\" (UID: \"f3f31704-4b1f-4604-aac3-6cb89d0f642e\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 18:49:49.278986 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:49:49.278952    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/f3f31704-4b1f-4604-aac3-6cb89d0f642e-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"f3f31704-4b1f-4604-aac3-6cb89d0f642e\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 18:49:49.279125 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:49:49.279008    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s4x5r\" (UniqueName: \"kubernetes.io/projected/f3f31704-4b1f-4604-aac3-6cb89d0f642e-kube-api-access-s4x5r\") pod \"alertmanager-main-0\" (UID: \"f3f31704-4b1f-4604-aac3-6cb89d0f642e\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 18:49:49.279125 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:49:49.279042    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/f3f31704-4b1f-4604-aac3-6cb89d0f642e-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"f3f31704-4b1f-4604-aac3-6cb89d0f642e\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 18:49:49.279125 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:49:49.279095    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/f3f31704-4b1f-4604-aac3-6cb89d0f642e-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"f3f31704-4b1f-4604-aac3-6cb89d0f642e\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 18:49:49.279125 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:49:49.279122    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/f3f31704-4b1f-4604-aac3-6cb89d0f642e-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"f3f31704-4b1f-4604-aac3-6cb89d0f642e\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 18:49:49.279360 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:49:49.279159    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/f3f31704-4b1f-4604-aac3-6cb89d0f642e-web-config\") pod \"alertmanager-main-0\" (UID: \"f3f31704-4b1f-4604-aac3-6cb89d0f642e\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 18:49:49.279360 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:49:49.279224    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/f3f31704-4b1f-4604-aac3-6cb89d0f642e-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"f3f31704-4b1f-4604-aac3-6cb89d0f642e\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 18:49:49.279360 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:49:49.279248    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/f3f31704-4b1f-4604-aac3-6cb89d0f642e-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"f3f31704-4b1f-4604-aac3-6cb89d0f642e\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 18:49:49.279360 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:49:49.279279    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/f3f31704-4b1f-4604-aac3-6cb89d0f642e-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"f3f31704-4b1f-4604-aac3-6cb89d0f642e\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 18:49:49.279565 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:49:49.279419    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/f3f31704-4b1f-4604-aac3-6cb89d0f642e-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"f3f31704-4b1f-4604-aac3-6cb89d0f642e\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 18:49:49.279774 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:49:49.279745    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f3f31704-4b1f-4604-aac3-6cb89d0f642e-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"f3f31704-4b1f-4604-aac3-6cb89d0f642e\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 18:49:49.282064 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:49:49.282039    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/f3f31704-4b1f-4604-aac3-6cb89d0f642e-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"f3f31704-4b1f-4604-aac3-6cb89d0f642e\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 18:49:49.282453 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:49:49.282353    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/f3f31704-4b1f-4604-aac3-6cb89d0f642e-config-out\") pod \"alertmanager-main-0\" (UID: \"f3f31704-4b1f-4604-aac3-6cb89d0f642e\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 18:49:49.282453 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:49:49.282448    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/f3f31704-4b1f-4604-aac3-6cb89d0f642e-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"f3f31704-4b1f-4604-aac3-6cb89d0f642e\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 18:49:49.282605 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:49:49.282505    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/f3f31704-4b1f-4604-aac3-6cb89d0f642e-config-volume\") pod \"alertmanager-main-0\" (UID: \"f3f31704-4b1f-4604-aac3-6cb89d0f642e\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 18:49:49.282605 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:49:49.282513    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/f3f31704-4b1f-4604-aac3-6cb89d0f642e-tls-assets\") pod \"alertmanager-main-0\" (UID: \"f3f31704-4b1f-4604-aac3-6cb89d0f642e\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 18:49:49.282722 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:49:49.282708    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/f3f31704-4b1f-4604-aac3-6cb89d0f642e-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"f3f31704-4b1f-4604-aac3-6cb89d0f642e\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 18:49:49.282849 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:49:49.282825    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/f3f31704-4b1f-4604-aac3-6cb89d0f642e-web-config\") pod \"alertmanager-main-0\" (UID: \"f3f31704-4b1f-4604-aac3-6cb89d0f642e\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 18:49:49.283039 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:49:49.283019    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/f3f31704-4b1f-4604-aac3-6cb89d0f642e-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"f3f31704-4b1f-4604-aac3-6cb89d0f642e\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 18:49:49.283166 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:49:49.283148    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/f3f31704-4b1f-4604-aac3-6cb89d0f642e-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"f3f31704-4b1f-4604-aac3-6cb89d0f642e\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 18:49:49.284822 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:49:49.284804    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/f3f31704-4b1f-4604-aac3-6cb89d0f642e-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"f3f31704-4b1f-4604-aac3-6cb89d0f642e\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 18:49:49.287833 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:49:49.287814    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-s4x5r\" (UniqueName: \"kubernetes.io/projected/f3f31704-4b1f-4604-aac3-6cb89d0f642e-kube-api-access-s4x5r\") pod \"alertmanager-main-0\" (UID: \"f3f31704-4b1f-4604-aac3-6cb89d0f642e\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 18:49:49.314009 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:49:49.313971    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Sep 16 18:49:49.607678 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:49:49.607595    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Sep 16 18:49:49.610551 ip-10-0-138-127 kubenswrapper[2581]: W0916 18:49:49.610521    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf3f31704_4b1f_4604_aac3_6cb89d0f642e.slice/crio-d481d0df34e02c4fa31f9e0ee28c0bc6ef53002318935feaa2dbda0113d21430 WatchSource:0}: Error finding container d481d0df34e02c4fa31f9e0ee28c0bc6ef53002318935feaa2dbda0113d21430: Status 404 returned error can't find the container with id d481d0df34e02c4fa31f9e0ee28c0bc6ef53002318935feaa2dbda0113d21430
Sep 16 18:49:49.940763 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:49:49.940667    2581 generic.go:356] "Generic (PLEG): container finished" podID="f3f31704-4b1f-4604-aac3-6cb89d0f642e" containerID="1aa77fa05c5f0e5a0abb25981a0d85b51687ddc868658594d96dd70e6f9aef90" exitCode=0
Sep 16 18:49:49.941225 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:49:49.940760    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"f3f31704-4b1f-4604-aac3-6cb89d0f642e","Type":"ContainerDied","Data":"1aa77fa05c5f0e5a0abb25981a0d85b51687ddc868658594d96dd70e6f9aef90"}
Sep 16 18:49:49.941225 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:49:49.940835    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"f3f31704-4b1f-4604-aac3-6cb89d0f642e","Type":"ContainerStarted","Data":"d481d0df34e02c4fa31f9e0ee28c0bc6ef53002318935feaa2dbda0113d21430"}
Sep 16 18:49:50.729103 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:49:50.729061    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="89fc85f4-ec94-4459-8311-f3dbf4681bf8" path="/var/lib/kubelet/pods/89fc85f4-ec94-4459-8311-f3dbf4681bf8/volumes"
Sep 16 18:49:50.948973 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:49:50.948941    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"f3f31704-4b1f-4604-aac3-6cb89d0f642e","Type":"ContainerStarted","Data":"6433c4551eeeed13994509e0a1c6c096439e517b1ba751559ae0e1b3a094093a"}
Sep 16 18:49:50.949345 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:49:50.949002    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"f3f31704-4b1f-4604-aac3-6cb89d0f642e","Type":"ContainerStarted","Data":"43a79fb565cea6e79f508650a354b68d5d4f18a75e31d3c24e96295b62fb5991"}
Sep 16 18:49:50.949345 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:49:50.949030    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"f3f31704-4b1f-4604-aac3-6cb89d0f642e","Type":"ContainerStarted","Data":"b4ad30ad6700f748fed1544dd181a475f382422c33d3a59252dbde0eedde2c5c"}
Sep 16 18:49:50.949345 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:49:50.949054    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"f3f31704-4b1f-4604-aac3-6cb89d0f642e","Type":"ContainerStarted","Data":"913d33e0e01bfe703046c14df95f31cbe23af9dbcc0b6e9ac60f009eaa719ecd"}
Sep 16 18:49:50.949345 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:49:50.949076    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"f3f31704-4b1f-4604-aac3-6cb89d0f642e","Type":"ContainerStarted","Data":"6d576b994a0722cbecf96146095f77a58a490c6dc5f551855265d9fc01c6b107"}
Sep 16 18:49:50.949345 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:49:50.949099    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"f3f31704-4b1f-4604-aac3-6cb89d0f642e","Type":"ContainerStarted","Data":"35943e5438319ee60b5eb4f8f718145a33db2391098af8d1d56319edcaf773ff"}
Sep 16 18:49:51.005345 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:49:51.005229    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/alertmanager-main-0" podStartSLOduration=2.986572098 podStartE2EDuration="3.005209122s" totalImagesPullingTime="0s" totalInitContainerRuntime="18.637024ms" isStatefulPod=false podCreationTimestamp="2026-09-16 18:49:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:49:51.003910071 +0000 UTC m=+260.823323981" watchObservedRunningTime="2026-09-16 18:49:51.005209122 +0000 UTC m=+260.824623028"
Sep 16 18:49:51.169483 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:49:51.169448    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 16 18:49:51.169769 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:49:51.169739    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="33280f86-4497-4343-829b-4781cbfd5e5f" containerName="prometheus" containerID="cri-o://83ac526d7a86604d68fe8d92bf00597bc299f57b92d72f39a7fb68e890df4232" gracePeriod=600
Sep 16 18:49:51.169925 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:49:51.169775    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="33280f86-4497-4343-829b-4781cbfd5e5f" containerName="thanos-sidecar" containerID="cri-o://f3007c04a6c7666ea52a294ed272da999f867f7ef844acc86315cb2a8cf1bec2" gracePeriod=600
Sep 16 18:49:51.169925 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:49:51.169802    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="33280f86-4497-4343-829b-4781cbfd5e5f" containerName="kube-rbac-proxy-thanos" containerID="cri-o://3f3dd1883e45a82010062071dbb5a547b0398022c662e0974bb6035db93c527b" gracePeriod=600
Sep 16 18:49:51.169925 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:49:51.169820    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="33280f86-4497-4343-829b-4781cbfd5e5f" containerName="kube-rbac-proxy-web" containerID="cri-o://314e4f5337c0ea1e4e5e6b86e35ae4ee43a7775322585a4488852c09df93923f" gracePeriod=600
Sep 16 18:49:51.169925 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:49:51.169774    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="33280f86-4497-4343-829b-4781cbfd5e5f" containerName="kube-rbac-proxy" containerID="cri-o://631c5ce2aac8760ec03de8d2609386e5bd65a6ab1a2ff3a684b3cc1c0d53f5a5" gracePeriod=600
Sep 16 18:49:51.170114 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:49:51.169785    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="33280f86-4497-4343-829b-4781cbfd5e5f" containerName="config-reloader" containerID="cri-o://fe8880e1d60a0122daff9a84c8c263336a34080fa9d654bae40ab0456d777281" gracePeriod=600
Sep 16 18:49:51.415300 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:49:51.415273    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:49:51.599768 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:49:51.599666    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/33280f86-4497-4343-829b-4781cbfd5e5f-prometheus-k8s-rulefiles-2\") pod \"33280f86-4497-4343-829b-4781cbfd5e5f\" (UID: \"33280f86-4497-4343-829b-4781cbfd5e5f\") "
Sep 16 18:49:51.599768 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:49:51.599729    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/33280f86-4497-4343-829b-4781cbfd5e5f-secret-prometheus-k8s-tls\") pod \"33280f86-4497-4343-829b-4781cbfd5e5f\" (UID: \"33280f86-4497-4343-829b-4781cbfd5e5f\") "
Sep 16 18:49:51.599768 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:49:51.599757    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/33280f86-4497-4343-829b-4781cbfd5e5f-tls-assets\") pod \"33280f86-4497-4343-829b-4781cbfd5e5f\" (UID: \"33280f86-4497-4343-829b-4781cbfd5e5f\") "
Sep 16 18:49:51.600050 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:49:51.599910    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/33280f86-4497-4343-829b-4781cbfd5e5f-thanos-prometheus-http-client-file\") pod \"33280f86-4497-4343-829b-4781cbfd5e5f\" (UID: \"33280f86-4497-4343-829b-4781cbfd5e5f\") "
Sep 16 18:49:51.600050 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:49:51.599940    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/33280f86-4497-4343-829b-4781cbfd5e5f-configmap-metrics-client-ca\") pod \"33280f86-4497-4343-829b-4781cbfd5e5f\" (UID: \"33280f86-4497-4343-829b-4781cbfd5e5f\") "
Sep 16 18:49:51.600050 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:49:51.599965    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/33280f86-4497-4343-829b-4781cbfd5e5f-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"33280f86-4497-4343-829b-4781cbfd5e5f\" (UID: \"33280f86-4497-4343-829b-4781cbfd5e5f\") "
Sep 16 18:49:51.600050 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:49:51.599983    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/33280f86-4497-4343-829b-4781cbfd5e5f-configmap-serving-certs-ca-bundle\") pod \"33280f86-4497-4343-829b-4781cbfd5e5f\" (UID: \"33280f86-4497-4343-829b-4781cbfd5e5f\") "
Sep 16 18:49:51.600050 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:49:51.600018    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/33280f86-4497-4343-829b-4781cbfd5e5f-config\") pod \"33280f86-4497-4343-829b-4781cbfd5e5f\" (UID: \"33280f86-4497-4343-829b-4781cbfd5e5f\") "
Sep 16 18:49:51.600050 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:49:51.600040    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/33280f86-4497-4343-829b-4781cbfd5e5f-config-out\") pod \"33280f86-4497-4343-829b-4781cbfd5e5f\" (UID: \"33280f86-4497-4343-829b-4781cbfd5e5f\") "
Sep 16 18:49:51.600372 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:49:51.600057    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/33280f86-4497-4343-829b-4781cbfd5e5f-prometheus-k8s-rulefiles-2" (OuterVolumeSpecName: "prometheus-k8s-rulefiles-2") pod "33280f86-4497-4343-829b-4781cbfd5e5f" (UID: "33280f86-4497-4343-829b-4781cbfd5e5f"). InnerVolumeSpecName "prometheus-k8s-rulefiles-2". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 18:49:51.600372 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:49:51.600073    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/33280f86-4497-4343-829b-4781cbfd5e5f-prometheus-k8s-db\") pod \"33280f86-4497-4343-829b-4781cbfd5e5f\" (UID: \"33280f86-4497-4343-829b-4781cbfd5e5f\") "
Sep 16 18:49:51.600372 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:49:51.600102    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/33280f86-4497-4343-829b-4781cbfd5e5f-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"33280f86-4497-4343-829b-4781cbfd5e5f\" (UID: \"33280f86-4497-4343-829b-4781cbfd5e5f\") "
Sep 16 18:49:51.600372 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:49:51.600130    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/33280f86-4497-4343-829b-4781cbfd5e5f-prometheus-k8s-rulefiles-0\") pod \"33280f86-4497-4343-829b-4781cbfd5e5f\" (UID: \"33280f86-4497-4343-829b-4781cbfd5e5f\") "
Sep 16 18:49:51.600372 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:49:51.600154    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9nk74\" (UniqueName: \"kubernetes.io/projected/33280f86-4497-4343-829b-4781cbfd5e5f-kube-api-access-9nk74\") pod \"33280f86-4497-4343-829b-4781cbfd5e5f\" (UID: \"33280f86-4497-4343-829b-4781cbfd5e5f\") "
Sep 16 18:49:51.600372 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:49:51.600181    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/33280f86-4497-4343-829b-4781cbfd5e5f-secret-grpc-tls\") pod \"33280f86-4497-4343-829b-4781cbfd5e5f\" (UID: \"33280f86-4497-4343-829b-4781cbfd5e5f\") "
Sep 16 18:49:51.600372 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:49:51.600232    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/33280f86-4497-4343-829b-4781cbfd5e5f-web-config\") pod \"33280f86-4497-4343-829b-4781cbfd5e5f\" (UID: \"33280f86-4497-4343-829b-4781cbfd5e5f\") "
Sep 16 18:49:51.600372 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:49:51.600270    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/33280f86-4497-4343-829b-4781cbfd5e5f-prometheus-k8s-rulefiles-1\") pod \"33280f86-4497-4343-829b-4781cbfd5e5f\" (UID: \"33280f86-4497-4343-829b-4781cbfd5e5f\") "
Sep 16 18:49:51.600372 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:49:51.600294    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/33280f86-4497-4343-829b-4781cbfd5e5f-secret-kube-rbac-proxy\") pod \"33280f86-4497-4343-829b-4781cbfd5e5f\" (UID: \"33280f86-4497-4343-829b-4781cbfd5e5f\") "
Sep 16 18:49:51.600372 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:49:51.600340    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/33280f86-4497-4343-829b-4781cbfd5e5f-secret-metrics-client-certs\") pod \"33280f86-4497-4343-829b-4781cbfd5e5f\" (UID: \"33280f86-4497-4343-829b-4781cbfd5e5f\") "
Sep 16 18:49:51.600372 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:49:51.600374    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/33280f86-4497-4343-829b-4781cbfd5e5f-configmap-kubelet-serving-ca-bundle\") pod \"33280f86-4497-4343-829b-4781cbfd5e5f\" (UID: \"33280f86-4497-4343-829b-4781cbfd5e5f\") "
Sep 16 18:49:51.601072 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:49:51.600419    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/33280f86-4497-4343-829b-4781cbfd5e5f-prometheus-trusted-ca-bundle\") pod \"33280f86-4497-4343-829b-4781cbfd5e5f\" (UID: \"33280f86-4497-4343-829b-4781cbfd5e5f\") "
Sep 16 18:49:51.601072 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:49:51.600678    2581 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/33280f86-4497-4343-829b-4781cbfd5e5f-prometheus-k8s-rulefiles-2\") on node \"ip-10-0-138-127.ec2.internal\" DevicePath \"\""
Sep 16 18:49:51.603788 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:49:51.600372    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/33280f86-4497-4343-829b-4781cbfd5e5f-configmap-metrics-client-ca" (OuterVolumeSpecName: "configmap-metrics-client-ca") pod "33280f86-4497-4343-829b-4781cbfd5e5f" (UID: "33280f86-4497-4343-829b-4781cbfd5e5f"). InnerVolumeSpecName "configmap-metrics-client-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 18:49:51.603788 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:49:51.600802    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/33280f86-4497-4343-829b-4781cbfd5e5f-configmap-serving-certs-ca-bundle" (OuterVolumeSpecName: "configmap-serving-certs-ca-bundle") pod "33280f86-4497-4343-829b-4781cbfd5e5f" (UID: "33280f86-4497-4343-829b-4781cbfd5e5f"). InnerVolumeSpecName "configmap-serving-certs-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 18:49:51.603788 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:49:51.600987    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/33280f86-4497-4343-829b-4781cbfd5e5f-prometheus-trusted-ca-bundle" (OuterVolumeSpecName: "prometheus-trusted-ca-bundle") pod "33280f86-4497-4343-829b-4781cbfd5e5f" (UID: "33280f86-4497-4343-829b-4781cbfd5e5f"). InnerVolumeSpecName "prometheus-trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 18:49:51.603788 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:49:51.601112    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/33280f86-4497-4343-829b-4781cbfd5e5f-prometheus-k8s-rulefiles-1" (OuterVolumeSpecName: "prometheus-k8s-rulefiles-1") pod "33280f86-4497-4343-829b-4781cbfd5e5f" (UID: "33280f86-4497-4343-829b-4781cbfd5e5f"). InnerVolumeSpecName "prometheus-k8s-rulefiles-1". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 18:49:51.603788 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:49:51.602049    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/33280f86-4497-4343-829b-4781cbfd5e5f-prometheus-k8s-rulefiles-0" (OuterVolumeSpecName: "prometheus-k8s-rulefiles-0") pod "33280f86-4497-4343-829b-4781cbfd5e5f" (UID: "33280f86-4497-4343-829b-4781cbfd5e5f"). InnerVolumeSpecName "prometheus-k8s-rulefiles-0". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 18:49:51.603788 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:49:51.602851    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/33280f86-4497-4343-829b-4781cbfd5e5f-secret-prometheus-k8s-tls" (OuterVolumeSpecName: "secret-prometheus-k8s-tls") pod "33280f86-4497-4343-829b-4781cbfd5e5f" (UID: "33280f86-4497-4343-829b-4781cbfd5e5f"). InnerVolumeSpecName "secret-prometheus-k8s-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 18:49:51.603788 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:49:51.602878    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/33280f86-4497-4343-829b-4781cbfd5e5f-thanos-prometheus-http-client-file" (OuterVolumeSpecName: "thanos-prometheus-http-client-file") pod "33280f86-4497-4343-829b-4781cbfd5e5f" (UID: "33280f86-4497-4343-829b-4781cbfd5e5f"). InnerVolumeSpecName "thanos-prometheus-http-client-file". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 18:49:51.603788 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:49:51.602892    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/33280f86-4497-4343-829b-4781cbfd5e5f-config" (OuterVolumeSpecName: "config") pod "33280f86-4497-4343-829b-4781cbfd5e5f" (UID: "33280f86-4497-4343-829b-4781cbfd5e5f"). InnerVolumeSpecName "config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 18:49:51.603788 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:49:51.602903    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/33280f86-4497-4343-829b-4781cbfd5e5f-tls-assets" (OuterVolumeSpecName: "tls-assets") pod "33280f86-4497-4343-829b-4781cbfd5e5f" (UID: "33280f86-4497-4343-829b-4781cbfd5e5f"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 18:49:51.603788 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:49:51.603257    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/33280f86-4497-4343-829b-4781cbfd5e5f-configmap-kubelet-serving-ca-bundle" (OuterVolumeSpecName: "configmap-kubelet-serving-ca-bundle") pod "33280f86-4497-4343-829b-4781cbfd5e5f" (UID: "33280f86-4497-4343-829b-4781cbfd5e5f"). InnerVolumeSpecName "configmap-kubelet-serving-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 18:49:51.603788 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:49:51.603263    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/33280f86-4497-4343-829b-4781cbfd5e5f-secret-prometheus-k8s-kube-rbac-proxy-web" (OuterVolumeSpecName: "secret-prometheus-k8s-kube-rbac-proxy-web") pod "33280f86-4497-4343-829b-4781cbfd5e5f" (UID: "33280f86-4497-4343-829b-4781cbfd5e5f"). InnerVolumeSpecName "secret-prometheus-k8s-kube-rbac-proxy-web". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 18:49:51.603788 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:49:51.603634    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/33280f86-4497-4343-829b-4781cbfd5e5f-secret-kube-rbac-proxy" (OuterVolumeSpecName: "secret-kube-rbac-proxy") pod "33280f86-4497-4343-829b-4781cbfd5e5f" (UID: "33280f86-4497-4343-829b-4781cbfd5e5f"). InnerVolumeSpecName "secret-kube-rbac-proxy". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 18:49:51.604558 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:49:51.604513    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/33280f86-4497-4343-829b-4781cbfd5e5f-prometheus-k8s-db" (OuterVolumeSpecName: "prometheus-k8s-db") pod "33280f86-4497-4343-829b-4781cbfd5e5f" (UID: "33280f86-4497-4343-829b-4781cbfd5e5f"). InnerVolumeSpecName "prometheus-k8s-db". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 18:49:51.604718 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:49:51.604693    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/33280f86-4497-4343-829b-4781cbfd5e5f-secret-prometheus-k8s-thanos-sidecar-tls" (OuterVolumeSpecName: "secret-prometheus-k8s-thanos-sidecar-tls") pod "33280f86-4497-4343-829b-4781cbfd5e5f" (UID: "33280f86-4497-4343-829b-4781cbfd5e5f"). InnerVolumeSpecName "secret-prometheus-k8s-thanos-sidecar-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 18:49:51.604855 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:49:51.604716    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/33280f86-4497-4343-829b-4781cbfd5e5f-kube-api-access-9nk74" (OuterVolumeSpecName: "kube-api-access-9nk74") pod "33280f86-4497-4343-829b-4781cbfd5e5f" (UID: "33280f86-4497-4343-829b-4781cbfd5e5f"). InnerVolumeSpecName "kube-api-access-9nk74". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 18:49:51.605401 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:49:51.605378    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/33280f86-4497-4343-829b-4781cbfd5e5f-secret-grpc-tls" (OuterVolumeSpecName: "secret-grpc-tls") pod "33280f86-4497-4343-829b-4781cbfd5e5f" (UID: "33280f86-4497-4343-829b-4781cbfd5e5f"). InnerVolumeSpecName "secret-grpc-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 18:49:51.605536 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:49:51.605411    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/33280f86-4497-4343-829b-4781cbfd5e5f-config-out" (OuterVolumeSpecName: "config-out") pod "33280f86-4497-4343-829b-4781cbfd5e5f" (UID: "33280f86-4497-4343-829b-4781cbfd5e5f"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 18:49:51.605536 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:49:51.605458    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/33280f86-4497-4343-829b-4781cbfd5e5f-secret-metrics-client-certs" (OuterVolumeSpecName: "secret-metrics-client-certs") pod "33280f86-4497-4343-829b-4781cbfd5e5f" (UID: "33280f86-4497-4343-829b-4781cbfd5e5f"). InnerVolumeSpecName "secret-metrics-client-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 18:49:51.615409 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:49:51.615384    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/33280f86-4497-4343-829b-4781cbfd5e5f-web-config" (OuterVolumeSpecName: "web-config") pod "33280f86-4497-4343-829b-4781cbfd5e5f" (UID: "33280f86-4497-4343-829b-4781cbfd5e5f"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 18:49:51.701174 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:49:51.701142    2581 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/33280f86-4497-4343-829b-4781cbfd5e5f-secret-prometheus-k8s-tls\") on node \"ip-10-0-138-127.ec2.internal\" DevicePath \"\""
Sep 16 18:49:51.701174 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:49:51.701169    2581 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/33280f86-4497-4343-829b-4781cbfd5e5f-tls-assets\") on node \"ip-10-0-138-127.ec2.internal\" DevicePath \"\""
Sep 16 18:49:51.701174 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:49:51.701179    2581 reconciler_common.go:299] "Volume detached for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/33280f86-4497-4343-829b-4781cbfd5e5f-thanos-prometheus-http-client-file\") on node \"ip-10-0-138-127.ec2.internal\" DevicePath \"\""
Sep 16 18:49:51.701423 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:49:51.701219    2581 reconciler_common.go:299] "Volume detached for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/33280f86-4497-4343-829b-4781cbfd5e5f-configmap-metrics-client-ca\") on node \"ip-10-0-138-127.ec2.internal\" DevicePath \"\""
Sep 16 18:49:51.701423 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:49:51.701233    2581 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/33280f86-4497-4343-829b-4781cbfd5e5f-secret-prometheus-k8s-thanos-sidecar-tls\") on node \"ip-10-0-138-127.ec2.internal\" DevicePath \"\""
Sep 16 18:49:51.701423 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:49:51.701249    2581 reconciler_common.go:299] "Volume detached for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/33280f86-4497-4343-829b-4781cbfd5e5f-configmap-serving-certs-ca-bundle\") on node \"ip-10-0-138-127.ec2.internal\" DevicePath \"\""
Sep 16 18:49:51.701423 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:49:51.701261    2581 reconciler_common.go:299] "Volume detached for volume \"config\" (UniqueName: \"kubernetes.io/secret/33280f86-4497-4343-829b-4781cbfd5e5f-config\") on node \"ip-10-0-138-127.ec2.internal\" DevicePath \"\""
Sep 16 18:49:51.701423 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:49:51.701273    2581 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/33280f86-4497-4343-829b-4781cbfd5e5f-config-out\") on node \"ip-10-0-138-127.ec2.internal\" DevicePath \"\""
Sep 16 18:49:51.701423 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:49:51.701281    2581 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/33280f86-4497-4343-829b-4781cbfd5e5f-prometheus-k8s-db\") on node \"ip-10-0-138-127.ec2.internal\" DevicePath \"\""
Sep 16 18:49:51.701423 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:49:51.701291    2581 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/33280f86-4497-4343-829b-4781cbfd5e5f-secret-prometheus-k8s-kube-rbac-proxy-web\") on node \"ip-10-0-138-127.ec2.internal\" DevicePath \"\""
Sep 16 18:49:51.701423 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:49:51.701300    2581 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/33280f86-4497-4343-829b-4781cbfd5e5f-prometheus-k8s-rulefiles-0\") on node \"ip-10-0-138-127.ec2.internal\" DevicePath \"\""
Sep 16 18:49:51.701423 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:49:51.701308    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9nk74\" (UniqueName: \"kubernetes.io/projected/33280f86-4497-4343-829b-4781cbfd5e5f-kube-api-access-9nk74\") on node \"ip-10-0-138-127.ec2.internal\" DevicePath \"\""
Sep 16 18:49:51.701423 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:49:51.701316    2581 reconciler_common.go:299] "Volume detached for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/33280f86-4497-4343-829b-4781cbfd5e5f-secret-grpc-tls\") on node \"ip-10-0-138-127.ec2.internal\" DevicePath \"\""
Sep 16 18:49:51.701423 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:49:51.701323    2581 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/33280f86-4497-4343-829b-4781cbfd5e5f-web-config\") on node \"ip-10-0-138-127.ec2.internal\" DevicePath \"\""
Sep 16 18:49:51.701423 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:49:51.701330    2581 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/33280f86-4497-4343-829b-4781cbfd5e5f-prometheus-k8s-rulefiles-1\") on node \"ip-10-0-138-127.ec2.internal\" DevicePath \"\""
Sep 16 18:49:51.701423 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:49:51.701338    2581 reconciler_common.go:299] "Volume detached for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/33280f86-4497-4343-829b-4781cbfd5e5f-secret-kube-rbac-proxy\") on node \"ip-10-0-138-127.ec2.internal\" DevicePath \"\""
Sep 16 18:49:51.701423 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:49:51.701347    2581 reconciler_common.go:299] "Volume detached for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/33280f86-4497-4343-829b-4781cbfd5e5f-secret-metrics-client-certs\") on node \"ip-10-0-138-127.ec2.internal\" DevicePath \"\""
Sep 16 18:49:51.701423 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:49:51.701355    2581 reconciler_common.go:299] "Volume detached for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/33280f86-4497-4343-829b-4781cbfd5e5f-configmap-kubelet-serving-ca-bundle\") on node \"ip-10-0-138-127.ec2.internal\" DevicePath \"\""
Sep 16 18:49:51.701423 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:49:51.701363    2581 reconciler_common.go:299] "Volume detached for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/33280f86-4497-4343-829b-4781cbfd5e5f-prometheus-trusted-ca-bundle\") on node \"ip-10-0-138-127.ec2.internal\" DevicePath \"\""
Sep 16 18:49:51.958211 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:49:51.958171    2581 generic.go:356] "Generic (PLEG): container finished" podID="33280f86-4497-4343-829b-4781cbfd5e5f" containerID="3f3dd1883e45a82010062071dbb5a547b0398022c662e0974bb6035db93c527b" exitCode=0
Sep 16 18:49:51.958211 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:49:51.958209    2581 generic.go:356] "Generic (PLEG): container finished" podID="33280f86-4497-4343-829b-4781cbfd5e5f" containerID="631c5ce2aac8760ec03de8d2609386e5bd65a6ab1a2ff3a684b3cc1c0d53f5a5" exitCode=0
Sep 16 18:49:51.958640 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:49:51.958219    2581 generic.go:356] "Generic (PLEG): container finished" podID="33280f86-4497-4343-829b-4781cbfd5e5f" containerID="314e4f5337c0ea1e4e5e6b86e35ae4ee43a7775322585a4488852c09df93923f" exitCode=0
Sep 16 18:49:51.958640 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:49:51.958228    2581 generic.go:356] "Generic (PLEG): container finished" podID="33280f86-4497-4343-829b-4781cbfd5e5f" containerID="f3007c04a6c7666ea52a294ed272da999f867f7ef844acc86315cb2a8cf1bec2" exitCode=0
Sep 16 18:49:51.958640 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:49:51.958235    2581 generic.go:356] "Generic (PLEG): container finished" podID="33280f86-4497-4343-829b-4781cbfd5e5f" containerID="fe8880e1d60a0122daff9a84c8c263336a34080fa9d654bae40ab0456d777281" exitCode=0
Sep 16 18:49:51.958640 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:49:51.958244    2581 generic.go:356] "Generic (PLEG): container finished" podID="33280f86-4497-4343-829b-4781cbfd5e5f" containerID="83ac526d7a86604d68fe8d92bf00597bc299f57b92d72f39a7fb68e890df4232" exitCode=0
Sep 16 18:49:51.958640 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:49:51.958238    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"33280f86-4497-4343-829b-4781cbfd5e5f","Type":"ContainerDied","Data":"3f3dd1883e45a82010062071dbb5a547b0398022c662e0974bb6035db93c527b"}
Sep 16 18:49:51.958640 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:49:51.958256    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:49:51.958640 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:49:51.958334    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"33280f86-4497-4343-829b-4781cbfd5e5f","Type":"ContainerDied","Data":"631c5ce2aac8760ec03de8d2609386e5bd65a6ab1a2ff3a684b3cc1c0d53f5a5"}
Sep 16 18:49:51.958640 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:49:51.958378    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"33280f86-4497-4343-829b-4781cbfd5e5f","Type":"ContainerDied","Data":"314e4f5337c0ea1e4e5e6b86e35ae4ee43a7775322585a4488852c09df93923f"}
Sep 16 18:49:51.958640 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:49:51.958406    2581 scope.go:117] "RemoveContainer" containerID="3f3dd1883e45a82010062071dbb5a547b0398022c662e0974bb6035db93c527b"
Sep 16 18:49:51.958640 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:49:51.958418    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"33280f86-4497-4343-829b-4781cbfd5e5f","Type":"ContainerDied","Data":"f3007c04a6c7666ea52a294ed272da999f867f7ef844acc86315cb2a8cf1bec2"}
Sep 16 18:49:51.958640 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:49:51.958463    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"33280f86-4497-4343-829b-4781cbfd5e5f","Type":"ContainerDied","Data":"fe8880e1d60a0122daff9a84c8c263336a34080fa9d654bae40ab0456d777281"}
Sep 16 18:49:51.958640 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:49:51.958506    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"33280f86-4497-4343-829b-4781cbfd5e5f","Type":"ContainerDied","Data":"83ac526d7a86604d68fe8d92bf00597bc299f57b92d72f39a7fb68e890df4232"}
Sep 16 18:49:51.958640 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:49:51.958549    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"33280f86-4497-4343-829b-4781cbfd5e5f","Type":"ContainerDied","Data":"b4b25c7a54e34f336cf21f069c208a6bf0973b30aca905dc2a5dced1936c5302"}
Sep 16 18:49:51.966910 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:49:51.966894    2581 scope.go:117] "RemoveContainer" containerID="631c5ce2aac8760ec03de8d2609386e5bd65a6ab1a2ff3a684b3cc1c0d53f5a5"
Sep 16 18:49:51.974904 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:49:51.974887    2581 scope.go:117] "RemoveContainer" containerID="314e4f5337c0ea1e4e5e6b86e35ae4ee43a7775322585a4488852c09df93923f"
Sep 16 18:49:51.982168 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:49:51.982139    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 16 18:49:51.982547 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:49:51.982530    2581 scope.go:117] "RemoveContainer" containerID="f3007c04a6c7666ea52a294ed272da999f867f7ef844acc86315cb2a8cf1bec2"
Sep 16 18:49:51.988629 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:49:51.988607    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 16 18:49:51.991099 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:49:51.991081    2581 scope.go:117] "RemoveContainer" containerID="fe8880e1d60a0122daff9a84c8c263336a34080fa9d654bae40ab0456d777281"
Sep 16 18:49:51.998449 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:49:51.998435    2581 scope.go:117] "RemoveContainer" containerID="83ac526d7a86604d68fe8d92bf00597bc299f57b92d72f39a7fb68e890df4232"
Sep 16 18:49:52.005724 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:49:52.005701    2581 scope.go:117] "RemoveContainer" containerID="b247a13bc56273cb77128be0df26ac1bd41f05879d206a7acaeb33fd20a62d0b"
Sep 16 18:49:52.012874 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:49:52.012853    2581 scope.go:117] "RemoveContainer" containerID="3f3dd1883e45a82010062071dbb5a547b0398022c662e0974bb6035db93c527b"
Sep 16 18:49:52.013180 ip-10-0-138-127 kubenswrapper[2581]: E0916 18:49:52.013160    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3f3dd1883e45a82010062071dbb5a547b0398022c662e0974bb6035db93c527b\": container with ID starting with 3f3dd1883e45a82010062071dbb5a547b0398022c662e0974bb6035db93c527b not found: ID does not exist" containerID="3f3dd1883e45a82010062071dbb5a547b0398022c662e0974bb6035db93c527b"
Sep 16 18:49:52.013264 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:49:52.013204    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3f3dd1883e45a82010062071dbb5a547b0398022c662e0974bb6035db93c527b"} err="failed to get container status \"3f3dd1883e45a82010062071dbb5a547b0398022c662e0974bb6035db93c527b\": rpc error: code = NotFound desc = could not find container \"3f3dd1883e45a82010062071dbb5a547b0398022c662e0974bb6035db93c527b\": container with ID starting with 3f3dd1883e45a82010062071dbb5a547b0398022c662e0974bb6035db93c527b not found: ID does not exist"
Sep 16 18:49:52.013264 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:49:52.013227    2581 scope.go:117] "RemoveContainer" containerID="631c5ce2aac8760ec03de8d2609386e5bd65a6ab1a2ff3a684b3cc1c0d53f5a5"
Sep 16 18:49:52.013504 ip-10-0-138-127 kubenswrapper[2581]: E0916 18:49:52.013486    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"631c5ce2aac8760ec03de8d2609386e5bd65a6ab1a2ff3a684b3cc1c0d53f5a5\": container with ID starting with 631c5ce2aac8760ec03de8d2609386e5bd65a6ab1a2ff3a684b3cc1c0d53f5a5 not found: ID does not exist" containerID="631c5ce2aac8760ec03de8d2609386e5bd65a6ab1a2ff3a684b3cc1c0d53f5a5"
Sep 16 18:49:52.013566 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:49:52.013513    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"631c5ce2aac8760ec03de8d2609386e5bd65a6ab1a2ff3a684b3cc1c0d53f5a5"} err="failed to get container status \"631c5ce2aac8760ec03de8d2609386e5bd65a6ab1a2ff3a684b3cc1c0d53f5a5\": rpc error: code = NotFound desc = could not find container \"631c5ce2aac8760ec03de8d2609386e5bd65a6ab1a2ff3a684b3cc1c0d53f5a5\": container with ID starting with 631c5ce2aac8760ec03de8d2609386e5bd65a6ab1a2ff3a684b3cc1c0d53f5a5 not found: ID does not exist"
Sep 16 18:49:52.013566 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:49:52.013538    2581 scope.go:117] "RemoveContainer" containerID="314e4f5337c0ea1e4e5e6b86e35ae4ee43a7775322585a4488852c09df93923f"
Sep 16 18:49:52.013830 ip-10-0-138-127 kubenswrapper[2581]: E0916 18:49:52.013811    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"314e4f5337c0ea1e4e5e6b86e35ae4ee43a7775322585a4488852c09df93923f\": container with ID starting with 314e4f5337c0ea1e4e5e6b86e35ae4ee43a7775322585a4488852c09df93923f not found: ID does not exist" containerID="314e4f5337c0ea1e4e5e6b86e35ae4ee43a7775322585a4488852c09df93923f"
Sep 16 18:49:52.013880 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:49:52.013840    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"314e4f5337c0ea1e4e5e6b86e35ae4ee43a7775322585a4488852c09df93923f"} err="failed to get container status \"314e4f5337c0ea1e4e5e6b86e35ae4ee43a7775322585a4488852c09df93923f\": rpc error: code = NotFound desc = could not find container \"314e4f5337c0ea1e4e5e6b86e35ae4ee43a7775322585a4488852c09df93923f\": container with ID starting with 314e4f5337c0ea1e4e5e6b86e35ae4ee43a7775322585a4488852c09df93923f not found: ID does not exist"
Sep 16 18:49:52.013880 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:49:52.013860    2581 scope.go:117] "RemoveContainer" containerID="f3007c04a6c7666ea52a294ed272da999f867f7ef844acc86315cb2a8cf1bec2"
Sep 16 18:49:52.014310 ip-10-0-138-127 kubenswrapper[2581]: E0916 18:49:52.014285    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f3007c04a6c7666ea52a294ed272da999f867f7ef844acc86315cb2a8cf1bec2\": container with ID starting with f3007c04a6c7666ea52a294ed272da999f867f7ef844acc86315cb2a8cf1bec2 not found: ID does not exist" containerID="f3007c04a6c7666ea52a294ed272da999f867f7ef844acc86315cb2a8cf1bec2"
Sep 16 18:49:52.014463 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:49:52.014340    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f3007c04a6c7666ea52a294ed272da999f867f7ef844acc86315cb2a8cf1bec2"} err="failed to get container status \"f3007c04a6c7666ea52a294ed272da999f867f7ef844acc86315cb2a8cf1bec2\": rpc error: code = NotFound desc = could not find container \"f3007c04a6c7666ea52a294ed272da999f867f7ef844acc86315cb2a8cf1bec2\": container with ID starting with f3007c04a6c7666ea52a294ed272da999f867f7ef844acc86315cb2a8cf1bec2 not found: ID does not exist"
Sep 16 18:49:52.014463 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:49:52.014363    2581 scope.go:117] "RemoveContainer" containerID="fe8880e1d60a0122daff9a84c8c263336a34080fa9d654bae40ab0456d777281"
Sep 16 18:49:52.014923 ip-10-0-138-127 kubenswrapper[2581]: E0916 18:49:52.014904    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fe8880e1d60a0122daff9a84c8c263336a34080fa9d654bae40ab0456d777281\": container with ID starting with fe8880e1d60a0122daff9a84c8c263336a34080fa9d654bae40ab0456d777281 not found: ID does not exist" containerID="fe8880e1d60a0122daff9a84c8c263336a34080fa9d654bae40ab0456d777281"
Sep 16 18:49:52.015010 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:49:52.014929    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fe8880e1d60a0122daff9a84c8c263336a34080fa9d654bae40ab0456d777281"} err="failed to get container status \"fe8880e1d60a0122daff9a84c8c263336a34080fa9d654bae40ab0456d777281\": rpc error: code = NotFound desc = could not find container \"fe8880e1d60a0122daff9a84c8c263336a34080fa9d654bae40ab0456d777281\": container with ID starting with fe8880e1d60a0122daff9a84c8c263336a34080fa9d654bae40ab0456d777281 not found: ID does not exist"
Sep 16 18:49:52.015010 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:49:52.014949    2581 scope.go:117] "RemoveContainer" containerID="83ac526d7a86604d68fe8d92bf00597bc299f57b92d72f39a7fb68e890df4232"
Sep 16 18:49:52.015249 ip-10-0-138-127 kubenswrapper[2581]: E0916 18:49:52.015233    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"83ac526d7a86604d68fe8d92bf00597bc299f57b92d72f39a7fb68e890df4232\": container with ID starting with 83ac526d7a86604d68fe8d92bf00597bc299f57b92d72f39a7fb68e890df4232 not found: ID does not exist" containerID="83ac526d7a86604d68fe8d92bf00597bc299f57b92d72f39a7fb68e890df4232"
Sep 16 18:49:52.015316 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:49:52.015256    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"83ac526d7a86604d68fe8d92bf00597bc299f57b92d72f39a7fb68e890df4232"} err="failed to get container status \"83ac526d7a86604d68fe8d92bf00597bc299f57b92d72f39a7fb68e890df4232\": rpc error: code = NotFound desc = could not find container \"83ac526d7a86604d68fe8d92bf00597bc299f57b92d72f39a7fb68e890df4232\": container with ID starting with 83ac526d7a86604d68fe8d92bf00597bc299f57b92d72f39a7fb68e890df4232 not found: ID does not exist"
Sep 16 18:49:52.015316 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:49:52.015274    2581 scope.go:117] "RemoveContainer" containerID="b247a13bc56273cb77128be0df26ac1bd41f05879d206a7acaeb33fd20a62d0b"
Sep 16 18:49:52.015591 ip-10-0-138-127 kubenswrapper[2581]: E0916 18:49:52.015573    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b247a13bc56273cb77128be0df26ac1bd41f05879d206a7acaeb33fd20a62d0b\": container with ID starting with b247a13bc56273cb77128be0df26ac1bd41f05879d206a7acaeb33fd20a62d0b not found: ID does not exist" containerID="b247a13bc56273cb77128be0df26ac1bd41f05879d206a7acaeb33fd20a62d0b"
Sep 16 18:49:52.015591 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:49:52.015595    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b247a13bc56273cb77128be0df26ac1bd41f05879d206a7acaeb33fd20a62d0b"} err="failed to get container status \"b247a13bc56273cb77128be0df26ac1bd41f05879d206a7acaeb33fd20a62d0b\": rpc error: code = NotFound desc = could not find container \"b247a13bc56273cb77128be0df26ac1bd41f05879d206a7acaeb33fd20a62d0b\": container with ID starting with b247a13bc56273cb77128be0df26ac1bd41f05879d206a7acaeb33fd20a62d0b not found: ID does not exist"
Sep 16 18:49:52.015702 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:49:52.015608    2581 scope.go:117] "RemoveContainer" containerID="3f3dd1883e45a82010062071dbb5a547b0398022c662e0974bb6035db93c527b"
Sep 16 18:49:52.015881 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:49:52.015861    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3f3dd1883e45a82010062071dbb5a547b0398022c662e0974bb6035db93c527b"} err="failed to get container status \"3f3dd1883e45a82010062071dbb5a547b0398022c662e0974bb6035db93c527b\": rpc error: code = NotFound desc = could not find container \"3f3dd1883e45a82010062071dbb5a547b0398022c662e0974bb6035db93c527b\": container with ID starting with 3f3dd1883e45a82010062071dbb5a547b0398022c662e0974bb6035db93c527b not found: ID does not exist"
Sep 16 18:49:52.015931 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:49:52.015883    2581 scope.go:117] "RemoveContainer" containerID="631c5ce2aac8760ec03de8d2609386e5bd65a6ab1a2ff3a684b3cc1c0d53f5a5"
Sep 16 18:49:52.015996 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:49:52.015974    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 16 18:49:52.016146 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:49:52.016130    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"631c5ce2aac8760ec03de8d2609386e5bd65a6ab1a2ff3a684b3cc1c0d53f5a5"} err="failed to get container status \"631c5ce2aac8760ec03de8d2609386e5bd65a6ab1a2ff3a684b3cc1c0d53f5a5\": rpc error: code = NotFound desc = could not find container \"631c5ce2aac8760ec03de8d2609386e5bd65a6ab1a2ff3a684b3cc1c0d53f5a5\": container with ID starting with 631c5ce2aac8760ec03de8d2609386e5bd65a6ab1a2ff3a684b3cc1c0d53f5a5 not found: ID does not exist"
Sep 16 18:49:52.016146 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:49:52.016146    2581 scope.go:117] "RemoveContainer" containerID="314e4f5337c0ea1e4e5e6b86e35ae4ee43a7775322585a4488852c09df93923f"
Sep 16 18:49:52.016330 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:49:52.016316    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="33280f86-4497-4343-829b-4781cbfd5e5f" containerName="kube-rbac-proxy"
Sep 16 18:49:52.016330 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:49:52.016331    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="33280f86-4497-4343-829b-4781cbfd5e5f" containerName="kube-rbac-proxy"
Sep 16 18:49:52.016439 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:49:52.016345    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="33280f86-4497-4343-829b-4781cbfd5e5f" containerName="init-config-reloader"
Sep 16 18:49:52.016439 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:49:52.016351    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="33280f86-4497-4343-829b-4781cbfd5e5f" containerName="init-config-reloader"
Sep 16 18:49:52.016439 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:49:52.016359    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="33280f86-4497-4343-829b-4781cbfd5e5f" containerName="thanos-sidecar"
Sep 16 18:49:52.016439 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:49:52.016364    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="33280f86-4497-4343-829b-4781cbfd5e5f" containerName="thanos-sidecar"
Sep 16 18:49:52.016439 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:49:52.016375    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="33280f86-4497-4343-829b-4781cbfd5e5f" containerName="kube-rbac-proxy-thanos"
Sep 16 18:49:52.016439 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:49:52.016383    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="33280f86-4497-4343-829b-4781cbfd5e5f" containerName="kube-rbac-proxy-thanos"
Sep 16 18:49:52.016439 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:49:52.016394    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="33280f86-4497-4343-829b-4781cbfd5e5f" containerName="kube-rbac-proxy-web"
Sep 16 18:49:52.016439 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:49:52.016399    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="33280f86-4497-4343-829b-4781cbfd5e5f" containerName="kube-rbac-proxy-web"
Sep 16 18:49:52.016439 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:49:52.016409    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="33280f86-4497-4343-829b-4781cbfd5e5f" containerName="prometheus"
Sep 16 18:49:52.016439 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:49:52.016414    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="33280f86-4497-4343-829b-4781cbfd5e5f" containerName="prometheus"
Sep 16 18:49:52.016439 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:49:52.016423    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="33280f86-4497-4343-829b-4781cbfd5e5f" containerName="config-reloader"
Sep 16 18:49:52.016439 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:49:52.016428    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="33280f86-4497-4343-829b-4781cbfd5e5f" containerName="config-reloader"
Sep 16 18:49:52.016439 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:49:52.016424    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"314e4f5337c0ea1e4e5e6b86e35ae4ee43a7775322585a4488852c09df93923f"} err="failed to get container status \"314e4f5337c0ea1e4e5e6b86e35ae4ee43a7775322585a4488852c09df93923f\": rpc error: code = NotFound desc = could not find container \"314e4f5337c0ea1e4e5e6b86e35ae4ee43a7775322585a4488852c09df93923f\": container with ID starting with 314e4f5337c0ea1e4e5e6b86e35ae4ee43a7775322585a4488852c09df93923f not found: ID does not exist"
Sep 16 18:49:52.016911 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:49:52.016448    2581 scope.go:117] "RemoveContainer" containerID="f3007c04a6c7666ea52a294ed272da999f867f7ef844acc86315cb2a8cf1bec2"
Sep 16 18:49:52.016911 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:49:52.016483    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="33280f86-4497-4343-829b-4781cbfd5e5f" containerName="kube-rbac-proxy-web"
Sep 16 18:49:52.016911 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:49:52.016493    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="33280f86-4497-4343-829b-4781cbfd5e5f" containerName="kube-rbac-proxy-thanos"
Sep 16 18:49:52.016911 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:49:52.016499    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="33280f86-4497-4343-829b-4781cbfd5e5f" containerName="thanos-sidecar"
Sep 16 18:49:52.016911 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:49:52.016506    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="33280f86-4497-4343-829b-4781cbfd5e5f" containerName="prometheus"
Sep 16 18:49:52.016911 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:49:52.016511    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="33280f86-4497-4343-829b-4781cbfd5e5f" containerName="kube-rbac-proxy"
Sep 16 18:49:52.016911 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:49:52.016515    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="33280f86-4497-4343-829b-4781cbfd5e5f" containerName="config-reloader"
Sep 16 18:49:52.016911 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:49:52.016717    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f3007c04a6c7666ea52a294ed272da999f867f7ef844acc86315cb2a8cf1bec2"} err="failed to get container status \"f3007c04a6c7666ea52a294ed272da999f867f7ef844acc86315cb2a8cf1bec2\": rpc error: code = NotFound desc = could not find container \"f3007c04a6c7666ea52a294ed272da999f867f7ef844acc86315cb2a8cf1bec2\": container with ID starting with f3007c04a6c7666ea52a294ed272da999f867f7ef844acc86315cb2a8cf1bec2 not found: ID does not exist"
Sep 16 18:49:52.016911 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:49:52.016739    2581 scope.go:117] "RemoveContainer" containerID="fe8880e1d60a0122daff9a84c8c263336a34080fa9d654bae40ab0456d777281"
Sep 16 18:49:52.017232 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:49:52.016975    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fe8880e1d60a0122daff9a84c8c263336a34080fa9d654bae40ab0456d777281"} err="failed to get container status \"fe8880e1d60a0122daff9a84c8c263336a34080fa9d654bae40ab0456d777281\": rpc error: code = NotFound desc = could not find container \"fe8880e1d60a0122daff9a84c8c263336a34080fa9d654bae40ab0456d777281\": container with ID starting with fe8880e1d60a0122daff9a84c8c263336a34080fa9d654bae40ab0456d777281 not found: ID does not exist"
Sep 16 18:49:52.017232 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:49:52.017009    2581 scope.go:117] "RemoveContainer" containerID="83ac526d7a86604d68fe8d92bf00597bc299f57b92d72f39a7fb68e890df4232"
Sep 16 18:49:52.017316 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:49:52.017256    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"83ac526d7a86604d68fe8d92bf00597bc299f57b92d72f39a7fb68e890df4232"} err="failed to get container status \"83ac526d7a86604d68fe8d92bf00597bc299f57b92d72f39a7fb68e890df4232\": rpc error: code = NotFound desc = could not find container \"83ac526d7a86604d68fe8d92bf00597bc299f57b92d72f39a7fb68e890df4232\": container with ID starting with 83ac526d7a86604d68fe8d92bf00597bc299f57b92d72f39a7fb68e890df4232 not found: ID does not exist"
Sep 16 18:49:52.017316 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:49:52.017271    2581 scope.go:117] "RemoveContainer" containerID="b247a13bc56273cb77128be0df26ac1bd41f05879d206a7acaeb33fd20a62d0b"
Sep 16 18:49:52.017555 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:49:52.017537    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b247a13bc56273cb77128be0df26ac1bd41f05879d206a7acaeb33fd20a62d0b"} err="failed to get container status \"b247a13bc56273cb77128be0df26ac1bd41f05879d206a7acaeb33fd20a62d0b\": rpc error: code = NotFound desc = could not find container \"b247a13bc56273cb77128be0df26ac1bd41f05879d206a7acaeb33fd20a62d0b\": container with ID starting with b247a13bc56273cb77128be0df26ac1bd41f05879d206a7acaeb33fd20a62d0b not found: ID does not exist"
Sep 16 18:49:52.017555 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:49:52.017557    2581 scope.go:117] "RemoveContainer" containerID="3f3dd1883e45a82010062071dbb5a547b0398022c662e0974bb6035db93c527b"
Sep 16 18:49:52.017789 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:49:52.017773    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3f3dd1883e45a82010062071dbb5a547b0398022c662e0974bb6035db93c527b"} err="failed to get container status \"3f3dd1883e45a82010062071dbb5a547b0398022c662e0974bb6035db93c527b\": rpc error: code = NotFound desc = could not find container \"3f3dd1883e45a82010062071dbb5a547b0398022c662e0974bb6035db93c527b\": container with ID starting with 3f3dd1883e45a82010062071dbb5a547b0398022c662e0974bb6035db93c527b not found: ID does not exist"
Sep 16 18:49:52.017833 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:49:52.017789    2581 scope.go:117] "RemoveContainer" containerID="631c5ce2aac8760ec03de8d2609386e5bd65a6ab1a2ff3a684b3cc1c0d53f5a5"
Sep 16 18:49:52.018020 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:49:52.017997    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"631c5ce2aac8760ec03de8d2609386e5bd65a6ab1a2ff3a684b3cc1c0d53f5a5"} err="failed to get container status \"631c5ce2aac8760ec03de8d2609386e5bd65a6ab1a2ff3a684b3cc1c0d53f5a5\": rpc error: code = NotFound desc = could not find container \"631c5ce2aac8760ec03de8d2609386e5bd65a6ab1a2ff3a684b3cc1c0d53f5a5\": container with ID starting with 631c5ce2aac8760ec03de8d2609386e5bd65a6ab1a2ff3a684b3cc1c0d53f5a5 not found: ID does not exist"
Sep 16 18:49:52.018020 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:49:52.018019    2581 scope.go:117] "RemoveContainer" containerID="314e4f5337c0ea1e4e5e6b86e35ae4ee43a7775322585a4488852c09df93923f"
Sep 16 18:49:52.018281 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:49:52.018256    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"314e4f5337c0ea1e4e5e6b86e35ae4ee43a7775322585a4488852c09df93923f"} err="failed to get container status \"314e4f5337c0ea1e4e5e6b86e35ae4ee43a7775322585a4488852c09df93923f\": rpc error: code = NotFound desc = could not find container \"314e4f5337c0ea1e4e5e6b86e35ae4ee43a7775322585a4488852c09df93923f\": container with ID starting with 314e4f5337c0ea1e4e5e6b86e35ae4ee43a7775322585a4488852c09df93923f not found: ID does not exist"
Sep 16 18:49:52.018281 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:49:52.018273    2581 scope.go:117] "RemoveContainer" containerID="f3007c04a6c7666ea52a294ed272da999f867f7ef844acc86315cb2a8cf1bec2"
Sep 16 18:49:52.018490 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:49:52.018476    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f3007c04a6c7666ea52a294ed272da999f867f7ef844acc86315cb2a8cf1bec2"} err="failed to get container status \"f3007c04a6c7666ea52a294ed272da999f867f7ef844acc86315cb2a8cf1bec2\": rpc error: code = NotFound desc = could not find container \"f3007c04a6c7666ea52a294ed272da999f867f7ef844acc86315cb2a8cf1bec2\": container with ID starting with f3007c04a6c7666ea52a294ed272da999f867f7ef844acc86315cb2a8cf1bec2 not found: ID does not exist"
Sep 16 18:49:52.018490 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:49:52.018490    2581 scope.go:117] "RemoveContainer" containerID="fe8880e1d60a0122daff9a84c8c263336a34080fa9d654bae40ab0456d777281"
Sep 16 18:49:52.018733 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:49:52.018716    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fe8880e1d60a0122daff9a84c8c263336a34080fa9d654bae40ab0456d777281"} err="failed to get container status \"fe8880e1d60a0122daff9a84c8c263336a34080fa9d654bae40ab0456d777281\": rpc error: code = NotFound desc = could not find container \"fe8880e1d60a0122daff9a84c8c263336a34080fa9d654bae40ab0456d777281\": container with ID starting with fe8880e1d60a0122daff9a84c8c263336a34080fa9d654bae40ab0456d777281 not found: ID does not exist"
Sep 16 18:49:52.018778 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:49:52.018733    2581 scope.go:117] "RemoveContainer" containerID="83ac526d7a86604d68fe8d92bf00597bc299f57b92d72f39a7fb68e890df4232"
Sep 16 18:49:52.018950 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:49:52.018933    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"83ac526d7a86604d68fe8d92bf00597bc299f57b92d72f39a7fb68e890df4232"} err="failed to get container status \"83ac526d7a86604d68fe8d92bf00597bc299f57b92d72f39a7fb68e890df4232\": rpc error: code = NotFound desc = could not find container \"83ac526d7a86604d68fe8d92bf00597bc299f57b92d72f39a7fb68e890df4232\": container with ID starting with 83ac526d7a86604d68fe8d92bf00597bc299f57b92d72f39a7fb68e890df4232 not found: ID does not exist"
Sep 16 18:49:52.018998 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:49:52.018950    2581 scope.go:117] "RemoveContainer" containerID="b247a13bc56273cb77128be0df26ac1bd41f05879d206a7acaeb33fd20a62d0b"
Sep 16 18:49:52.019183 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:49:52.019155    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b247a13bc56273cb77128be0df26ac1bd41f05879d206a7acaeb33fd20a62d0b"} err="failed to get container status \"b247a13bc56273cb77128be0df26ac1bd41f05879d206a7acaeb33fd20a62d0b\": rpc error: code = NotFound desc = could not find container \"b247a13bc56273cb77128be0df26ac1bd41f05879d206a7acaeb33fd20a62d0b\": container with ID starting with b247a13bc56273cb77128be0df26ac1bd41f05879d206a7acaeb33fd20a62d0b not found: ID does not exist"
Sep 16 18:49:52.019258 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:49:52.019186    2581 scope.go:117] "RemoveContainer" containerID="3f3dd1883e45a82010062071dbb5a547b0398022c662e0974bb6035db93c527b"
Sep 16 18:49:52.019451 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:49:52.019434    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3f3dd1883e45a82010062071dbb5a547b0398022c662e0974bb6035db93c527b"} err="failed to get container status \"3f3dd1883e45a82010062071dbb5a547b0398022c662e0974bb6035db93c527b\": rpc error: code = NotFound desc = could not find container \"3f3dd1883e45a82010062071dbb5a547b0398022c662e0974bb6035db93c527b\": container with ID starting with 3f3dd1883e45a82010062071dbb5a547b0398022c662e0974bb6035db93c527b not found: ID does not exist"
Sep 16 18:49:52.019503 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:49:52.019452    2581 scope.go:117] "RemoveContainer" containerID="631c5ce2aac8760ec03de8d2609386e5bd65a6ab1a2ff3a684b3cc1c0d53f5a5"
Sep 16 18:49:52.019690 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:49:52.019674    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"631c5ce2aac8760ec03de8d2609386e5bd65a6ab1a2ff3a684b3cc1c0d53f5a5"} err="failed to get container status \"631c5ce2aac8760ec03de8d2609386e5bd65a6ab1a2ff3a684b3cc1c0d53f5a5\": rpc error: code = NotFound desc = could not find container \"631c5ce2aac8760ec03de8d2609386e5bd65a6ab1a2ff3a684b3cc1c0d53f5a5\": container with ID starting with 631c5ce2aac8760ec03de8d2609386e5bd65a6ab1a2ff3a684b3cc1c0d53f5a5 not found: ID does not exist"
Sep 16 18:49:52.019690 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:49:52.019690    2581 scope.go:117] "RemoveContainer" containerID="314e4f5337c0ea1e4e5e6b86e35ae4ee43a7775322585a4488852c09df93923f"
Sep 16 18:49:52.019915 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:49:52.019901    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"314e4f5337c0ea1e4e5e6b86e35ae4ee43a7775322585a4488852c09df93923f"} err="failed to get container status \"314e4f5337c0ea1e4e5e6b86e35ae4ee43a7775322585a4488852c09df93923f\": rpc error: code = NotFound desc = could not find container \"314e4f5337c0ea1e4e5e6b86e35ae4ee43a7775322585a4488852c09df93923f\": container with ID starting with 314e4f5337c0ea1e4e5e6b86e35ae4ee43a7775322585a4488852c09df93923f not found: ID does not exist"
Sep 16 18:49:52.019915 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:49:52.019915    2581 scope.go:117] "RemoveContainer" containerID="f3007c04a6c7666ea52a294ed272da999f867f7ef844acc86315cb2a8cf1bec2"
Sep 16 18:49:52.020140 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:49:52.020119    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f3007c04a6c7666ea52a294ed272da999f867f7ef844acc86315cb2a8cf1bec2"} err="failed to get container status \"f3007c04a6c7666ea52a294ed272da999f867f7ef844acc86315cb2a8cf1bec2\": rpc error: code = NotFound desc = could not find container \"f3007c04a6c7666ea52a294ed272da999f867f7ef844acc86315cb2a8cf1bec2\": container with ID starting with f3007c04a6c7666ea52a294ed272da999f867f7ef844acc86315cb2a8cf1bec2 not found: ID does not exist"
Sep 16 18:49:52.020178 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:49:52.020142    2581 scope.go:117] "RemoveContainer" containerID="fe8880e1d60a0122daff9a84c8c263336a34080fa9d654bae40ab0456d777281"
Sep 16 18:49:52.020405 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:49:52.020386    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fe8880e1d60a0122daff9a84c8c263336a34080fa9d654bae40ab0456d777281"} err="failed to get container status \"fe8880e1d60a0122daff9a84c8c263336a34080fa9d654bae40ab0456d777281\": rpc error: code = NotFound desc = could not find container \"fe8880e1d60a0122daff9a84c8c263336a34080fa9d654bae40ab0456d777281\": container with ID starting with fe8880e1d60a0122daff9a84c8c263336a34080fa9d654bae40ab0456d777281 not found: ID does not exist"
Sep 16 18:49:52.020449 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:49:52.020406    2581 scope.go:117] "RemoveContainer" containerID="83ac526d7a86604d68fe8d92bf00597bc299f57b92d72f39a7fb68e890df4232"
Sep 16 18:49:52.020672 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:49:52.020657    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"83ac526d7a86604d68fe8d92bf00597bc299f57b92d72f39a7fb68e890df4232"} err="failed to get container status \"83ac526d7a86604d68fe8d92bf00597bc299f57b92d72f39a7fb68e890df4232\": rpc error: code = NotFound desc = could not find container \"83ac526d7a86604d68fe8d92bf00597bc299f57b92d72f39a7fb68e890df4232\": container with ID starting with 83ac526d7a86604d68fe8d92bf00597bc299f57b92d72f39a7fb68e890df4232 not found: ID does not exist"
Sep 16 18:49:52.020707 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:49:52.020673    2581 scope.go:117] "RemoveContainer" containerID="b247a13bc56273cb77128be0df26ac1bd41f05879d206a7acaeb33fd20a62d0b"
Sep 16 18:49:52.020896 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:49:52.020880    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b247a13bc56273cb77128be0df26ac1bd41f05879d206a7acaeb33fd20a62d0b"} err="failed to get container status \"b247a13bc56273cb77128be0df26ac1bd41f05879d206a7acaeb33fd20a62d0b\": rpc error: code = NotFound desc = could not find container \"b247a13bc56273cb77128be0df26ac1bd41f05879d206a7acaeb33fd20a62d0b\": container with ID starting with b247a13bc56273cb77128be0df26ac1bd41f05879d206a7acaeb33fd20a62d0b not found: ID does not exist"
Sep 16 18:49:52.020935 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:49:52.020898    2581 scope.go:117] "RemoveContainer" containerID="3f3dd1883e45a82010062071dbb5a547b0398022c662e0974bb6035db93c527b"
Sep 16 18:49:52.021126 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:49:52.021109    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3f3dd1883e45a82010062071dbb5a547b0398022c662e0974bb6035db93c527b"} err="failed to get container status \"3f3dd1883e45a82010062071dbb5a547b0398022c662e0974bb6035db93c527b\": rpc error: code = NotFound desc = could not find container \"3f3dd1883e45a82010062071dbb5a547b0398022c662e0974bb6035db93c527b\": container with ID starting with 3f3dd1883e45a82010062071dbb5a547b0398022c662e0974bb6035db93c527b not found: ID does not exist"
Sep 16 18:49:52.021162 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:49:52.021127    2581 scope.go:117] "RemoveContainer" containerID="631c5ce2aac8760ec03de8d2609386e5bd65a6ab1a2ff3a684b3cc1c0d53f5a5"
Sep 16 18:49:52.021413 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:49:52.021391    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"631c5ce2aac8760ec03de8d2609386e5bd65a6ab1a2ff3a684b3cc1c0d53f5a5"} err="failed to get container status \"631c5ce2aac8760ec03de8d2609386e5bd65a6ab1a2ff3a684b3cc1c0d53f5a5\": rpc error: code = NotFound desc = could not find container \"631c5ce2aac8760ec03de8d2609386e5bd65a6ab1a2ff3a684b3cc1c0d53f5a5\": container with ID starting with 631c5ce2aac8760ec03de8d2609386e5bd65a6ab1a2ff3a684b3cc1c0d53f5a5 not found: ID does not exist"
Sep 16 18:49:52.021467 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:49:52.021413    2581 scope.go:117] "RemoveContainer" containerID="314e4f5337c0ea1e4e5e6b86e35ae4ee43a7775322585a4488852c09df93923f"
Sep 16 18:49:52.021659 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:49:52.021640    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"314e4f5337c0ea1e4e5e6b86e35ae4ee43a7775322585a4488852c09df93923f"} err="failed to get container status \"314e4f5337c0ea1e4e5e6b86e35ae4ee43a7775322585a4488852c09df93923f\": rpc error: code = NotFound desc = could not find container \"314e4f5337c0ea1e4e5e6b86e35ae4ee43a7775322585a4488852c09df93923f\": container with ID starting with 314e4f5337c0ea1e4e5e6b86e35ae4ee43a7775322585a4488852c09df93923f not found: ID does not exist"
Sep 16 18:49:52.021739 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:49:52.021661    2581 scope.go:117] "RemoveContainer" containerID="f3007c04a6c7666ea52a294ed272da999f867f7ef844acc86315cb2a8cf1bec2"
Sep 16 18:49:52.021953 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:49:52.021938    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:49:52.021994 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:49:52.021953    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f3007c04a6c7666ea52a294ed272da999f867f7ef844acc86315cb2a8cf1bec2"} err="failed to get container status \"f3007c04a6c7666ea52a294ed272da999f867f7ef844acc86315cb2a8cf1bec2\": rpc error: code = NotFound desc = could not find container \"f3007c04a6c7666ea52a294ed272da999f867f7ef844acc86315cb2a8cf1bec2\": container with ID starting with f3007c04a6c7666ea52a294ed272da999f867f7ef844acc86315cb2a8cf1bec2 not found: ID does not exist"
Sep 16 18:49:52.021994 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:49:52.021971    2581 scope.go:117] "RemoveContainer" containerID="fe8880e1d60a0122daff9a84c8c263336a34080fa9d654bae40ab0456d777281"
Sep 16 18:49:52.022348 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:49:52.022237    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fe8880e1d60a0122daff9a84c8c263336a34080fa9d654bae40ab0456d777281"} err="failed to get container status \"fe8880e1d60a0122daff9a84c8c263336a34080fa9d654bae40ab0456d777281\": rpc error: code = NotFound desc = could not find container \"fe8880e1d60a0122daff9a84c8c263336a34080fa9d654bae40ab0456d777281\": container with ID starting with fe8880e1d60a0122daff9a84c8c263336a34080fa9d654bae40ab0456d777281 not found: ID does not exist"
Sep 16 18:49:52.022348 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:49:52.022268    2581 scope.go:117] "RemoveContainer" containerID="83ac526d7a86604d68fe8d92bf00597bc299f57b92d72f39a7fb68e890df4232"
Sep 16 18:49:52.022617 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:49:52.022596    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"83ac526d7a86604d68fe8d92bf00597bc299f57b92d72f39a7fb68e890df4232"} err="failed to get container status \"83ac526d7a86604d68fe8d92bf00597bc299f57b92d72f39a7fb68e890df4232\": rpc error: code = NotFound desc = could not find container \"83ac526d7a86604d68fe8d92bf00597bc299f57b92d72f39a7fb68e890df4232\": container with ID starting with 83ac526d7a86604d68fe8d92bf00597bc299f57b92d72f39a7fb68e890df4232 not found: ID does not exist"
Sep 16 18:49:52.022687 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:49:52.022620    2581 scope.go:117] "RemoveContainer" containerID="b247a13bc56273cb77128be0df26ac1bd41f05879d206a7acaeb33fd20a62d0b"
Sep 16 18:49:52.022883 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:49:52.022860    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b247a13bc56273cb77128be0df26ac1bd41f05879d206a7acaeb33fd20a62d0b"} err="failed to get container status \"b247a13bc56273cb77128be0df26ac1bd41f05879d206a7acaeb33fd20a62d0b\": rpc error: code = NotFound desc = could not find container \"b247a13bc56273cb77128be0df26ac1bd41f05879d206a7acaeb33fd20a62d0b\": container with ID starting with b247a13bc56273cb77128be0df26ac1bd41f05879d206a7acaeb33fd20a62d0b not found: ID does not exist"
Sep 16 18:49:52.022925 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:49:52.022889    2581 scope.go:117] "RemoveContainer" containerID="3f3dd1883e45a82010062071dbb5a547b0398022c662e0974bb6035db93c527b"
Sep 16 18:49:52.023149 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:49:52.023121    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3f3dd1883e45a82010062071dbb5a547b0398022c662e0974bb6035db93c527b"} err="failed to get container status \"3f3dd1883e45a82010062071dbb5a547b0398022c662e0974bb6035db93c527b\": rpc error: code = NotFound desc = could not find container \"3f3dd1883e45a82010062071dbb5a547b0398022c662e0974bb6035db93c527b\": container with ID starting with 3f3dd1883e45a82010062071dbb5a547b0398022c662e0974bb6035db93c527b not found: ID does not exist"
Sep 16 18:49:52.023237 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:49:52.023150    2581 scope.go:117] "RemoveContainer" containerID="631c5ce2aac8760ec03de8d2609386e5bd65a6ab1a2ff3a684b3cc1c0d53f5a5"
Sep 16 18:49:52.023424 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:49:52.023405    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"631c5ce2aac8760ec03de8d2609386e5bd65a6ab1a2ff3a684b3cc1c0d53f5a5"} err="failed to get container status \"631c5ce2aac8760ec03de8d2609386e5bd65a6ab1a2ff3a684b3cc1c0d53f5a5\": rpc error: code = NotFound desc = could not find container \"631c5ce2aac8760ec03de8d2609386e5bd65a6ab1a2ff3a684b3cc1c0d53f5a5\": container with ID starting with 631c5ce2aac8760ec03de8d2609386e5bd65a6ab1a2ff3a684b3cc1c0d53f5a5 not found: ID does not exist"
Sep 16 18:49:52.023492 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:49:52.023426    2581 scope.go:117] "RemoveContainer" containerID="314e4f5337c0ea1e4e5e6b86e35ae4ee43a7775322585a4488852c09df93923f"
Sep 16 18:49:52.023717 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:49:52.023697    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"314e4f5337c0ea1e4e5e6b86e35ae4ee43a7775322585a4488852c09df93923f"} err="failed to get container status \"314e4f5337c0ea1e4e5e6b86e35ae4ee43a7775322585a4488852c09df93923f\": rpc error: code = NotFound desc = could not find container \"314e4f5337c0ea1e4e5e6b86e35ae4ee43a7775322585a4488852c09df93923f\": container with ID starting with 314e4f5337c0ea1e4e5e6b86e35ae4ee43a7775322585a4488852c09df93923f not found: ID does not exist"
Sep 16 18:49:52.023762 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:49:52.023718    2581 scope.go:117] "RemoveContainer" containerID="f3007c04a6c7666ea52a294ed272da999f867f7ef844acc86315cb2a8cf1bec2"
Sep 16 18:49:52.023952 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:49:52.023935    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f3007c04a6c7666ea52a294ed272da999f867f7ef844acc86315cb2a8cf1bec2"} err="failed to get container status \"f3007c04a6c7666ea52a294ed272da999f867f7ef844acc86315cb2a8cf1bec2\": rpc error: code = NotFound desc = could not find container \"f3007c04a6c7666ea52a294ed272da999f867f7ef844acc86315cb2a8cf1bec2\": container with ID starting with f3007c04a6c7666ea52a294ed272da999f867f7ef844acc86315cb2a8cf1bec2 not found: ID does not exist"
Sep 16 18:49:52.024022 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:49:52.023953    2581 scope.go:117] "RemoveContainer" containerID="fe8880e1d60a0122daff9a84c8c263336a34080fa9d654bae40ab0456d777281"
Sep 16 18:49:52.024188 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:49:52.024171    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fe8880e1d60a0122daff9a84c8c263336a34080fa9d654bae40ab0456d777281"} err="failed to get container status \"fe8880e1d60a0122daff9a84c8c263336a34080fa9d654bae40ab0456d777281\": rpc error: code = NotFound desc = could not find container \"fe8880e1d60a0122daff9a84c8c263336a34080fa9d654bae40ab0456d777281\": container with ID starting with fe8880e1d60a0122daff9a84c8c263336a34080fa9d654bae40ab0456d777281 not found: ID does not exist"
Sep 16 18:49:52.024266 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:49:52.024189    2581 scope.go:117] "RemoveContainer" containerID="83ac526d7a86604d68fe8d92bf00597bc299f57b92d72f39a7fb68e890df4232"
Sep 16 18:49:52.026904 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:49:52.025118    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"83ac526d7a86604d68fe8d92bf00597bc299f57b92d72f39a7fb68e890df4232"} err="failed to get container status \"83ac526d7a86604d68fe8d92bf00597bc299f57b92d72f39a7fb68e890df4232\": rpc error: code = NotFound desc = could not find container \"83ac526d7a86604d68fe8d92bf00597bc299f57b92d72f39a7fb68e890df4232\": container with ID starting with 83ac526d7a86604d68fe8d92bf00597bc299f57b92d72f39a7fb68e890df4232 not found: ID does not exist"
Sep 16 18:49:52.026904 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:49:52.025148    2581 scope.go:117] "RemoveContainer" containerID="b247a13bc56273cb77128be0df26ac1bd41f05879d206a7acaeb33fd20a62d0b"
Sep 16 18:49:52.026904 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:49:52.025342    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\""
Sep 16 18:49:52.026904 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:49:52.025355    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\""
Sep 16 18:49:52.026904 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:49:52.025664    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\""
Sep 16 18:49:52.026904 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:49:52.025723    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\""
Sep 16 18:49:52.026904 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:49:52.026242    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-6ijsm9j35gbld\""
Sep 16 18:49:52.026904 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:49:52.026274    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\""
Sep 16 18:49:52.026904 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:49:52.026399    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\""
Sep 16 18:49:52.026904 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:49:52.026530    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-828mq\""
Sep 16 18:49:52.026904 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:49:52.026843    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b247a13bc56273cb77128be0df26ac1bd41f05879d206a7acaeb33fd20a62d0b"} err="failed to get container status \"b247a13bc56273cb77128be0df26ac1bd41f05879d206a7acaeb33fd20a62d0b\": rpc error: code = NotFound desc = could not find container \"b247a13bc56273cb77128be0df26ac1bd41f05879d206a7acaeb33fd20a62d0b\": container with ID starting with b247a13bc56273cb77128be0df26ac1bd41f05879d206a7acaeb33fd20a62d0b not found: ID does not exist"
Sep 16 18:49:52.027850 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:49:52.026994    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\""
Sep 16 18:49:52.027850 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:49:52.027184    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\""
Sep 16 18:49:52.027850 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:49:52.027363    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\""
Sep 16 18:49:52.028509 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:49:52.028041    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\""
Sep 16 18:49:52.028509 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:49:52.028278    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\""
Sep 16 18:49:52.028509 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:49:52.028470    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\""
Sep 16 18:49:52.031670 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:49:52.030774    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\""
Sep 16 18:49:52.033843 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:49:52.033822    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\""
Sep 16 18:49:52.041990 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:49:52.041961    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 16 18:49:52.205041 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:49:52.204953    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/4629d3f4-54ea-4bdf-9b75-650bbc2299d6-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"4629d3f4-54ea-4bdf-9b75-650bbc2299d6\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:49:52.205041 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:49:52.204990    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/4629d3f4-54ea-4bdf-9b75-650bbc2299d6-config\") pod \"prometheus-k8s-0\" (UID: \"4629d3f4-54ea-4bdf-9b75-650bbc2299d6\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:49:52.205041 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:49:52.205013    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/4629d3f4-54ea-4bdf-9b75-650bbc2299d6-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"4629d3f4-54ea-4bdf-9b75-650bbc2299d6\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:49:52.205041 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:49:52.205032    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/4629d3f4-54ea-4bdf-9b75-650bbc2299d6-web-config\") pod \"prometheus-k8s-0\" (UID: \"4629d3f4-54ea-4bdf-9b75-650bbc2299d6\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:49:52.205329 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:49:52.205054    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4629d3f4-54ea-4bdf-9b75-650bbc2299d6-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"4629d3f4-54ea-4bdf-9b75-650bbc2299d6\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:49:52.205329 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:49:52.205073    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/4629d3f4-54ea-4bdf-9b75-650bbc2299d6-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"4629d3f4-54ea-4bdf-9b75-650bbc2299d6\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:49:52.205329 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:49:52.205089    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zvtqm\" (UniqueName: \"kubernetes.io/projected/4629d3f4-54ea-4bdf-9b75-650bbc2299d6-kube-api-access-zvtqm\") pod \"prometheus-k8s-0\" (UID: \"4629d3f4-54ea-4bdf-9b75-650bbc2299d6\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:49:52.205329 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:49:52.205117    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/4629d3f4-54ea-4bdf-9b75-650bbc2299d6-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"4629d3f4-54ea-4bdf-9b75-650bbc2299d6\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:49:52.205329 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:49:52.205139    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/4629d3f4-54ea-4bdf-9b75-650bbc2299d6-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"4629d3f4-54ea-4bdf-9b75-650bbc2299d6\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:49:52.205329 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:49:52.205157    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/4629d3f4-54ea-4bdf-9b75-650bbc2299d6-config-out\") pod \"prometheus-k8s-0\" (UID: \"4629d3f4-54ea-4bdf-9b75-650bbc2299d6\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:49:52.205329 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:49:52.205172    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/4629d3f4-54ea-4bdf-9b75-650bbc2299d6-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"4629d3f4-54ea-4bdf-9b75-650bbc2299d6\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:49:52.205329 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:49:52.205235    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/4629d3f4-54ea-4bdf-9b75-650bbc2299d6-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"4629d3f4-54ea-4bdf-9b75-650bbc2299d6\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:49:52.205329 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:49:52.205254    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/4629d3f4-54ea-4bdf-9b75-650bbc2299d6-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"4629d3f4-54ea-4bdf-9b75-650bbc2299d6\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:49:52.205329 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:49:52.205271    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4629d3f4-54ea-4bdf-9b75-650bbc2299d6-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"4629d3f4-54ea-4bdf-9b75-650bbc2299d6\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:49:52.205329 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:49:52.205286    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/4629d3f4-54ea-4bdf-9b75-650bbc2299d6-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"4629d3f4-54ea-4bdf-9b75-650bbc2299d6\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:49:52.205639 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:49:52.205310    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/4629d3f4-54ea-4bdf-9b75-650bbc2299d6-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"4629d3f4-54ea-4bdf-9b75-650bbc2299d6\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:49:52.205639 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:49:52.205361    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4629d3f4-54ea-4bdf-9b75-650bbc2299d6-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"4629d3f4-54ea-4bdf-9b75-650bbc2299d6\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:49:52.205639 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:49:52.205398    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/4629d3f4-54ea-4bdf-9b75-650bbc2299d6-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"4629d3f4-54ea-4bdf-9b75-650bbc2299d6\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:49:52.205639 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:49:52.205412    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/4629d3f4-54ea-4bdf-9b75-650bbc2299d6-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"4629d3f4-54ea-4bdf-9b75-650bbc2299d6\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:49:52.205639 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:49:52.205437    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/4629d3f4-54ea-4bdf-9b75-650bbc2299d6-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"4629d3f4-54ea-4bdf-9b75-650bbc2299d6\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:49:52.305842 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:49:52.305800    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/4629d3f4-54ea-4bdf-9b75-650bbc2299d6-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"4629d3f4-54ea-4bdf-9b75-650bbc2299d6\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:49:52.305842 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:49:52.305843    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4629d3f4-54ea-4bdf-9b75-650bbc2299d6-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"4629d3f4-54ea-4bdf-9b75-650bbc2299d6\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:49:52.306094 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:49:52.305867    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/4629d3f4-54ea-4bdf-9b75-650bbc2299d6-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"4629d3f4-54ea-4bdf-9b75-650bbc2299d6\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:49:52.306094 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:49:52.305990    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/4629d3f4-54ea-4bdf-9b75-650bbc2299d6-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"4629d3f4-54ea-4bdf-9b75-650bbc2299d6\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:49:52.306094 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:49:52.306036    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/4629d3f4-54ea-4bdf-9b75-650bbc2299d6-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"4629d3f4-54ea-4bdf-9b75-650bbc2299d6\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:49:52.306094 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:49:52.306069    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/4629d3f4-54ea-4bdf-9b75-650bbc2299d6-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"4629d3f4-54ea-4bdf-9b75-650bbc2299d6\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:49:52.306326 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:49:52.306097    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/4629d3f4-54ea-4bdf-9b75-650bbc2299d6-config\") pod \"prometheus-k8s-0\" (UID: \"4629d3f4-54ea-4bdf-9b75-650bbc2299d6\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:49:52.306326 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:49:52.306131    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/4629d3f4-54ea-4bdf-9b75-650bbc2299d6-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"4629d3f4-54ea-4bdf-9b75-650bbc2299d6\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:49:52.306326 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:49:52.306162    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/4629d3f4-54ea-4bdf-9b75-650bbc2299d6-web-config\") pod \"prometheus-k8s-0\" (UID: \"4629d3f4-54ea-4bdf-9b75-650bbc2299d6\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:49:52.306326 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:49:52.306185    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4629d3f4-54ea-4bdf-9b75-650bbc2299d6-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"4629d3f4-54ea-4bdf-9b75-650bbc2299d6\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:49:52.306326 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:49:52.306245    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/4629d3f4-54ea-4bdf-9b75-650bbc2299d6-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"4629d3f4-54ea-4bdf-9b75-650bbc2299d6\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:49:52.306326 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:49:52.306270    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zvtqm\" (UniqueName: \"kubernetes.io/projected/4629d3f4-54ea-4bdf-9b75-650bbc2299d6-kube-api-access-zvtqm\") pod \"prometheus-k8s-0\" (UID: \"4629d3f4-54ea-4bdf-9b75-650bbc2299d6\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:49:52.306326 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:49:52.306306    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/4629d3f4-54ea-4bdf-9b75-650bbc2299d6-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"4629d3f4-54ea-4bdf-9b75-650bbc2299d6\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:49:52.306738 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:49:52.306330    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/4629d3f4-54ea-4bdf-9b75-650bbc2299d6-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"4629d3f4-54ea-4bdf-9b75-650bbc2299d6\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:49:52.306738 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:49:52.306359    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/4629d3f4-54ea-4bdf-9b75-650bbc2299d6-config-out\") pod \"prometheus-k8s-0\" (UID: \"4629d3f4-54ea-4bdf-9b75-650bbc2299d6\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:49:52.306738 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:49:52.306383    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/4629d3f4-54ea-4bdf-9b75-650bbc2299d6-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"4629d3f4-54ea-4bdf-9b75-650bbc2299d6\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:49:52.306738 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:49:52.306418    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/4629d3f4-54ea-4bdf-9b75-650bbc2299d6-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"4629d3f4-54ea-4bdf-9b75-650bbc2299d6\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:49:52.306738 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:49:52.306446    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/4629d3f4-54ea-4bdf-9b75-650bbc2299d6-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"4629d3f4-54ea-4bdf-9b75-650bbc2299d6\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:49:52.306738 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:49:52.306472    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4629d3f4-54ea-4bdf-9b75-650bbc2299d6-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"4629d3f4-54ea-4bdf-9b75-650bbc2299d6\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:49:52.306738 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:49:52.306497    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/4629d3f4-54ea-4bdf-9b75-650bbc2299d6-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"4629d3f4-54ea-4bdf-9b75-650bbc2299d6\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:49:52.306738 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:49:52.306617    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4629d3f4-54ea-4bdf-9b75-650bbc2299d6-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"4629d3f4-54ea-4bdf-9b75-650bbc2299d6\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:49:52.307218 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:49:52.307024    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/4629d3f4-54ea-4bdf-9b75-650bbc2299d6-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"4629d3f4-54ea-4bdf-9b75-650bbc2299d6\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:49:52.307546 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:49:52.307523    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/4629d3f4-54ea-4bdf-9b75-650bbc2299d6-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"4629d3f4-54ea-4bdf-9b75-650bbc2299d6\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:49:52.309302 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:49:52.309273    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/4629d3f4-54ea-4bdf-9b75-650bbc2299d6-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"4629d3f4-54ea-4bdf-9b75-650bbc2299d6\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:49:52.309515 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:49:52.309490    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/4629d3f4-54ea-4bdf-9b75-650bbc2299d6-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"4629d3f4-54ea-4bdf-9b75-650bbc2299d6\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:49:52.309941 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:49:52.309761    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/4629d3f4-54ea-4bdf-9b75-650bbc2299d6-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"4629d3f4-54ea-4bdf-9b75-650bbc2299d6\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:49:52.309941 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:49:52.309796    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/4629d3f4-54ea-4bdf-9b75-650bbc2299d6-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"4629d3f4-54ea-4bdf-9b75-650bbc2299d6\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:49:52.309941 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:49:52.309837    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/4629d3f4-54ea-4bdf-9b75-650bbc2299d6-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"4629d3f4-54ea-4bdf-9b75-650bbc2299d6\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:49:52.309941 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:49:52.309898    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/4629d3f4-54ea-4bdf-9b75-650bbc2299d6-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"4629d3f4-54ea-4bdf-9b75-650bbc2299d6\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:49:52.310213 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:49:52.310039    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/4629d3f4-54ea-4bdf-9b75-650bbc2299d6-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"4629d3f4-54ea-4bdf-9b75-650bbc2299d6\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:49:52.310563 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:49:52.310536    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4629d3f4-54ea-4bdf-9b75-650bbc2299d6-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"4629d3f4-54ea-4bdf-9b75-650bbc2299d6\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:49:52.310637 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:49:52.310588    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/4629d3f4-54ea-4bdf-9b75-650bbc2299d6-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"4629d3f4-54ea-4bdf-9b75-650bbc2299d6\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:49:52.310827 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:49:52.310806    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/4629d3f4-54ea-4bdf-9b75-650bbc2299d6-config-out\") pod \"prometheus-k8s-0\" (UID: \"4629d3f4-54ea-4bdf-9b75-650bbc2299d6\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:49:52.311093 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:49:52.311069    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4629d3f4-54ea-4bdf-9b75-650bbc2299d6-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"4629d3f4-54ea-4bdf-9b75-650bbc2299d6\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:49:52.311249 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:49:52.311228    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/4629d3f4-54ea-4bdf-9b75-650bbc2299d6-config\") pod \"prometheus-k8s-0\" (UID: \"4629d3f4-54ea-4bdf-9b75-650bbc2299d6\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:49:52.312557 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:49:52.312534    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/4629d3f4-54ea-4bdf-9b75-650bbc2299d6-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"4629d3f4-54ea-4bdf-9b75-650bbc2299d6\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:49:52.312703 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:49:52.312673    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/4629d3f4-54ea-4bdf-9b75-650bbc2299d6-web-config\") pod \"prometheus-k8s-0\" (UID: \"4629d3f4-54ea-4bdf-9b75-650bbc2299d6\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:49:52.313040 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:49:52.313018    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/4629d3f4-54ea-4bdf-9b75-650bbc2299d6-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"4629d3f4-54ea-4bdf-9b75-650bbc2299d6\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:49:52.313115 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:49:52.313088    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/4629d3f4-54ea-4bdf-9b75-650bbc2299d6-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"4629d3f4-54ea-4bdf-9b75-650bbc2299d6\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:49:52.319210 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:49:52.319170    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zvtqm\" (UniqueName: \"kubernetes.io/projected/4629d3f4-54ea-4bdf-9b75-650bbc2299d6-kube-api-access-zvtqm\") pod \"prometheus-k8s-0\" (UID: \"4629d3f4-54ea-4bdf-9b75-650bbc2299d6\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:49:52.336181 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:49:52.336139    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:49:52.622919 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:49:52.622889    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 16 18:49:52.626396 ip-10-0-138-127 kubenswrapper[2581]: W0916 18:49:52.626370    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4629d3f4_54ea_4bdf_9b75_650bbc2299d6.slice/crio-eb55835f903750c9ea6f4fa3114047181a990340b2b22736bfc9b5e1ec9dccde WatchSource:0}: Error finding container eb55835f903750c9ea6f4fa3114047181a990340b2b22736bfc9b5e1ec9dccde: Status 404 returned error can't find the container with id eb55835f903750c9ea6f4fa3114047181a990340b2b22736bfc9b5e1ec9dccde
Sep 16 18:49:52.727134 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:49:52.727103    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="33280f86-4497-4343-829b-4781cbfd5e5f" path="/var/lib/kubelet/pods/33280f86-4497-4343-829b-4781cbfd5e5f/volumes"
Sep 16 18:49:52.964176 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:49:52.964089    2581 generic.go:356] "Generic (PLEG): container finished" podID="4629d3f4-54ea-4bdf-9b75-650bbc2299d6" containerID="d334c87998960cf12f0f7e6c7f878f137ac0c90206d8fc70837ca3c820662a57" exitCode=0
Sep 16 18:49:52.964616 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:49:52.964179    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"4629d3f4-54ea-4bdf-9b75-650bbc2299d6","Type":"ContainerDied","Data":"d334c87998960cf12f0f7e6c7f878f137ac0c90206d8fc70837ca3c820662a57"}
Sep 16 18:49:52.964616 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:49:52.964277    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"4629d3f4-54ea-4bdf-9b75-650bbc2299d6","Type":"ContainerStarted","Data":"eb55835f903750c9ea6f4fa3114047181a990340b2b22736bfc9b5e1ec9dccde"}
Sep 16 18:49:53.978818 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:49:53.978779    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"4629d3f4-54ea-4bdf-9b75-650bbc2299d6","Type":"ContainerStarted","Data":"de08c518a66a1dc9d466ac36ecd8fcdff79a3a29b8b2123660b062cf5595e203"}
Sep 16 18:49:53.979264 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:49:53.978868    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"4629d3f4-54ea-4bdf-9b75-650bbc2299d6","Type":"ContainerStarted","Data":"e2e51842668236413b9a29a6cd8860185cafec8f4f3d3dd54366a705ed512526"}
Sep 16 18:49:53.979264 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:49:53.978901    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"4629d3f4-54ea-4bdf-9b75-650bbc2299d6","Type":"ContainerStarted","Data":"0c26d6ddf7c83d67953d96ab78fa4b27c6807a614b4f399447b8a6ae344a5b5a"}
Sep 16 18:49:53.979264 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:49:53.978927    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"4629d3f4-54ea-4bdf-9b75-650bbc2299d6","Type":"ContainerStarted","Data":"fbc213e5cdef73ac547fddffacba89264a473ff25598d31709b21303d6e43f1f"}
Sep 16 18:49:53.979264 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:49:53.978951    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"4629d3f4-54ea-4bdf-9b75-650bbc2299d6","Type":"ContainerStarted","Data":"1812e2692893fc9459681aa810a30ea0c9b8d3f35937faadffc155c231224bbe"}
Sep 16 18:49:53.979264 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:49:53.978992    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"4629d3f4-54ea-4bdf-9b75-650bbc2299d6","Type":"ContainerStarted","Data":"56f5b10e8e212c6c1eb09e3d05dfe7d19602aa6fd369a74550ae703a6cbe4b16"}
Sep 16 18:49:54.011019 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:49:54.010961    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-k8s-0" podStartSLOduration=2.991071397 podStartE2EDuration="3.010940837s" totalImagesPullingTime="0s" totalInitContainerRuntime="19.86944ms" isStatefulPod=false podCreationTimestamp="2026-09-16 18:49:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:49:54.009819117 +0000 UTC m=+263.829233025" watchObservedRunningTime="2026-09-16 18:49:54.010940837 +0000 UTC m=+263.830354747"
Sep 16 18:49:57.337331 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:49:57.337282    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:50:30.614008 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:50:30.613982    2581 kubelet.go:1648] "Image garbage collection succeeded"
Sep 16 18:50:52.336876 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:50:52.336824    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:50:52.353336 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:50:52.353309    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:50:53.224665 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:50:53.224628    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:55:30.479848 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:55:30.479812    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/s3-init-f6mqj"]
Sep 16 18:55:30.483035 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:55:30.483019    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/s3-init-f6mqj"
Sep 16 18:55:30.485457 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:55:30.485434    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"mlpipeline-s3-artifact\""
Sep 16 18:55:30.485457 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:55:30.485442    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve\"/\"openshift-service-ca.crt\""
Sep 16 18:55:30.485652 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:55:30.485524    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"hf-token\""
Sep 16 18:55:30.485652 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:55:30.485542    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve\"/\"kube-root-ca.crt\""
Sep 16 18:55:30.485652 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:55:30.485559    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"default-dockercfg-45jfk\""
Sep 16 18:55:30.491701 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:55:30.491682    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/s3-init-f6mqj"]
Sep 16 18:55:30.535064 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:55:30.535031    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c9b793f9-ac61-43c7-a947-06ca75687644-model-cache\") pod \"s3-init-f6mqj\" (UID: \"c9b793f9-ac61-43c7-a947-06ca75687644\") " pod="kserve/s3-init-f6mqj"
Sep 16 18:55:30.535064 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:55:30.535067    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j6f2w\" (UniqueName: \"kubernetes.io/projected/c9b793f9-ac61-43c7-a947-06ca75687644-kube-api-access-j6f2w\") pod \"s3-init-f6mqj\" (UID: \"c9b793f9-ac61-43c7-a947-06ca75687644\") " pod="kserve/s3-init-f6mqj"
Sep 16 18:55:30.635506 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:55:30.635478    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c9b793f9-ac61-43c7-a947-06ca75687644-model-cache\") pod \"s3-init-f6mqj\" (UID: \"c9b793f9-ac61-43c7-a947-06ca75687644\") " pod="kserve/s3-init-f6mqj"
Sep 16 18:55:30.635506 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:55:30.635509    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j6f2w\" (UniqueName: \"kubernetes.io/projected/c9b793f9-ac61-43c7-a947-06ca75687644-kube-api-access-j6f2w\") pod \"s3-init-f6mqj\" (UID: \"c9b793f9-ac61-43c7-a947-06ca75687644\") " pod="kserve/s3-init-f6mqj"
Sep 16 18:55:30.635879 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:55:30.635853    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c9b793f9-ac61-43c7-a947-06ca75687644-model-cache\") pod \"s3-init-f6mqj\" (UID: \"c9b793f9-ac61-43c7-a947-06ca75687644\") " pod="kserve/s3-init-f6mqj"
Sep 16 18:55:30.643039 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:55:30.643020    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve\"/\"kube-root-ca.crt\""
Sep 16 18:55:30.653722 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:55:30.653703    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve\"/\"openshift-service-ca.crt\""
Sep 16 18:55:30.664465 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:55:30.664448    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-j6f2w\" (UniqueName: \"kubernetes.io/projected/c9b793f9-ac61-43c7-a947-06ca75687644-kube-api-access-j6f2w\") pod \"s3-init-f6mqj\" (UID: \"c9b793f9-ac61-43c7-a947-06ca75687644\") " pod="kserve/s3-init-f6mqj"
Sep 16 18:55:30.801549 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:55:30.801523    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"default-dockercfg-45jfk\""
Sep 16 18:55:30.809969 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:55:30.809944    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/s3-init-f6mqj"
Sep 16 18:55:31.080422 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:55:31.080343    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/s3-init-f6mqj"]
Sep 16 18:55:31.082944 ip-10-0-138-127 kubenswrapper[2581]: W0916 18:55:31.082918    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podc9b793f9_ac61_43c7_a947_06ca75687644.slice/crio-8d386774bfa40255d740811763099c5ce29af931a02b92c473f1ea2ac6db7887 WatchSource:0}: Error finding container 8d386774bfa40255d740811763099c5ce29af931a02b92c473f1ea2ac6db7887: Status 404 returned error can't find the container with id 8d386774bfa40255d740811763099c5ce29af931a02b92c473f1ea2ac6db7887
Sep 16 18:55:31.085043 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:55:31.085026    2581 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 16 18:55:31.262819 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:55:31.262787    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-f6mqj" event={"ID":"c9b793f9-ac61-43c7-a947-06ca75687644","Type":"ContainerStarted","Data":"8d386774bfa40255d740811763099c5ce29af931a02b92c473f1ea2ac6db7887"}
Sep 16 18:55:34.788700 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:55:34.788678    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"hf-token\""
Sep 16 18:55:35.283076 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:55:35.283038    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-f6mqj" event={"ID":"c9b793f9-ac61-43c7-a947-06ca75687644","Type":"ContainerStarted","Data":"3780e89f0c72b6547a95ab7e05b0c63863d0f9ae1f8e59b1debe413ed3e90f55"}
Sep 16 18:55:37.292888 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:55:37.292851    2581 generic.go:356] "Generic (PLEG): container finished" podID="c9b793f9-ac61-43c7-a947-06ca75687644" containerID="3780e89f0c72b6547a95ab7e05b0c63863d0f9ae1f8e59b1debe413ed3e90f55" exitCode=0
Sep 16 18:55:37.293310 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:55:37.292923    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-f6mqj" event={"ID":"c9b793f9-ac61-43c7-a947-06ca75687644","Type":"ContainerDied","Data":"3780e89f0c72b6547a95ab7e05b0c63863d0f9ae1f8e59b1debe413ed3e90f55"}
Sep 16 18:55:41.618257 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:55:41.618239    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"mlpipeline-s3-artifact\""
Sep 16 18:55:42.318692 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:55:42.318658    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-f6mqj" event={"ID":"c9b793f9-ac61-43c7-a947-06ca75687644","Type":"ContainerStarted","Data":"fcba8c021ec0d3b1ada4f971fb198788ad8aa7a00cadd98a6f381657d2ac994d"}
Sep 16 18:55:42.337013 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:55:42.336953    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/s3-init-f6mqj" podStartSLOduration=2.647893378 podStartE2EDuration="12.336934284s" totalImagesPullingTime="8.016930627s" totalInitContainerRuntime="1.672110279s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:55:30 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:55:42.335031825 +0000 UTC m=+612.154445735" watchObservedRunningTime="2026-09-16 18:55:42.336934284 +0000 UTC m=+612.156348194"
Sep 16 18:56:47.583469 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:56:47.583432    2581 generic.go:356] "Generic (PLEG): container finished" podID="c9b793f9-ac61-43c7-a947-06ca75687644" containerID="fcba8c021ec0d3b1ada4f971fb198788ad8aa7a00cadd98a6f381657d2ac994d" exitCode=0
Sep 16 18:56:47.583991 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:56:47.583505    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-f6mqj" event={"ID":"c9b793f9-ac61-43c7-a947-06ca75687644","Type":"ContainerDied","Data":"fcba8c021ec0d3b1ada4f971fb198788ad8aa7a00cadd98a6f381657d2ac994d"}
Sep 16 18:56:48.711241 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:56:48.711216    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/s3-init-f6mqj"
Sep 16 18:56:48.790561 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:56:48.790522    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-j6f2w\" (UniqueName: \"kubernetes.io/projected/c9b793f9-ac61-43c7-a947-06ca75687644-kube-api-access-j6f2w\") pod \"c9b793f9-ac61-43c7-a947-06ca75687644\" (UID: \"c9b793f9-ac61-43c7-a947-06ca75687644\") "
Sep 16 18:56:48.790731 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:56:48.790597    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c9b793f9-ac61-43c7-a947-06ca75687644-model-cache\") pod \"c9b793f9-ac61-43c7-a947-06ca75687644\" (UID: \"c9b793f9-ac61-43c7-a947-06ca75687644\") "
Sep 16 18:56:48.792934 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:56:48.792907    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c9b793f9-ac61-43c7-a947-06ca75687644-kube-api-access-j6f2w" (OuterVolumeSpecName: "kube-api-access-j6f2w") pod "c9b793f9-ac61-43c7-a947-06ca75687644" (UID: "c9b793f9-ac61-43c7-a947-06ca75687644"). InnerVolumeSpecName "kube-api-access-j6f2w". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 18:56:48.810982 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:56:48.810943    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c9b793f9-ac61-43c7-a947-06ca75687644-model-cache" (OuterVolumeSpecName: "model-cache") pod "c9b793f9-ac61-43c7-a947-06ca75687644" (UID: "c9b793f9-ac61-43c7-a947-06ca75687644"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 18:56:48.891625 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:56:48.891509    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j6f2w\" (UniqueName: \"kubernetes.io/projected/c9b793f9-ac61-43c7-a947-06ca75687644-kube-api-access-j6f2w\") on node \"ip-10-0-138-127.ec2.internal\" DevicePath \"\""
Sep 16 18:56:48.891625 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:56:48.891567    2581 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c9b793f9-ac61-43c7-a947-06ca75687644-model-cache\") on node \"ip-10-0-138-127.ec2.internal\" DevicePath \"\""
Sep 16 18:56:49.593042 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:56:49.593010    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-f6mqj" event={"ID":"c9b793f9-ac61-43c7-a947-06ca75687644","Type":"ContainerDied","Data":"8d386774bfa40255d740811763099c5ce29af931a02b92c473f1ea2ac6db7887"}
Sep 16 18:56:49.593042 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:56:49.593030    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/s3-init-f6mqj"
Sep 16 18:56:49.593042 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:56:49.593048    2581 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8d386774bfa40255d740811763099c5ce29af931a02b92c473f1ea2ac6db7887"
Sep 16 18:57:01.557683 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:57:01.557652    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-raw-sklearn-batcher-b88cf-predictor-6f597c578b-zzd8b"]
Sep 16 18:57:01.558178 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:57:01.557986    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c9b793f9-ac61-43c7-a947-06ca75687644" containerName="s3-init"
Sep 16 18:57:01.558178 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:57:01.557997    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="c9b793f9-ac61-43c7-a947-06ca75687644" containerName="s3-init"
Sep 16 18:57:01.558178 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:57:01.558016    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c9b793f9-ac61-43c7-a947-06ca75687644" containerName="download-hf-model"
Sep 16 18:57:01.558178 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:57:01.558022    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="c9b793f9-ac61-43c7-a947-06ca75687644" containerName="download-hf-model"
Sep 16 18:57:01.558178 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:57:01.558068    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="c9b793f9-ac61-43c7-a947-06ca75687644" containerName="s3-init"
Sep 16 18:57:01.561152 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:57:01.561136    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-b88cf-predictor-6f597c578b-zzd8b"
Sep 16 18:57:01.564178 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:57:01.564157    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-raw-sklearn-batcher-b88cf-predictor-serving-cert\""
Sep 16 18:57:01.564297 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:57:01.564159    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Sep 16 18:57:01.564520 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:57:01.564501    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-c8lqg\""
Sep 16 18:57:01.564914 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:57:01.564899    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Sep 16 18:57:01.564995 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:57:01.564943    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Sep 16 18:57:01.565188 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:57:01.565172    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-raw-sklearn-batcher-b88cf-kube-rbac-proxy-sar-config\""
Sep 16 18:57:01.565288 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:57:01.565271    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Sep 16 18:57:01.577563 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:57:01.577535    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-raw-sklearn-batcher-b88cf-predictor-6f597c578b-zzd8b"]
Sep 16 18:57:01.700830 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:57:01.700793    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/a81cbc02-9a43-40df-9a7a-fb53b8d0ed74-cabundle-cert\") pod \"isvc-raw-sklearn-batcher-b88cf-predictor-6f597c578b-zzd8b\" (UID: \"a81cbc02-9a43-40df-9a7a-fb53b8d0ed74\") " pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-b88cf-predictor-6f597c578b-zzd8b"
Sep 16 18:57:01.701018 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:57:01.700883    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a81cbc02-9a43-40df-9a7a-fb53b8d0ed74-proxy-tls\") pod \"isvc-raw-sklearn-batcher-b88cf-predictor-6f597c578b-zzd8b\" (UID: \"a81cbc02-9a43-40df-9a7a-fb53b8d0ed74\") " pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-b88cf-predictor-6f597c578b-zzd8b"
Sep 16 18:57:01.701018 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:57:01.700935    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a81cbc02-9a43-40df-9a7a-fb53b8d0ed74-kserve-provision-location\") pod \"isvc-raw-sklearn-batcher-b88cf-predictor-6f597c578b-zzd8b\" (UID: \"a81cbc02-9a43-40df-9a7a-fb53b8d0ed74\") " pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-b88cf-predictor-6f597c578b-zzd8b"
Sep 16 18:57:01.701018 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:57:01.700996    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lkbrm\" (UniqueName: \"kubernetes.io/projected/a81cbc02-9a43-40df-9a7a-fb53b8d0ed74-kube-api-access-lkbrm\") pod \"isvc-raw-sklearn-batcher-b88cf-predictor-6f597c578b-zzd8b\" (UID: \"a81cbc02-9a43-40df-9a7a-fb53b8d0ed74\") " pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-b88cf-predictor-6f597c578b-zzd8b"
Sep 16 18:57:01.701138 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:57:01.701038    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a81cbc02-9a43-40df-9a7a-fb53b8d0ed74-kube-rbac-proxy-sar-config\") pod \"isvc-raw-sklearn-batcher-b88cf-predictor-6f597c578b-zzd8b\" (UID: \"a81cbc02-9a43-40df-9a7a-fb53b8d0ed74\") " pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-b88cf-predictor-6f597c578b-zzd8b"
Sep 16 18:57:01.726172 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:57:01.726142    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-predictor-75f85bc9f9-hfctg"]
Sep 16 18:57:01.729716 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:57:01.729699    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-75f85bc9f9-hfctg"
Sep 16 18:57:01.732373 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:57:01.732353    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-rollback-kube-rbac-proxy-sar-config\""
Sep 16 18:57:01.732373 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:57:01.732352    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-rollback-predictor-serving-cert\""
Sep 16 18:57:01.753820 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:57:01.753794    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-predictor-75f85bc9f9-hfctg"]
Sep 16 18:57:01.801993 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:57:01.801953    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a81cbc02-9a43-40df-9a7a-fb53b8d0ed74-proxy-tls\") pod \"isvc-raw-sklearn-batcher-b88cf-predictor-6f597c578b-zzd8b\" (UID: \"a81cbc02-9a43-40df-9a7a-fb53b8d0ed74\") " pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-b88cf-predictor-6f597c578b-zzd8b"
Sep 16 18:57:01.802223 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:57:01.802018    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a81cbc02-9a43-40df-9a7a-fb53b8d0ed74-kserve-provision-location\") pod \"isvc-raw-sklearn-batcher-b88cf-predictor-6f597c578b-zzd8b\" (UID: \"a81cbc02-9a43-40df-9a7a-fb53b8d0ed74\") " pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-b88cf-predictor-6f597c578b-zzd8b"
Sep 16 18:57:01.802223 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:57:01.802058    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lkbrm\" (UniqueName: \"kubernetes.io/projected/a81cbc02-9a43-40df-9a7a-fb53b8d0ed74-kube-api-access-lkbrm\") pod \"isvc-raw-sklearn-batcher-b88cf-predictor-6f597c578b-zzd8b\" (UID: \"a81cbc02-9a43-40df-9a7a-fb53b8d0ed74\") " pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-b88cf-predictor-6f597c578b-zzd8b"
Sep 16 18:57:01.802223 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:57:01.802087    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a81cbc02-9a43-40df-9a7a-fb53b8d0ed74-kube-rbac-proxy-sar-config\") pod \"isvc-raw-sklearn-batcher-b88cf-predictor-6f597c578b-zzd8b\" (UID: \"a81cbc02-9a43-40df-9a7a-fb53b8d0ed74\") " pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-b88cf-predictor-6f597c578b-zzd8b"
Sep 16 18:57:01.802442 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:57:01.802229    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/a81cbc02-9a43-40df-9a7a-fb53b8d0ed74-cabundle-cert\") pod \"isvc-raw-sklearn-batcher-b88cf-predictor-6f597c578b-zzd8b\" (UID: \"a81cbc02-9a43-40df-9a7a-fb53b8d0ed74\") " pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-b88cf-predictor-6f597c578b-zzd8b"
Sep 16 18:57:01.802566 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:57:01.802525    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a81cbc02-9a43-40df-9a7a-fb53b8d0ed74-kserve-provision-location\") pod \"isvc-raw-sklearn-batcher-b88cf-predictor-6f597c578b-zzd8b\" (UID: \"a81cbc02-9a43-40df-9a7a-fb53b8d0ed74\") " pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-b88cf-predictor-6f597c578b-zzd8b"
Sep 16 18:57:01.802907 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:57:01.802882    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a81cbc02-9a43-40df-9a7a-fb53b8d0ed74-kube-rbac-proxy-sar-config\") pod \"isvc-raw-sklearn-batcher-b88cf-predictor-6f597c578b-zzd8b\" (UID: \"a81cbc02-9a43-40df-9a7a-fb53b8d0ed74\") " pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-b88cf-predictor-6f597c578b-zzd8b"
Sep 16 18:57:01.802907 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:57:01.802894    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/a81cbc02-9a43-40df-9a7a-fb53b8d0ed74-cabundle-cert\") pod \"isvc-raw-sklearn-batcher-b88cf-predictor-6f597c578b-zzd8b\" (UID: \"a81cbc02-9a43-40df-9a7a-fb53b8d0ed74\") " pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-b88cf-predictor-6f597c578b-zzd8b"
Sep 16 18:57:01.804568 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:57:01.804551    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a81cbc02-9a43-40df-9a7a-fb53b8d0ed74-proxy-tls\") pod \"isvc-raw-sklearn-batcher-b88cf-predictor-6f597c578b-zzd8b\" (UID: \"a81cbc02-9a43-40df-9a7a-fb53b8d0ed74\") " pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-b88cf-predictor-6f597c578b-zzd8b"
Sep 16 18:57:01.813477 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:57:01.813420    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lkbrm\" (UniqueName: \"kubernetes.io/projected/a81cbc02-9a43-40df-9a7a-fb53b8d0ed74-kube-api-access-lkbrm\") pod \"isvc-raw-sklearn-batcher-b88cf-predictor-6f597c578b-zzd8b\" (UID: \"a81cbc02-9a43-40df-9a7a-fb53b8d0ed74\") " pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-b88cf-predictor-6f597c578b-zzd8b"
Sep 16 18:57:01.871175 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:57:01.871134    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-b88cf-predictor-6f597c578b-zzd8b"
Sep 16 18:57:01.903702 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:57:01.903653    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/b17e8459-9e27-44f6-82a0-f1c8bf30d48b-kube-rbac-proxy-sar-config\") pod \"isvc-canary-rollback-predictor-75f85bc9f9-hfctg\" (UID: \"b17e8459-9e27-44f6-82a0-f1c8bf30d48b\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-75f85bc9f9-hfctg"
Sep 16 18:57:01.903702 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:57:01.903695    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b17e8459-9e27-44f6-82a0-f1c8bf30d48b-proxy-tls\") pod \"isvc-canary-rollback-predictor-75f85bc9f9-hfctg\" (UID: \"b17e8459-9e27-44f6-82a0-f1c8bf30d48b\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-75f85bc9f9-hfctg"
Sep 16 18:57:01.903968 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:57:01.903724    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b17e8459-9e27-44f6-82a0-f1c8bf30d48b-kserve-provision-location\") pod \"isvc-canary-rollback-predictor-75f85bc9f9-hfctg\" (UID: \"b17e8459-9e27-44f6-82a0-f1c8bf30d48b\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-75f85bc9f9-hfctg"
Sep 16 18:57:01.903968 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:57:01.903792    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j5xnl\" (UniqueName: \"kubernetes.io/projected/b17e8459-9e27-44f6-82a0-f1c8bf30d48b-kube-api-access-j5xnl\") pod \"isvc-canary-rollback-predictor-75f85bc9f9-hfctg\" (UID: \"b17e8459-9e27-44f6-82a0-f1c8bf30d48b\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-75f85bc9f9-hfctg"
Sep 16 18:57:01.903968 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:57:01.903830    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/b17e8459-9e27-44f6-82a0-f1c8bf30d48b-cabundle-cert\") pod \"isvc-canary-rollback-predictor-75f85bc9f9-hfctg\" (UID: \"b17e8459-9e27-44f6-82a0-f1c8bf30d48b\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-75f85bc9f9-hfctg"
Sep 16 18:57:02.004563 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:57:02.004514    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/b17e8459-9e27-44f6-82a0-f1c8bf30d48b-kube-rbac-proxy-sar-config\") pod \"isvc-canary-rollback-predictor-75f85bc9f9-hfctg\" (UID: \"b17e8459-9e27-44f6-82a0-f1c8bf30d48b\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-75f85bc9f9-hfctg"
Sep 16 18:57:02.004772 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:57:02.004573    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b17e8459-9e27-44f6-82a0-f1c8bf30d48b-proxy-tls\") pod \"isvc-canary-rollback-predictor-75f85bc9f9-hfctg\" (UID: \"b17e8459-9e27-44f6-82a0-f1c8bf30d48b\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-75f85bc9f9-hfctg"
Sep 16 18:57:02.004772 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:57:02.004616    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b17e8459-9e27-44f6-82a0-f1c8bf30d48b-kserve-provision-location\") pod \"isvc-canary-rollback-predictor-75f85bc9f9-hfctg\" (UID: \"b17e8459-9e27-44f6-82a0-f1c8bf30d48b\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-75f85bc9f9-hfctg"
Sep 16 18:57:02.004772 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:57:02.004668    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j5xnl\" (UniqueName: \"kubernetes.io/projected/b17e8459-9e27-44f6-82a0-f1c8bf30d48b-kube-api-access-j5xnl\") pod \"isvc-canary-rollback-predictor-75f85bc9f9-hfctg\" (UID: \"b17e8459-9e27-44f6-82a0-f1c8bf30d48b\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-75f85bc9f9-hfctg"
Sep 16 18:57:02.004772 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:57:02.004694    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/b17e8459-9e27-44f6-82a0-f1c8bf30d48b-cabundle-cert\") pod \"isvc-canary-rollback-predictor-75f85bc9f9-hfctg\" (UID: \"b17e8459-9e27-44f6-82a0-f1c8bf30d48b\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-75f85bc9f9-hfctg"
Sep 16 18:57:02.005129 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:57:02.005103    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b17e8459-9e27-44f6-82a0-f1c8bf30d48b-kserve-provision-location\") pod \"isvc-canary-rollback-predictor-75f85bc9f9-hfctg\" (UID: \"b17e8459-9e27-44f6-82a0-f1c8bf30d48b\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-75f85bc9f9-hfctg"
Sep 16 18:57:02.005306 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:57:02.005285    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/b17e8459-9e27-44f6-82a0-f1c8bf30d48b-kube-rbac-proxy-sar-config\") pod \"isvc-canary-rollback-predictor-75f85bc9f9-hfctg\" (UID: \"b17e8459-9e27-44f6-82a0-f1c8bf30d48b\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-75f85bc9f9-hfctg"
Sep 16 18:57:02.005398 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:57:02.005379    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/b17e8459-9e27-44f6-82a0-f1c8bf30d48b-cabundle-cert\") pod \"isvc-canary-rollback-predictor-75f85bc9f9-hfctg\" (UID: \"b17e8459-9e27-44f6-82a0-f1c8bf30d48b\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-75f85bc9f9-hfctg"
Sep 16 18:57:02.007206 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:57:02.007170    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b17e8459-9e27-44f6-82a0-f1c8bf30d48b-proxy-tls\") pod \"isvc-canary-rollback-predictor-75f85bc9f9-hfctg\" (UID: \"b17e8459-9e27-44f6-82a0-f1c8bf30d48b\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-75f85bc9f9-hfctg"
Sep 16 18:57:02.014173 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:57:02.014151    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-j5xnl\" (UniqueName: \"kubernetes.io/projected/b17e8459-9e27-44f6-82a0-f1c8bf30d48b-kube-api-access-j5xnl\") pod \"isvc-canary-rollback-predictor-75f85bc9f9-hfctg\" (UID: \"b17e8459-9e27-44f6-82a0-f1c8bf30d48b\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-75f85bc9f9-hfctg"
Sep 16 18:57:02.039899 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:57:02.039865    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-75f85bc9f9-hfctg"
Sep 16 18:57:02.147333 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:57:02.147297    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-raw-sklearn-batcher-b88cf-predictor-6f597c578b-zzd8b"]
Sep 16 18:57:02.151165 ip-10-0-138-127 kubenswrapper[2581]: W0916 18:57:02.151132    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda81cbc02_9a43_40df_9a7a_fb53b8d0ed74.slice/crio-597abb2513796a0d7accded6d7298bd2d585da8763029539a1484fac77c337ea WatchSource:0}: Error finding container 597abb2513796a0d7accded6d7298bd2d585da8763029539a1484fac77c337ea: Status 404 returned error can't find the container with id 597abb2513796a0d7accded6d7298bd2d585da8763029539a1484fac77c337ea
Sep 16 18:57:02.328401 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:57:02.328376    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-predictor-75f85bc9f9-hfctg"]
Sep 16 18:57:02.331417 ip-10-0-138-127 kubenswrapper[2581]: W0916 18:57:02.331381    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb17e8459_9e27_44f6_82a0_f1c8bf30d48b.slice/crio-ed8b4a7ae79798379038d78dc797cd903b448774f82a522e770df21e21c6f3eb WatchSource:0}: Error finding container ed8b4a7ae79798379038d78dc797cd903b448774f82a522e770df21e21c6f3eb: Status 404 returned error can't find the container with id ed8b4a7ae79798379038d78dc797cd903b448774f82a522e770df21e21c6f3eb
Sep 16 18:57:02.646951 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:57:02.646852    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-75f85bc9f9-hfctg" event={"ID":"b17e8459-9e27-44f6-82a0-f1c8bf30d48b","Type":"ContainerStarted","Data":"4cf5778c6b02e8c6c17f4c0db06f2324a9d897a530774acc4ad6131dc8164e34"}
Sep 16 18:57:02.647388 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:57:02.646959    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-75f85bc9f9-hfctg" event={"ID":"b17e8459-9e27-44f6-82a0-f1c8bf30d48b","Type":"ContainerStarted","Data":"ed8b4a7ae79798379038d78dc797cd903b448774f82a522e770df21e21c6f3eb"}
Sep 16 18:57:02.648702 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:57:02.648674    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-b88cf-predictor-6f597c578b-zzd8b" event={"ID":"a81cbc02-9a43-40df-9a7a-fb53b8d0ed74","Type":"ContainerStarted","Data":"d35aa5bcc401a5f827aacd509b60ae310fc989951e6dc303ed094f214c4ccb62"}
Sep 16 18:57:02.648838 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:57:02.648728    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-b88cf-predictor-6f597c578b-zzd8b" event={"ID":"a81cbc02-9a43-40df-9a7a-fb53b8d0ed74","Type":"ContainerStarted","Data":"597abb2513796a0d7accded6d7298bd2d585da8763029539a1484fac77c337ea"}
Sep 16 18:57:06.667944 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:57:06.667856    2581 generic.go:356] "Generic (PLEG): container finished" podID="a81cbc02-9a43-40df-9a7a-fb53b8d0ed74" containerID="d35aa5bcc401a5f827aacd509b60ae310fc989951e6dc303ed094f214c4ccb62" exitCode=0
Sep 16 18:57:06.667944 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:57:06.667930    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-b88cf-predictor-6f597c578b-zzd8b" event={"ID":"a81cbc02-9a43-40df-9a7a-fb53b8d0ed74","Type":"ContainerDied","Data":"d35aa5bcc401a5f827aacd509b60ae310fc989951e6dc303ed094f214c4ccb62"}
Sep 16 18:57:06.669644 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:57:06.669613    2581 generic.go:356] "Generic (PLEG): container finished" podID="b17e8459-9e27-44f6-82a0-f1c8bf30d48b" containerID="4cf5778c6b02e8c6c17f4c0db06f2324a9d897a530774acc4ad6131dc8164e34" exitCode=0
Sep 16 18:57:06.669752 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:57:06.669653    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-75f85bc9f9-hfctg" event={"ID":"b17e8459-9e27-44f6-82a0-f1c8bf30d48b","Type":"ContainerDied","Data":"4cf5778c6b02e8c6c17f4c0db06f2324a9d897a530774acc4ad6131dc8164e34"}
Sep 16 18:57:22.792025 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:57:22.791989    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-75f85bc9f9-hfctg" event={"ID":"b17e8459-9e27-44f6-82a0-f1c8bf30d48b","Type":"ContainerStarted","Data":"cec2e87528d736155874f241e395ba11290513d7a462cda2bf79df6ab3d6d422"}
Sep 16 18:57:22.793831 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:57:22.793811    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-b88cf-predictor-6f597c578b-zzd8b" event={"ID":"a81cbc02-9a43-40df-9a7a-fb53b8d0ed74","Type":"ContainerStarted","Data":"0cbb2c96007f87d82f48a3d56a1fd2e9b9b1cbf4b67e23dc86d56ad304ee2d0e"}
Sep 16 18:57:25.812440 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:57:25.812391    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-b88cf-predictor-6f597c578b-zzd8b" event={"ID":"a81cbc02-9a43-40df-9a7a-fb53b8d0ed74","Type":"ContainerStarted","Data":"c9b55a427c44320832682e4a60470866aeab296465631b4c6ca4c10caa2f23d4"}
Sep 16 18:57:25.815009 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:57:25.814953    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-75f85bc9f9-hfctg" event={"ID":"b17e8459-9e27-44f6-82a0-f1c8bf30d48b","Type":"ContainerStarted","Data":"ae9b007e97b9ece3e72e0c0058e40ddbf0931cdb6b5ccd642a12baa97081a202"}
Sep 16 18:57:25.815174 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:57:25.815112    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-75f85bc9f9-hfctg"
Sep 16 18:57:25.815272 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:57:25.815253    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-75f85bc9f9-hfctg"
Sep 16 18:57:25.816794 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:57:25.816756    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-75f85bc9f9-hfctg" podUID="b17e8459-9e27-44f6-82a0-f1c8bf30d48b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.26:8080: connect: connection refused"
Sep 16 18:57:25.839031 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:57:25.838982    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-75f85bc9f9-hfctg" podStartSLOduration=2.420071614 podStartE2EDuration="24.838965348s" totalImagesPullingTime="18.505991954s" totalInitContainerRuntime="3.91290178s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:57:01 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:57:25.837082018 +0000 UTC m=+715.656495928" watchObservedRunningTime="2026-09-16 18:57:25.838965348 +0000 UTC m=+715.658379323"
Sep 16 18:57:26.819860 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:57:26.819811    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-75f85bc9f9-hfctg" podUID="b17e8459-9e27-44f6-82a0-f1c8bf30d48b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.26:8080: connect: connection refused"
Sep 16 18:57:27.826501 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:57:27.826469    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-b88cf-predictor-6f597c578b-zzd8b" event={"ID":"a81cbc02-9a43-40df-9a7a-fb53b8d0ed74","Type":"ContainerStarted","Data":"d63b026583466c72366b547f05d90b8073c81b492695d00130d20a364c208055"}
Sep 16 18:57:27.826964 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:57:27.826703    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-b88cf-predictor-6f597c578b-zzd8b"
Sep 16 18:57:27.826964 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:57:27.826854    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-b88cf-predictor-6f597c578b-zzd8b"
Sep 16 18:57:27.828121 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:57:27.828092    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-b88cf-predictor-6f597c578b-zzd8b" podUID="a81cbc02-9a43-40df-9a7a-fb53b8d0ed74" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.25:8080: connect: connection refused"
Sep 16 18:57:27.851184 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:57:27.851120    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-b88cf-predictor-6f597c578b-zzd8b" podStartSLOduration=2.625080046 podStartE2EDuration="26.851096045s" totalImagesPullingTime="20.376902002s" totalInitContainerRuntime="3.849113997s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:57:01 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:57:27.850475993 +0000 UTC m=+717.669889928" watchObservedRunningTime="2026-09-16 18:57:27.851096045 +0000 UTC m=+717.670509953"
Sep 16 18:57:28.830853 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:57:28.830812    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-b88cf-predictor-6f597c578b-zzd8b"
Sep 16 18:57:28.831329 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:57:28.830990    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-b88cf-predictor-6f597c578b-zzd8b" podUID="a81cbc02-9a43-40df-9a7a-fb53b8d0ed74" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.25:8080: connect: connection refused"
Sep 16 18:57:28.832215 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:57:28.832166    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-b88cf-predictor-6f597c578b-zzd8b" podUID="a81cbc02-9a43-40df-9a7a-fb53b8d0ed74" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 16 18:57:29.836137 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:57:29.836096    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-b88cf-predictor-6f597c578b-zzd8b" podUID="a81cbc02-9a43-40df-9a7a-fb53b8d0ed74" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.25:8080: connect: connection refused"
Sep 16 18:57:29.836605 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:57:29.836583    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-b88cf-predictor-6f597c578b-zzd8b" podUID="a81cbc02-9a43-40df-9a7a-fb53b8d0ed74" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 16 18:57:31.824444 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:57:31.824414    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-75f85bc9f9-hfctg"
Sep 16 18:57:31.825026 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:57:31.825000    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-75f85bc9f9-hfctg" podUID="b17e8459-9e27-44f6-82a0-f1c8bf30d48b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.26:8080: connect: connection refused"
Sep 16 18:57:34.840979 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:57:34.840943    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-b88cf-predictor-6f597c578b-zzd8b"
Sep 16 18:57:34.841631 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:57:34.841602    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-b88cf-predictor-6f597c578b-zzd8b" podUID="a81cbc02-9a43-40df-9a7a-fb53b8d0ed74" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.25:8080: connect: connection refused"
Sep 16 18:57:34.842031 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:57:34.842009    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-b88cf-predictor-6f597c578b-zzd8b" podUID="a81cbc02-9a43-40df-9a7a-fb53b8d0ed74" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 16 18:57:41.825349 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:57:41.825304    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-75f85bc9f9-hfctg" podUID="b17e8459-9e27-44f6-82a0-f1c8bf30d48b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.26:8080: connect: connection refused"
Sep 16 18:57:44.841705 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:57:44.841657    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-b88cf-predictor-6f597c578b-zzd8b" podUID="a81cbc02-9a43-40df-9a7a-fb53b8d0ed74" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.25:8080: connect: connection refused"
Sep 16 18:57:44.842185 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:57:44.842157    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-b88cf-predictor-6f597c578b-zzd8b" podUID="a81cbc02-9a43-40df-9a7a-fb53b8d0ed74" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 16 18:57:51.825458 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:57:51.825411    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-75f85bc9f9-hfctg" podUID="b17e8459-9e27-44f6-82a0-f1c8bf30d48b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.26:8080: connect: connection refused"
Sep 16 18:57:54.841937 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:57:54.841889    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-b88cf-predictor-6f597c578b-zzd8b" podUID="a81cbc02-9a43-40df-9a7a-fb53b8d0ed74" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.25:8080: connect: connection refused"
Sep 16 18:57:54.842456 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:57:54.842351    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-b88cf-predictor-6f597c578b-zzd8b" podUID="a81cbc02-9a43-40df-9a7a-fb53b8d0ed74" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 16 18:58:01.825629 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:58:01.825586    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-75f85bc9f9-hfctg" podUID="b17e8459-9e27-44f6-82a0-f1c8bf30d48b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.26:8080: connect: connection refused"
Sep 16 18:58:04.841680 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:58:04.841633    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-b88cf-predictor-6f597c578b-zzd8b" podUID="a81cbc02-9a43-40df-9a7a-fb53b8d0ed74" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.25:8080: connect: connection refused"
Sep 16 18:58:04.842077 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:58:04.842030    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-b88cf-predictor-6f597c578b-zzd8b" podUID="a81cbc02-9a43-40df-9a7a-fb53b8d0ed74" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 16 18:58:11.825078 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:58:11.825037    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-75f85bc9f9-hfctg" podUID="b17e8459-9e27-44f6-82a0-f1c8bf30d48b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.26:8080: connect: connection refused"
Sep 16 18:58:14.842405 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:58:14.842361    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-b88cf-predictor-6f597c578b-zzd8b" podUID="a81cbc02-9a43-40df-9a7a-fb53b8d0ed74" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.25:8080: connect: connection refused"
Sep 16 18:58:14.842896 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:58:14.842872    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-b88cf-predictor-6f597c578b-zzd8b" podUID="a81cbc02-9a43-40df-9a7a-fb53b8d0ed74" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 16 18:58:21.825177 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:58:21.825137    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-75f85bc9f9-hfctg" podUID="b17e8459-9e27-44f6-82a0-f1c8bf30d48b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.26:8080: connect: connection refused"
Sep 16 18:58:24.842465 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:58:24.842427    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-b88cf-predictor-6f597c578b-zzd8b" podUID="a81cbc02-9a43-40df-9a7a-fb53b8d0ed74" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.25:8080: connect: connection refused"
Sep 16 18:58:24.842926 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:58:24.842883    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-b88cf-predictor-6f597c578b-zzd8b" podUID="a81cbc02-9a43-40df-9a7a-fb53b8d0ed74" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 16 18:58:31.825249 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:58:31.825185    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-75f85bc9f9-hfctg" podUID="b17e8459-9e27-44f6-82a0-f1c8bf30d48b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.26:8080: connect: connection refused"
Sep 16 18:58:34.841698 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:58:34.841649    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-b88cf-predictor-6f597c578b-zzd8b" podUID="a81cbc02-9a43-40df-9a7a-fb53b8d0ed74" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.25:8080: connect: connection refused"
Sep 16 18:58:34.842149 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:58:34.842096    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-b88cf-predictor-6f597c578b-zzd8b" podUID="a81cbc02-9a43-40df-9a7a-fb53b8d0ed74" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 16 18:58:41.550073 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:58:41.550034    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-predictor-75f85bc9f9-hfctg"]
Sep 16 18:58:41.550690 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:58:41.550361    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-75f85bc9f9-hfctg" podUID="b17e8459-9e27-44f6-82a0-f1c8bf30d48b" containerName="kserve-container" containerID="cri-o://cec2e87528d736155874f241e395ba11290513d7a462cda2bf79df6ab3d6d422" gracePeriod=30
Sep 16 18:58:41.550690 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:58:41.550403    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-75f85bc9f9-hfctg" podUID="b17e8459-9e27-44f6-82a0-f1c8bf30d48b" containerName="kube-rbac-proxy" containerID="cri-o://ae9b007e97b9ece3e72e0c0058e40ddbf0931cdb6b5ccd642a12baa97081a202" gracePeriod=30
Sep 16 18:58:41.551617 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:58:41.551482    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-75f85bc9f9-hfctg"
Sep 16 18:58:41.820389 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:58:41.820283    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-75f85bc9f9-hfctg" podUID="b17e8459-9e27-44f6-82a0-f1c8bf30d48b" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.26:8643/healthz\": dial tcp 10.132.0.26:8643: connect: connection refused"
Sep 16 18:58:42.149359 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:58:42.149274    2581 generic.go:356] "Generic (PLEG): container finished" podID="b17e8459-9e27-44f6-82a0-f1c8bf30d48b" containerID="ae9b007e97b9ece3e72e0c0058e40ddbf0931cdb6b5ccd642a12baa97081a202" exitCode=2
Sep 16 18:58:42.149359 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:58:42.149349    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-75f85bc9f9-hfctg" event={"ID":"b17e8459-9e27-44f6-82a0-f1c8bf30d48b","Type":"ContainerDied","Data":"ae9b007e97b9ece3e72e0c0058e40ddbf0931cdb6b5ccd642a12baa97081a202"}
Sep 16 18:58:44.843175 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:58:44.843146    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-b88cf-predictor-6f597c578b-zzd8b"
Sep 16 18:58:44.843617 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:58:44.843419    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-b88cf-predictor-6f597c578b-zzd8b"
Sep 16 18:58:46.591460 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:58:46.591430    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-75f85bc9f9-hfctg"
Sep 16 18:58:46.744690 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:58:46.744603    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b17e8459-9e27-44f6-82a0-f1c8bf30d48b-proxy-tls\") pod \"b17e8459-9e27-44f6-82a0-f1c8bf30d48b\" (UID: \"b17e8459-9e27-44f6-82a0-f1c8bf30d48b\") "
Sep 16 18:58:46.744690 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:58:46.744637    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b17e8459-9e27-44f6-82a0-f1c8bf30d48b-kserve-provision-location\") pod \"b17e8459-9e27-44f6-82a0-f1c8bf30d48b\" (UID: \"b17e8459-9e27-44f6-82a0-f1c8bf30d48b\") "
Sep 16 18:58:46.744927 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:58:46.744699    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/b17e8459-9e27-44f6-82a0-f1c8bf30d48b-kube-rbac-proxy-sar-config\") pod \"b17e8459-9e27-44f6-82a0-f1c8bf30d48b\" (UID: \"b17e8459-9e27-44f6-82a0-f1c8bf30d48b\") "
Sep 16 18:58:46.744927 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:58:46.744737    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-j5xnl\" (UniqueName: \"kubernetes.io/projected/b17e8459-9e27-44f6-82a0-f1c8bf30d48b-kube-api-access-j5xnl\") pod \"b17e8459-9e27-44f6-82a0-f1c8bf30d48b\" (UID: \"b17e8459-9e27-44f6-82a0-f1c8bf30d48b\") "
Sep 16 18:58:46.744927 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:58:46.744834    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/b17e8459-9e27-44f6-82a0-f1c8bf30d48b-cabundle-cert\") pod \"b17e8459-9e27-44f6-82a0-f1c8bf30d48b\" (UID: \"b17e8459-9e27-44f6-82a0-f1c8bf30d48b\") "
Sep 16 18:58:46.745135 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:58:46.745068    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b17e8459-9e27-44f6-82a0-f1c8bf30d48b-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "b17e8459-9e27-44f6-82a0-f1c8bf30d48b" (UID: "b17e8459-9e27-44f6-82a0-f1c8bf30d48b"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 18:58:46.745283 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:58:46.745069    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b17e8459-9e27-44f6-82a0-f1c8bf30d48b-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "b17e8459-9e27-44f6-82a0-f1c8bf30d48b" (UID: "b17e8459-9e27-44f6-82a0-f1c8bf30d48b"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 18:58:46.745283 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:58:46.745241    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b17e8459-9e27-44f6-82a0-f1c8bf30d48b-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "b17e8459-9e27-44f6-82a0-f1c8bf30d48b" (UID: "b17e8459-9e27-44f6-82a0-f1c8bf30d48b"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 18:58:46.747078 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:58:46.747055    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b17e8459-9e27-44f6-82a0-f1c8bf30d48b-kube-api-access-j5xnl" (OuterVolumeSpecName: "kube-api-access-j5xnl") pod "b17e8459-9e27-44f6-82a0-f1c8bf30d48b" (UID: "b17e8459-9e27-44f6-82a0-f1c8bf30d48b"). InnerVolumeSpecName "kube-api-access-j5xnl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 18:58:46.747170 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:58:46.747080    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b17e8459-9e27-44f6-82a0-f1c8bf30d48b-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "b17e8459-9e27-44f6-82a0-f1c8bf30d48b" (UID: "b17e8459-9e27-44f6-82a0-f1c8bf30d48b"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 18:58:46.846027 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:58:46.845990    2581 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/b17e8459-9e27-44f6-82a0-f1c8bf30d48b-cabundle-cert\") on node \"ip-10-0-138-127.ec2.internal\" DevicePath \"\""
Sep 16 18:58:46.846027 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:58:46.846020    2581 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b17e8459-9e27-44f6-82a0-f1c8bf30d48b-proxy-tls\") on node \"ip-10-0-138-127.ec2.internal\" DevicePath \"\""
Sep 16 18:58:46.846027 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:58:46.846029    2581 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b17e8459-9e27-44f6-82a0-f1c8bf30d48b-kserve-provision-location\") on node \"ip-10-0-138-127.ec2.internal\" DevicePath \"\""
Sep 16 18:58:46.846027 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:58:46.846038    2581 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/b17e8459-9e27-44f6-82a0-f1c8bf30d48b-kube-rbac-proxy-sar-config\") on node \"ip-10-0-138-127.ec2.internal\" DevicePath \"\""
Sep 16 18:58:46.846325 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:58:46.846047    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j5xnl\" (UniqueName: \"kubernetes.io/projected/b17e8459-9e27-44f6-82a0-f1c8bf30d48b-kube-api-access-j5xnl\") on node \"ip-10-0-138-127.ec2.internal\" DevicePath \"\""
Sep 16 18:58:47.174959 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:58:47.174923    2581 generic.go:356] "Generic (PLEG): container finished" podID="b17e8459-9e27-44f6-82a0-f1c8bf30d48b" containerID="cec2e87528d736155874f241e395ba11290513d7a462cda2bf79df6ab3d6d422" exitCode=0
Sep 16 18:58:47.175250 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:58:47.174975    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-75f85bc9f9-hfctg" event={"ID":"b17e8459-9e27-44f6-82a0-f1c8bf30d48b","Type":"ContainerDied","Data":"cec2e87528d736155874f241e395ba11290513d7a462cda2bf79df6ab3d6d422"}
Sep 16 18:58:47.175250 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:58:47.174993    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-75f85bc9f9-hfctg"
Sep 16 18:58:47.175250 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:58:47.175024    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-75f85bc9f9-hfctg" event={"ID":"b17e8459-9e27-44f6-82a0-f1c8bf30d48b","Type":"ContainerDied","Data":"ed8b4a7ae79798379038d78dc797cd903b448774f82a522e770df21e21c6f3eb"}
Sep 16 18:58:47.175250 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:58:47.175052    2581 scope.go:117] "RemoveContainer" containerID="ae9b007e97b9ece3e72e0c0058e40ddbf0931cdb6b5ccd642a12baa97081a202"
Sep 16 18:58:47.184239 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:58:47.184210    2581 scope.go:117] "RemoveContainer" containerID="cec2e87528d736155874f241e395ba11290513d7a462cda2bf79df6ab3d6d422"
Sep 16 18:58:47.192147 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:58:47.192126    2581 scope.go:117] "RemoveContainer" containerID="4cf5778c6b02e8c6c17f4c0db06f2324a9d897a530774acc4ad6131dc8164e34"
Sep 16 18:58:47.200054 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:58:47.200030    2581 scope.go:117] "RemoveContainer" containerID="ae9b007e97b9ece3e72e0c0058e40ddbf0931cdb6b5ccd642a12baa97081a202"
Sep 16 18:58:47.200513 ip-10-0-138-127 kubenswrapper[2581]: E0916 18:58:47.200493    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ae9b007e97b9ece3e72e0c0058e40ddbf0931cdb6b5ccd642a12baa97081a202\": container with ID starting with ae9b007e97b9ece3e72e0c0058e40ddbf0931cdb6b5ccd642a12baa97081a202 not found: ID does not exist" containerID="ae9b007e97b9ece3e72e0c0058e40ddbf0931cdb6b5ccd642a12baa97081a202"
Sep 16 18:58:47.200590 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:58:47.200521    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ae9b007e97b9ece3e72e0c0058e40ddbf0931cdb6b5ccd642a12baa97081a202"} err="failed to get container status \"ae9b007e97b9ece3e72e0c0058e40ddbf0931cdb6b5ccd642a12baa97081a202\": rpc error: code = NotFound desc = could not find container \"ae9b007e97b9ece3e72e0c0058e40ddbf0931cdb6b5ccd642a12baa97081a202\": container with ID starting with ae9b007e97b9ece3e72e0c0058e40ddbf0931cdb6b5ccd642a12baa97081a202 not found: ID does not exist"
Sep 16 18:58:47.200590 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:58:47.200541    2581 scope.go:117] "RemoveContainer" containerID="cec2e87528d736155874f241e395ba11290513d7a462cda2bf79df6ab3d6d422"
Sep 16 18:58:47.200816 ip-10-0-138-127 kubenswrapper[2581]: E0916 18:58:47.200801    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cec2e87528d736155874f241e395ba11290513d7a462cda2bf79df6ab3d6d422\": container with ID starting with cec2e87528d736155874f241e395ba11290513d7a462cda2bf79df6ab3d6d422 not found: ID does not exist" containerID="cec2e87528d736155874f241e395ba11290513d7a462cda2bf79df6ab3d6d422"
Sep 16 18:58:47.200862 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:58:47.200820    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cec2e87528d736155874f241e395ba11290513d7a462cda2bf79df6ab3d6d422"} err="failed to get container status \"cec2e87528d736155874f241e395ba11290513d7a462cda2bf79df6ab3d6d422\": rpc error: code = NotFound desc = could not find container \"cec2e87528d736155874f241e395ba11290513d7a462cda2bf79df6ab3d6d422\": container with ID starting with cec2e87528d736155874f241e395ba11290513d7a462cda2bf79df6ab3d6d422 not found: ID does not exist"
Sep 16 18:58:47.200862 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:58:47.200833    2581 scope.go:117] "RemoveContainer" containerID="4cf5778c6b02e8c6c17f4c0db06f2324a9d897a530774acc4ad6131dc8164e34"
Sep 16 18:58:47.201120 ip-10-0-138-127 kubenswrapper[2581]: E0916 18:58:47.201104    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4cf5778c6b02e8c6c17f4c0db06f2324a9d897a530774acc4ad6131dc8164e34\": container with ID starting with 4cf5778c6b02e8c6c17f4c0db06f2324a9d897a530774acc4ad6131dc8164e34 not found: ID does not exist" containerID="4cf5778c6b02e8c6c17f4c0db06f2324a9d897a530774acc4ad6131dc8164e34"
Sep 16 18:58:47.201173 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:58:47.201122    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4cf5778c6b02e8c6c17f4c0db06f2324a9d897a530774acc4ad6131dc8164e34"} err="failed to get container status \"4cf5778c6b02e8c6c17f4c0db06f2324a9d897a530774acc4ad6131dc8164e34\": rpc error: code = NotFound desc = could not find container \"4cf5778c6b02e8c6c17f4c0db06f2324a9d897a530774acc4ad6131dc8164e34\": container with ID starting with 4cf5778c6b02e8c6c17f4c0db06f2324a9d897a530774acc4ad6131dc8164e34 not found: ID does not exist"
Sep 16 18:58:47.201960 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:58:47.201940    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-predictor-75f85bc9f9-hfctg"]
Sep 16 18:58:47.208778 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:58:47.208750    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-predictor-75f85bc9f9-hfctg"]
Sep 16 18:58:48.727388 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:58:48.727357    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b17e8459-9e27-44f6-82a0-f1c8bf30d48b" path="/var/lib/kubelet/pods/b17e8459-9e27-44f6-82a0-f1c8bf30d48b/volumes"
Sep 16 18:58:51.496111 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:58:51.496076    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-predictor-75f85bc9f9-mqmdb"]
Sep 16 18:58:51.496526 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:58:51.496442    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b17e8459-9e27-44f6-82a0-f1c8bf30d48b" containerName="kube-rbac-proxy"
Sep 16 18:58:51.496526 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:58:51.496453    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="b17e8459-9e27-44f6-82a0-f1c8bf30d48b" containerName="kube-rbac-proxy"
Sep 16 18:58:51.496526 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:58:51.496474    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b17e8459-9e27-44f6-82a0-f1c8bf30d48b" containerName="storage-initializer"
Sep 16 18:58:51.496526 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:58:51.496480    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="b17e8459-9e27-44f6-82a0-f1c8bf30d48b" containerName="storage-initializer"
Sep 16 18:58:51.496526 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:58:51.496487    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b17e8459-9e27-44f6-82a0-f1c8bf30d48b" containerName="kserve-container"
Sep 16 18:58:51.496526 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:58:51.496494    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="b17e8459-9e27-44f6-82a0-f1c8bf30d48b" containerName="kserve-container"
Sep 16 18:58:51.496739 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:58:51.496539    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="b17e8459-9e27-44f6-82a0-f1c8bf30d48b" containerName="kserve-container"
Sep 16 18:58:51.496739 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:58:51.496547    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="b17e8459-9e27-44f6-82a0-f1c8bf30d48b" containerName="kube-rbac-proxy"
Sep 16 18:58:51.501308 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:58:51.501288    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-75f85bc9f9-mqmdb"
Sep 16 18:58:51.503480 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:58:51.503456    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-rollback-kube-rbac-proxy-sar-config\""
Sep 16 18:58:51.503598 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:58:51.503521    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-rollback-predictor-serving-cert\""
Sep 16 18:58:51.510902 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:58:51.510881    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-predictor-75f85bc9f9-mqmdb"]
Sep 16 18:58:51.583691 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:58:51.583651    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/b1ee0ded-03f5-4504-8a9d-84c6d7d745cb-cabundle-cert\") pod \"isvc-canary-rollback-predictor-75f85bc9f9-mqmdb\" (UID: \"b1ee0ded-03f5-4504-8a9d-84c6d7d745cb\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-75f85bc9f9-mqmdb"
Sep 16 18:58:51.583691 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:58:51.583694    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/b1ee0ded-03f5-4504-8a9d-84c6d7d745cb-kube-rbac-proxy-sar-config\") pod \"isvc-canary-rollback-predictor-75f85bc9f9-mqmdb\" (UID: \"b1ee0ded-03f5-4504-8a9d-84c6d7d745cb\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-75f85bc9f9-mqmdb"
Sep 16 18:58:51.583902 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:58:51.583781    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b1ee0ded-03f5-4504-8a9d-84c6d7d745cb-proxy-tls\") pod \"isvc-canary-rollback-predictor-75f85bc9f9-mqmdb\" (UID: \"b1ee0ded-03f5-4504-8a9d-84c6d7d745cb\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-75f85bc9f9-mqmdb"
Sep 16 18:58:51.583902 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:58:51.583811    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xpj8l\" (UniqueName: \"kubernetes.io/projected/b1ee0ded-03f5-4504-8a9d-84c6d7d745cb-kube-api-access-xpj8l\") pod \"isvc-canary-rollback-predictor-75f85bc9f9-mqmdb\" (UID: \"b1ee0ded-03f5-4504-8a9d-84c6d7d745cb\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-75f85bc9f9-mqmdb"
Sep 16 18:58:51.583902 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:58:51.583887    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b1ee0ded-03f5-4504-8a9d-84c6d7d745cb-kserve-provision-location\") pod \"isvc-canary-rollback-predictor-75f85bc9f9-mqmdb\" (UID: \"b1ee0ded-03f5-4504-8a9d-84c6d7d745cb\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-75f85bc9f9-mqmdb"
Sep 16 18:58:51.685149 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:58:51.685097    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/b1ee0ded-03f5-4504-8a9d-84c6d7d745cb-cabundle-cert\") pod \"isvc-canary-rollback-predictor-75f85bc9f9-mqmdb\" (UID: \"b1ee0ded-03f5-4504-8a9d-84c6d7d745cb\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-75f85bc9f9-mqmdb"
Sep 16 18:58:51.685149 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:58:51.685152    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/b1ee0ded-03f5-4504-8a9d-84c6d7d745cb-kube-rbac-proxy-sar-config\") pod \"isvc-canary-rollback-predictor-75f85bc9f9-mqmdb\" (UID: \"b1ee0ded-03f5-4504-8a9d-84c6d7d745cb\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-75f85bc9f9-mqmdb"
Sep 16 18:58:51.685414 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:58:51.685188    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b1ee0ded-03f5-4504-8a9d-84c6d7d745cb-proxy-tls\") pod \"isvc-canary-rollback-predictor-75f85bc9f9-mqmdb\" (UID: \"b1ee0ded-03f5-4504-8a9d-84c6d7d745cb\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-75f85bc9f9-mqmdb"
Sep 16 18:58:51.685414 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:58:51.685235    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xpj8l\" (UniqueName: \"kubernetes.io/projected/b1ee0ded-03f5-4504-8a9d-84c6d7d745cb-kube-api-access-xpj8l\") pod \"isvc-canary-rollback-predictor-75f85bc9f9-mqmdb\" (UID: \"b1ee0ded-03f5-4504-8a9d-84c6d7d745cb\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-75f85bc9f9-mqmdb"
Sep 16 18:58:51.685414 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:58:51.685280    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b1ee0ded-03f5-4504-8a9d-84c6d7d745cb-kserve-provision-location\") pod \"isvc-canary-rollback-predictor-75f85bc9f9-mqmdb\" (UID: \"b1ee0ded-03f5-4504-8a9d-84c6d7d745cb\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-75f85bc9f9-mqmdb"
Sep 16 18:58:51.685660 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:58:51.685642    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b1ee0ded-03f5-4504-8a9d-84c6d7d745cb-kserve-provision-location\") pod \"isvc-canary-rollback-predictor-75f85bc9f9-mqmdb\" (UID: \"b1ee0ded-03f5-4504-8a9d-84c6d7d745cb\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-75f85bc9f9-mqmdb"
Sep 16 18:58:51.685901 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:58:51.685882    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/b1ee0ded-03f5-4504-8a9d-84c6d7d745cb-kube-rbac-proxy-sar-config\") pod \"isvc-canary-rollback-predictor-75f85bc9f9-mqmdb\" (UID: \"b1ee0ded-03f5-4504-8a9d-84c6d7d745cb\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-75f85bc9f9-mqmdb"
Sep 16 18:58:51.685953 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:58:51.685908    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/b1ee0ded-03f5-4504-8a9d-84c6d7d745cb-cabundle-cert\") pod \"isvc-canary-rollback-predictor-75f85bc9f9-mqmdb\" (UID: \"b1ee0ded-03f5-4504-8a9d-84c6d7d745cb\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-75f85bc9f9-mqmdb"
Sep 16 18:58:51.687884 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:58:51.687868    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b1ee0ded-03f5-4504-8a9d-84c6d7d745cb-proxy-tls\") pod \"isvc-canary-rollback-predictor-75f85bc9f9-mqmdb\" (UID: \"b1ee0ded-03f5-4504-8a9d-84c6d7d745cb\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-75f85bc9f9-mqmdb"
Sep 16 18:58:51.699582 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:58:51.699517    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xpj8l\" (UniqueName: \"kubernetes.io/projected/b1ee0ded-03f5-4504-8a9d-84c6d7d745cb-kube-api-access-xpj8l\") pod \"isvc-canary-rollback-predictor-75f85bc9f9-mqmdb\" (UID: \"b1ee0ded-03f5-4504-8a9d-84c6d7d745cb\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-75f85bc9f9-mqmdb"
Sep 16 18:58:51.811799 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:58:51.811710    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-75f85bc9f9-mqmdb"
Sep 16 18:58:52.090680 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:58:52.090657    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-predictor-75f85bc9f9-mqmdb"]
Sep 16 18:58:52.092962 ip-10-0-138-127 kubenswrapper[2581]: W0916 18:58:52.092919    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb1ee0ded_03f5_4504_8a9d_84c6d7d745cb.slice/crio-f7445801a433a63ed1445336f4b3c714e6e5dee770db9619d90bb62a567642c8 WatchSource:0}: Error finding container f7445801a433a63ed1445336f4b3c714e6e5dee770db9619d90bb62a567642c8: Status 404 returned error can't find the container with id f7445801a433a63ed1445336f4b3c714e6e5dee770db9619d90bb62a567642c8
Sep 16 18:58:52.198156 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:58:52.198122    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-75f85bc9f9-mqmdb" event={"ID":"b1ee0ded-03f5-4504-8a9d-84c6d7d745cb","Type":"ContainerStarted","Data":"233b873351a33cbe599372b8a2770642a839e9c5e63dc74b371fdbdf699a3a6b"}
Sep 16 18:58:52.198332 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:58:52.198177    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-75f85bc9f9-mqmdb" event={"ID":"b1ee0ded-03f5-4504-8a9d-84c6d7d745cb","Type":"ContainerStarted","Data":"f7445801a433a63ed1445336f4b3c714e6e5dee770db9619d90bb62a567642c8"}
Sep 16 18:58:56.215438 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:58:56.215400    2581 generic.go:356] "Generic (PLEG): container finished" podID="b1ee0ded-03f5-4504-8a9d-84c6d7d745cb" containerID="233b873351a33cbe599372b8a2770642a839e9c5e63dc74b371fdbdf699a3a6b" exitCode=0
Sep 16 18:58:56.215839 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:58:56.215475    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-75f85bc9f9-mqmdb" event={"ID":"b1ee0ded-03f5-4504-8a9d-84c6d7d745cb","Type":"ContainerDied","Data":"233b873351a33cbe599372b8a2770642a839e9c5e63dc74b371fdbdf699a3a6b"}
Sep 16 18:58:56.459659 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:58:56.459566    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-raw-sklearn-batcher-b88cf-predictor-6f597c578b-zzd8b"]
Sep 16 18:58:56.460301 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:58:56.459861    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-b88cf-predictor-6f597c578b-zzd8b" podUID="a81cbc02-9a43-40df-9a7a-fb53b8d0ed74" containerName="kserve-container" containerID="cri-o://0cbb2c96007f87d82f48a3d56a1fd2e9b9b1cbf4b67e23dc86d56ad304ee2d0e" gracePeriod=30
Sep 16 18:58:56.460301 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:58:56.459902    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-b88cf-predictor-6f597c578b-zzd8b" podUID="a81cbc02-9a43-40df-9a7a-fb53b8d0ed74" containerName="agent" containerID="cri-o://d63b026583466c72366b547f05d90b8073c81b492695d00130d20a364c208055" gracePeriod=30
Sep 16 18:58:56.460301 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:58:56.460064    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-b88cf-predictor-6f597c578b-zzd8b" podUID="a81cbc02-9a43-40df-9a7a-fb53b8d0ed74" containerName="kube-rbac-proxy" containerID="cri-o://c9b55a427c44320832682e4a60470866aeab296465631b4c6ca4c10caa2f23d4" gracePeriod=30
Sep 16 18:58:56.803372 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:58:56.803337    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-graph-raw-1ba12-predictor-74579d997-hmg4m"]
Sep 16 18:58:56.806861 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:58:56.806845    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-1ba12-predictor-74579d997-hmg4m"
Sep 16 18:58:56.811468 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:58:56.811446    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-xgboost-graph-raw-1ba12-predictor-serving-cert\""
Sep 16 18:58:56.811733 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:58:56.811717    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-xgboost-graph-raw-1ba12-kube-rbac-proxy-sar-config\""
Sep 16 18:58:56.825698 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:58:56.825668    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-graph-raw-1ba12-predictor-74579d997-hmg4m"]
Sep 16 18:58:56.832869 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:58:56.832841    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/1035c64c-18a1-43f0-9e88-c06f78ad4591-cabundle-cert\") pod \"isvc-xgboost-graph-raw-1ba12-predictor-74579d997-hmg4m\" (UID: \"1035c64c-18a1-43f0-9e88-c06f78ad4591\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-1ba12-predictor-74579d997-hmg4m"
Sep 16 18:58:56.832999 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:58:56.832879    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/1035c64c-18a1-43f0-9e88-c06f78ad4591-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-graph-raw-1ba12-predictor-74579d997-hmg4m\" (UID: \"1035c64c-18a1-43f0-9e88-c06f78ad4591\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-1ba12-predictor-74579d997-hmg4m"
Sep 16 18:58:56.833218 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:58:56.833177    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/1035c64c-18a1-43f0-9e88-c06f78ad4591-proxy-tls\") pod \"isvc-xgboost-graph-raw-1ba12-predictor-74579d997-hmg4m\" (UID: \"1035c64c-18a1-43f0-9e88-c06f78ad4591\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-1ba12-predictor-74579d997-hmg4m"
Sep 16 18:58:56.833334 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:58:56.833309    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/1035c64c-18a1-43f0-9e88-c06f78ad4591-kserve-provision-location\") pod \"isvc-xgboost-graph-raw-1ba12-predictor-74579d997-hmg4m\" (UID: \"1035c64c-18a1-43f0-9e88-c06f78ad4591\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-1ba12-predictor-74579d997-hmg4m"
Sep 16 18:58:56.833457 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:58:56.833434    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bjmkd\" (UniqueName: \"kubernetes.io/projected/1035c64c-18a1-43f0-9e88-c06f78ad4591-kube-api-access-bjmkd\") pod \"isvc-xgboost-graph-raw-1ba12-predictor-74579d997-hmg4m\" (UID: \"1035c64c-18a1-43f0-9e88-c06f78ad4591\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-1ba12-predictor-74579d997-hmg4m"
Sep 16 18:58:56.934414 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:58:56.934367    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/1035c64c-18a1-43f0-9e88-c06f78ad4591-kserve-provision-location\") pod \"isvc-xgboost-graph-raw-1ba12-predictor-74579d997-hmg4m\" (UID: \"1035c64c-18a1-43f0-9e88-c06f78ad4591\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-1ba12-predictor-74579d997-hmg4m"
Sep 16 18:58:56.934414 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:58:56.934416    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bjmkd\" (UniqueName: \"kubernetes.io/projected/1035c64c-18a1-43f0-9e88-c06f78ad4591-kube-api-access-bjmkd\") pod \"isvc-xgboost-graph-raw-1ba12-predictor-74579d997-hmg4m\" (UID: \"1035c64c-18a1-43f0-9e88-c06f78ad4591\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-1ba12-predictor-74579d997-hmg4m"
Sep 16 18:58:56.934674 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:58:56.934489    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/1035c64c-18a1-43f0-9e88-c06f78ad4591-cabundle-cert\") pod \"isvc-xgboost-graph-raw-1ba12-predictor-74579d997-hmg4m\" (UID: \"1035c64c-18a1-43f0-9e88-c06f78ad4591\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-1ba12-predictor-74579d997-hmg4m"
Sep 16 18:58:56.934674 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:58:56.934515    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/1035c64c-18a1-43f0-9e88-c06f78ad4591-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-graph-raw-1ba12-predictor-74579d997-hmg4m\" (UID: \"1035c64c-18a1-43f0-9e88-c06f78ad4591\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-1ba12-predictor-74579d997-hmg4m"
Sep 16 18:58:56.934674 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:58:56.934549    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/1035c64c-18a1-43f0-9e88-c06f78ad4591-proxy-tls\") pod \"isvc-xgboost-graph-raw-1ba12-predictor-74579d997-hmg4m\" (UID: \"1035c64c-18a1-43f0-9e88-c06f78ad4591\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-1ba12-predictor-74579d997-hmg4m"
Sep 16 18:58:56.934674 ip-10-0-138-127 kubenswrapper[2581]: E0916 18:58:56.934672    2581 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-xgboost-graph-raw-1ba12-predictor-serving-cert: secret "isvc-xgboost-graph-raw-1ba12-predictor-serving-cert" not found
Sep 16 18:58:56.934870 ip-10-0-138-127 kubenswrapper[2581]: E0916 18:58:56.934743    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1035c64c-18a1-43f0-9e88-c06f78ad4591-proxy-tls podName:1035c64c-18a1-43f0-9e88-c06f78ad4591 nodeName:}" failed. No retries permitted until 2026-09-16 18:58:57.434722523 +0000 UTC m=+807.254136426 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/1035c64c-18a1-43f0-9e88-c06f78ad4591-proxy-tls") pod "isvc-xgboost-graph-raw-1ba12-predictor-74579d997-hmg4m" (UID: "1035c64c-18a1-43f0-9e88-c06f78ad4591") : secret "isvc-xgboost-graph-raw-1ba12-predictor-serving-cert" not found
Sep 16 18:58:56.934870 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:58:56.934819    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/1035c64c-18a1-43f0-9e88-c06f78ad4591-kserve-provision-location\") pod \"isvc-xgboost-graph-raw-1ba12-predictor-74579d997-hmg4m\" (UID: \"1035c64c-18a1-43f0-9e88-c06f78ad4591\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-1ba12-predictor-74579d997-hmg4m"
Sep 16 18:58:56.935110 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:58:56.935090    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/1035c64c-18a1-43f0-9e88-c06f78ad4591-cabundle-cert\") pod \"isvc-xgboost-graph-raw-1ba12-predictor-74579d997-hmg4m\" (UID: \"1035c64c-18a1-43f0-9e88-c06f78ad4591\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-1ba12-predictor-74579d997-hmg4m"
Sep 16 18:58:56.935145 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:58:56.935104    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/1035c64c-18a1-43f0-9e88-c06f78ad4591-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-graph-raw-1ba12-predictor-74579d997-hmg4m\" (UID: \"1035c64c-18a1-43f0-9e88-c06f78ad4591\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-1ba12-predictor-74579d997-hmg4m"
Sep 16 18:58:56.947108 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:58:56.947073    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bjmkd\" (UniqueName: \"kubernetes.io/projected/1035c64c-18a1-43f0-9e88-c06f78ad4591-kube-api-access-bjmkd\") pod \"isvc-xgboost-graph-raw-1ba12-predictor-74579d997-hmg4m\" (UID: \"1035c64c-18a1-43f0-9e88-c06f78ad4591\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-1ba12-predictor-74579d997-hmg4m"
Sep 16 18:58:57.222269 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:58:57.222157    2581 generic.go:356] "Generic (PLEG): container finished" podID="a81cbc02-9a43-40df-9a7a-fb53b8d0ed74" containerID="c9b55a427c44320832682e4a60470866aeab296465631b4c6ca4c10caa2f23d4" exitCode=2
Sep 16 18:58:57.222269 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:58:57.222234    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-b88cf-predictor-6f597c578b-zzd8b" event={"ID":"a81cbc02-9a43-40df-9a7a-fb53b8d0ed74","Type":"ContainerDied","Data":"c9b55a427c44320832682e4a60470866aeab296465631b4c6ca4c10caa2f23d4"}
Sep 16 18:58:57.224505 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:58:57.224480    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-75f85bc9f9-mqmdb" event={"ID":"b1ee0ded-03f5-4504-8a9d-84c6d7d745cb","Type":"ContainerStarted","Data":"5a4f4c2f9817feee14e00a700b2b97cfaab54c9dfa8898b0e52e6e340f33e344"}
Sep 16 18:58:57.224609 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:58:57.224532    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-75f85bc9f9-mqmdb" event={"ID":"b1ee0ded-03f5-4504-8a9d-84c6d7d745cb","Type":"ContainerStarted","Data":"1cd42316a3c1d724bc4a802e13f0f14f26366c2a91269eb8aacc5a23d534a29a"}
Sep 16 18:58:57.224779 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:58:57.224766    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-75f85bc9f9-mqmdb"
Sep 16 18:58:57.250572 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:58:57.250521    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-75f85bc9f9-mqmdb" podStartSLOduration=2.285507093 podStartE2EDuration="6.250506771s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.964999678s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:58:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:58:57.249656886 +0000 UTC m=+807.069070796" watchObservedRunningTime="2026-09-16 18:58:57.250506771 +0000 UTC m=+807.069920681"
Sep 16 18:58:57.439714 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:58:57.439680    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/1035c64c-18a1-43f0-9e88-c06f78ad4591-proxy-tls\") pod \"isvc-xgboost-graph-raw-1ba12-predictor-74579d997-hmg4m\" (UID: \"1035c64c-18a1-43f0-9e88-c06f78ad4591\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-1ba12-predictor-74579d997-hmg4m"
Sep 16 18:58:57.442253 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:58:57.442232    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/1035c64c-18a1-43f0-9e88-c06f78ad4591-proxy-tls\") pod \"isvc-xgboost-graph-raw-1ba12-predictor-74579d997-hmg4m\" (UID: \"1035c64c-18a1-43f0-9e88-c06f78ad4591\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-1ba12-predictor-74579d997-hmg4m"
Sep 16 18:58:57.717582 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:58:57.717535    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-1ba12-predictor-74579d997-hmg4m"
Sep 16 18:58:57.994005 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:58:57.993978    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-graph-raw-1ba12-predictor-74579d997-hmg4m"]
Sep 16 18:58:57.996068 ip-10-0-138-127 kubenswrapper[2581]: W0916 18:58:57.996039    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1035c64c_18a1_43f0_9e88_c06f78ad4591.slice/crio-d084f58fa9eb5ab06659dd759d3caf7fa1a427eb3abdf05f88dcfc2d2cfb66eb WatchSource:0}: Error finding container d084f58fa9eb5ab06659dd759d3caf7fa1a427eb3abdf05f88dcfc2d2cfb66eb: Status 404 returned error can't find the container with id d084f58fa9eb5ab06659dd759d3caf7fa1a427eb3abdf05f88dcfc2d2cfb66eb
Sep 16 18:58:58.231450 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:58:58.231413    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-1ba12-predictor-74579d997-hmg4m" event={"ID":"1035c64c-18a1-43f0-9e88-c06f78ad4591","Type":"ContainerStarted","Data":"1f7c53029145d867d0aacedf653ed854345e42c2e325ffc785c4d62d1c512998"}
Sep 16 18:58:58.231884 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:58:58.231485    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-1ba12-predictor-74579d997-hmg4m" event={"ID":"1035c64c-18a1-43f0-9e88-c06f78ad4591","Type":"ContainerStarted","Data":"d084f58fa9eb5ab06659dd759d3caf7fa1a427eb3abdf05f88dcfc2d2cfb66eb"}
Sep 16 18:58:58.231884 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:58:58.231663    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-75f85bc9f9-mqmdb"
Sep 16 18:58:58.232655 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:58:58.232632    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-75f85bc9f9-mqmdb" podUID="b1ee0ded-03f5-4504-8a9d-84c6d7d745cb" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.27:8080: connect: connection refused"
Sep 16 18:58:59.239678 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:58:59.239640    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-75f85bc9f9-mqmdb" podUID="b1ee0ded-03f5-4504-8a9d-84c6d7d745cb" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.27:8080: connect: connection refused"
Sep 16 18:58:59.836722 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:58:59.836670    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-b88cf-predictor-6f597c578b-zzd8b" podUID="a81cbc02-9a43-40df-9a7a-fb53b8d0ed74" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.25:8643/healthz\": dial tcp 10.132.0.25:8643: connect: connection refused"
Sep 16 18:59:02.255017 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:59:02.254981    2581 generic.go:356] "Generic (PLEG): container finished" podID="a81cbc02-9a43-40df-9a7a-fb53b8d0ed74" containerID="0cbb2c96007f87d82f48a3d56a1fd2e9b9b1cbf4b67e23dc86d56ad304ee2d0e" exitCode=0
Sep 16 18:59:02.255445 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:59:02.255052    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-b88cf-predictor-6f597c578b-zzd8b" event={"ID":"a81cbc02-9a43-40df-9a7a-fb53b8d0ed74","Type":"ContainerDied","Data":"0cbb2c96007f87d82f48a3d56a1fd2e9b9b1cbf4b67e23dc86d56ad304ee2d0e"}
Sep 16 18:59:02.256837 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:59:02.256813    2581 generic.go:356] "Generic (PLEG): container finished" podID="1035c64c-18a1-43f0-9e88-c06f78ad4591" containerID="1f7c53029145d867d0aacedf653ed854345e42c2e325ffc785c4d62d1c512998" exitCode=0
Sep 16 18:59:02.256961 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:59:02.256872    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-1ba12-predictor-74579d997-hmg4m" event={"ID":"1035c64c-18a1-43f0-9e88-c06f78ad4591","Type":"ContainerDied","Data":"1f7c53029145d867d0aacedf653ed854345e42c2e325ffc785c4d62d1c512998"}
Sep 16 18:59:04.245924 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:59:04.245889    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-75f85bc9f9-mqmdb"
Sep 16 18:59:04.246637 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:59:04.246604    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-75f85bc9f9-mqmdb" podUID="b1ee0ded-03f5-4504-8a9d-84c6d7d745cb" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.27:8080: connect: connection refused"
Sep 16 18:59:04.836758 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:59:04.836703    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-b88cf-predictor-6f597c578b-zzd8b" podUID="a81cbc02-9a43-40df-9a7a-fb53b8d0ed74" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.25:8643/healthz\": dial tcp 10.132.0.25:8643: connect: connection refused"
Sep 16 18:59:04.842046 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:59:04.842006    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-b88cf-predictor-6f597c578b-zzd8b" podUID="a81cbc02-9a43-40df-9a7a-fb53b8d0ed74" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.25:8080: connect: connection refused"
Sep 16 18:59:04.842513 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:59:04.842484    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-b88cf-predictor-6f597c578b-zzd8b" podUID="a81cbc02-9a43-40df-9a7a-fb53b8d0ed74" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 16 18:59:09.836994 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:59:09.836946    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-b88cf-predictor-6f597c578b-zzd8b" podUID="a81cbc02-9a43-40df-9a7a-fb53b8d0ed74" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.25:8643/healthz\": dial tcp 10.132.0.25:8643: connect: connection refused"
Sep 16 18:59:09.837512 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:59:09.837118    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-b88cf-predictor-6f597c578b-zzd8b"
Sep 16 18:59:14.246510 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:59:14.246464    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-75f85bc9f9-mqmdb" podUID="b1ee0ded-03f5-4504-8a9d-84c6d7d745cb" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.27:8080: connect: connection refused"
Sep 16 18:59:14.836645 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:59:14.836595    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-b88cf-predictor-6f597c578b-zzd8b" podUID="a81cbc02-9a43-40df-9a7a-fb53b8d0ed74" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.25:8643/healthz\": dial tcp 10.132.0.25:8643: connect: connection refused"
Sep 16 18:59:14.842541 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:59:14.842496    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-b88cf-predictor-6f597c578b-zzd8b" podUID="a81cbc02-9a43-40df-9a7a-fb53b8d0ed74" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.25:8080: connect: connection refused"
Sep 16 18:59:14.842956 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:59:14.842907    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-b88cf-predictor-6f597c578b-zzd8b" podUID="a81cbc02-9a43-40df-9a7a-fb53b8d0ed74" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 16 18:59:19.837046 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:59:19.837004    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-b88cf-predictor-6f597c578b-zzd8b" podUID="a81cbc02-9a43-40df-9a7a-fb53b8d0ed74" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.25:8643/healthz\": dial tcp 10.132.0.25:8643: connect: connection refused"
Sep 16 18:59:20.352647 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:59:20.352608    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-1ba12-predictor-74579d997-hmg4m" event={"ID":"1035c64c-18a1-43f0-9e88-c06f78ad4591","Type":"ContainerStarted","Data":"471ef77688652749185a328029f5dc37ff482014e210663fcca15481687a71f4"}
Sep 16 18:59:20.352827 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:59:20.352725    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-1ba12-predictor-74579d997-hmg4m" event={"ID":"1035c64c-18a1-43f0-9e88-c06f78ad4591","Type":"ContainerStarted","Data":"a51ba3abb020aba40b477770c8e51ede18f48bac355266a9a0fb2d774d4b0f5b"}
Sep 16 18:59:20.352971 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:59:20.352952    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-1ba12-predictor-74579d997-hmg4m"
Sep 16 18:59:20.377421 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:59:20.377369    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-1ba12-predictor-74579d997-hmg4m" podStartSLOduration=2.952246997 podStartE2EDuration="24.3773518s" totalImagesPullingTime="17.426176316s" totalInitContainerRuntime="3.998928487s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:58:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:59:20.376218068 +0000 UTC m=+830.195631972" watchObservedRunningTime="2026-09-16 18:59:20.3773518 +0000 UTC m=+830.196765709"
Sep 16 18:59:21.357970 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:59:21.357935    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-1ba12-predictor-74579d997-hmg4m"
Sep 16 18:59:21.359563 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:59:21.359533    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-1ba12-predictor-74579d997-hmg4m" podUID="1035c64c-18a1-43f0-9e88-c06f78ad4591" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.28:8080: connect: connection refused"
Sep 16 18:59:22.362269 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:59:22.362235    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-1ba12-predictor-74579d997-hmg4m" podUID="1035c64c-18a1-43f0-9e88-c06f78ad4591" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.28:8080: connect: connection refused"
Sep 16 18:59:24.246718 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:59:24.246681    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-75f85bc9f9-mqmdb" podUID="b1ee0ded-03f5-4504-8a9d-84c6d7d745cb" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.27:8080: connect: connection refused"
Sep 16 18:59:24.837166 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:59:24.837121    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-b88cf-predictor-6f597c578b-zzd8b" podUID="a81cbc02-9a43-40df-9a7a-fb53b8d0ed74" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.25:8643/healthz\": dial tcp 10.132.0.25:8643: connect: connection refused"
Sep 16 18:59:24.841561 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:59:24.841514    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-b88cf-predictor-6f597c578b-zzd8b" podUID="a81cbc02-9a43-40df-9a7a-fb53b8d0ed74" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.25:8080: connect: connection refused"
Sep 16 18:59:24.841706 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:59:24.841691    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-b88cf-predictor-6f597c578b-zzd8b"
Sep 16 18:59:24.841961 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:59:24.841931    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-b88cf-predictor-6f597c578b-zzd8b" podUID="a81cbc02-9a43-40df-9a7a-fb53b8d0ed74" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 16 18:59:24.842072 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:59:24.842057    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-b88cf-predictor-6f597c578b-zzd8b"
Sep 16 18:59:26.604963 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:59:26.604937    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-b88cf-predictor-6f597c578b-zzd8b"
Sep 16 18:59:26.718060 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:59:26.717964    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a81cbc02-9a43-40df-9a7a-fb53b8d0ed74-proxy-tls\") pod \"a81cbc02-9a43-40df-9a7a-fb53b8d0ed74\" (UID: \"a81cbc02-9a43-40df-9a7a-fb53b8d0ed74\") "
Sep 16 18:59:26.718060 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:59:26.718001    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a81cbc02-9a43-40df-9a7a-fb53b8d0ed74-kube-rbac-proxy-sar-config\") pod \"a81cbc02-9a43-40df-9a7a-fb53b8d0ed74\" (UID: \"a81cbc02-9a43-40df-9a7a-fb53b8d0ed74\") "
Sep 16 18:59:26.718060 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:59:26.718036    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a81cbc02-9a43-40df-9a7a-fb53b8d0ed74-kserve-provision-location\") pod \"a81cbc02-9a43-40df-9a7a-fb53b8d0ed74\" (UID: \"a81cbc02-9a43-40df-9a7a-fb53b8d0ed74\") "
Sep 16 18:59:26.718383 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:59:26.718083    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/a81cbc02-9a43-40df-9a7a-fb53b8d0ed74-cabundle-cert\") pod \"a81cbc02-9a43-40df-9a7a-fb53b8d0ed74\" (UID: \"a81cbc02-9a43-40df-9a7a-fb53b8d0ed74\") "
Sep 16 18:59:26.718383 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:59:26.718105    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-lkbrm\" (UniqueName: \"kubernetes.io/projected/a81cbc02-9a43-40df-9a7a-fb53b8d0ed74-kube-api-access-lkbrm\") pod \"a81cbc02-9a43-40df-9a7a-fb53b8d0ed74\" (UID: \"a81cbc02-9a43-40df-9a7a-fb53b8d0ed74\") "
Sep 16 18:59:26.718497 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:59:26.718416    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a81cbc02-9a43-40df-9a7a-fb53b8d0ed74-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "a81cbc02-9a43-40df-9a7a-fb53b8d0ed74" (UID: "a81cbc02-9a43-40df-9a7a-fb53b8d0ed74"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 18:59:26.718497 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:59:26.718430    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a81cbc02-9a43-40df-9a7a-fb53b8d0ed74-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "a81cbc02-9a43-40df-9a7a-fb53b8d0ed74" (UID: "a81cbc02-9a43-40df-9a7a-fb53b8d0ed74"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 18:59:26.718568 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:59:26.718494    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a81cbc02-9a43-40df-9a7a-fb53b8d0ed74-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "a81cbc02-9a43-40df-9a7a-fb53b8d0ed74" (UID: "a81cbc02-9a43-40df-9a7a-fb53b8d0ed74"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 18:59:26.720495 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:59:26.720473    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a81cbc02-9a43-40df-9a7a-fb53b8d0ed74-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "a81cbc02-9a43-40df-9a7a-fb53b8d0ed74" (UID: "a81cbc02-9a43-40df-9a7a-fb53b8d0ed74"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 18:59:26.720613 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:59:26.720496    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a81cbc02-9a43-40df-9a7a-fb53b8d0ed74-kube-api-access-lkbrm" (OuterVolumeSpecName: "kube-api-access-lkbrm") pod "a81cbc02-9a43-40df-9a7a-fb53b8d0ed74" (UID: "a81cbc02-9a43-40df-9a7a-fb53b8d0ed74"). InnerVolumeSpecName "kube-api-access-lkbrm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 18:59:26.819507 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:59:26.819466    2581 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a81cbc02-9a43-40df-9a7a-fb53b8d0ed74-proxy-tls\") on node \"ip-10-0-138-127.ec2.internal\" DevicePath \"\""
Sep 16 18:59:26.819507 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:59:26.819500    2581 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a81cbc02-9a43-40df-9a7a-fb53b8d0ed74-kube-rbac-proxy-sar-config\") on node \"ip-10-0-138-127.ec2.internal\" DevicePath \"\""
Sep 16 18:59:26.819507 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:59:26.819515    2581 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a81cbc02-9a43-40df-9a7a-fb53b8d0ed74-kserve-provision-location\") on node \"ip-10-0-138-127.ec2.internal\" DevicePath \"\""
Sep 16 18:59:26.819790 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:59:26.819524    2581 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/a81cbc02-9a43-40df-9a7a-fb53b8d0ed74-cabundle-cert\") on node \"ip-10-0-138-127.ec2.internal\" DevicePath \"\""
Sep 16 18:59:26.819790 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:59:26.819533    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lkbrm\" (UniqueName: \"kubernetes.io/projected/a81cbc02-9a43-40df-9a7a-fb53b8d0ed74-kube-api-access-lkbrm\") on node \"ip-10-0-138-127.ec2.internal\" DevicePath \"\""
Sep 16 18:59:27.366241 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:59:27.366212    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-1ba12-predictor-74579d997-hmg4m"
Sep 16 18:59:27.366727 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:59:27.366702    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-1ba12-predictor-74579d997-hmg4m" podUID="1035c64c-18a1-43f0-9e88-c06f78ad4591" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.28:8080: connect: connection refused"
Sep 16 18:59:27.387915 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:59:27.387877    2581 generic.go:356] "Generic (PLEG): container finished" podID="a81cbc02-9a43-40df-9a7a-fb53b8d0ed74" containerID="d63b026583466c72366b547f05d90b8073c81b492695d00130d20a364c208055" exitCode=0
Sep 16 18:59:27.388073 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:59:27.387931    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-b88cf-predictor-6f597c578b-zzd8b"
Sep 16 18:59:27.388073 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:59:27.387970    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-b88cf-predictor-6f597c578b-zzd8b" event={"ID":"a81cbc02-9a43-40df-9a7a-fb53b8d0ed74","Type":"ContainerDied","Data":"d63b026583466c72366b547f05d90b8073c81b492695d00130d20a364c208055"}
Sep 16 18:59:27.388073 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:59:27.388039    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-b88cf-predictor-6f597c578b-zzd8b" event={"ID":"a81cbc02-9a43-40df-9a7a-fb53b8d0ed74","Type":"ContainerDied","Data":"597abb2513796a0d7accded6d7298bd2d585da8763029539a1484fac77c337ea"}
Sep 16 18:59:27.388220 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:59:27.388085    2581 scope.go:117] "RemoveContainer" containerID="d63b026583466c72366b547f05d90b8073c81b492695d00130d20a364c208055"
Sep 16 18:59:27.396494 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:59:27.396475    2581 scope.go:117] "RemoveContainer" containerID="c9b55a427c44320832682e4a60470866aeab296465631b4c6ca4c10caa2f23d4"
Sep 16 18:59:27.404618 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:59:27.404594    2581 scope.go:117] "RemoveContainer" containerID="0cbb2c96007f87d82f48a3d56a1fd2e9b9b1cbf4b67e23dc86d56ad304ee2d0e"
Sep 16 18:59:27.405210 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:59:27.405164    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-raw-sklearn-batcher-b88cf-predictor-6f597c578b-zzd8b"]
Sep 16 18:59:27.409081 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:59:27.409049    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-raw-sklearn-batcher-b88cf-predictor-6f597c578b-zzd8b"]
Sep 16 18:59:27.412554 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:59:27.412534    2581 scope.go:117] "RemoveContainer" containerID="d35aa5bcc401a5f827aacd509b60ae310fc989951e6dc303ed094f214c4ccb62"
Sep 16 18:59:27.420220 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:59:27.420179    2581 scope.go:117] "RemoveContainer" containerID="d63b026583466c72366b547f05d90b8073c81b492695d00130d20a364c208055"
Sep 16 18:59:27.420513 ip-10-0-138-127 kubenswrapper[2581]: E0916 18:59:27.420495    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d63b026583466c72366b547f05d90b8073c81b492695d00130d20a364c208055\": container with ID starting with d63b026583466c72366b547f05d90b8073c81b492695d00130d20a364c208055 not found: ID does not exist" containerID="d63b026583466c72366b547f05d90b8073c81b492695d00130d20a364c208055"
Sep 16 18:59:27.420567 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:59:27.420523    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d63b026583466c72366b547f05d90b8073c81b492695d00130d20a364c208055"} err="failed to get container status \"d63b026583466c72366b547f05d90b8073c81b492695d00130d20a364c208055\": rpc error: code = NotFound desc = could not find container \"d63b026583466c72366b547f05d90b8073c81b492695d00130d20a364c208055\": container with ID starting with d63b026583466c72366b547f05d90b8073c81b492695d00130d20a364c208055 not found: ID does not exist"
Sep 16 18:59:27.420567 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:59:27.420542    2581 scope.go:117] "RemoveContainer" containerID="c9b55a427c44320832682e4a60470866aeab296465631b4c6ca4c10caa2f23d4"
Sep 16 18:59:27.420819 ip-10-0-138-127 kubenswrapper[2581]: E0916 18:59:27.420802    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c9b55a427c44320832682e4a60470866aeab296465631b4c6ca4c10caa2f23d4\": container with ID starting with c9b55a427c44320832682e4a60470866aeab296465631b4c6ca4c10caa2f23d4 not found: ID does not exist" containerID="c9b55a427c44320832682e4a60470866aeab296465631b4c6ca4c10caa2f23d4"
Sep 16 18:59:27.420857 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:59:27.420829    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c9b55a427c44320832682e4a60470866aeab296465631b4c6ca4c10caa2f23d4"} err="failed to get container status \"c9b55a427c44320832682e4a60470866aeab296465631b4c6ca4c10caa2f23d4\": rpc error: code = NotFound desc = could not find container \"c9b55a427c44320832682e4a60470866aeab296465631b4c6ca4c10caa2f23d4\": container with ID starting with c9b55a427c44320832682e4a60470866aeab296465631b4c6ca4c10caa2f23d4 not found: ID does not exist"
Sep 16 18:59:27.420857 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:59:27.420845    2581 scope.go:117] "RemoveContainer" containerID="0cbb2c96007f87d82f48a3d56a1fd2e9b9b1cbf4b67e23dc86d56ad304ee2d0e"
Sep 16 18:59:27.421093 ip-10-0-138-127 kubenswrapper[2581]: E0916 18:59:27.421074    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0cbb2c96007f87d82f48a3d56a1fd2e9b9b1cbf4b67e23dc86d56ad304ee2d0e\": container with ID starting with 0cbb2c96007f87d82f48a3d56a1fd2e9b9b1cbf4b67e23dc86d56ad304ee2d0e not found: ID does not exist" containerID="0cbb2c96007f87d82f48a3d56a1fd2e9b9b1cbf4b67e23dc86d56ad304ee2d0e"
Sep 16 18:59:27.421136 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:59:27.421098    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0cbb2c96007f87d82f48a3d56a1fd2e9b9b1cbf4b67e23dc86d56ad304ee2d0e"} err="failed to get container status \"0cbb2c96007f87d82f48a3d56a1fd2e9b9b1cbf4b67e23dc86d56ad304ee2d0e\": rpc error: code = NotFound desc = could not find container \"0cbb2c96007f87d82f48a3d56a1fd2e9b9b1cbf4b67e23dc86d56ad304ee2d0e\": container with ID starting with 0cbb2c96007f87d82f48a3d56a1fd2e9b9b1cbf4b67e23dc86d56ad304ee2d0e not found: ID does not exist"
Sep 16 18:59:27.421136 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:59:27.421113    2581 scope.go:117] "RemoveContainer" containerID="d35aa5bcc401a5f827aacd509b60ae310fc989951e6dc303ed094f214c4ccb62"
Sep 16 18:59:27.421384 ip-10-0-138-127 kubenswrapper[2581]: E0916 18:59:27.421368    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d35aa5bcc401a5f827aacd509b60ae310fc989951e6dc303ed094f214c4ccb62\": container with ID starting with d35aa5bcc401a5f827aacd509b60ae310fc989951e6dc303ed094f214c4ccb62 not found: ID does not exist" containerID="d35aa5bcc401a5f827aacd509b60ae310fc989951e6dc303ed094f214c4ccb62"
Sep 16 18:59:27.421452 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:59:27.421405    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d35aa5bcc401a5f827aacd509b60ae310fc989951e6dc303ed094f214c4ccb62"} err="failed to get container status \"d35aa5bcc401a5f827aacd509b60ae310fc989951e6dc303ed094f214c4ccb62\": rpc error: code = NotFound desc = could not find container \"d35aa5bcc401a5f827aacd509b60ae310fc989951e6dc303ed094f214c4ccb62\": container with ID starting with d35aa5bcc401a5f827aacd509b60ae310fc989951e6dc303ed094f214c4ccb62 not found: ID does not exist"
Sep 16 18:59:28.728542 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:59:28.728510    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a81cbc02-9a43-40df-9a7a-fb53b8d0ed74" path="/var/lib/kubelet/pods/a81cbc02-9a43-40df-9a7a-fb53b8d0ed74/volumes"
Sep 16 18:59:34.246514 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:59:34.246469    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-75f85bc9f9-mqmdb" podUID="b1ee0ded-03f5-4504-8a9d-84c6d7d745cb" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.27:8080: connect: connection refused"
Sep 16 18:59:37.366700 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:59:37.366660    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-1ba12-predictor-74579d997-hmg4m" podUID="1035c64c-18a1-43f0-9e88-c06f78ad4591" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.28:8080: connect: connection refused"
Sep 16 18:59:44.247509 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:59:44.247463    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-75f85bc9f9-mqmdb" podUID="b1ee0ded-03f5-4504-8a9d-84c6d7d745cb" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.27:8080: connect: connection refused"
Sep 16 18:59:47.367298 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:59:47.367252    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-1ba12-predictor-74579d997-hmg4m" podUID="1035c64c-18a1-43f0-9e88-c06f78ad4591" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.28:8080: connect: connection refused"
Sep 16 18:59:54.247354 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:59:54.247310    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-75f85bc9f9-mqmdb" podUID="b1ee0ded-03f5-4504-8a9d-84c6d7d745cb" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.27:8080: connect: connection refused"
Sep 16 18:59:57.366882 ip-10-0-138-127 kubenswrapper[2581]: I0916 18:59:57.366842    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-1ba12-predictor-74579d997-hmg4m" podUID="1035c64c-18a1-43f0-9e88-c06f78ad4591" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.28:8080: connect: connection refused"
Sep 16 19:00:04.246748 ip-10-0-138-127 kubenswrapper[2581]: I0916 19:00:04.246707    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-75f85bc9f9-mqmdb" podUID="b1ee0ded-03f5-4504-8a9d-84c6d7d745cb" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.27:8080: connect: connection refused"
Sep 16 19:00:07.367377 ip-10-0-138-127 kubenswrapper[2581]: I0916 19:00:07.367334    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-1ba12-predictor-74579d997-hmg4m" podUID="1035c64c-18a1-43f0-9e88-c06f78ad4591" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.28:8080: connect: connection refused"
Sep 16 19:00:13.723049 ip-10-0-138-127 kubenswrapper[2581]: I0916 19:00:13.722994    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-75f85bc9f9-mqmdb" podUID="b1ee0ded-03f5-4504-8a9d-84c6d7d745cb" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.27:8080: connect: connection refused"
Sep 16 19:00:17.367586 ip-10-0-138-127 kubenswrapper[2581]: I0916 19:00:17.367531    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-1ba12-predictor-74579d997-hmg4m" podUID="1035c64c-18a1-43f0-9e88-c06f78ad4591" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.28:8080: connect: connection refused"
Sep 16 19:00:23.723614 ip-10-0-138-127 kubenswrapper[2581]: I0916 19:00:23.723523    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-75f85bc9f9-mqmdb"
Sep 16 19:00:27.367300 ip-10-0-138-127 kubenswrapper[2581]: I0916 19:00:27.367274    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-1ba12-predictor-74579d997-hmg4m"
Sep 16 19:00:31.590478 ip-10-0-138-127 kubenswrapper[2581]: I0916 19:00:31.590444    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-predictor-75f85bc9f9-mqmdb"]
Sep 16 19:00:31.590917 ip-10-0-138-127 kubenswrapper[2581]: I0916 19:00:31.590684    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-75f85bc9f9-mqmdb" podUID="b1ee0ded-03f5-4504-8a9d-84c6d7d745cb" containerName="kserve-container" containerID="cri-o://1cd42316a3c1d724bc4a802e13f0f14f26366c2a91269eb8aacc5a23d534a29a" gracePeriod=30
Sep 16 19:00:31.590917 ip-10-0-138-127 kubenswrapper[2581]: I0916 19:00:31.590709    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-75f85bc9f9-mqmdb" podUID="b1ee0ded-03f5-4504-8a9d-84c6d7d745cb" containerName="kube-rbac-proxy" containerID="cri-o://5a4f4c2f9817feee14e00a700b2b97cfaab54c9dfa8898b0e52e6e340f33e344" gracePeriod=30
Sep 16 19:00:32.669570 ip-10-0-138-127 kubenswrapper[2581]: I0916 19:00:32.669536    2581 generic.go:356] "Generic (PLEG): container finished" podID="b1ee0ded-03f5-4504-8a9d-84c6d7d745cb" containerID="5a4f4c2f9817feee14e00a700b2b97cfaab54c9dfa8898b0e52e6e340f33e344" exitCode=2
Sep 16 19:00:32.669952 ip-10-0-138-127 kubenswrapper[2581]: I0916 19:00:32.669605    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-75f85bc9f9-mqmdb" event={"ID":"b1ee0ded-03f5-4504-8a9d-84c6d7d745cb","Type":"ContainerDied","Data":"5a4f4c2f9817feee14e00a700b2b97cfaab54c9dfa8898b0e52e6e340f33e344"}
Sep 16 19:00:33.723022 ip-10-0-138-127 kubenswrapper[2581]: I0916 19:00:33.722977    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-75f85bc9f9-mqmdb" podUID="b1ee0ded-03f5-4504-8a9d-84c6d7d745cb" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.27:8080: connect: connection refused"
Sep 16 19:00:34.240157 ip-10-0-138-127 kubenswrapper[2581]: I0916 19:00:34.240117    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-75f85bc9f9-mqmdb" podUID="b1ee0ded-03f5-4504-8a9d-84c6d7d745cb" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.27:8643/healthz\": dial tcp 10.132.0.27:8643: connect: connection refused"
Sep 16 19:00:36.430989 ip-10-0-138-127 kubenswrapper[2581]: I0916 19:00:36.430962    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-75f85bc9f9-mqmdb"
Sep 16 19:00:36.532659 ip-10-0-138-127 kubenswrapper[2581]: I0916 19:00:36.532627    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/b1ee0ded-03f5-4504-8a9d-84c6d7d745cb-kube-rbac-proxy-sar-config\") pod \"b1ee0ded-03f5-4504-8a9d-84c6d7d745cb\" (UID: \"b1ee0ded-03f5-4504-8a9d-84c6d7d745cb\") "
Sep 16 19:00:36.532826 ip-10-0-138-127 kubenswrapper[2581]: I0916 19:00:36.532666    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b1ee0ded-03f5-4504-8a9d-84c6d7d745cb-kserve-provision-location\") pod \"b1ee0ded-03f5-4504-8a9d-84c6d7d745cb\" (UID: \"b1ee0ded-03f5-4504-8a9d-84c6d7d745cb\") "
Sep 16 19:00:36.532826 ip-10-0-138-127 kubenswrapper[2581]: I0916 19:00:36.532695    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/b1ee0ded-03f5-4504-8a9d-84c6d7d745cb-cabundle-cert\") pod \"b1ee0ded-03f5-4504-8a9d-84c6d7d745cb\" (UID: \"b1ee0ded-03f5-4504-8a9d-84c6d7d745cb\") "
Sep 16 19:00:36.532826 ip-10-0-138-127 kubenswrapper[2581]: I0916 19:00:36.532717    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b1ee0ded-03f5-4504-8a9d-84c6d7d745cb-proxy-tls\") pod \"b1ee0ded-03f5-4504-8a9d-84c6d7d745cb\" (UID: \"b1ee0ded-03f5-4504-8a9d-84c6d7d745cb\") "
Sep 16 19:00:36.532826 ip-10-0-138-127 kubenswrapper[2581]: I0916 19:00:36.532764    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xpj8l\" (UniqueName: \"kubernetes.io/projected/b1ee0ded-03f5-4504-8a9d-84c6d7d745cb-kube-api-access-xpj8l\") pod \"b1ee0ded-03f5-4504-8a9d-84c6d7d745cb\" (UID: \"b1ee0ded-03f5-4504-8a9d-84c6d7d745cb\") "
Sep 16 19:00:36.533024 ip-10-0-138-127 kubenswrapper[2581]: I0916 19:00:36.532997    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b1ee0ded-03f5-4504-8a9d-84c6d7d745cb-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "b1ee0ded-03f5-4504-8a9d-84c6d7d745cb" (UID: "b1ee0ded-03f5-4504-8a9d-84c6d7d745cb"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:00:36.533086 ip-10-0-138-127 kubenswrapper[2581]: I0916 19:00:36.533062    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b1ee0ded-03f5-4504-8a9d-84c6d7d745cb-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "b1ee0ded-03f5-4504-8a9d-84c6d7d745cb" (UID: "b1ee0ded-03f5-4504-8a9d-84c6d7d745cb"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 19:00:36.533143 ip-10-0-138-127 kubenswrapper[2581]: I0916 19:00:36.533059    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b1ee0ded-03f5-4504-8a9d-84c6d7d745cb-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "b1ee0ded-03f5-4504-8a9d-84c6d7d745cb" (UID: "b1ee0ded-03f5-4504-8a9d-84c6d7d745cb"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 19:00:36.534945 ip-10-0-138-127 kubenswrapper[2581]: I0916 19:00:36.534923    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b1ee0ded-03f5-4504-8a9d-84c6d7d745cb-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "b1ee0ded-03f5-4504-8a9d-84c6d7d745cb" (UID: "b1ee0ded-03f5-4504-8a9d-84c6d7d745cb"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 19:00:36.535151 ip-10-0-138-127 kubenswrapper[2581]: I0916 19:00:36.535128    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b1ee0ded-03f5-4504-8a9d-84c6d7d745cb-kube-api-access-xpj8l" (OuterVolumeSpecName: "kube-api-access-xpj8l") pod "b1ee0ded-03f5-4504-8a9d-84c6d7d745cb" (UID: "b1ee0ded-03f5-4504-8a9d-84c6d7d745cb"). InnerVolumeSpecName "kube-api-access-xpj8l". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 19:00:36.634091 ip-10-0-138-127 kubenswrapper[2581]: I0916 19:00:36.634052    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xpj8l\" (UniqueName: \"kubernetes.io/projected/b1ee0ded-03f5-4504-8a9d-84c6d7d745cb-kube-api-access-xpj8l\") on node \"ip-10-0-138-127.ec2.internal\" DevicePath \"\""
Sep 16 19:00:36.634091 ip-10-0-138-127 kubenswrapper[2581]: I0916 19:00:36.634081    2581 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/b1ee0ded-03f5-4504-8a9d-84c6d7d745cb-kube-rbac-proxy-sar-config\") on node \"ip-10-0-138-127.ec2.internal\" DevicePath \"\""
Sep 16 19:00:36.634091 ip-10-0-138-127 kubenswrapper[2581]: I0916 19:00:36.634091    2581 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b1ee0ded-03f5-4504-8a9d-84c6d7d745cb-kserve-provision-location\") on node \"ip-10-0-138-127.ec2.internal\" DevicePath \"\""
Sep 16 19:00:36.634091 ip-10-0-138-127 kubenswrapper[2581]: I0916 19:00:36.634100    2581 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/b1ee0ded-03f5-4504-8a9d-84c6d7d745cb-cabundle-cert\") on node \"ip-10-0-138-127.ec2.internal\" DevicePath \"\""
Sep 16 19:00:36.634476 ip-10-0-138-127 kubenswrapper[2581]: I0916 19:00:36.634110    2581 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b1ee0ded-03f5-4504-8a9d-84c6d7d745cb-proxy-tls\") on node \"ip-10-0-138-127.ec2.internal\" DevicePath \"\""
Sep 16 19:00:36.688809 ip-10-0-138-127 kubenswrapper[2581]: I0916 19:00:36.688777    2581 generic.go:356] "Generic (PLEG): container finished" podID="b1ee0ded-03f5-4504-8a9d-84c6d7d745cb" containerID="1cd42316a3c1d724bc4a802e13f0f14f26366c2a91269eb8aacc5a23d534a29a" exitCode=0
Sep 16 19:00:36.688985 ip-10-0-138-127 kubenswrapper[2581]: I0916 19:00:36.688842    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-75f85bc9f9-mqmdb"
Sep 16 19:00:36.688985 ip-10-0-138-127 kubenswrapper[2581]: I0916 19:00:36.688868    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-75f85bc9f9-mqmdb" event={"ID":"b1ee0ded-03f5-4504-8a9d-84c6d7d745cb","Type":"ContainerDied","Data":"1cd42316a3c1d724bc4a802e13f0f14f26366c2a91269eb8aacc5a23d534a29a"}
Sep 16 19:00:36.688985 ip-10-0-138-127 kubenswrapper[2581]: I0916 19:00:36.688932    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-75f85bc9f9-mqmdb" event={"ID":"b1ee0ded-03f5-4504-8a9d-84c6d7d745cb","Type":"ContainerDied","Data":"f7445801a433a63ed1445336f4b3c714e6e5dee770db9619d90bb62a567642c8"}
Sep 16 19:00:36.688985 ip-10-0-138-127 kubenswrapper[2581]: I0916 19:00:36.688966    2581 scope.go:117] "RemoveContainer" containerID="5a4f4c2f9817feee14e00a700b2b97cfaab54c9dfa8898b0e52e6e340f33e344"
Sep 16 19:00:36.698120 ip-10-0-138-127 kubenswrapper[2581]: I0916 19:00:36.698099    2581 scope.go:117] "RemoveContainer" containerID="1cd42316a3c1d724bc4a802e13f0f14f26366c2a91269eb8aacc5a23d534a29a"
Sep 16 19:00:36.706428 ip-10-0-138-127 kubenswrapper[2581]: I0916 19:00:36.706412    2581 scope.go:117] "RemoveContainer" containerID="233b873351a33cbe599372b8a2770642a839e9c5e63dc74b371fdbdf699a3a6b"
Sep 16 19:00:36.714477 ip-10-0-138-127 kubenswrapper[2581]: I0916 19:00:36.714453    2581 scope.go:117] "RemoveContainer" containerID="5a4f4c2f9817feee14e00a700b2b97cfaab54c9dfa8898b0e52e6e340f33e344"
Sep 16 19:00:36.714792 ip-10-0-138-127 kubenswrapper[2581]: E0916 19:00:36.714769    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5a4f4c2f9817feee14e00a700b2b97cfaab54c9dfa8898b0e52e6e340f33e344\": container with ID starting with 5a4f4c2f9817feee14e00a700b2b97cfaab54c9dfa8898b0e52e6e340f33e344 not found: ID does not exist" containerID="5a4f4c2f9817feee14e00a700b2b97cfaab54c9dfa8898b0e52e6e340f33e344"
Sep 16 19:00:36.714894 ip-10-0-138-127 kubenswrapper[2581]: I0916 19:00:36.714799    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5a4f4c2f9817feee14e00a700b2b97cfaab54c9dfa8898b0e52e6e340f33e344"} err="failed to get container status \"5a4f4c2f9817feee14e00a700b2b97cfaab54c9dfa8898b0e52e6e340f33e344\": rpc error: code = NotFound desc = could not find container \"5a4f4c2f9817feee14e00a700b2b97cfaab54c9dfa8898b0e52e6e340f33e344\": container with ID starting with 5a4f4c2f9817feee14e00a700b2b97cfaab54c9dfa8898b0e52e6e340f33e344 not found: ID does not exist"
Sep 16 19:00:36.714894 ip-10-0-138-127 kubenswrapper[2581]: I0916 19:00:36.714816    2581 scope.go:117] "RemoveContainer" containerID="1cd42316a3c1d724bc4a802e13f0f14f26366c2a91269eb8aacc5a23d534a29a"
Sep 16 19:00:36.715085 ip-10-0-138-127 kubenswrapper[2581]: E0916 19:00:36.715065    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1cd42316a3c1d724bc4a802e13f0f14f26366c2a91269eb8aacc5a23d534a29a\": container with ID starting with 1cd42316a3c1d724bc4a802e13f0f14f26366c2a91269eb8aacc5a23d534a29a not found: ID does not exist" containerID="1cd42316a3c1d724bc4a802e13f0f14f26366c2a91269eb8aacc5a23d534a29a"
Sep 16 19:00:36.715211 ip-10-0-138-127 kubenswrapper[2581]: I0916 19:00:36.715087    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1cd42316a3c1d724bc4a802e13f0f14f26366c2a91269eb8aacc5a23d534a29a"} err="failed to get container status \"1cd42316a3c1d724bc4a802e13f0f14f26366c2a91269eb8aacc5a23d534a29a\": rpc error: code = NotFound desc = could not find container \"1cd42316a3c1d724bc4a802e13f0f14f26366c2a91269eb8aacc5a23d534a29a\": container with ID starting with 1cd42316a3c1d724bc4a802e13f0f14f26366c2a91269eb8aacc5a23d534a29a not found: ID does not exist"
Sep 16 19:00:36.715211 ip-10-0-138-127 kubenswrapper[2581]: I0916 19:00:36.715101    2581 scope.go:117] "RemoveContainer" containerID="233b873351a33cbe599372b8a2770642a839e9c5e63dc74b371fdbdf699a3a6b"
Sep 16 19:00:36.715466 ip-10-0-138-127 kubenswrapper[2581]: E0916 19:00:36.715445    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"233b873351a33cbe599372b8a2770642a839e9c5e63dc74b371fdbdf699a3a6b\": container with ID starting with 233b873351a33cbe599372b8a2770642a839e9c5e63dc74b371fdbdf699a3a6b not found: ID does not exist" containerID="233b873351a33cbe599372b8a2770642a839e9c5e63dc74b371fdbdf699a3a6b"
Sep 16 19:00:36.715557 ip-10-0-138-127 kubenswrapper[2581]: I0916 19:00:36.715470    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"233b873351a33cbe599372b8a2770642a839e9c5e63dc74b371fdbdf699a3a6b"} err="failed to get container status \"233b873351a33cbe599372b8a2770642a839e9c5e63dc74b371fdbdf699a3a6b\": rpc error: code = NotFound desc = could not find container \"233b873351a33cbe599372b8a2770642a839e9c5e63dc74b371fdbdf699a3a6b\": container with ID starting with 233b873351a33cbe599372b8a2770642a839e9c5e63dc74b371fdbdf699a3a6b not found: ID does not exist"
Sep 16 19:00:36.727514 ip-10-0-138-127 kubenswrapper[2581]: I0916 19:00:36.727487    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-predictor-75f85bc9f9-mqmdb"]
Sep 16 19:00:36.727658 ip-10-0-138-127 kubenswrapper[2581]: I0916 19:00:36.727522    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-predictor-75f85bc9f9-mqmdb"]
Sep 16 19:00:36.870129 ip-10-0-138-127 kubenswrapper[2581]: I0916 19:00:36.870045    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-canary-stop-predictor-6cbd7d9f59-ks6sw"]
Sep 16 19:00:36.870389 ip-10-0-138-127 kubenswrapper[2581]: I0916 19:00:36.870377    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b1ee0ded-03f5-4504-8a9d-84c6d7d745cb" containerName="storage-initializer"
Sep 16 19:00:36.870439 ip-10-0-138-127 kubenswrapper[2581]: I0916 19:00:36.870391    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="b1ee0ded-03f5-4504-8a9d-84c6d7d745cb" containerName="storage-initializer"
Sep 16 19:00:36.870439 ip-10-0-138-127 kubenswrapper[2581]: I0916 19:00:36.870405    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a81cbc02-9a43-40df-9a7a-fb53b8d0ed74" containerName="kserve-container"
Sep 16 19:00:36.870439 ip-10-0-138-127 kubenswrapper[2581]: I0916 19:00:36.870410    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="a81cbc02-9a43-40df-9a7a-fb53b8d0ed74" containerName="kserve-container"
Sep 16 19:00:36.870439 ip-10-0-138-127 kubenswrapper[2581]: I0916 19:00:36.870422    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a81cbc02-9a43-40df-9a7a-fb53b8d0ed74" containerName="agent"
Sep 16 19:00:36.870439 ip-10-0-138-127 kubenswrapper[2581]: I0916 19:00:36.870427    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="a81cbc02-9a43-40df-9a7a-fb53b8d0ed74" containerName="agent"
Sep 16 19:00:36.870439 ip-10-0-138-127 kubenswrapper[2581]: I0916 19:00:36.870435    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b1ee0ded-03f5-4504-8a9d-84c6d7d745cb" containerName="kube-rbac-proxy"
Sep 16 19:00:36.870439 ip-10-0-138-127 kubenswrapper[2581]: I0916 19:00:36.870440    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="b1ee0ded-03f5-4504-8a9d-84c6d7d745cb" containerName="kube-rbac-proxy"
Sep 16 19:00:36.870648 ip-10-0-138-127 kubenswrapper[2581]: I0916 19:00:36.870446    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a81cbc02-9a43-40df-9a7a-fb53b8d0ed74" containerName="kube-rbac-proxy"
Sep 16 19:00:36.870648 ip-10-0-138-127 kubenswrapper[2581]: I0916 19:00:36.870451    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="a81cbc02-9a43-40df-9a7a-fb53b8d0ed74" containerName="kube-rbac-proxy"
Sep 16 19:00:36.870648 ip-10-0-138-127 kubenswrapper[2581]: I0916 19:00:36.870461    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a81cbc02-9a43-40df-9a7a-fb53b8d0ed74" containerName="storage-initializer"
Sep 16 19:00:36.870648 ip-10-0-138-127 kubenswrapper[2581]: I0916 19:00:36.870467    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="a81cbc02-9a43-40df-9a7a-fb53b8d0ed74" containerName="storage-initializer"
Sep 16 19:00:36.870648 ip-10-0-138-127 kubenswrapper[2581]: I0916 19:00:36.870474    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b1ee0ded-03f5-4504-8a9d-84c6d7d745cb" containerName="kserve-container"
Sep 16 19:00:36.870648 ip-10-0-138-127 kubenswrapper[2581]: I0916 19:00:36.870479    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="b1ee0ded-03f5-4504-8a9d-84c6d7d745cb" containerName="kserve-container"
Sep 16 19:00:36.870648 ip-10-0-138-127 kubenswrapper[2581]: I0916 19:00:36.870528    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="b1ee0ded-03f5-4504-8a9d-84c6d7d745cb" containerName="kserve-container"
Sep 16 19:00:36.870648 ip-10-0-138-127 kubenswrapper[2581]: I0916 19:00:36.870536    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="a81cbc02-9a43-40df-9a7a-fb53b8d0ed74" containerName="kserve-container"
Sep 16 19:00:36.870648 ip-10-0-138-127 kubenswrapper[2581]: I0916 19:00:36.870541    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="b1ee0ded-03f5-4504-8a9d-84c6d7d745cb" containerName="kube-rbac-proxy"
Sep 16 19:00:36.870648 ip-10-0-138-127 kubenswrapper[2581]: I0916 19:00:36.870549    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="a81cbc02-9a43-40df-9a7a-fb53b8d0ed74" containerName="kube-rbac-proxy"
Sep 16 19:00:36.870648 ip-10-0-138-127 kubenswrapper[2581]: I0916 19:00:36.870554    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="a81cbc02-9a43-40df-9a7a-fb53b8d0ed74" containerName="agent"
Sep 16 19:00:36.875003 ip-10-0-138-127 kubenswrapper[2581]: I0916 19:00:36.874981    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-6cbd7d9f59-ks6sw"
Sep 16 19:00:36.878434 ip-10-0-138-127 kubenswrapper[2581]: I0916 19:00:36.878415    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-stop-kube-rbac-proxy-sar-config\""
Sep 16 19:00:36.878548 ip-10-0-138-127 kubenswrapper[2581]: I0916 19:00:36.878423    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-stop-predictor-serving-cert\""
Sep 16 19:00:36.925596 ip-10-0-138-127 kubenswrapper[2581]: I0916 19:00:36.925562    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-stop-predictor-6cbd7d9f59-ks6sw"]
Sep 16 19:00:36.936778 ip-10-0-138-127 kubenswrapper[2581]: I0916 19:00:36.936740    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tcrld\" (UniqueName: \"kubernetes.io/projected/64110e03-5cab-44aa-bdd3-f7a937cf46f2-kube-api-access-tcrld\") pod \"isvc-canary-stop-predictor-6cbd7d9f59-ks6sw\" (UID: \"64110e03-5cab-44aa-bdd3-f7a937cf46f2\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-6cbd7d9f59-ks6sw"
Sep 16 19:00:36.936778 ip-10-0-138-127 kubenswrapper[2581]: I0916 19:00:36.936776    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/64110e03-5cab-44aa-bdd3-f7a937cf46f2-kserve-provision-location\") pod \"isvc-canary-stop-predictor-6cbd7d9f59-ks6sw\" (UID: \"64110e03-5cab-44aa-bdd3-f7a937cf46f2\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-6cbd7d9f59-ks6sw"
Sep 16 19:00:36.936980 ip-10-0-138-127 kubenswrapper[2581]: I0916 19:00:36.936830    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/64110e03-5cab-44aa-bdd3-f7a937cf46f2-proxy-tls\") pod \"isvc-canary-stop-predictor-6cbd7d9f59-ks6sw\" (UID: \"64110e03-5cab-44aa-bdd3-f7a937cf46f2\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-6cbd7d9f59-ks6sw"
Sep 16 19:00:36.936980 ip-10-0-138-127 kubenswrapper[2581]: I0916 19:00:36.936871    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/64110e03-5cab-44aa-bdd3-f7a937cf46f2-cabundle-cert\") pod \"isvc-canary-stop-predictor-6cbd7d9f59-ks6sw\" (UID: \"64110e03-5cab-44aa-bdd3-f7a937cf46f2\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-6cbd7d9f59-ks6sw"
Sep 16 19:00:36.936980 ip-10-0-138-127 kubenswrapper[2581]: I0916 19:00:36.936910    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/64110e03-5cab-44aa-bdd3-f7a937cf46f2-kube-rbac-proxy-sar-config\") pod \"isvc-canary-stop-predictor-6cbd7d9f59-ks6sw\" (UID: \"64110e03-5cab-44aa-bdd3-f7a937cf46f2\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-6cbd7d9f59-ks6sw"
Sep 16 19:00:37.038161 ip-10-0-138-127 kubenswrapper[2581]: I0916 19:00:37.038117    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/64110e03-5cab-44aa-bdd3-f7a937cf46f2-kserve-provision-location\") pod \"isvc-canary-stop-predictor-6cbd7d9f59-ks6sw\" (UID: \"64110e03-5cab-44aa-bdd3-f7a937cf46f2\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-6cbd7d9f59-ks6sw"
Sep 16 19:00:37.038161 ip-10-0-138-127 kubenswrapper[2581]: I0916 19:00:37.038161    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/64110e03-5cab-44aa-bdd3-f7a937cf46f2-proxy-tls\") pod \"isvc-canary-stop-predictor-6cbd7d9f59-ks6sw\" (UID: \"64110e03-5cab-44aa-bdd3-f7a937cf46f2\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-6cbd7d9f59-ks6sw"
Sep 16 19:00:37.038432 ip-10-0-138-127 kubenswrapper[2581]: I0916 19:00:37.038223    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/64110e03-5cab-44aa-bdd3-f7a937cf46f2-cabundle-cert\") pod \"isvc-canary-stop-predictor-6cbd7d9f59-ks6sw\" (UID: \"64110e03-5cab-44aa-bdd3-f7a937cf46f2\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-6cbd7d9f59-ks6sw"
Sep 16 19:00:37.038432 ip-10-0-138-127 kubenswrapper[2581]: I0916 19:00:37.038270    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/64110e03-5cab-44aa-bdd3-f7a937cf46f2-kube-rbac-proxy-sar-config\") pod \"isvc-canary-stop-predictor-6cbd7d9f59-ks6sw\" (UID: \"64110e03-5cab-44aa-bdd3-f7a937cf46f2\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-6cbd7d9f59-ks6sw"
Sep 16 19:00:37.038432 ip-10-0-138-127 kubenswrapper[2581]: I0916 19:00:37.038339    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tcrld\" (UniqueName: \"kubernetes.io/projected/64110e03-5cab-44aa-bdd3-f7a937cf46f2-kube-api-access-tcrld\") pod \"isvc-canary-stop-predictor-6cbd7d9f59-ks6sw\" (UID: \"64110e03-5cab-44aa-bdd3-f7a937cf46f2\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-6cbd7d9f59-ks6sw"
Sep 16 19:00:37.038657 ip-10-0-138-127 kubenswrapper[2581]: I0916 19:00:37.038634    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/64110e03-5cab-44aa-bdd3-f7a937cf46f2-kserve-provision-location\") pod \"isvc-canary-stop-predictor-6cbd7d9f59-ks6sw\" (UID: \"64110e03-5cab-44aa-bdd3-f7a937cf46f2\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-6cbd7d9f59-ks6sw"
Sep 16 19:00:37.038908 ip-10-0-138-127 kubenswrapper[2581]: I0916 19:00:37.038876    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/64110e03-5cab-44aa-bdd3-f7a937cf46f2-cabundle-cert\") pod \"isvc-canary-stop-predictor-6cbd7d9f59-ks6sw\" (UID: \"64110e03-5cab-44aa-bdd3-f7a937cf46f2\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-6cbd7d9f59-ks6sw"
Sep 16 19:00:37.039031 ip-10-0-138-127 kubenswrapper[2581]: I0916 19:00:37.038942    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/64110e03-5cab-44aa-bdd3-f7a937cf46f2-kube-rbac-proxy-sar-config\") pod \"isvc-canary-stop-predictor-6cbd7d9f59-ks6sw\" (UID: \"64110e03-5cab-44aa-bdd3-f7a937cf46f2\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-6cbd7d9f59-ks6sw"
Sep 16 19:00:37.040881 ip-10-0-138-127 kubenswrapper[2581]: I0916 19:00:37.040864    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/64110e03-5cab-44aa-bdd3-f7a937cf46f2-proxy-tls\") pod \"isvc-canary-stop-predictor-6cbd7d9f59-ks6sw\" (UID: \"64110e03-5cab-44aa-bdd3-f7a937cf46f2\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-6cbd7d9f59-ks6sw"
Sep 16 19:00:37.050137 ip-10-0-138-127 kubenswrapper[2581]: I0916 19:00:37.050109    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tcrld\" (UniqueName: \"kubernetes.io/projected/64110e03-5cab-44aa-bdd3-f7a937cf46f2-kube-api-access-tcrld\") pod \"isvc-canary-stop-predictor-6cbd7d9f59-ks6sw\" (UID: \"64110e03-5cab-44aa-bdd3-f7a937cf46f2\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-6cbd7d9f59-ks6sw"
Sep 16 19:00:37.184429 ip-10-0-138-127 kubenswrapper[2581]: I0916 19:00:37.184332    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-6cbd7d9f59-ks6sw"
Sep 16 19:00:37.464800 ip-10-0-138-127 kubenswrapper[2581]: I0916 19:00:37.464634    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-stop-predictor-6cbd7d9f59-ks6sw"]
Sep 16 19:00:37.467337 ip-10-0-138-127 kubenswrapper[2581]: W0916 19:00:37.467308    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod64110e03_5cab_44aa_bdd3_f7a937cf46f2.slice/crio-72ca1a35b0c977a738a03be4b07acc36a0816fab11299622f4d9da7a329fa813 WatchSource:0}: Error finding container 72ca1a35b0c977a738a03be4b07acc36a0816fab11299622f4d9da7a329fa813: Status 404 returned error can't find the container with id 72ca1a35b0c977a738a03be4b07acc36a0816fab11299622f4d9da7a329fa813
Sep 16 19:00:37.695550 ip-10-0-138-127 kubenswrapper[2581]: I0916 19:00:37.695509    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-6cbd7d9f59-ks6sw" event={"ID":"64110e03-5cab-44aa-bdd3-f7a937cf46f2","Type":"ContainerStarted","Data":"7b8963f2daef3b7f9e5667ebe18aeab1e516cfb080d0adc82b7e14f47f9c0923"}
Sep 16 19:00:37.695737 ip-10-0-138-127 kubenswrapper[2581]: I0916 19:00:37.695600    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-6cbd7d9f59-ks6sw" event={"ID":"64110e03-5cab-44aa-bdd3-f7a937cf46f2","Type":"ContainerStarted","Data":"72ca1a35b0c977a738a03be4b07acc36a0816fab11299622f4d9da7a329fa813"}
Sep 16 19:00:38.727611 ip-10-0-138-127 kubenswrapper[2581]: I0916 19:00:38.727578    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b1ee0ded-03f5-4504-8a9d-84c6d7d745cb" path="/var/lib/kubelet/pods/b1ee0ded-03f5-4504-8a9d-84c6d7d745cb/volumes"
Sep 16 19:00:41.716391 ip-10-0-138-127 kubenswrapper[2581]: I0916 19:00:41.716355    2581 generic.go:356] "Generic (PLEG): container finished" podID="64110e03-5cab-44aa-bdd3-f7a937cf46f2" containerID="7b8963f2daef3b7f9e5667ebe18aeab1e516cfb080d0adc82b7e14f47f9c0923" exitCode=0
Sep 16 19:00:41.716844 ip-10-0-138-127 kubenswrapper[2581]: I0916 19:00:41.716435    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-6cbd7d9f59-ks6sw" event={"ID":"64110e03-5cab-44aa-bdd3-f7a937cf46f2","Type":"ContainerDied","Data":"7b8963f2daef3b7f9e5667ebe18aeab1e516cfb080d0adc82b7e14f47f9c0923"}
Sep 16 19:00:42.726637 ip-10-0-138-127 kubenswrapper[2581]: I0916 19:00:42.726604    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-6cbd7d9f59-ks6sw" event={"ID":"64110e03-5cab-44aa-bdd3-f7a937cf46f2","Type":"ContainerStarted","Data":"ddf713e4ecda7f018ff13e9c3ff9b03b9df879757b83f062ef0e25c1c883ced6"}
Sep 16 19:00:42.727021 ip-10-0-138-127 kubenswrapper[2581]: I0916 19:00:42.726663    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-6cbd7d9f59-ks6sw" event={"ID":"64110e03-5cab-44aa-bdd3-f7a937cf46f2","Type":"ContainerStarted","Data":"53bdee0638892d715499027d504d4004f701a1a3760118579cf72b01605e3d97"}
Sep 16 19:00:42.727021 ip-10-0-138-127 kubenswrapper[2581]: I0916 19:00:42.726851    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-6cbd7d9f59-ks6sw"
Sep 16 19:00:42.758727 ip-10-0-138-127 kubenswrapper[2581]: I0916 19:00:42.758678    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-6cbd7d9f59-ks6sw" podStartSLOduration=2.715021027 podStartE2EDuration="6.758662849s" totalImagesPullingTime="0s" totalInitContainerRuntime="4.043641822s" isStatefulPod=false podCreationTimestamp="2026-09-16 19:00:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:00:42.757586562 +0000 UTC m=+912.577000463" watchObservedRunningTime="2026-09-16 19:00:42.758662849 +0000 UTC m=+912.578076757"
Sep 16 19:00:43.729662 ip-10-0-138-127 kubenswrapper[2581]: I0916 19:00:43.729632    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-6cbd7d9f59-ks6sw"
Sep 16 19:00:43.730910 ip-10-0-138-127 kubenswrapper[2581]: I0916 19:00:43.730881    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-6cbd7d9f59-ks6sw" podUID="64110e03-5cab-44aa-bdd3-f7a937cf46f2" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.29:8080: connect: connection refused"
Sep 16 19:00:44.734458 ip-10-0-138-127 kubenswrapper[2581]: I0916 19:00:44.734414    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-6cbd7d9f59-ks6sw" podUID="64110e03-5cab-44aa-bdd3-f7a937cf46f2" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.29:8080: connect: connection refused"
Sep 16 19:00:46.924500 ip-10-0-138-127 kubenswrapper[2581]: I0916 19:00:46.924466    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-graph-raw-1ba12-predictor-74579d997-hmg4m"]
Sep 16 19:00:46.924931 ip-10-0-138-127 kubenswrapper[2581]: I0916 19:00:46.924697    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-1ba12-predictor-74579d997-hmg4m" podUID="1035c64c-18a1-43f0-9e88-c06f78ad4591" containerName="kserve-container" containerID="cri-o://a51ba3abb020aba40b477770c8e51ede18f48bac355266a9a0fb2d774d4b0f5b" gracePeriod=30
Sep 16 19:00:46.924931 ip-10-0-138-127 kubenswrapper[2581]: I0916 19:00:46.924760    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-1ba12-predictor-74579d997-hmg4m" podUID="1035c64c-18a1-43f0-9e88-c06f78ad4591" containerName="kube-rbac-proxy" containerID="cri-o://471ef77688652749185a328029f5dc37ff482014e210663fcca15481687a71f4" gracePeriod=30
Sep 16 19:00:47.058366 ip-10-0-138-127 kubenswrapper[2581]: I0916 19:00:47.058330    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-4959b-predictor-7544df7b5d-899mk"]
Sep 16 19:00:47.062646 ip-10-0-138-127 kubenswrapper[2581]: I0916 19:00:47.062627    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-4959b-predictor-7544df7b5d-899mk"
Sep 16 19:00:47.066717 ip-10-0-138-127 kubenswrapper[2581]: I0916 19:00:47.066693    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-graph-raw-hpa-4959b-predictor-serving-cert\""
Sep 16 19:00:47.066974 ip-10-0-138-127 kubenswrapper[2581]: I0916 19:00:47.066954    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-graph-raw-hpa-4959b-kube-rbac-proxy-sar-config\""
Sep 16 19:00:47.130120 ip-10-0-138-127 kubenswrapper[2581]: I0916 19:00:47.130073    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a76ee980-74ce-4370-a7f8-3e204b8bbb5c-kserve-provision-location\") pod \"isvc-sklearn-graph-raw-hpa-4959b-predictor-7544df7b5d-899mk\" (UID: \"a76ee980-74ce-4370-a7f8-3e204b8bbb5c\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-4959b-predictor-7544df7b5d-899mk"
Sep 16 19:00:47.130331 ip-10-0-138-127 kubenswrapper[2581]: I0916 19:00:47.130128    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a76ee980-74ce-4370-a7f8-3e204b8bbb5c-proxy-tls\") pod \"isvc-sklearn-graph-raw-hpa-4959b-predictor-7544df7b5d-899mk\" (UID: \"a76ee980-74ce-4370-a7f8-3e204b8bbb5c\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-4959b-predictor-7544df7b5d-899mk"
Sep 16 19:00:47.130331 ip-10-0-138-127 kubenswrapper[2581]: I0916 19:00:47.130226    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/a76ee980-74ce-4370-a7f8-3e204b8bbb5c-cabundle-cert\") pod \"isvc-sklearn-graph-raw-hpa-4959b-predictor-7544df7b5d-899mk\" (UID: \"a76ee980-74ce-4370-a7f8-3e204b8bbb5c\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-4959b-predictor-7544df7b5d-899mk"
Sep 16 19:00:47.130331 ip-10-0-138-127 kubenswrapper[2581]: I0916 19:00:47.130303    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vrvqx\" (UniqueName: \"kubernetes.io/projected/a76ee980-74ce-4370-a7f8-3e204b8bbb5c-kube-api-access-vrvqx\") pod \"isvc-sklearn-graph-raw-hpa-4959b-predictor-7544df7b5d-899mk\" (UID: \"a76ee980-74ce-4370-a7f8-3e204b8bbb5c\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-4959b-predictor-7544df7b5d-899mk"
Sep 16 19:00:47.130493 ip-10-0-138-127 kubenswrapper[2581]: I0916 19:00:47.130343    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a76ee980-74ce-4370-a7f8-3e204b8bbb5c-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-graph-raw-hpa-4959b-predictor-7544df7b5d-899mk\" (UID: \"a76ee980-74ce-4370-a7f8-3e204b8bbb5c\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-4959b-predictor-7544df7b5d-899mk"
Sep 16 19:00:47.156859 ip-10-0-138-127 kubenswrapper[2581]: I0916 19:00:47.156823    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-4959b-predictor-7544df7b5d-899mk"]
Sep 16 19:00:47.231840 ip-10-0-138-127 kubenswrapper[2581]: I0916 19:00:47.231751    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/a76ee980-74ce-4370-a7f8-3e204b8bbb5c-cabundle-cert\") pod \"isvc-sklearn-graph-raw-hpa-4959b-predictor-7544df7b5d-899mk\" (UID: \"a76ee980-74ce-4370-a7f8-3e204b8bbb5c\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-4959b-predictor-7544df7b5d-899mk"
Sep 16 19:00:47.231840 ip-10-0-138-127 kubenswrapper[2581]: I0916 19:00:47.231803    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vrvqx\" (UniqueName: \"kubernetes.io/projected/a76ee980-74ce-4370-a7f8-3e204b8bbb5c-kube-api-access-vrvqx\") pod \"isvc-sklearn-graph-raw-hpa-4959b-predictor-7544df7b5d-899mk\" (UID: \"a76ee980-74ce-4370-a7f8-3e204b8bbb5c\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-4959b-predictor-7544df7b5d-899mk"
Sep 16 19:00:47.231840 ip-10-0-138-127 kubenswrapper[2581]: I0916 19:00:47.231835    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a76ee980-74ce-4370-a7f8-3e204b8bbb5c-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-graph-raw-hpa-4959b-predictor-7544df7b5d-899mk\" (UID: \"a76ee980-74ce-4370-a7f8-3e204b8bbb5c\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-4959b-predictor-7544df7b5d-899mk"
Sep 16 19:00:47.232101 ip-10-0-138-127 kubenswrapper[2581]: I0916 19:00:47.231866    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a76ee980-74ce-4370-a7f8-3e204b8bbb5c-kserve-provision-location\") pod \"isvc-sklearn-graph-raw-hpa-4959b-predictor-7544df7b5d-899mk\" (UID: \"a76ee980-74ce-4370-a7f8-3e204b8bbb5c\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-4959b-predictor-7544df7b5d-899mk"
Sep 16 19:00:47.232101 ip-10-0-138-127 kubenswrapper[2581]: I0916 19:00:47.231887    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a76ee980-74ce-4370-a7f8-3e204b8bbb5c-proxy-tls\") pod \"isvc-sklearn-graph-raw-hpa-4959b-predictor-7544df7b5d-899mk\" (UID: \"a76ee980-74ce-4370-a7f8-3e204b8bbb5c\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-4959b-predictor-7544df7b5d-899mk"
Sep 16 19:00:47.232101 ip-10-0-138-127 kubenswrapper[2581]: E0916 19:00:47.231993    2581 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-4959b-predictor-serving-cert: secret "isvc-sklearn-graph-raw-hpa-4959b-predictor-serving-cert" not found
Sep 16 19:00:47.232101 ip-10-0-138-127 kubenswrapper[2581]: E0916 19:00:47.232057    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a76ee980-74ce-4370-a7f8-3e204b8bbb5c-proxy-tls podName:a76ee980-74ce-4370-a7f8-3e204b8bbb5c nodeName:}" failed. No retries permitted until 2026-09-16 19:00:47.732035459 +0000 UTC m=+917.551449351 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/a76ee980-74ce-4370-a7f8-3e204b8bbb5c-proxy-tls") pod "isvc-sklearn-graph-raw-hpa-4959b-predictor-7544df7b5d-899mk" (UID: "a76ee980-74ce-4370-a7f8-3e204b8bbb5c") : secret "isvc-sklearn-graph-raw-hpa-4959b-predictor-serving-cert" not found
Sep 16 19:00:47.232378 ip-10-0-138-127 kubenswrapper[2581]: I0916 19:00:47.232351    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a76ee980-74ce-4370-a7f8-3e204b8bbb5c-kserve-provision-location\") pod \"isvc-sklearn-graph-raw-hpa-4959b-predictor-7544df7b5d-899mk\" (UID: \"a76ee980-74ce-4370-a7f8-3e204b8bbb5c\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-4959b-predictor-7544df7b5d-899mk"
Sep 16 19:00:47.232550 ip-10-0-138-127 kubenswrapper[2581]: I0916 19:00:47.232535    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/a76ee980-74ce-4370-a7f8-3e204b8bbb5c-cabundle-cert\") pod \"isvc-sklearn-graph-raw-hpa-4959b-predictor-7544df7b5d-899mk\" (UID: \"a76ee980-74ce-4370-a7f8-3e204b8bbb5c\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-4959b-predictor-7544df7b5d-899mk"
Sep 16 19:00:47.232586 ip-10-0-138-127 kubenswrapper[2581]: I0916 19:00:47.232547    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a76ee980-74ce-4370-a7f8-3e204b8bbb5c-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-graph-raw-hpa-4959b-predictor-7544df7b5d-899mk\" (UID: \"a76ee980-74ce-4370-a7f8-3e204b8bbb5c\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-4959b-predictor-7544df7b5d-899mk"
Sep 16 19:00:47.243334 ip-10-0-138-127 kubenswrapper[2581]: I0916 19:00:47.243308    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vrvqx\" (UniqueName: \"kubernetes.io/projected/a76ee980-74ce-4370-a7f8-3e204b8bbb5c-kube-api-access-vrvqx\") pod \"isvc-sklearn-graph-raw-hpa-4959b-predictor-7544df7b5d-899mk\" (UID: \"a76ee980-74ce-4370-a7f8-3e204b8bbb5c\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-4959b-predictor-7544df7b5d-899mk"
Sep 16 19:00:47.268411 ip-10-0-138-127 kubenswrapper[2581]: I0916 19:00:47.268374    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-4959b-predictor-7554894dd5-5wf2g"]
Sep 16 19:00:47.272133 ip-10-0-138-127 kubenswrapper[2581]: I0916 19:00:47.272116    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-4959b-predictor-7554894dd5-5wf2g"
Sep 16 19:00:47.274816 ip-10-0-138-127 kubenswrapper[2581]: I0916 19:00:47.274782    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-xgboost-graph-raw-hpa-4959b-predictor-serving-cert\""
Sep 16 19:00:47.274816 ip-10-0-138-127 kubenswrapper[2581]: I0916 19:00:47.274795    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-xgboost-graph-raw-hpa-4959b-kube-rbac-proxy-sar-config\""
Sep 16 19:00:47.291578 ip-10-0-138-127 kubenswrapper[2581]: I0916 19:00:47.291544    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-4959b-predictor-7554894dd5-5wf2g"]
Sep 16 19:00:47.333167 ip-10-0-138-127 kubenswrapper[2581]: I0916 19:00:47.333137    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-csbz7\" (UniqueName: \"kubernetes.io/projected/0be1ad70-7a20-4b06-a2cc-a6f3a5024e8d-kube-api-access-csbz7\") pod \"isvc-xgboost-graph-raw-hpa-4959b-predictor-7554894dd5-5wf2g\" (UID: \"0be1ad70-7a20-4b06-a2cc-a6f3a5024e8d\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-4959b-predictor-7554894dd5-5wf2g"
Sep 16 19:00:47.333354 ip-10-0-138-127 kubenswrapper[2581]: I0916 19:00:47.333234    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/0be1ad70-7a20-4b06-a2cc-a6f3a5024e8d-cabundle-cert\") pod \"isvc-xgboost-graph-raw-hpa-4959b-predictor-7554894dd5-5wf2g\" (UID: \"0be1ad70-7a20-4b06-a2cc-a6f3a5024e8d\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-4959b-predictor-7554894dd5-5wf2g"
Sep 16 19:00:47.333354 ip-10-0-138-127 kubenswrapper[2581]: I0916 19:00:47.333267    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/0be1ad70-7a20-4b06-a2cc-a6f3a5024e8d-proxy-tls\") pod \"isvc-xgboost-graph-raw-hpa-4959b-predictor-7554894dd5-5wf2g\" (UID: \"0be1ad70-7a20-4b06-a2cc-a6f3a5024e8d\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-4959b-predictor-7554894dd5-5wf2g"
Sep 16 19:00:47.333354 ip-10-0-138-127 kubenswrapper[2581]: I0916 19:00:47.333282    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/0be1ad70-7a20-4b06-a2cc-a6f3a5024e8d-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-graph-raw-hpa-4959b-predictor-7554894dd5-5wf2g\" (UID: \"0be1ad70-7a20-4b06-a2cc-a6f3a5024e8d\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-4959b-predictor-7554894dd5-5wf2g"
Sep 16 19:00:47.333354 ip-10-0-138-127 kubenswrapper[2581]: I0916 19:00:47.333297    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/0be1ad70-7a20-4b06-a2cc-a6f3a5024e8d-kserve-provision-location\") pod \"isvc-xgboost-graph-raw-hpa-4959b-predictor-7554894dd5-5wf2g\" (UID: \"0be1ad70-7a20-4b06-a2cc-a6f3a5024e8d\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-4959b-predictor-7554894dd5-5wf2g"
Sep 16 19:00:47.362906 ip-10-0-138-127 kubenswrapper[2581]: I0916 19:00:47.362863    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-1ba12-predictor-74579d997-hmg4m" podUID="1035c64c-18a1-43f0-9e88-c06f78ad4591" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.28:8643/healthz\": dial tcp 10.132.0.28:8643: connect: connection refused"
Sep 16 19:00:47.367182 ip-10-0-138-127 kubenswrapper[2581]: I0916 19:00:47.367153    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-1ba12-predictor-74579d997-hmg4m" podUID="1035c64c-18a1-43f0-9e88-c06f78ad4591" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.28:8080: connect: connection refused"
Sep 16 19:00:47.434223 ip-10-0-138-127 kubenswrapper[2581]: I0916 19:00:47.434175    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/0be1ad70-7a20-4b06-a2cc-a6f3a5024e8d-cabundle-cert\") pod \"isvc-xgboost-graph-raw-hpa-4959b-predictor-7554894dd5-5wf2g\" (UID: \"0be1ad70-7a20-4b06-a2cc-a6f3a5024e8d\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-4959b-predictor-7554894dd5-5wf2g"
Sep 16 19:00:47.434392 ip-10-0-138-127 kubenswrapper[2581]: I0916 19:00:47.434253    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/0be1ad70-7a20-4b06-a2cc-a6f3a5024e8d-proxy-tls\") pod \"isvc-xgboost-graph-raw-hpa-4959b-predictor-7554894dd5-5wf2g\" (UID: \"0be1ad70-7a20-4b06-a2cc-a6f3a5024e8d\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-4959b-predictor-7554894dd5-5wf2g"
Sep 16 19:00:47.434392 ip-10-0-138-127 kubenswrapper[2581]: I0916 19:00:47.434280    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/0be1ad70-7a20-4b06-a2cc-a6f3a5024e8d-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-graph-raw-hpa-4959b-predictor-7554894dd5-5wf2g\" (UID: \"0be1ad70-7a20-4b06-a2cc-a6f3a5024e8d\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-4959b-predictor-7554894dd5-5wf2g"
Sep 16 19:00:47.434392 ip-10-0-138-127 kubenswrapper[2581]: I0916 19:00:47.434305    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/0be1ad70-7a20-4b06-a2cc-a6f3a5024e8d-kserve-provision-location\") pod \"isvc-xgboost-graph-raw-hpa-4959b-predictor-7554894dd5-5wf2g\" (UID: \"0be1ad70-7a20-4b06-a2cc-a6f3a5024e8d\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-4959b-predictor-7554894dd5-5wf2g"
Sep 16 19:00:47.434579 ip-10-0-138-127 kubenswrapper[2581]: E0916 19:00:47.434427    2581 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-4959b-predictor-serving-cert: secret "isvc-xgboost-graph-raw-hpa-4959b-predictor-serving-cert" not found
Sep 16 19:00:47.434579 ip-10-0-138-127 kubenswrapper[2581]: I0916 19:00:47.434482    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-csbz7\" (UniqueName: \"kubernetes.io/projected/0be1ad70-7a20-4b06-a2cc-a6f3a5024e8d-kube-api-access-csbz7\") pod \"isvc-xgboost-graph-raw-hpa-4959b-predictor-7554894dd5-5wf2g\" (UID: \"0be1ad70-7a20-4b06-a2cc-a6f3a5024e8d\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-4959b-predictor-7554894dd5-5wf2g"
Sep 16 19:00:47.434579 ip-10-0-138-127 kubenswrapper[2581]: E0916 19:00:47.434497    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0be1ad70-7a20-4b06-a2cc-a6f3a5024e8d-proxy-tls podName:0be1ad70-7a20-4b06-a2cc-a6f3a5024e8d nodeName:}" failed. No retries permitted until 2026-09-16 19:00:47.934479958 +0000 UTC m=+917.753893868 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/0be1ad70-7a20-4b06-a2cc-a6f3a5024e8d-proxy-tls") pod "isvc-xgboost-graph-raw-hpa-4959b-predictor-7554894dd5-5wf2g" (UID: "0be1ad70-7a20-4b06-a2cc-a6f3a5024e8d") : secret "isvc-xgboost-graph-raw-hpa-4959b-predictor-serving-cert" not found
Sep 16 19:00:47.434774 ip-10-0-138-127 kubenswrapper[2581]: I0916 19:00:47.434754    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/0be1ad70-7a20-4b06-a2cc-a6f3a5024e8d-kserve-provision-location\") pod \"isvc-xgboost-graph-raw-hpa-4959b-predictor-7554894dd5-5wf2g\" (UID: \"0be1ad70-7a20-4b06-a2cc-a6f3a5024e8d\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-4959b-predictor-7554894dd5-5wf2g"
Sep 16 19:00:47.435030 ip-10-0-138-127 kubenswrapper[2581]: I0916 19:00:47.435013    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/0be1ad70-7a20-4b06-a2cc-a6f3a5024e8d-cabundle-cert\") pod \"isvc-xgboost-graph-raw-hpa-4959b-predictor-7554894dd5-5wf2g\" (UID: \"0be1ad70-7a20-4b06-a2cc-a6f3a5024e8d\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-4959b-predictor-7554894dd5-5wf2g"
Sep 16 19:00:47.435072 ip-10-0-138-127 kubenswrapper[2581]: I0916 19:00:47.435051    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/0be1ad70-7a20-4b06-a2cc-a6f3a5024e8d-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-graph-raw-hpa-4959b-predictor-7554894dd5-5wf2g\" (UID: \"0be1ad70-7a20-4b06-a2cc-a6f3a5024e8d\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-4959b-predictor-7554894dd5-5wf2g"
Sep 16 19:00:47.444892 ip-10-0-138-127 kubenswrapper[2581]: I0916 19:00:47.444872    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-csbz7\" (UniqueName: \"kubernetes.io/projected/0be1ad70-7a20-4b06-a2cc-a6f3a5024e8d-kube-api-access-csbz7\") pod \"isvc-xgboost-graph-raw-hpa-4959b-predictor-7554894dd5-5wf2g\" (UID: \"0be1ad70-7a20-4b06-a2cc-a6f3a5024e8d\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-4959b-predictor-7554894dd5-5wf2g"
Sep 16 19:00:47.737178 ip-10-0-138-127 kubenswrapper[2581]: I0916 19:00:47.737126    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a76ee980-74ce-4370-a7f8-3e204b8bbb5c-proxy-tls\") pod \"isvc-sklearn-graph-raw-hpa-4959b-predictor-7544df7b5d-899mk\" (UID: \"a76ee980-74ce-4370-a7f8-3e204b8bbb5c\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-4959b-predictor-7544df7b5d-899mk"
Sep 16 19:00:47.739740 ip-10-0-138-127 kubenswrapper[2581]: I0916 19:00:47.739708    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a76ee980-74ce-4370-a7f8-3e204b8bbb5c-proxy-tls\") pod \"isvc-sklearn-graph-raw-hpa-4959b-predictor-7544df7b5d-899mk\" (UID: \"a76ee980-74ce-4370-a7f8-3e204b8bbb5c\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-4959b-predictor-7544df7b5d-899mk"
Sep 16 19:00:47.748642 ip-10-0-138-127 kubenswrapper[2581]: I0916 19:00:47.748609    2581 generic.go:356] "Generic (PLEG): container finished" podID="1035c64c-18a1-43f0-9e88-c06f78ad4591" containerID="471ef77688652749185a328029f5dc37ff482014e210663fcca15481687a71f4" exitCode=2
Sep 16 19:00:47.748789 ip-10-0-138-127 kubenswrapper[2581]: I0916 19:00:47.748665    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-1ba12-predictor-74579d997-hmg4m" event={"ID":"1035c64c-18a1-43f0-9e88-c06f78ad4591","Type":"ContainerDied","Data":"471ef77688652749185a328029f5dc37ff482014e210663fcca15481687a71f4"}
Sep 16 19:00:47.939410 ip-10-0-138-127 kubenswrapper[2581]: I0916 19:00:47.939370    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/0be1ad70-7a20-4b06-a2cc-a6f3a5024e8d-proxy-tls\") pod \"isvc-xgboost-graph-raw-hpa-4959b-predictor-7554894dd5-5wf2g\" (UID: \"0be1ad70-7a20-4b06-a2cc-a6f3a5024e8d\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-4959b-predictor-7554894dd5-5wf2g"
Sep 16 19:00:47.942030 ip-10-0-138-127 kubenswrapper[2581]: I0916 19:00:47.942007    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/0be1ad70-7a20-4b06-a2cc-a6f3a5024e8d-proxy-tls\") pod \"isvc-xgboost-graph-raw-hpa-4959b-predictor-7554894dd5-5wf2g\" (UID: \"0be1ad70-7a20-4b06-a2cc-a6f3a5024e8d\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-4959b-predictor-7554894dd5-5wf2g"
Sep 16 19:00:47.972882 ip-10-0-138-127 kubenswrapper[2581]: I0916 19:00:47.972842    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-4959b-predictor-7544df7b5d-899mk"
Sep 16 19:00:48.182537 ip-10-0-138-127 kubenswrapper[2581]: I0916 19:00:48.182497    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-4959b-predictor-7554894dd5-5wf2g"
Sep 16 19:00:48.259685 ip-10-0-138-127 kubenswrapper[2581]: I0916 19:00:48.259660    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-4959b-predictor-7544df7b5d-899mk"]
Sep 16 19:00:48.262701 ip-10-0-138-127 kubenswrapper[2581]: W0916 19:00:48.262670    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda76ee980_74ce_4370_a7f8_3e204b8bbb5c.slice/crio-9720c69816809d11589b2740ffadad9cd6c845750e85f636beb26670b9b9770d WatchSource:0}: Error finding container 9720c69816809d11589b2740ffadad9cd6c845750e85f636beb26670b9b9770d: Status 404 returned error can't find the container with id 9720c69816809d11589b2740ffadad9cd6c845750e85f636beb26670b9b9770d
Sep 16 19:00:48.466428 ip-10-0-138-127 kubenswrapper[2581]: I0916 19:00:48.466401    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-4959b-predictor-7554894dd5-5wf2g"]
Sep 16 19:00:48.468445 ip-10-0-138-127 kubenswrapper[2581]: W0916 19:00:48.468422    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0be1ad70_7a20_4b06_a2cc_a6f3a5024e8d.slice/crio-2edbc02580a069e7df1728d0149f186610932332ef14cb3ff4e530bd3e543e20 WatchSource:0}: Error finding container 2edbc02580a069e7df1728d0149f186610932332ef14cb3ff4e530bd3e543e20: Status 404 returned error can't find the container with id 2edbc02580a069e7df1728d0149f186610932332ef14cb3ff4e530bd3e543e20
Sep 16 19:00:48.756303 ip-10-0-138-127 kubenswrapper[2581]: I0916 19:00:48.756261    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-4959b-predictor-7554894dd5-5wf2g" event={"ID":"0be1ad70-7a20-4b06-a2cc-a6f3a5024e8d","Type":"ContainerStarted","Data":"4b9822b37c73d77de4beb2ea431a9be774f32023268cbc9036d1c3e2160d1ba4"}
Sep 16 19:00:48.756498 ip-10-0-138-127 kubenswrapper[2581]: I0916 19:00:48.756346    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-4959b-predictor-7554894dd5-5wf2g" event={"ID":"0be1ad70-7a20-4b06-a2cc-a6f3a5024e8d","Type":"ContainerStarted","Data":"2edbc02580a069e7df1728d0149f186610932332ef14cb3ff4e530bd3e543e20"}
Sep 16 19:00:48.758891 ip-10-0-138-127 kubenswrapper[2581]: I0916 19:00:48.758852    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-4959b-predictor-7544df7b5d-899mk" event={"ID":"a76ee980-74ce-4370-a7f8-3e204b8bbb5c","Type":"ContainerStarted","Data":"a279cdfa735f47d4b0ab58401066250ee75d8c09e19a5462533a20aa46f47d9f"}
Sep 16 19:00:48.759046 ip-10-0-138-127 kubenswrapper[2581]: I0916 19:00:48.758930    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-4959b-predictor-7544df7b5d-899mk" event={"ID":"a76ee980-74ce-4370-a7f8-3e204b8bbb5c","Type":"ContainerStarted","Data":"9720c69816809d11589b2740ffadad9cd6c845750e85f636beb26670b9b9770d"}
Sep 16 19:00:49.744869 ip-10-0-138-127 kubenswrapper[2581]: I0916 19:00:49.744837    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-6cbd7d9f59-ks6sw"
Sep 16 19:00:49.745469 ip-10-0-138-127 kubenswrapper[2581]: I0916 19:00:49.745439    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-6cbd7d9f59-ks6sw" podUID="64110e03-5cab-44aa-bdd3-f7a937cf46f2" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.29:8080: connect: connection refused"
Sep 16 19:00:50.973663 ip-10-0-138-127 kubenswrapper[2581]: I0916 19:00:50.973639    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-1ba12-predictor-74579d997-hmg4m"
Sep 16 19:00:51.070237 ip-10-0-138-127 kubenswrapper[2581]: I0916 19:00:51.070173    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/1035c64c-18a1-43f0-9e88-c06f78ad4591-kube-rbac-proxy-sar-config\") pod \"1035c64c-18a1-43f0-9e88-c06f78ad4591\" (UID: \"1035c64c-18a1-43f0-9e88-c06f78ad4591\") "
Sep 16 19:00:51.070237 ip-10-0-138-127 kubenswrapper[2581]: I0916 19:00:51.070248    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-bjmkd\" (UniqueName: \"kubernetes.io/projected/1035c64c-18a1-43f0-9e88-c06f78ad4591-kube-api-access-bjmkd\") pod \"1035c64c-18a1-43f0-9e88-c06f78ad4591\" (UID: \"1035c64c-18a1-43f0-9e88-c06f78ad4591\") "
Sep 16 19:00:51.070474 ip-10-0-138-127 kubenswrapper[2581]: I0916 19:00:51.070325    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/1035c64c-18a1-43f0-9e88-c06f78ad4591-kserve-provision-location\") pod \"1035c64c-18a1-43f0-9e88-c06f78ad4591\" (UID: \"1035c64c-18a1-43f0-9e88-c06f78ad4591\") "
Sep 16 19:00:51.070474 ip-10-0-138-127 kubenswrapper[2581]: I0916 19:00:51.070376    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/1035c64c-18a1-43f0-9e88-c06f78ad4591-proxy-tls\") pod \"1035c64c-18a1-43f0-9e88-c06f78ad4591\" (UID: \"1035c64c-18a1-43f0-9e88-c06f78ad4591\") "
Sep 16 19:00:51.070474 ip-10-0-138-127 kubenswrapper[2581]: I0916 19:00:51.070431    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/1035c64c-18a1-43f0-9e88-c06f78ad4591-cabundle-cert\") pod \"1035c64c-18a1-43f0-9e88-c06f78ad4591\" (UID: \"1035c64c-18a1-43f0-9e88-c06f78ad4591\") "
Sep 16 19:00:51.070630 ip-10-0-138-127 kubenswrapper[2581]: I0916 19:00:51.070602    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1035c64c-18a1-43f0-9e88-c06f78ad4591-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "1035c64c-18a1-43f0-9e88-c06f78ad4591" (UID: "1035c64c-18a1-43f0-9e88-c06f78ad4591"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 19:00:51.070630 ip-10-0-138-127 kubenswrapper[2581]: I0916 19:00:51.070602    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1035c64c-18a1-43f0-9e88-c06f78ad4591-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "1035c64c-18a1-43f0-9e88-c06f78ad4591" (UID: "1035c64c-18a1-43f0-9e88-c06f78ad4591"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:00:51.070746 ip-10-0-138-127 kubenswrapper[2581]: I0916 19:00:51.070692    2581 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/1035c64c-18a1-43f0-9e88-c06f78ad4591-kube-rbac-proxy-sar-config\") on node \"ip-10-0-138-127.ec2.internal\" DevicePath \"\""
Sep 16 19:00:51.070746 ip-10-0-138-127 kubenswrapper[2581]: I0916 19:00:51.070710    2581 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/1035c64c-18a1-43f0-9e88-c06f78ad4591-kserve-provision-location\") on node \"ip-10-0-138-127.ec2.internal\" DevicePath \"\""
Sep 16 19:00:51.070837 ip-10-0-138-127 kubenswrapper[2581]: I0916 19:00:51.070813    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1035c64c-18a1-43f0-9e88-c06f78ad4591-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "1035c64c-18a1-43f0-9e88-c06f78ad4591" (UID: "1035c64c-18a1-43f0-9e88-c06f78ad4591"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 19:00:51.072599 ip-10-0-138-127 kubenswrapper[2581]: I0916 19:00:51.072578    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1035c64c-18a1-43f0-9e88-c06f78ad4591-kube-api-access-bjmkd" (OuterVolumeSpecName: "kube-api-access-bjmkd") pod "1035c64c-18a1-43f0-9e88-c06f78ad4591" (UID: "1035c64c-18a1-43f0-9e88-c06f78ad4591"). InnerVolumeSpecName "kube-api-access-bjmkd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 19:00:51.072687 ip-10-0-138-127 kubenswrapper[2581]: I0916 19:00:51.072625    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1035c64c-18a1-43f0-9e88-c06f78ad4591-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "1035c64c-18a1-43f0-9e88-c06f78ad4591" (UID: "1035c64c-18a1-43f0-9e88-c06f78ad4591"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 19:00:51.171672 ip-10-0-138-127 kubenswrapper[2581]: I0916 19:00:51.171631    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bjmkd\" (UniqueName: \"kubernetes.io/projected/1035c64c-18a1-43f0-9e88-c06f78ad4591-kube-api-access-bjmkd\") on node \"ip-10-0-138-127.ec2.internal\" DevicePath \"\""
Sep 16 19:00:51.171672 ip-10-0-138-127 kubenswrapper[2581]: I0916 19:00:51.171662    2581 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/1035c64c-18a1-43f0-9e88-c06f78ad4591-proxy-tls\") on node \"ip-10-0-138-127.ec2.internal\" DevicePath \"\""
Sep 16 19:00:51.171672 ip-10-0-138-127 kubenswrapper[2581]: I0916 19:00:51.171672    2581 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/1035c64c-18a1-43f0-9e88-c06f78ad4591-cabundle-cert\") on node \"ip-10-0-138-127.ec2.internal\" DevicePath \"\""
Sep 16 19:00:51.776350 ip-10-0-138-127 kubenswrapper[2581]: I0916 19:00:51.776316    2581 generic.go:356] "Generic (PLEG): container finished" podID="1035c64c-18a1-43f0-9e88-c06f78ad4591" containerID="a51ba3abb020aba40b477770c8e51ede18f48bac355266a9a0fb2d774d4b0f5b" exitCode=0
Sep 16 19:00:51.776547 ip-10-0-138-127 kubenswrapper[2581]: I0916 19:00:51.776382    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-1ba12-predictor-74579d997-hmg4m"
Sep 16 19:00:51.776547 ip-10-0-138-127 kubenswrapper[2581]: I0916 19:00:51.776385    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-1ba12-predictor-74579d997-hmg4m" event={"ID":"1035c64c-18a1-43f0-9e88-c06f78ad4591","Type":"ContainerDied","Data":"a51ba3abb020aba40b477770c8e51ede18f48bac355266a9a0fb2d774d4b0f5b"}
Sep 16 19:00:51.776547 ip-10-0-138-127 kubenswrapper[2581]: I0916 19:00:51.776511    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-1ba12-predictor-74579d997-hmg4m" event={"ID":"1035c64c-18a1-43f0-9e88-c06f78ad4591","Type":"ContainerDied","Data":"d084f58fa9eb5ab06659dd759d3caf7fa1a427eb3abdf05f88dcfc2d2cfb66eb"}
Sep 16 19:00:51.776547 ip-10-0-138-127 kubenswrapper[2581]: I0916 19:00:51.776537    2581 scope.go:117] "RemoveContainer" containerID="471ef77688652749185a328029f5dc37ff482014e210663fcca15481687a71f4"
Sep 16 19:00:51.785891 ip-10-0-138-127 kubenswrapper[2581]: I0916 19:00:51.785871    2581 scope.go:117] "RemoveContainer" containerID="a51ba3abb020aba40b477770c8e51ede18f48bac355266a9a0fb2d774d4b0f5b"
Sep 16 19:00:51.793585 ip-10-0-138-127 kubenswrapper[2581]: I0916 19:00:51.793561    2581 scope.go:117] "RemoveContainer" containerID="1f7c53029145d867d0aacedf653ed854345e42c2e325ffc785c4d62d1c512998"
Sep 16 19:00:51.800926 ip-10-0-138-127 kubenswrapper[2581]: I0916 19:00:51.800900    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-graph-raw-1ba12-predictor-74579d997-hmg4m"]
Sep 16 19:00:51.802103 ip-10-0-138-127 kubenswrapper[2581]: I0916 19:00:51.802086    2581 scope.go:117] "RemoveContainer" containerID="471ef77688652749185a328029f5dc37ff482014e210663fcca15481687a71f4"
Sep 16 19:00:51.802416 ip-10-0-138-127 kubenswrapper[2581]: E0916 19:00:51.802397    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"471ef77688652749185a328029f5dc37ff482014e210663fcca15481687a71f4\": container with ID starting with 471ef77688652749185a328029f5dc37ff482014e210663fcca15481687a71f4 not found: ID does not exist" containerID="471ef77688652749185a328029f5dc37ff482014e210663fcca15481687a71f4"
Sep 16 19:00:51.802469 ip-10-0-138-127 kubenswrapper[2581]: I0916 19:00:51.802426    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"471ef77688652749185a328029f5dc37ff482014e210663fcca15481687a71f4"} err="failed to get container status \"471ef77688652749185a328029f5dc37ff482014e210663fcca15481687a71f4\": rpc error: code = NotFound desc = could not find container \"471ef77688652749185a328029f5dc37ff482014e210663fcca15481687a71f4\": container with ID starting with 471ef77688652749185a328029f5dc37ff482014e210663fcca15481687a71f4 not found: ID does not exist"
Sep 16 19:00:51.802469 ip-10-0-138-127 kubenswrapper[2581]: I0916 19:00:51.802445    2581 scope.go:117] "RemoveContainer" containerID="a51ba3abb020aba40b477770c8e51ede18f48bac355266a9a0fb2d774d4b0f5b"
Sep 16 19:00:51.802708 ip-10-0-138-127 kubenswrapper[2581]: E0916 19:00:51.802691    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a51ba3abb020aba40b477770c8e51ede18f48bac355266a9a0fb2d774d4b0f5b\": container with ID starting with a51ba3abb020aba40b477770c8e51ede18f48bac355266a9a0fb2d774d4b0f5b not found: ID does not exist" containerID="a51ba3abb020aba40b477770c8e51ede18f48bac355266a9a0fb2d774d4b0f5b"
Sep 16 19:00:51.802746 ip-10-0-138-127 kubenswrapper[2581]: I0916 19:00:51.802716    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a51ba3abb020aba40b477770c8e51ede18f48bac355266a9a0fb2d774d4b0f5b"} err="failed to get container status \"a51ba3abb020aba40b477770c8e51ede18f48bac355266a9a0fb2d774d4b0f5b\": rpc error: code = NotFound desc = could not find container \"a51ba3abb020aba40b477770c8e51ede18f48bac355266a9a0fb2d774d4b0f5b\": container with ID starting with a51ba3abb020aba40b477770c8e51ede18f48bac355266a9a0fb2d774d4b0f5b not found: ID does not exist"
Sep 16 19:00:51.802746 ip-10-0-138-127 kubenswrapper[2581]: I0916 19:00:51.802731    2581 scope.go:117] "RemoveContainer" containerID="1f7c53029145d867d0aacedf653ed854345e42c2e325ffc785c4d62d1c512998"
Sep 16 19:00:51.802980 ip-10-0-138-127 kubenswrapper[2581]: E0916 19:00:51.802963    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1f7c53029145d867d0aacedf653ed854345e42c2e325ffc785c4d62d1c512998\": container with ID starting with 1f7c53029145d867d0aacedf653ed854345e42c2e325ffc785c4d62d1c512998 not found: ID does not exist" containerID="1f7c53029145d867d0aacedf653ed854345e42c2e325ffc785c4d62d1c512998"
Sep 16 19:00:51.803020 ip-10-0-138-127 kubenswrapper[2581]: I0916 19:00:51.802984    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1f7c53029145d867d0aacedf653ed854345e42c2e325ffc785c4d62d1c512998"} err="failed to get container status \"1f7c53029145d867d0aacedf653ed854345e42c2e325ffc785c4d62d1c512998\": rpc error: code = NotFound desc = could not find container \"1f7c53029145d867d0aacedf653ed854345e42c2e325ffc785c4d62d1c512998\": container with ID starting with 1f7c53029145d867d0aacedf653ed854345e42c2e325ffc785c4d62d1c512998 not found: ID does not exist"
Sep 16 19:00:51.806437 ip-10-0-138-127 kubenswrapper[2581]: I0916 19:00:51.806416    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-graph-raw-1ba12-predictor-74579d997-hmg4m"]
Sep 16 19:00:52.727476 ip-10-0-138-127 kubenswrapper[2581]: I0916 19:00:52.727444    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1035c64c-18a1-43f0-9e88-c06f78ad4591" path="/var/lib/kubelet/pods/1035c64c-18a1-43f0-9e88-c06f78ad4591/volumes"
Sep 16 19:00:52.782345 ip-10-0-138-127 kubenswrapper[2581]: I0916 19:00:52.782312    2581 generic.go:356] "Generic (PLEG): container finished" podID="0be1ad70-7a20-4b06-a2cc-a6f3a5024e8d" containerID="4b9822b37c73d77de4beb2ea431a9be774f32023268cbc9036d1c3e2160d1ba4" exitCode=0
Sep 16 19:00:52.782494 ip-10-0-138-127 kubenswrapper[2581]: I0916 19:00:52.782393    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-4959b-predictor-7554894dd5-5wf2g" event={"ID":"0be1ad70-7a20-4b06-a2cc-a6f3a5024e8d","Type":"ContainerDied","Data":"4b9822b37c73d77de4beb2ea431a9be774f32023268cbc9036d1c3e2160d1ba4"}
Sep 16 19:00:52.784043 ip-10-0-138-127 kubenswrapper[2581]: I0916 19:00:52.784023    2581 generic.go:356] "Generic (PLEG): container finished" podID="a76ee980-74ce-4370-a7f8-3e204b8bbb5c" containerID="a279cdfa735f47d4b0ab58401066250ee75d8c09e19a5462533a20aa46f47d9f" exitCode=0
Sep 16 19:00:52.784112 ip-10-0-138-127 kubenswrapper[2581]: I0916 19:00:52.784095    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-4959b-predictor-7544df7b5d-899mk" event={"ID":"a76ee980-74ce-4370-a7f8-3e204b8bbb5c","Type":"ContainerDied","Data":"a279cdfa735f47d4b0ab58401066250ee75d8c09e19a5462533a20aa46f47d9f"}
Sep 16 19:00:53.792337 ip-10-0-138-127 kubenswrapper[2581]: I0916 19:00:53.792295    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-4959b-predictor-7554894dd5-5wf2g" event={"ID":"0be1ad70-7a20-4b06-a2cc-a6f3a5024e8d","Type":"ContainerStarted","Data":"300e8513c073ec17efc2ca1146d065144462ab09f6cb55681b6b436c9b60f186"}
Sep 16 19:00:53.792828 ip-10-0-138-127 kubenswrapper[2581]: I0916 19:00:53.792418    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-4959b-predictor-7554894dd5-5wf2g" event={"ID":"0be1ad70-7a20-4b06-a2cc-a6f3a5024e8d","Type":"ContainerStarted","Data":"7a262fb50f01d23f6cca728c240b1d40ed3f86a7c571a367d27704e4d4ad3441"}
Sep 16 19:00:53.792828 ip-10-0-138-127 kubenswrapper[2581]: I0916 19:00:53.792644    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-4959b-predictor-7554894dd5-5wf2g"
Sep 16 19:00:53.794573 ip-10-0-138-127 kubenswrapper[2581]: I0916 19:00:53.794551    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-4959b-predictor-7544df7b5d-899mk" event={"ID":"a76ee980-74ce-4370-a7f8-3e204b8bbb5c","Type":"ContainerStarted","Data":"dd806a67ebe2cd99ba28fd6afda9efa0de9fc81f948baae00048b25857972633"}
Sep 16 19:00:53.794689 ip-10-0-138-127 kubenswrapper[2581]: I0916 19:00:53.794596    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-4959b-predictor-7544df7b5d-899mk" event={"ID":"a76ee980-74ce-4370-a7f8-3e204b8bbb5c","Type":"ContainerStarted","Data":"0831b63490f6606b0db244c4e889045ea00ffb22586c8f872c7d0537465b00dc"}
Sep 16 19:00:53.794891 ip-10-0-138-127 kubenswrapper[2581]: I0916 19:00:53.794872    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-4959b-predictor-7544df7b5d-899mk"
Sep 16 19:00:53.794929 ip-10-0-138-127 kubenswrapper[2581]: I0916 19:00:53.794918    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-4959b-predictor-7544df7b5d-899mk"
Sep 16 19:00:53.796241 ip-10-0-138-127 kubenswrapper[2581]: I0916 19:00:53.796218    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-4959b-predictor-7544df7b5d-899mk" podUID="a76ee980-74ce-4370-a7f8-3e204b8bbb5c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.30:8080: connect: connection refused"
Sep 16 19:00:53.818521 ip-10-0-138-127 kubenswrapper[2581]: I0916 19:00:53.818468    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-4959b-predictor-7554894dd5-5wf2g" podStartSLOduration=3.021360875 podStartE2EDuration="6.818452612s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.797091737s" isStatefulPod=false podCreationTimestamp="2026-09-16 19:00:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:00:53.817168066 +0000 UTC m=+923.636581976" watchObservedRunningTime="2026-09-16 19:00:53.818452612 +0000 UTC m=+923.637866521"
Sep 16 19:00:53.841921 ip-10-0-138-127 kubenswrapper[2581]: I0916 19:00:53.841868    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-4959b-predictor-7544df7b5d-899mk" podStartSLOduration=2.978525437 podStartE2EDuration="6.841850644s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.863325207s" isStatefulPod=false podCreationTimestamp="2026-09-16 19:00:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:00:53.840776403 +0000 UTC m=+923.660190315" watchObservedRunningTime="2026-09-16 19:00:53.841850644 +0000 UTC m=+923.661264553"
Sep 16 19:00:54.799982 ip-10-0-138-127 kubenswrapper[2581]: I0916 19:00:54.799954    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-4959b-predictor-7554894dd5-5wf2g"
Sep 16 19:00:54.800591 ip-10-0-138-127 kubenswrapper[2581]: I0916 19:00:54.800027    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-4959b-predictor-7544df7b5d-899mk" podUID="a76ee980-74ce-4370-a7f8-3e204b8bbb5c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.30:8080: connect: connection refused"
Sep 16 19:00:54.801258 ip-10-0-138-127 kubenswrapper[2581]: I0916 19:00:54.801235    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-4959b-predictor-7554894dd5-5wf2g" podUID="0be1ad70-7a20-4b06-a2cc-a6f3a5024e8d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.31:8080: connect: connection refused"
Sep 16 19:00:55.804644 ip-10-0-138-127 kubenswrapper[2581]: I0916 19:00:55.804607    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-4959b-predictor-7554894dd5-5wf2g" podUID="0be1ad70-7a20-4b06-a2cc-a6f3a5024e8d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.31:8080: connect: connection refused"
Sep 16 19:00:59.745465 ip-10-0-138-127 kubenswrapper[2581]: I0916 19:00:59.745417    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-6cbd7d9f59-ks6sw" podUID="64110e03-5cab-44aa-bdd3-f7a937cf46f2" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.29:8080: connect: connection refused"
Sep 16 19:00:59.804772 ip-10-0-138-127 kubenswrapper[2581]: I0916 19:00:59.804742    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-4959b-predictor-7544df7b5d-899mk"
Sep 16 19:00:59.805401 ip-10-0-138-127 kubenswrapper[2581]: I0916 19:00:59.805370    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-4959b-predictor-7544df7b5d-899mk" podUID="a76ee980-74ce-4370-a7f8-3e204b8bbb5c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.30:8080: connect: connection refused"
Sep 16 19:01:00.809525 ip-10-0-138-127 kubenswrapper[2581]: I0916 19:01:00.809492    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-4959b-predictor-7554894dd5-5wf2g"
Sep 16 19:01:00.809959 ip-10-0-138-127 kubenswrapper[2581]: I0916 19:01:00.809931    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-4959b-predictor-7554894dd5-5wf2g" podUID="0be1ad70-7a20-4b06-a2cc-a6f3a5024e8d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.31:8080: connect: connection refused"
Sep 16 19:01:09.745891 ip-10-0-138-127 kubenswrapper[2581]: I0916 19:01:09.745842    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-6cbd7d9f59-ks6sw" podUID="64110e03-5cab-44aa-bdd3-f7a937cf46f2" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.29:8080: connect: connection refused"
Sep 16 19:01:09.805508 ip-10-0-138-127 kubenswrapper[2581]: I0916 19:01:09.805469    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-4959b-predictor-7544df7b5d-899mk" podUID="a76ee980-74ce-4370-a7f8-3e204b8bbb5c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.30:8080: connect: connection refused"
Sep 16 19:01:10.810006 ip-10-0-138-127 kubenswrapper[2581]: I0916 19:01:10.809963    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-4959b-predictor-7554894dd5-5wf2g" podUID="0be1ad70-7a20-4b06-a2cc-a6f3a5024e8d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.31:8080: connect: connection refused"
Sep 16 19:01:19.745962 ip-10-0-138-127 kubenswrapper[2581]: I0916 19:01:19.745917    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-6cbd7d9f59-ks6sw" podUID="64110e03-5cab-44aa-bdd3-f7a937cf46f2" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.29:8080: connect: connection refused"
Sep 16 19:01:19.805307 ip-10-0-138-127 kubenswrapper[2581]: I0916 19:01:19.805269    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-4959b-predictor-7544df7b5d-899mk" podUID="a76ee980-74ce-4370-a7f8-3e204b8bbb5c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.30:8080: connect: connection refused"
Sep 16 19:01:20.813551 ip-10-0-138-127 kubenswrapper[2581]: I0916 19:01:20.813503    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-4959b-predictor-7554894dd5-5wf2g" podUID="0be1ad70-7a20-4b06-a2cc-a6f3a5024e8d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.31:8080: connect: connection refused"
Sep 16 19:01:29.745670 ip-10-0-138-127 kubenswrapper[2581]: I0916 19:01:29.745622    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-6cbd7d9f59-ks6sw" podUID="64110e03-5cab-44aa-bdd3-f7a937cf46f2" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.29:8080: connect: connection refused"
Sep 16 19:01:29.805411 ip-10-0-138-127 kubenswrapper[2581]: I0916 19:01:29.805372    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-4959b-predictor-7544df7b5d-899mk" podUID="a76ee980-74ce-4370-a7f8-3e204b8bbb5c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.30:8080: connect: connection refused"
Sep 16 19:01:30.810425 ip-10-0-138-127 kubenswrapper[2581]: I0916 19:01:30.810380    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-4959b-predictor-7554894dd5-5wf2g" podUID="0be1ad70-7a20-4b06-a2cc-a6f3a5024e8d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.31:8080: connect: connection refused"
Sep 16 19:01:39.746374 ip-10-0-138-127 kubenswrapper[2581]: I0916 19:01:39.746327    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-6cbd7d9f59-ks6sw" podUID="64110e03-5cab-44aa-bdd3-f7a937cf46f2" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.29:8080: connect: connection refused"
Sep 16 19:01:39.805963 ip-10-0-138-127 kubenswrapper[2581]: I0916 19:01:39.805923    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-4959b-predictor-7544df7b5d-899mk" podUID="a76ee980-74ce-4370-a7f8-3e204b8bbb5c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.30:8080: connect: connection refused"
Sep 16 19:01:40.810014 ip-10-0-138-127 kubenswrapper[2581]: I0916 19:01:40.809974    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-4959b-predictor-7554894dd5-5wf2g" podUID="0be1ad70-7a20-4b06-a2cc-a6f3a5024e8d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.31:8080: connect: connection refused"
Sep 16 19:01:49.745504 ip-10-0-138-127 kubenswrapper[2581]: I0916 19:01:49.745464    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-6cbd7d9f59-ks6sw" podUID="64110e03-5cab-44aa-bdd3-f7a937cf46f2" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.29:8080: connect: connection refused"
Sep 16 19:01:49.805421 ip-10-0-138-127 kubenswrapper[2581]: I0916 19:01:49.805381    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-4959b-predictor-7544df7b5d-899mk" podUID="a76ee980-74ce-4370-a7f8-3e204b8bbb5c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.30:8080: connect: connection refused"
Sep 16 19:01:50.810170 ip-10-0-138-127 kubenswrapper[2581]: I0916 19:01:50.810130    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-4959b-predictor-7554894dd5-5wf2g" podUID="0be1ad70-7a20-4b06-a2cc-a6f3a5024e8d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.31:8080: connect: connection refused"
Sep 16 19:01:59.361568 ip-10-0-138-127 kubenswrapper[2581]: I0916 19:01:59.361477    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-stop-predictor-6cbd7d9f59-ks6sw"]
Sep 16 19:01:59.362002 ip-10-0-138-127 kubenswrapper[2581]: I0916 19:01:59.361791    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-6cbd7d9f59-ks6sw" podUID="64110e03-5cab-44aa-bdd3-f7a937cf46f2" containerName="kserve-container" containerID="cri-o://53bdee0638892d715499027d504d4004f701a1a3760118579cf72b01605e3d97" gracePeriod=30
Sep 16 19:01:59.362002 ip-10-0-138-127 kubenswrapper[2581]: I0916 19:01:59.361860    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-6cbd7d9f59-ks6sw" podUID="64110e03-5cab-44aa-bdd3-f7a937cf46f2" containerName="kube-rbac-proxy" containerID="cri-o://ddf713e4ecda7f018ff13e9c3ff9b03b9df879757b83f062ef0e25c1c883ced6" gracePeriod=30
Sep 16 19:01:59.362645 ip-10-0-138-127 kubenswrapper[2581]: I0916 19:01:59.362353    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-6cbd7d9f59-ks6sw"
Sep 16 19:01:59.734920 ip-10-0-138-127 kubenswrapper[2581]: I0916 19:01:59.734828    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-6cbd7d9f59-ks6sw" podUID="64110e03-5cab-44aa-bdd3-f7a937cf46f2" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.29:8643/healthz\": dial tcp 10.132.0.29:8643: connect: connection refused"
Sep 16 19:01:59.805937 ip-10-0-138-127 kubenswrapper[2581]: I0916 19:01:59.805885    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-4959b-predictor-7544df7b5d-899mk" podUID="a76ee980-74ce-4370-a7f8-3e204b8bbb5c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.30:8080: connect: connection refused"
Sep 16 19:02:00.122489 ip-10-0-138-127 kubenswrapper[2581]: I0916 19:02:00.122457    2581 generic.go:356] "Generic (PLEG): container finished" podID="64110e03-5cab-44aa-bdd3-f7a937cf46f2" containerID="ddf713e4ecda7f018ff13e9c3ff9b03b9df879757b83f062ef0e25c1c883ced6" exitCode=2
Sep 16 19:02:00.122669 ip-10-0-138-127 kubenswrapper[2581]: I0916 19:02:00.122529    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-6cbd7d9f59-ks6sw" event={"ID":"64110e03-5cab-44aa-bdd3-f7a937cf46f2","Type":"ContainerDied","Data":"ddf713e4ecda7f018ff13e9c3ff9b03b9df879757b83f062ef0e25c1c883ced6"}
Sep 16 19:02:00.811161 ip-10-0-138-127 kubenswrapper[2581]: I0916 19:02:00.811134    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-4959b-predictor-7554894dd5-5wf2g"
Sep 16 19:02:04.413454 ip-10-0-138-127 kubenswrapper[2581]: I0916 19:02:04.413430    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-6cbd7d9f59-ks6sw"
Sep 16 19:02:04.519111 ip-10-0-138-127 kubenswrapper[2581]: I0916 19:02:04.519084    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/64110e03-5cab-44aa-bdd3-f7a937cf46f2-proxy-tls\") pod \"64110e03-5cab-44aa-bdd3-f7a937cf46f2\" (UID: \"64110e03-5cab-44aa-bdd3-f7a937cf46f2\") "
Sep 16 19:02:04.519306 ip-10-0-138-127 kubenswrapper[2581]: I0916 19:02:04.519125    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/64110e03-5cab-44aa-bdd3-f7a937cf46f2-kube-rbac-proxy-sar-config\") pod \"64110e03-5cab-44aa-bdd3-f7a937cf46f2\" (UID: \"64110e03-5cab-44aa-bdd3-f7a937cf46f2\") "
Sep 16 19:02:04.519306 ip-10-0-138-127 kubenswrapper[2581]: I0916 19:02:04.519149    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/64110e03-5cab-44aa-bdd3-f7a937cf46f2-cabundle-cert\") pod \"64110e03-5cab-44aa-bdd3-f7a937cf46f2\" (UID: \"64110e03-5cab-44aa-bdd3-f7a937cf46f2\") "
Sep 16 19:02:04.519306 ip-10-0-138-127 kubenswrapper[2581]: I0916 19:02:04.519221    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-tcrld\" (UniqueName: \"kubernetes.io/projected/64110e03-5cab-44aa-bdd3-f7a937cf46f2-kube-api-access-tcrld\") pod \"64110e03-5cab-44aa-bdd3-f7a937cf46f2\" (UID: \"64110e03-5cab-44aa-bdd3-f7a937cf46f2\") "
Sep 16 19:02:04.519306 ip-10-0-138-127 kubenswrapper[2581]: I0916 19:02:04.519251    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/64110e03-5cab-44aa-bdd3-f7a937cf46f2-kserve-provision-location\") pod \"64110e03-5cab-44aa-bdd3-f7a937cf46f2\" (UID: \"64110e03-5cab-44aa-bdd3-f7a937cf46f2\") "
Sep 16 19:02:04.519681 ip-10-0-138-127 kubenswrapper[2581]: I0916 19:02:04.519651    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/64110e03-5cab-44aa-bdd3-f7a937cf46f2-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "64110e03-5cab-44aa-bdd3-f7a937cf46f2" (UID: "64110e03-5cab-44aa-bdd3-f7a937cf46f2"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 19:02:04.519749 ip-10-0-138-127 kubenswrapper[2581]: I0916 19:02:04.519651    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/64110e03-5cab-44aa-bdd3-f7a937cf46f2-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "64110e03-5cab-44aa-bdd3-f7a937cf46f2" (UID: "64110e03-5cab-44aa-bdd3-f7a937cf46f2"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:02:04.519749 ip-10-0-138-127 kubenswrapper[2581]: I0916 19:02:04.519657    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/64110e03-5cab-44aa-bdd3-f7a937cf46f2-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "64110e03-5cab-44aa-bdd3-f7a937cf46f2" (UID: "64110e03-5cab-44aa-bdd3-f7a937cf46f2"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 19:02:04.521514 ip-10-0-138-127 kubenswrapper[2581]: I0916 19:02:04.521486    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/64110e03-5cab-44aa-bdd3-f7a937cf46f2-kube-api-access-tcrld" (OuterVolumeSpecName: "kube-api-access-tcrld") pod "64110e03-5cab-44aa-bdd3-f7a937cf46f2" (UID: "64110e03-5cab-44aa-bdd3-f7a937cf46f2"). InnerVolumeSpecName "kube-api-access-tcrld". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 19:02:04.521602 ip-10-0-138-127 kubenswrapper[2581]: I0916 19:02:04.521526    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/64110e03-5cab-44aa-bdd3-f7a937cf46f2-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "64110e03-5cab-44aa-bdd3-f7a937cf46f2" (UID: "64110e03-5cab-44aa-bdd3-f7a937cf46f2"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 19:02:04.620075 ip-10-0-138-127 kubenswrapper[2581]: I0916 19:02:04.620014    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tcrld\" (UniqueName: \"kubernetes.io/projected/64110e03-5cab-44aa-bdd3-f7a937cf46f2-kube-api-access-tcrld\") on node \"ip-10-0-138-127.ec2.internal\" DevicePath \"\""
Sep 16 19:02:04.620075 ip-10-0-138-127 kubenswrapper[2581]: I0916 19:02:04.620066    2581 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/64110e03-5cab-44aa-bdd3-f7a937cf46f2-kserve-provision-location\") on node \"ip-10-0-138-127.ec2.internal\" DevicePath \"\""
Sep 16 19:02:04.620075 ip-10-0-138-127 kubenswrapper[2581]: I0916 19:02:04.620077    2581 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/64110e03-5cab-44aa-bdd3-f7a937cf46f2-proxy-tls\") on node \"ip-10-0-138-127.ec2.internal\" DevicePath \"\""
Sep 16 19:02:04.620075 ip-10-0-138-127 kubenswrapper[2581]: I0916 19:02:04.620086    2581 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/64110e03-5cab-44aa-bdd3-f7a937cf46f2-kube-rbac-proxy-sar-config\") on node \"ip-10-0-138-127.ec2.internal\" DevicePath \"\""
Sep 16 19:02:04.620075 ip-10-0-138-127 kubenswrapper[2581]: I0916 19:02:04.620095    2581 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/64110e03-5cab-44aa-bdd3-f7a937cf46f2-cabundle-cert\") on node \"ip-10-0-138-127.ec2.internal\" DevicePath \"\""
Sep 16 19:02:05.149000 ip-10-0-138-127 kubenswrapper[2581]: I0916 19:02:05.148964    2581 generic.go:356] "Generic (PLEG): container finished" podID="64110e03-5cab-44aa-bdd3-f7a937cf46f2" containerID="53bdee0638892d715499027d504d4004f701a1a3760118579cf72b01605e3d97" exitCode=0
Sep 16 19:02:05.149229 ip-10-0-138-127 kubenswrapper[2581]: I0916 19:02:05.149018    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-6cbd7d9f59-ks6sw"
Sep 16 19:02:05.149229 ip-10-0-138-127 kubenswrapper[2581]: I0916 19:02:05.149055    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-6cbd7d9f59-ks6sw" event={"ID":"64110e03-5cab-44aa-bdd3-f7a937cf46f2","Type":"ContainerDied","Data":"53bdee0638892d715499027d504d4004f701a1a3760118579cf72b01605e3d97"}
Sep 16 19:02:05.149229 ip-10-0-138-127 kubenswrapper[2581]: I0916 19:02:05.149122    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-6cbd7d9f59-ks6sw" event={"ID":"64110e03-5cab-44aa-bdd3-f7a937cf46f2","Type":"ContainerDied","Data":"72ca1a35b0c977a738a03be4b07acc36a0816fab11299622f4d9da7a329fa813"}
Sep 16 19:02:05.149229 ip-10-0-138-127 kubenswrapper[2581]: I0916 19:02:05.149149    2581 scope.go:117] "RemoveContainer" containerID="ddf713e4ecda7f018ff13e9c3ff9b03b9df879757b83f062ef0e25c1c883ced6"
Sep 16 19:02:05.157855 ip-10-0-138-127 kubenswrapper[2581]: I0916 19:02:05.157836    2581 scope.go:117] "RemoveContainer" containerID="53bdee0638892d715499027d504d4004f701a1a3760118579cf72b01605e3d97"
Sep 16 19:02:05.165710 ip-10-0-138-127 kubenswrapper[2581]: I0916 19:02:05.165689    2581 scope.go:117] "RemoveContainer" containerID="7b8963f2daef3b7f9e5667ebe18aeab1e516cfb080d0adc82b7e14f47f9c0923"
Sep 16 19:02:05.171128 ip-10-0-138-127 kubenswrapper[2581]: I0916 19:02:05.171099    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-stop-predictor-6cbd7d9f59-ks6sw"]
Sep 16 19:02:05.174562 ip-10-0-138-127 kubenswrapper[2581]: I0916 19:02:05.174542    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-stop-predictor-6cbd7d9f59-ks6sw"]
Sep 16 19:02:05.174965 ip-10-0-138-127 kubenswrapper[2581]: I0916 19:02:05.174936    2581 scope.go:117] "RemoveContainer" containerID="ddf713e4ecda7f018ff13e9c3ff9b03b9df879757b83f062ef0e25c1c883ced6"
Sep 16 19:02:05.175254 ip-10-0-138-127 kubenswrapper[2581]: E0916 19:02:05.175232    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ddf713e4ecda7f018ff13e9c3ff9b03b9df879757b83f062ef0e25c1c883ced6\": container with ID starting with ddf713e4ecda7f018ff13e9c3ff9b03b9df879757b83f062ef0e25c1c883ced6 not found: ID does not exist" containerID="ddf713e4ecda7f018ff13e9c3ff9b03b9df879757b83f062ef0e25c1c883ced6"
Sep 16 19:02:05.175356 ip-10-0-138-127 kubenswrapper[2581]: I0916 19:02:05.175267    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ddf713e4ecda7f018ff13e9c3ff9b03b9df879757b83f062ef0e25c1c883ced6"} err="failed to get container status \"ddf713e4ecda7f018ff13e9c3ff9b03b9df879757b83f062ef0e25c1c883ced6\": rpc error: code = NotFound desc = could not find container \"ddf713e4ecda7f018ff13e9c3ff9b03b9df879757b83f062ef0e25c1c883ced6\": container with ID starting with ddf713e4ecda7f018ff13e9c3ff9b03b9df879757b83f062ef0e25c1c883ced6 not found: ID does not exist"
Sep 16 19:02:05.175356 ip-10-0-138-127 kubenswrapper[2581]: I0916 19:02:05.175285    2581 scope.go:117] "RemoveContainer" containerID="53bdee0638892d715499027d504d4004f701a1a3760118579cf72b01605e3d97"
Sep 16 19:02:05.175526 ip-10-0-138-127 kubenswrapper[2581]: E0916 19:02:05.175510    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"53bdee0638892d715499027d504d4004f701a1a3760118579cf72b01605e3d97\": container with ID starting with 53bdee0638892d715499027d504d4004f701a1a3760118579cf72b01605e3d97 not found: ID does not exist" containerID="53bdee0638892d715499027d504d4004f701a1a3760118579cf72b01605e3d97"
Sep 16 19:02:05.175572 ip-10-0-138-127 kubenswrapper[2581]: I0916 19:02:05.175530    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"53bdee0638892d715499027d504d4004f701a1a3760118579cf72b01605e3d97"} err="failed to get container status \"53bdee0638892d715499027d504d4004f701a1a3760118579cf72b01605e3d97\": rpc error: code = NotFound desc = could not find container \"53bdee0638892d715499027d504d4004f701a1a3760118579cf72b01605e3d97\": container with ID starting with 53bdee0638892d715499027d504d4004f701a1a3760118579cf72b01605e3d97 not found: ID does not exist"
Sep 16 19:02:05.175572 ip-10-0-138-127 kubenswrapper[2581]: I0916 19:02:05.175543    2581 scope.go:117] "RemoveContainer" containerID="7b8963f2daef3b7f9e5667ebe18aeab1e516cfb080d0adc82b7e14f47f9c0923"
Sep 16 19:02:05.175787 ip-10-0-138-127 kubenswrapper[2581]: E0916 19:02:05.175771    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7b8963f2daef3b7f9e5667ebe18aeab1e516cfb080d0adc82b7e14f47f9c0923\": container with ID starting with 7b8963f2daef3b7f9e5667ebe18aeab1e516cfb080d0adc82b7e14f47f9c0923 not found: ID does not exist" containerID="7b8963f2daef3b7f9e5667ebe18aeab1e516cfb080d0adc82b7e14f47f9c0923"
Sep 16 19:02:05.175838 ip-10-0-138-127 kubenswrapper[2581]: I0916 19:02:05.175793    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7b8963f2daef3b7f9e5667ebe18aeab1e516cfb080d0adc82b7e14f47f9c0923"} err="failed to get container status \"7b8963f2daef3b7f9e5667ebe18aeab1e516cfb080d0adc82b7e14f47f9c0923\": rpc error: code = NotFound desc = could not find container \"7b8963f2daef3b7f9e5667ebe18aeab1e516cfb080d0adc82b7e14f47f9c0923\": container with ID starting with 7b8963f2daef3b7f9e5667ebe18aeab1e516cfb080d0adc82b7e14f47f9c0923 not found: ID does not exist"
Sep 16 19:02:06.727311 ip-10-0-138-127 kubenswrapper[2581]: I0916 19:02:06.727274    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="64110e03-5cab-44aa-bdd3-f7a937cf46f2" path="/var/lib/kubelet/pods/64110e03-5cab-44aa-bdd3-f7a937cf46f2/volumes"
Sep 16 19:02:08.723289 ip-10-0-138-127 kubenswrapper[2581]: I0916 19:02:08.723252    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-4959b-predictor-7544df7b5d-899mk" podUID="a76ee980-74ce-4370-a7f8-3e204b8bbb5c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.30:8080: connect: connection refused"
Sep 16 19:02:18.727078 ip-10-0-138-127 kubenswrapper[2581]: I0916 19:02:18.727051    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-4959b-predictor-7544df7b5d-899mk"
Sep 16 19:02:47.419955 ip-10-0-138-127 kubenswrapper[2581]: I0916 19:02:47.419915    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-4959b-predictor-7544df7b5d-899mk"]
Sep 16 19:02:47.420558 ip-10-0-138-127 kubenswrapper[2581]: I0916 19:02:47.420177    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-4959b-predictor-7544df7b5d-899mk" podUID="a76ee980-74ce-4370-a7f8-3e204b8bbb5c" containerName="kserve-container" containerID="cri-o://0831b63490f6606b0db244c4e889045ea00ffb22586c8f872c7d0537465b00dc" gracePeriod=30
Sep 16 19:02:47.420558 ip-10-0-138-127 kubenswrapper[2581]: I0916 19:02:47.420232    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-4959b-predictor-7544df7b5d-899mk" podUID="a76ee980-74ce-4370-a7f8-3e204b8bbb5c" containerName="kube-rbac-proxy" containerID="cri-o://dd806a67ebe2cd99ba28fd6afda9efa0de9fc81f948baae00048b25857972633" gracePeriod=30
Sep 16 19:02:47.536548 ip-10-0-138-127 kubenswrapper[2581]: I0916 19:02:47.536519    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-4959b-predictor-7554894dd5-5wf2g"]
Sep 16 19:02:47.536821 ip-10-0-138-127 kubenswrapper[2581]: I0916 19:02:47.536796    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-4959b-predictor-7554894dd5-5wf2g" podUID="0be1ad70-7a20-4b06-a2cc-a6f3a5024e8d" containerName="kube-rbac-proxy" containerID="cri-o://300e8513c073ec17efc2ca1146d065144462ab09f6cb55681b6b436c9b60f186" gracePeriod=30
Sep 16 19:02:47.536906 ip-10-0-138-127 kubenswrapper[2581]: I0916 19:02:47.536775    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-4959b-predictor-7554894dd5-5wf2g" podUID="0be1ad70-7a20-4b06-a2cc-a6f3a5024e8d" containerName="kserve-container" containerID="cri-o://7a262fb50f01d23f6cca728c240b1d40ed3f86a7c571a367d27704e4d4ad3441" gracePeriod=30
Sep 16 19:02:48.332301 ip-10-0-138-127 kubenswrapper[2581]: I0916 19:02:48.332260    2581 generic.go:356] "Generic (PLEG): container finished" podID="0be1ad70-7a20-4b06-a2cc-a6f3a5024e8d" containerID="300e8513c073ec17efc2ca1146d065144462ab09f6cb55681b6b436c9b60f186" exitCode=2
Sep 16 19:02:48.332488 ip-10-0-138-127 kubenswrapper[2581]: I0916 19:02:48.332334    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-4959b-predictor-7554894dd5-5wf2g" event={"ID":"0be1ad70-7a20-4b06-a2cc-a6f3a5024e8d","Type":"ContainerDied","Data":"300e8513c073ec17efc2ca1146d065144462ab09f6cb55681b6b436c9b60f186"}
Sep 16 19:02:48.334527 ip-10-0-138-127 kubenswrapper[2581]: I0916 19:02:48.334503    2581 generic.go:356] "Generic (PLEG): container finished" podID="a76ee980-74ce-4370-a7f8-3e204b8bbb5c" containerID="dd806a67ebe2cd99ba28fd6afda9efa0de9fc81f948baae00048b25857972633" exitCode=2
Sep 16 19:02:48.334652 ip-10-0-138-127 kubenswrapper[2581]: I0916 19:02:48.334555    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-4959b-predictor-7544df7b5d-899mk" event={"ID":"a76ee980-74ce-4370-a7f8-3e204b8bbb5c","Type":"ContainerDied","Data":"dd806a67ebe2cd99ba28fd6afda9efa0de9fc81f948baae00048b25857972633"}
Sep 16 19:02:48.723837 ip-10-0-138-127 kubenswrapper[2581]: I0916 19:02:48.723742    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-4959b-predictor-7544df7b5d-899mk" podUID="a76ee980-74ce-4370-a7f8-3e204b8bbb5c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.30:8080: connect: connection refused"
Sep 16 19:02:49.800619 ip-10-0-138-127 kubenswrapper[2581]: I0916 19:02:49.800570    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-4959b-predictor-7544df7b5d-899mk" podUID="a76ee980-74ce-4370-a7f8-3e204b8bbb5c" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.30:8643/healthz\": dial tcp 10.132.0.30:8643: connect: connection refused"
Sep 16 19:02:50.805581 ip-10-0-138-127 kubenswrapper[2581]: I0916 19:02:50.805537    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-4959b-predictor-7554894dd5-5wf2g" podUID="0be1ad70-7a20-4b06-a2cc-a6f3a5024e8d" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.31:8643/healthz\": dial tcp 10.132.0.31:8643: connect: connection refused"
Sep 16 19:02:50.810925 ip-10-0-138-127 kubenswrapper[2581]: I0916 19:02:50.810894    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-4959b-predictor-7554894dd5-5wf2g" podUID="0be1ad70-7a20-4b06-a2cc-a6f3a5024e8d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.31:8080: connect: connection refused"
Sep 16 19:02:51.589546 ip-10-0-138-127 kubenswrapper[2581]: I0916 19:02:51.589516    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-4959b-predictor-7554894dd5-5wf2g"
Sep 16 19:02:51.721438 ip-10-0-138-127 kubenswrapper[2581]: I0916 19:02:51.721349    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/0be1ad70-7a20-4b06-a2cc-a6f3a5024e8d-cabundle-cert\") pod \"0be1ad70-7a20-4b06-a2cc-a6f3a5024e8d\" (UID: \"0be1ad70-7a20-4b06-a2cc-a6f3a5024e8d\") "
Sep 16 19:02:51.721438 ip-10-0-138-127 kubenswrapper[2581]: I0916 19:02:51.721418    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-csbz7\" (UniqueName: \"kubernetes.io/projected/0be1ad70-7a20-4b06-a2cc-a6f3a5024e8d-kube-api-access-csbz7\") pod \"0be1ad70-7a20-4b06-a2cc-a6f3a5024e8d\" (UID: \"0be1ad70-7a20-4b06-a2cc-a6f3a5024e8d\") "
Sep 16 19:02:51.721681 ip-10-0-138-127 kubenswrapper[2581]: I0916 19:02:51.721443    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/0be1ad70-7a20-4b06-a2cc-a6f3a5024e8d-kserve-provision-location\") pod \"0be1ad70-7a20-4b06-a2cc-a6f3a5024e8d\" (UID: \"0be1ad70-7a20-4b06-a2cc-a6f3a5024e8d\") "
Sep 16 19:02:51.721681 ip-10-0-138-127 kubenswrapper[2581]: I0916 19:02:51.721515    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/0be1ad70-7a20-4b06-a2cc-a6f3a5024e8d-proxy-tls\") pod \"0be1ad70-7a20-4b06-a2cc-a6f3a5024e8d\" (UID: \"0be1ad70-7a20-4b06-a2cc-a6f3a5024e8d\") "
Sep 16 19:02:51.721681 ip-10-0-138-127 kubenswrapper[2581]: I0916 19:02:51.721532    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/0be1ad70-7a20-4b06-a2cc-a6f3a5024e8d-kube-rbac-proxy-sar-config\") pod \"0be1ad70-7a20-4b06-a2cc-a6f3a5024e8d\" (UID: \"0be1ad70-7a20-4b06-a2cc-a6f3a5024e8d\") "
Sep 16 19:02:51.721849 ip-10-0-138-127 kubenswrapper[2581]: I0916 19:02:51.721767    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0be1ad70-7a20-4b06-a2cc-a6f3a5024e8d-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "0be1ad70-7a20-4b06-a2cc-a6f3a5024e8d" (UID: "0be1ad70-7a20-4b06-a2cc-a6f3a5024e8d"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 19:02:51.721913 ip-10-0-138-127 kubenswrapper[2581]: I0916 19:02:51.721893    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0be1ad70-7a20-4b06-a2cc-a6f3a5024e8d-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "0be1ad70-7a20-4b06-a2cc-a6f3a5024e8d" (UID: "0be1ad70-7a20-4b06-a2cc-a6f3a5024e8d"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:02:51.722004 ip-10-0-138-127 kubenswrapper[2581]: I0916 19:02:51.721983    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0be1ad70-7a20-4b06-a2cc-a6f3a5024e8d-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "0be1ad70-7a20-4b06-a2cc-a6f3a5024e8d" (UID: "0be1ad70-7a20-4b06-a2cc-a6f3a5024e8d"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 19:02:51.723817 ip-10-0-138-127 kubenswrapper[2581]: I0916 19:02:51.723798    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0be1ad70-7a20-4b06-a2cc-a6f3a5024e8d-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "0be1ad70-7a20-4b06-a2cc-a6f3a5024e8d" (UID: "0be1ad70-7a20-4b06-a2cc-a6f3a5024e8d"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 19:02:51.723817 ip-10-0-138-127 kubenswrapper[2581]: I0916 19:02:51.723810    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0be1ad70-7a20-4b06-a2cc-a6f3a5024e8d-kube-api-access-csbz7" (OuterVolumeSpecName: "kube-api-access-csbz7") pod "0be1ad70-7a20-4b06-a2cc-a6f3a5024e8d" (UID: "0be1ad70-7a20-4b06-a2cc-a6f3a5024e8d"). InnerVolumeSpecName "kube-api-access-csbz7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 19:02:51.822384 ip-10-0-138-127 kubenswrapper[2581]: I0916 19:02:51.822341    2581 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/0be1ad70-7a20-4b06-a2cc-a6f3a5024e8d-proxy-tls\") on node \"ip-10-0-138-127.ec2.internal\" DevicePath \"\""
Sep 16 19:02:51.822384 ip-10-0-138-127 kubenswrapper[2581]: I0916 19:02:51.822375    2581 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/0be1ad70-7a20-4b06-a2cc-a6f3a5024e8d-kube-rbac-proxy-sar-config\") on node \"ip-10-0-138-127.ec2.internal\" DevicePath \"\""
Sep 16 19:02:51.822384 ip-10-0-138-127 kubenswrapper[2581]: I0916 19:02:51.822388    2581 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/0be1ad70-7a20-4b06-a2cc-a6f3a5024e8d-cabundle-cert\") on node \"ip-10-0-138-127.ec2.internal\" DevicePath \"\""
Sep 16 19:02:51.822843 ip-10-0-138-127 kubenswrapper[2581]: I0916 19:02:51.822402    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-csbz7\" (UniqueName: \"kubernetes.io/projected/0be1ad70-7a20-4b06-a2cc-a6f3a5024e8d-kube-api-access-csbz7\") on node \"ip-10-0-138-127.ec2.internal\" DevicePath \"\""
Sep 16 19:02:51.822843 ip-10-0-138-127 kubenswrapper[2581]: I0916 19:02:51.822414    2581 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/0be1ad70-7a20-4b06-a2cc-a6f3a5024e8d-kserve-provision-location\") on node \"ip-10-0-138-127.ec2.internal\" DevicePath \"\""
Sep 16 19:02:52.357528 ip-10-0-138-127 kubenswrapper[2581]: I0916 19:02:52.357496    2581 generic.go:356] "Generic (PLEG): container finished" podID="0be1ad70-7a20-4b06-a2cc-a6f3a5024e8d" containerID="7a262fb50f01d23f6cca728c240b1d40ed3f86a7c571a367d27704e4d4ad3441" exitCode=0
Sep 16 19:02:52.357717 ip-10-0-138-127 kubenswrapper[2581]: I0916 19:02:52.357552    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-4959b-predictor-7554894dd5-5wf2g"
Sep 16 19:02:52.357717 ip-10-0-138-127 kubenswrapper[2581]: I0916 19:02:52.357580    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-4959b-predictor-7554894dd5-5wf2g" event={"ID":"0be1ad70-7a20-4b06-a2cc-a6f3a5024e8d","Type":"ContainerDied","Data":"7a262fb50f01d23f6cca728c240b1d40ed3f86a7c571a367d27704e4d4ad3441"}
Sep 16 19:02:52.357717 ip-10-0-138-127 kubenswrapper[2581]: I0916 19:02:52.357658    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-4959b-predictor-7554894dd5-5wf2g" event={"ID":"0be1ad70-7a20-4b06-a2cc-a6f3a5024e8d","Type":"ContainerDied","Data":"2edbc02580a069e7df1728d0149f186610932332ef14cb3ff4e530bd3e543e20"}
Sep 16 19:02:52.357717 ip-10-0-138-127 kubenswrapper[2581]: I0916 19:02:52.357693    2581 scope.go:117] "RemoveContainer" containerID="300e8513c073ec17efc2ca1146d065144462ab09f6cb55681b6b436c9b60f186"
Sep 16 19:02:52.360207 ip-10-0-138-127 kubenswrapper[2581]: I0916 19:02:52.360160    2581 generic.go:356] "Generic (PLEG): container finished" podID="a76ee980-74ce-4370-a7f8-3e204b8bbb5c" containerID="0831b63490f6606b0db244c4e889045ea00ffb22586c8f872c7d0537465b00dc" exitCode=0
Sep 16 19:02:52.360327 ip-10-0-138-127 kubenswrapper[2581]: I0916 19:02:52.360227    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-4959b-predictor-7544df7b5d-899mk" event={"ID":"a76ee980-74ce-4370-a7f8-3e204b8bbb5c","Type":"ContainerDied","Data":"0831b63490f6606b0db244c4e889045ea00ffb22586c8f872c7d0537465b00dc"}
Sep 16 19:02:52.370214 ip-10-0-138-127 kubenswrapper[2581]: I0916 19:02:52.370178    2581 scope.go:117] "RemoveContainer" containerID="7a262fb50f01d23f6cca728c240b1d40ed3f86a7c571a367d27704e4d4ad3441"
Sep 16 19:02:52.380140 ip-10-0-138-127 kubenswrapper[2581]: I0916 19:02:52.380120    2581 scope.go:117] "RemoveContainer" containerID="4b9822b37c73d77de4beb2ea431a9be774f32023268cbc9036d1c3e2160d1ba4"
Sep 16 19:02:52.385294 ip-10-0-138-127 kubenswrapper[2581]: I0916 19:02:52.385248    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-4959b-predictor-7554894dd5-5wf2g"]
Sep 16 19:02:52.387168 ip-10-0-138-127 kubenswrapper[2581]: I0916 19:02:52.387144    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-4959b-predictor-7554894dd5-5wf2g"]
Sep 16 19:02:52.390918 ip-10-0-138-127 kubenswrapper[2581]: I0916 19:02:52.390894    2581 scope.go:117] "RemoveContainer" containerID="300e8513c073ec17efc2ca1146d065144462ab09f6cb55681b6b436c9b60f186"
Sep 16 19:02:52.391290 ip-10-0-138-127 kubenswrapper[2581]: E0916 19:02:52.391268    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"300e8513c073ec17efc2ca1146d065144462ab09f6cb55681b6b436c9b60f186\": container with ID starting with 300e8513c073ec17efc2ca1146d065144462ab09f6cb55681b6b436c9b60f186 not found: ID does not exist" containerID="300e8513c073ec17efc2ca1146d065144462ab09f6cb55681b6b436c9b60f186"
Sep 16 19:02:52.391381 ip-10-0-138-127 kubenswrapper[2581]: I0916 19:02:52.391309    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"300e8513c073ec17efc2ca1146d065144462ab09f6cb55681b6b436c9b60f186"} err="failed to get container status \"300e8513c073ec17efc2ca1146d065144462ab09f6cb55681b6b436c9b60f186\": rpc error: code = NotFound desc = could not find container \"300e8513c073ec17efc2ca1146d065144462ab09f6cb55681b6b436c9b60f186\": container with ID starting with 300e8513c073ec17efc2ca1146d065144462ab09f6cb55681b6b436c9b60f186 not found: ID does not exist"
Sep 16 19:02:52.391381 ip-10-0-138-127 kubenswrapper[2581]: I0916 19:02:52.391329    2581 scope.go:117] "RemoveContainer" containerID="7a262fb50f01d23f6cca728c240b1d40ed3f86a7c571a367d27704e4d4ad3441"
Sep 16 19:02:52.391667 ip-10-0-138-127 kubenswrapper[2581]: E0916 19:02:52.391647    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7a262fb50f01d23f6cca728c240b1d40ed3f86a7c571a367d27704e4d4ad3441\": container with ID starting with 7a262fb50f01d23f6cca728c240b1d40ed3f86a7c571a367d27704e4d4ad3441 not found: ID does not exist" containerID="7a262fb50f01d23f6cca728c240b1d40ed3f86a7c571a367d27704e4d4ad3441"
Sep 16 19:02:52.391741 ip-10-0-138-127 kubenswrapper[2581]: I0916 19:02:52.391676    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7a262fb50f01d23f6cca728c240b1d40ed3f86a7c571a367d27704e4d4ad3441"} err="failed to get container status \"7a262fb50f01d23f6cca728c240b1d40ed3f86a7c571a367d27704e4d4ad3441\": rpc error: code = NotFound desc = could not find container \"7a262fb50f01d23f6cca728c240b1d40ed3f86a7c571a367d27704e4d4ad3441\": container with ID starting with 7a262fb50f01d23f6cca728c240b1d40ed3f86a7c571a367d27704e4d4ad3441 not found: ID does not exist"
Sep 16 19:02:52.391741 ip-10-0-138-127 kubenswrapper[2581]: I0916 19:02:52.391700    2581 scope.go:117] "RemoveContainer" containerID="4b9822b37c73d77de4beb2ea431a9be774f32023268cbc9036d1c3e2160d1ba4"
Sep 16 19:02:52.392027 ip-10-0-138-127 kubenswrapper[2581]: E0916 19:02:52.392011    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4b9822b37c73d77de4beb2ea431a9be774f32023268cbc9036d1c3e2160d1ba4\": container with ID starting with 4b9822b37c73d77de4beb2ea431a9be774f32023268cbc9036d1c3e2160d1ba4 not found: ID does not exist" containerID="4b9822b37c73d77de4beb2ea431a9be774f32023268cbc9036d1c3e2160d1ba4"
Sep 16 19:02:52.392066 ip-10-0-138-127 kubenswrapper[2581]: I0916 19:02:52.392033    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4b9822b37c73d77de4beb2ea431a9be774f32023268cbc9036d1c3e2160d1ba4"} err="failed to get container status \"4b9822b37c73d77de4beb2ea431a9be774f32023268cbc9036d1c3e2160d1ba4\": rpc error: code = NotFound desc = could not find container \"4b9822b37c73d77de4beb2ea431a9be774f32023268cbc9036d1c3e2160d1ba4\": container with ID starting with 4b9822b37c73d77de4beb2ea431a9be774f32023268cbc9036d1c3e2160d1ba4 not found: ID does not exist"
Sep 16 19:02:52.488230 ip-10-0-138-127 kubenswrapper[2581]: I0916 19:02:52.488174    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-4959b-predictor-7544df7b5d-899mk"
Sep 16 19:02:52.631275 ip-10-0-138-127 kubenswrapper[2581]: I0916 19:02:52.631230    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/a76ee980-74ce-4370-a7f8-3e204b8bbb5c-cabundle-cert\") pod \"a76ee980-74ce-4370-a7f8-3e204b8bbb5c\" (UID: \"a76ee980-74ce-4370-a7f8-3e204b8bbb5c\") "
Sep 16 19:02:52.631481 ip-10-0-138-127 kubenswrapper[2581]: I0916 19:02:52.631307    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a76ee980-74ce-4370-a7f8-3e204b8bbb5c-proxy-tls\") pod \"a76ee980-74ce-4370-a7f8-3e204b8bbb5c\" (UID: \"a76ee980-74ce-4370-a7f8-3e204b8bbb5c\") "
Sep 16 19:02:52.631481 ip-10-0-138-127 kubenswrapper[2581]: I0916 19:02:52.631358    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vrvqx\" (UniqueName: \"kubernetes.io/projected/a76ee980-74ce-4370-a7f8-3e204b8bbb5c-kube-api-access-vrvqx\") pod \"a76ee980-74ce-4370-a7f8-3e204b8bbb5c\" (UID: \"a76ee980-74ce-4370-a7f8-3e204b8bbb5c\") "
Sep 16 19:02:52.631481 ip-10-0-138-127 kubenswrapper[2581]: I0916 19:02:52.631413    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a76ee980-74ce-4370-a7f8-3e204b8bbb5c-kube-rbac-proxy-sar-config\") pod \"a76ee980-74ce-4370-a7f8-3e204b8bbb5c\" (UID: \"a76ee980-74ce-4370-a7f8-3e204b8bbb5c\") "
Sep 16 19:02:52.631481 ip-10-0-138-127 kubenswrapper[2581]: I0916 19:02:52.631446    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a76ee980-74ce-4370-a7f8-3e204b8bbb5c-kserve-provision-location\") pod \"a76ee980-74ce-4370-a7f8-3e204b8bbb5c\" (UID: \"a76ee980-74ce-4370-a7f8-3e204b8bbb5c\") "
Sep 16 19:02:52.631731 ip-10-0-138-127 kubenswrapper[2581]: I0916 19:02:52.631703    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a76ee980-74ce-4370-a7f8-3e204b8bbb5c-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "a76ee980-74ce-4370-a7f8-3e204b8bbb5c" (UID: "a76ee980-74ce-4370-a7f8-3e204b8bbb5c"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 19:02:52.631800 ip-10-0-138-127 kubenswrapper[2581]: I0916 19:02:52.631775    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a76ee980-74ce-4370-a7f8-3e204b8bbb5c-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "a76ee980-74ce-4370-a7f8-3e204b8bbb5c" (UID: "a76ee980-74ce-4370-a7f8-3e204b8bbb5c"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:02:52.631800 ip-10-0-138-127 kubenswrapper[2581]: I0916 19:02:52.631782    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a76ee980-74ce-4370-a7f8-3e204b8bbb5c-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "a76ee980-74ce-4370-a7f8-3e204b8bbb5c" (UID: "a76ee980-74ce-4370-a7f8-3e204b8bbb5c"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 19:02:52.633706 ip-10-0-138-127 kubenswrapper[2581]: I0916 19:02:52.633685    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a76ee980-74ce-4370-a7f8-3e204b8bbb5c-kube-api-access-vrvqx" (OuterVolumeSpecName: "kube-api-access-vrvqx") pod "a76ee980-74ce-4370-a7f8-3e204b8bbb5c" (UID: "a76ee980-74ce-4370-a7f8-3e204b8bbb5c"). InnerVolumeSpecName "kube-api-access-vrvqx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 19:02:52.633783 ip-10-0-138-127 kubenswrapper[2581]: I0916 19:02:52.633705    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a76ee980-74ce-4370-a7f8-3e204b8bbb5c-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "a76ee980-74ce-4370-a7f8-3e204b8bbb5c" (UID: "a76ee980-74ce-4370-a7f8-3e204b8bbb5c"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 19:02:52.727034 ip-10-0-138-127 kubenswrapper[2581]: I0916 19:02:52.726999    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0be1ad70-7a20-4b06-a2cc-a6f3a5024e8d" path="/var/lib/kubelet/pods/0be1ad70-7a20-4b06-a2cc-a6f3a5024e8d/volumes"
Sep 16 19:02:52.732028 ip-10-0-138-127 kubenswrapper[2581]: I0916 19:02:52.732000    2581 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/a76ee980-74ce-4370-a7f8-3e204b8bbb5c-cabundle-cert\") on node \"ip-10-0-138-127.ec2.internal\" DevicePath \"\""
Sep 16 19:02:52.732028 ip-10-0-138-127 kubenswrapper[2581]: I0916 19:02:52.732028    2581 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a76ee980-74ce-4370-a7f8-3e204b8bbb5c-proxy-tls\") on node \"ip-10-0-138-127.ec2.internal\" DevicePath \"\""
Sep 16 19:02:52.732187 ip-10-0-138-127 kubenswrapper[2581]: I0916 19:02:52.732043    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vrvqx\" (UniqueName: \"kubernetes.io/projected/a76ee980-74ce-4370-a7f8-3e204b8bbb5c-kube-api-access-vrvqx\") on node \"ip-10-0-138-127.ec2.internal\" DevicePath \"\""
Sep 16 19:02:52.732187 ip-10-0-138-127 kubenswrapper[2581]: I0916 19:02:52.732056    2581 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a76ee980-74ce-4370-a7f8-3e204b8bbb5c-kube-rbac-proxy-sar-config\") on node \"ip-10-0-138-127.ec2.internal\" DevicePath \"\""
Sep 16 19:02:52.732187 ip-10-0-138-127 kubenswrapper[2581]: I0916 19:02:52.732068    2581 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a76ee980-74ce-4370-a7f8-3e204b8bbb5c-kserve-provision-location\") on node \"ip-10-0-138-127.ec2.internal\" DevicePath \"\""
Sep 16 19:02:53.366272 ip-10-0-138-127 kubenswrapper[2581]: I0916 19:02:53.366239    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-4959b-predictor-7544df7b5d-899mk"
Sep 16 19:02:53.366709 ip-10-0-138-127 kubenswrapper[2581]: I0916 19:02:53.366239    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-4959b-predictor-7544df7b5d-899mk" event={"ID":"a76ee980-74ce-4370-a7f8-3e204b8bbb5c","Type":"ContainerDied","Data":"9720c69816809d11589b2740ffadad9cd6c845750e85f636beb26670b9b9770d"}
Sep 16 19:02:53.366709 ip-10-0-138-127 kubenswrapper[2581]: I0916 19:02:53.366418    2581 scope.go:117] "RemoveContainer" containerID="dd806a67ebe2cd99ba28fd6afda9efa0de9fc81f948baae00048b25857972633"
Sep 16 19:02:53.374788 ip-10-0-138-127 kubenswrapper[2581]: I0916 19:02:53.374766    2581 scope.go:117] "RemoveContainer" containerID="0831b63490f6606b0db244c4e889045ea00ffb22586c8f872c7d0537465b00dc"
Sep 16 19:02:53.382967 ip-10-0-138-127 kubenswrapper[2581]: I0916 19:02:53.382949    2581 scope.go:117] "RemoveContainer" containerID="a279cdfa735f47d4b0ab58401066250ee75d8c09e19a5462533a20aa46f47d9f"
Sep 16 19:02:53.386470 ip-10-0-138-127 kubenswrapper[2581]: I0916 19:02:53.386447    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-4959b-predictor-7544df7b5d-899mk"]
Sep 16 19:02:53.392047 ip-10-0-138-127 kubenswrapper[2581]: I0916 19:02:53.392025    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-4959b-predictor-7544df7b5d-899mk"]
Sep 16 19:02:54.727321 ip-10-0-138-127 kubenswrapper[2581]: I0916 19:02:54.727286    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a76ee980-74ce-4370-a7f8-3e204b8bbb5c" path="/var/lib/kubelet/pods/a76ee980-74ce-4370-a7f8-3e204b8bbb5c/volumes"
Sep 16 19:03:07.405982 ip-10-0-138-127 kubenswrapper[2581]: I0916 19:03:07.405942    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-logger-raw-87fdd-predictor-f5488758-zhz5c"]
Sep 16 19:03:07.406487 ip-10-0-138-127 kubenswrapper[2581]: I0916 19:03:07.406414    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1035c64c-18a1-43f0-9e88-c06f78ad4591" containerName="storage-initializer"
Sep 16 19:03:07.406487 ip-10-0-138-127 kubenswrapper[2581]: I0916 19:03:07.406432    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="1035c64c-18a1-43f0-9e88-c06f78ad4591" containerName="storage-initializer"
Sep 16 19:03:07.406487 ip-10-0-138-127 kubenswrapper[2581]: I0916 19:03:07.406443    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0be1ad70-7a20-4b06-a2cc-a6f3a5024e8d" containerName="storage-initializer"
Sep 16 19:03:07.406487 ip-10-0-138-127 kubenswrapper[2581]: I0916 19:03:07.406450    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="0be1ad70-7a20-4b06-a2cc-a6f3a5024e8d" containerName="storage-initializer"
Sep 16 19:03:07.406487 ip-10-0-138-127 kubenswrapper[2581]: I0916 19:03:07.406466    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1035c64c-18a1-43f0-9e88-c06f78ad4591" containerName="kserve-container"
Sep 16 19:03:07.406487 ip-10-0-138-127 kubenswrapper[2581]: I0916 19:03:07.406475    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="1035c64c-18a1-43f0-9e88-c06f78ad4591" containerName="kserve-container"
Sep 16 19:03:07.406487 ip-10-0-138-127 kubenswrapper[2581]: I0916 19:03:07.406484    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1035c64c-18a1-43f0-9e88-c06f78ad4591" containerName="kube-rbac-proxy"
Sep 16 19:03:07.406487 ip-10-0-138-127 kubenswrapper[2581]: I0916 19:03:07.406492    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="1035c64c-18a1-43f0-9e88-c06f78ad4591" containerName="kube-rbac-proxy"
Sep 16 19:03:07.406894 ip-10-0-138-127 kubenswrapper[2581]: I0916 19:03:07.406503    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="64110e03-5cab-44aa-bdd3-f7a937cf46f2" containerName="storage-initializer"
Sep 16 19:03:07.406894 ip-10-0-138-127 kubenswrapper[2581]: I0916 19:03:07.406510    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="64110e03-5cab-44aa-bdd3-f7a937cf46f2" containerName="storage-initializer"
Sep 16 19:03:07.406894 ip-10-0-138-127 kubenswrapper[2581]: I0916 19:03:07.406521    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0be1ad70-7a20-4b06-a2cc-a6f3a5024e8d" containerName="kube-rbac-proxy"
Sep 16 19:03:07.406894 ip-10-0-138-127 kubenswrapper[2581]: I0916 19:03:07.406529    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="0be1ad70-7a20-4b06-a2cc-a6f3a5024e8d" containerName="kube-rbac-proxy"
Sep 16 19:03:07.406894 ip-10-0-138-127 kubenswrapper[2581]: I0916 19:03:07.406543    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="64110e03-5cab-44aa-bdd3-f7a937cf46f2" containerName="kube-rbac-proxy"
Sep 16 19:03:07.406894 ip-10-0-138-127 kubenswrapper[2581]: I0916 19:03:07.406552    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="64110e03-5cab-44aa-bdd3-f7a937cf46f2" containerName="kube-rbac-proxy"
Sep 16 19:03:07.406894 ip-10-0-138-127 kubenswrapper[2581]: I0916 19:03:07.406564    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0be1ad70-7a20-4b06-a2cc-a6f3a5024e8d" containerName="kserve-container"
Sep 16 19:03:07.406894 ip-10-0-138-127 kubenswrapper[2581]: I0916 19:03:07.406572    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="0be1ad70-7a20-4b06-a2cc-a6f3a5024e8d" containerName="kserve-container"
Sep 16 19:03:07.406894 ip-10-0-138-127 kubenswrapper[2581]: I0916 19:03:07.406582    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a76ee980-74ce-4370-a7f8-3e204b8bbb5c" containerName="kserve-container"
Sep 16 19:03:07.406894 ip-10-0-138-127 kubenswrapper[2581]: I0916 19:03:07.406590    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="a76ee980-74ce-4370-a7f8-3e204b8bbb5c" containerName="kserve-container"
Sep 16 19:03:07.406894 ip-10-0-138-127 kubenswrapper[2581]: I0916 19:03:07.406600    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a76ee980-74ce-4370-a7f8-3e204b8bbb5c" containerName="kube-rbac-proxy"
Sep 16 19:03:07.406894 ip-10-0-138-127 kubenswrapper[2581]: I0916 19:03:07.406608    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="a76ee980-74ce-4370-a7f8-3e204b8bbb5c" containerName="kube-rbac-proxy"
Sep 16 19:03:07.406894 ip-10-0-138-127 kubenswrapper[2581]: I0916 19:03:07.406618    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="64110e03-5cab-44aa-bdd3-f7a937cf46f2" containerName="kserve-container"
Sep 16 19:03:07.406894 ip-10-0-138-127 kubenswrapper[2581]: I0916 19:03:07.406629    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="64110e03-5cab-44aa-bdd3-f7a937cf46f2" containerName="kserve-container"
Sep 16 19:03:07.406894 ip-10-0-138-127 kubenswrapper[2581]: I0916 19:03:07.406641    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a76ee980-74ce-4370-a7f8-3e204b8bbb5c" containerName="storage-initializer"
Sep 16 19:03:07.406894 ip-10-0-138-127 kubenswrapper[2581]: I0916 19:03:07.406649    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="a76ee980-74ce-4370-a7f8-3e204b8bbb5c" containerName="storage-initializer"
Sep 16 19:03:07.406894 ip-10-0-138-127 kubenswrapper[2581]: I0916 19:03:07.406726    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="64110e03-5cab-44aa-bdd3-f7a937cf46f2" containerName="kserve-container"
Sep 16 19:03:07.406894 ip-10-0-138-127 kubenswrapper[2581]: I0916 19:03:07.406737    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="0be1ad70-7a20-4b06-a2cc-a6f3a5024e8d" containerName="kserve-container"
Sep 16 19:03:07.406894 ip-10-0-138-127 kubenswrapper[2581]: I0916 19:03:07.406745    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="0be1ad70-7a20-4b06-a2cc-a6f3a5024e8d" containerName="kube-rbac-proxy"
Sep 16 19:03:07.406894 ip-10-0-138-127 kubenswrapper[2581]: I0916 19:03:07.406754    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="a76ee980-74ce-4370-a7f8-3e204b8bbb5c" containerName="kube-rbac-proxy"
Sep 16 19:03:07.406894 ip-10-0-138-127 kubenswrapper[2581]: I0916 19:03:07.406763    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="1035c64c-18a1-43f0-9e88-c06f78ad4591" containerName="kserve-container"
Sep 16 19:03:07.406894 ip-10-0-138-127 kubenswrapper[2581]: I0916 19:03:07.406772    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="64110e03-5cab-44aa-bdd3-f7a937cf46f2" containerName="kube-rbac-proxy"
Sep 16 19:03:07.406894 ip-10-0-138-127 kubenswrapper[2581]: I0916 19:03:07.406779    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="a76ee980-74ce-4370-a7f8-3e204b8bbb5c" containerName="kserve-container"
Sep 16 19:03:07.406894 ip-10-0-138-127 kubenswrapper[2581]: I0916 19:03:07.406787    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="1035c64c-18a1-43f0-9e88-c06f78ad4591" containerName="kube-rbac-proxy"
Sep 16 19:03:07.410477 ip-10-0-138-127 kubenswrapper[2581]: I0916 19:03:07.410456    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-logger-raw-87fdd-predictor-f5488758-zhz5c"
Sep 16 19:03:07.414872 ip-10-0-138-127 kubenswrapper[2581]: I0916 19:03:07.414847    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-logger-raw-87fdd-kube-rbac-proxy-sar-config\""
Sep 16 19:03:07.415019 ip-10-0-138-127 kubenswrapper[2581]: I0916 19:03:07.414899    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-logger-raw-87fdd-predictor-serving-cert\""
Sep 16 19:03:07.415019 ip-10-0-138-127 kubenswrapper[2581]: I0916 19:03:07.414996    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Sep 16 19:03:07.415528 ip-10-0-138-127 kubenswrapper[2581]: I0916 19:03:07.415511    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Sep 16 19:03:07.415528 ip-10-0-138-127 kubenswrapper[2581]: I0916 19:03:07.415521    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Sep 16 19:03:07.416011 ip-10-0-138-127 kubenswrapper[2581]: I0916 19:03:07.415995    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Sep 16 19:03:07.416088 ip-10-0-138-127 kubenswrapper[2581]: I0916 19:03:07.416024    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-c8lqg\""
Sep 16 19:03:07.422961 ip-10-0-138-127 kubenswrapper[2581]: I0916 19:03:07.422936    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-logger-raw-87fdd-predictor-f5488758-zhz5c"]
Sep 16 19:03:07.445511 ip-10-0-138-127 kubenswrapper[2581]: I0916 19:03:07.445480    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2hlkn\" (UniqueName: \"kubernetes.io/projected/f8801259-429e-44ec-a099-9a9d4395e96f-kube-api-access-2hlkn\") pod \"isvc-logger-raw-87fdd-predictor-f5488758-zhz5c\" (UID: \"f8801259-429e-44ec-a099-9a9d4395e96f\") " pod="kserve-ci-e2e-test/isvc-logger-raw-87fdd-predictor-f5488758-zhz5c"
Sep 16 19:03:07.445511 ip-10-0-138-127 kubenswrapper[2581]: I0916 19:03:07.445517    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f8801259-429e-44ec-a099-9a9d4395e96f-kserve-provision-location\") pod \"isvc-logger-raw-87fdd-predictor-f5488758-zhz5c\" (UID: \"f8801259-429e-44ec-a099-9a9d4395e96f\") " pod="kserve-ci-e2e-test/isvc-logger-raw-87fdd-predictor-f5488758-zhz5c"
Sep 16 19:03:07.445736 ip-10-0-138-127 kubenswrapper[2581]: I0916 19:03:07.445550    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f8801259-429e-44ec-a099-9a9d4395e96f-proxy-tls\") pod \"isvc-logger-raw-87fdd-predictor-f5488758-zhz5c\" (UID: \"f8801259-429e-44ec-a099-9a9d4395e96f\") " pod="kserve-ci-e2e-test/isvc-logger-raw-87fdd-predictor-f5488758-zhz5c"
Sep 16 19:03:07.445736 ip-10-0-138-127 kubenswrapper[2581]: I0916 19:03:07.445572    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/f8801259-429e-44ec-a099-9a9d4395e96f-kube-rbac-proxy-sar-config\") pod \"isvc-logger-raw-87fdd-predictor-f5488758-zhz5c\" (UID: \"f8801259-429e-44ec-a099-9a9d4395e96f\") " pod="kserve-ci-e2e-test/isvc-logger-raw-87fdd-predictor-f5488758-zhz5c"
Sep 16 19:03:07.445736 ip-10-0-138-127 kubenswrapper[2581]: I0916 19:03:07.445598    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/f8801259-429e-44ec-a099-9a9d4395e96f-cabundle-cert\") pod \"isvc-logger-raw-87fdd-predictor-f5488758-zhz5c\" (UID: \"f8801259-429e-44ec-a099-9a9d4395e96f\") " pod="kserve-ci-e2e-test/isvc-logger-raw-87fdd-predictor-f5488758-zhz5c"
Sep 16 19:03:07.546575 ip-10-0-138-127 kubenswrapper[2581]: I0916 19:03:07.546535    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2hlkn\" (UniqueName: \"kubernetes.io/projected/f8801259-429e-44ec-a099-9a9d4395e96f-kube-api-access-2hlkn\") pod \"isvc-logger-raw-87fdd-predictor-f5488758-zhz5c\" (UID: \"f8801259-429e-44ec-a099-9a9d4395e96f\") " pod="kserve-ci-e2e-test/isvc-logger-raw-87fdd-predictor-f5488758-zhz5c"
Sep 16 19:03:07.546575 ip-10-0-138-127 kubenswrapper[2581]: I0916 19:03:07.546575    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f8801259-429e-44ec-a099-9a9d4395e96f-kserve-provision-location\") pod \"isvc-logger-raw-87fdd-predictor-f5488758-zhz5c\" (UID: \"f8801259-429e-44ec-a099-9a9d4395e96f\") " pod="kserve-ci-e2e-test/isvc-logger-raw-87fdd-predictor-f5488758-zhz5c"
Sep 16 19:03:07.546799 ip-10-0-138-127 kubenswrapper[2581]: I0916 19:03:07.546605    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f8801259-429e-44ec-a099-9a9d4395e96f-proxy-tls\") pod \"isvc-logger-raw-87fdd-predictor-f5488758-zhz5c\" (UID: \"f8801259-429e-44ec-a099-9a9d4395e96f\") " pod="kserve-ci-e2e-test/isvc-logger-raw-87fdd-predictor-f5488758-zhz5c"
Sep 16 19:03:07.546799 ip-10-0-138-127 kubenswrapper[2581]: I0916 19:03:07.546624    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/f8801259-429e-44ec-a099-9a9d4395e96f-kube-rbac-proxy-sar-config\") pod \"isvc-logger-raw-87fdd-predictor-f5488758-zhz5c\" (UID: \"f8801259-429e-44ec-a099-9a9d4395e96f\") " pod="kserve-ci-e2e-test/isvc-logger-raw-87fdd-predictor-f5488758-zhz5c"
Sep 16 19:03:07.546799 ip-10-0-138-127 kubenswrapper[2581]: I0916 19:03:07.546665    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/f8801259-429e-44ec-a099-9a9d4395e96f-cabundle-cert\") pod \"isvc-logger-raw-87fdd-predictor-f5488758-zhz5c\" (UID: \"f8801259-429e-44ec-a099-9a9d4395e96f\") " pod="kserve-ci-e2e-test/isvc-logger-raw-87fdd-predictor-f5488758-zhz5c"
Sep 16 19:03:07.546799 ip-10-0-138-127 kubenswrapper[2581]: E0916 19:03:07.546753    2581 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-logger-raw-87fdd-predictor-serving-cert: secret "isvc-logger-raw-87fdd-predictor-serving-cert" not found
Sep 16 19:03:07.546989 ip-10-0-138-127 kubenswrapper[2581]: E0916 19:03:07.546858    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f8801259-429e-44ec-a099-9a9d4395e96f-proxy-tls podName:f8801259-429e-44ec-a099-9a9d4395e96f nodeName:}" failed. No retries permitted until 2026-09-16 19:03:08.04683674 +0000 UTC m=+1057.866250636 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/f8801259-429e-44ec-a099-9a9d4395e96f-proxy-tls") pod "isvc-logger-raw-87fdd-predictor-f5488758-zhz5c" (UID: "f8801259-429e-44ec-a099-9a9d4395e96f") : secret "isvc-logger-raw-87fdd-predictor-serving-cert" not found
Sep 16 19:03:07.547039 ip-10-0-138-127 kubenswrapper[2581]: I0916 19:03:07.546995    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f8801259-429e-44ec-a099-9a9d4395e96f-kserve-provision-location\") pod \"isvc-logger-raw-87fdd-predictor-f5488758-zhz5c\" (UID: \"f8801259-429e-44ec-a099-9a9d4395e96f\") " pod="kserve-ci-e2e-test/isvc-logger-raw-87fdd-predictor-f5488758-zhz5c"
Sep 16 19:03:07.547477 ip-10-0-138-127 kubenswrapper[2581]: I0916 19:03:07.547456    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/f8801259-429e-44ec-a099-9a9d4395e96f-kube-rbac-proxy-sar-config\") pod \"isvc-logger-raw-87fdd-predictor-f5488758-zhz5c\" (UID: \"f8801259-429e-44ec-a099-9a9d4395e96f\") " pod="kserve-ci-e2e-test/isvc-logger-raw-87fdd-predictor-f5488758-zhz5c"
Sep 16 19:03:07.547520 ip-10-0-138-127 kubenswrapper[2581]: I0916 19:03:07.547456    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/f8801259-429e-44ec-a099-9a9d4395e96f-cabundle-cert\") pod \"isvc-logger-raw-87fdd-predictor-f5488758-zhz5c\" (UID: \"f8801259-429e-44ec-a099-9a9d4395e96f\") " pod="kserve-ci-e2e-test/isvc-logger-raw-87fdd-predictor-f5488758-zhz5c"
Sep 16 19:03:07.556329 ip-10-0-138-127 kubenswrapper[2581]: I0916 19:03:07.556304    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2hlkn\" (UniqueName: \"kubernetes.io/projected/f8801259-429e-44ec-a099-9a9d4395e96f-kube-api-access-2hlkn\") pod \"isvc-logger-raw-87fdd-predictor-f5488758-zhz5c\" (UID: \"f8801259-429e-44ec-a099-9a9d4395e96f\") " pod="kserve-ci-e2e-test/isvc-logger-raw-87fdd-predictor-f5488758-zhz5c"
Sep 16 19:03:08.052310 ip-10-0-138-127 kubenswrapper[2581]: I0916 19:03:08.052251    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f8801259-429e-44ec-a099-9a9d4395e96f-proxy-tls\") pod \"isvc-logger-raw-87fdd-predictor-f5488758-zhz5c\" (UID: \"f8801259-429e-44ec-a099-9a9d4395e96f\") " pod="kserve-ci-e2e-test/isvc-logger-raw-87fdd-predictor-f5488758-zhz5c"
Sep 16 19:03:08.054817 ip-10-0-138-127 kubenswrapper[2581]: I0916 19:03:08.054785    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f8801259-429e-44ec-a099-9a9d4395e96f-proxy-tls\") pod \"isvc-logger-raw-87fdd-predictor-f5488758-zhz5c\" (UID: \"f8801259-429e-44ec-a099-9a9d4395e96f\") " pod="kserve-ci-e2e-test/isvc-logger-raw-87fdd-predictor-f5488758-zhz5c"
Sep 16 19:03:08.321242 ip-10-0-138-127 kubenswrapper[2581]: I0916 19:03:08.321122    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-logger-raw-87fdd-predictor-f5488758-zhz5c"
Sep 16 19:03:08.605842 ip-10-0-138-127 kubenswrapper[2581]: I0916 19:03:08.605815    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-logger-raw-87fdd-predictor-f5488758-zhz5c"]
Sep 16 19:03:08.608763 ip-10-0-138-127 kubenswrapper[2581]: W0916 19:03:08.608738    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf8801259_429e_44ec_a099_9a9d4395e96f.slice/crio-96ac9b4f6efaec78e0d263251797b569e1960f76a1e31049106f904464a9df7d WatchSource:0}: Error finding container 96ac9b4f6efaec78e0d263251797b569e1960f76a1e31049106f904464a9df7d: Status 404 returned error can't find the container with id 96ac9b4f6efaec78e0d263251797b569e1960f76a1e31049106f904464a9df7d
Sep 16 19:03:09.435476 ip-10-0-138-127 kubenswrapper[2581]: I0916 19:03:09.435437    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-raw-87fdd-predictor-f5488758-zhz5c" event={"ID":"f8801259-429e-44ec-a099-9a9d4395e96f","Type":"ContainerStarted","Data":"4a3fed54c3387165aea5f008982bb415a51280c755383b65a2b38d5118db5fbb"}
Sep 16 19:03:09.435694 ip-10-0-138-127 kubenswrapper[2581]: I0916 19:03:09.435506    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-raw-87fdd-predictor-f5488758-zhz5c" event={"ID":"f8801259-429e-44ec-a099-9a9d4395e96f","Type":"ContainerStarted","Data":"96ac9b4f6efaec78e0d263251797b569e1960f76a1e31049106f904464a9df7d"}
Sep 16 19:03:13.453461 ip-10-0-138-127 kubenswrapper[2581]: I0916 19:03:13.453425    2581 generic.go:356] "Generic (PLEG): container finished" podID="f8801259-429e-44ec-a099-9a9d4395e96f" containerID="4a3fed54c3387165aea5f008982bb415a51280c755383b65a2b38d5118db5fbb" exitCode=0
Sep 16 19:03:13.453867 ip-10-0-138-127 kubenswrapper[2581]: I0916 19:03:13.453498    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-raw-87fdd-predictor-f5488758-zhz5c" event={"ID":"f8801259-429e-44ec-a099-9a9d4395e96f","Type":"ContainerDied","Data":"4a3fed54c3387165aea5f008982bb415a51280c755383b65a2b38d5118db5fbb"}
Sep 16 19:03:14.461253 ip-10-0-138-127 kubenswrapper[2581]: I0916 19:03:14.461212    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-raw-87fdd-predictor-f5488758-zhz5c" event={"ID":"f8801259-429e-44ec-a099-9a9d4395e96f","Type":"ContainerStarted","Data":"cab7dbb1e8d742e6788fb0c4f11033ecfe9838f32b49f6e78c996023787e3f31"}
Sep 16 19:03:14.461657 ip-10-0-138-127 kubenswrapper[2581]: I0916 19:03:14.461314    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-raw-87fdd-predictor-f5488758-zhz5c" event={"ID":"f8801259-429e-44ec-a099-9a9d4395e96f","Type":"ContainerStarted","Data":"e34a585d8fe6d34030701b5b148c47129f2ba68517395b2caf75481bbd29b435"}
Sep 16 19:03:14.461657 ip-10-0-138-127 kubenswrapper[2581]: I0916 19:03:14.461339    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-raw-87fdd-predictor-f5488758-zhz5c" event={"ID":"f8801259-429e-44ec-a099-9a9d4395e96f","Type":"ContainerStarted","Data":"ed469c25248b4e759ae2e72b86e54eb49ee521231623f7525772535c7a3792a3"}
Sep 16 19:03:14.461657 ip-10-0-138-127 kubenswrapper[2581]: I0916 19:03:14.461635    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-logger-raw-87fdd-predictor-f5488758-zhz5c"
Sep 16 19:03:14.461789 ip-10-0-138-127 kubenswrapper[2581]: I0916 19:03:14.461700    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-logger-raw-87fdd-predictor-f5488758-zhz5c"
Sep 16 19:03:14.461789 ip-10-0-138-127 kubenswrapper[2581]: I0916 19:03:14.461734    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-logger-raw-87fdd-predictor-f5488758-zhz5c"
Sep 16 19:03:14.463234 ip-10-0-138-127 kubenswrapper[2581]: I0916 19:03:14.463178    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-87fdd-predictor-f5488758-zhz5c" podUID="f8801259-429e-44ec-a099-9a9d4395e96f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.32:8080: connect: connection refused"
Sep 16 19:03:14.463916 ip-10-0-138-127 kubenswrapper[2581]: I0916 19:03:14.463890    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-87fdd-predictor-f5488758-zhz5c" podUID="f8801259-429e-44ec-a099-9a9d4395e96f" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 16 19:03:14.485705 ip-10-0-138-127 kubenswrapper[2581]: I0916 19:03:14.485657    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-logger-raw-87fdd-predictor-f5488758-zhz5c" podStartSLOduration=3.383848535 podStartE2EDuration="7.485638503s" totalImagesPullingTime="0s" totalInitContainerRuntime="4.101789968s" isStatefulPod=false podCreationTimestamp="2026-09-16 19:03:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:03:14.484264531 +0000 UTC m=+1064.303678442" watchObservedRunningTime="2026-09-16 19:03:14.485638503 +0000 UTC m=+1064.305052413"
Sep 16 19:03:15.465956 ip-10-0-138-127 kubenswrapper[2581]: I0916 19:03:15.465914    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-87fdd-predictor-f5488758-zhz5c" podUID="f8801259-429e-44ec-a099-9a9d4395e96f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.32:8080: connect: connection refused"
Sep 16 19:03:15.466370 ip-10-0-138-127 kubenswrapper[2581]: I0916 19:03:15.466344    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-87fdd-predictor-f5488758-zhz5c" podUID="f8801259-429e-44ec-a099-9a9d4395e96f" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 16 19:03:20.471340 ip-10-0-138-127 kubenswrapper[2581]: I0916 19:03:20.471308    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-logger-raw-87fdd-predictor-f5488758-zhz5c"
Sep 16 19:03:20.472005 ip-10-0-138-127 kubenswrapper[2581]: I0916 19:03:20.471926    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-87fdd-predictor-f5488758-zhz5c" podUID="f8801259-429e-44ec-a099-9a9d4395e96f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.32:8080: connect: connection refused"
Sep 16 19:03:20.472254 ip-10-0-138-127 kubenswrapper[2581]: I0916 19:03:20.472231    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-87fdd-predictor-f5488758-zhz5c" podUID="f8801259-429e-44ec-a099-9a9d4395e96f" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 16 19:03:30.472440 ip-10-0-138-127 kubenswrapper[2581]: I0916 19:03:30.472335    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-87fdd-predictor-f5488758-zhz5c" podUID="f8801259-429e-44ec-a099-9a9d4395e96f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.32:8080: connect: connection refused"
Sep 16 19:03:30.472838 ip-10-0-138-127 kubenswrapper[2581]: I0916 19:03:30.472813    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-87fdd-predictor-f5488758-zhz5c" podUID="f8801259-429e-44ec-a099-9a9d4395e96f" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 16 19:03:40.472684 ip-10-0-138-127 kubenswrapper[2581]: I0916 19:03:40.472638    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-87fdd-predictor-f5488758-zhz5c" podUID="f8801259-429e-44ec-a099-9a9d4395e96f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.32:8080: connect: connection refused"
Sep 16 19:03:40.473156 ip-10-0-138-127 kubenswrapper[2581]: I0916 19:03:40.473039    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-87fdd-predictor-f5488758-zhz5c" podUID="f8801259-429e-44ec-a099-9a9d4395e96f" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 16 19:03:50.471971 ip-10-0-138-127 kubenswrapper[2581]: I0916 19:03:50.471925    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-87fdd-predictor-f5488758-zhz5c" podUID="f8801259-429e-44ec-a099-9a9d4395e96f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.32:8080: connect: connection refused"
Sep 16 19:03:50.472458 ip-10-0-138-127 kubenswrapper[2581]: I0916 19:03:50.472430    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-87fdd-predictor-f5488758-zhz5c" podUID="f8801259-429e-44ec-a099-9a9d4395e96f" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 16 19:04:00.471992 ip-10-0-138-127 kubenswrapper[2581]: I0916 19:04:00.471946    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-87fdd-predictor-f5488758-zhz5c" podUID="f8801259-429e-44ec-a099-9a9d4395e96f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.32:8080: connect: connection refused"
Sep 16 19:04:00.472520 ip-10-0-138-127 kubenswrapper[2581]: I0916 19:04:00.472446    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-87fdd-predictor-f5488758-zhz5c" podUID="f8801259-429e-44ec-a099-9a9d4395e96f" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 16 19:04:02.320386 ip-10-0-138-127 kubenswrapper[2581]: I0916 19:04:02.320348    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-secondary-8f4cb6-predictor-74cf44d648-mj256"]
Sep 16 19:04:02.323868 ip-10-0-138-127 kubenswrapper[2581]: I0916 19:04:02.323850    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-secondary-8f4cb6-predictor-74cf44d648-mj256"
Sep 16 19:04:02.327050 ip-10-0-138-127 kubenswrapper[2581]: I0916 19:04:02.327026    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-secondary-8f4cb6-predictor-serving-cert\""
Sep 16 19:04:02.327397 ip-10-0-138-127 kubenswrapper[2581]: I0916 19:04:02.327380    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"invalid-s3-sa-8f4cb6-dockercfg-pc4k2\""
Sep 16 19:04:02.327690 ip-10-0-138-127 kubenswrapper[2581]: I0916 19:04:02.327677    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-secondary-8f4cb6-kube-rbac-proxy-sar-config\""
Sep 16 19:04:02.327881 ip-10-0-138-127 kubenswrapper[2581]: I0916 19:04:02.327864    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"invalid-s3-secret-8f4cb6\""
Sep 16 19:04:02.344107 ip-10-0-138-127 kubenswrapper[2581]: I0916 19:04:02.344082    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-secondary-8f4cb6-predictor-74cf44d648-mj256"]
Sep 16 19:04:02.423070 ip-10-0-138-127 kubenswrapper[2581]: I0916 19:04:02.423029    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a9ec2f2e-fb50-4533-b8bc-382f7c5139d5-kube-rbac-proxy-sar-config\") pod \"isvc-secondary-8f4cb6-predictor-74cf44d648-mj256\" (UID: \"a9ec2f2e-fb50-4533-b8bc-382f7c5139d5\") " pod="kserve-ci-e2e-test/isvc-secondary-8f4cb6-predictor-74cf44d648-mj256"
Sep 16 19:04:02.423297 ip-10-0-138-127 kubenswrapper[2581]: I0916 19:04:02.423115    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a9ec2f2e-fb50-4533-b8bc-382f7c5139d5-kserve-provision-location\") pod \"isvc-secondary-8f4cb6-predictor-74cf44d648-mj256\" (UID: \"a9ec2f2e-fb50-4533-b8bc-382f7c5139d5\") " pod="kserve-ci-e2e-test/isvc-secondary-8f4cb6-predictor-74cf44d648-mj256"
Sep 16 19:04:02.423297 ip-10-0-138-127 kubenswrapper[2581]: I0916 19:04:02.423142    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7r64w\" (UniqueName: \"kubernetes.io/projected/a9ec2f2e-fb50-4533-b8bc-382f7c5139d5-kube-api-access-7r64w\") pod \"isvc-secondary-8f4cb6-predictor-74cf44d648-mj256\" (UID: \"a9ec2f2e-fb50-4533-b8bc-382f7c5139d5\") " pod="kserve-ci-e2e-test/isvc-secondary-8f4cb6-predictor-74cf44d648-mj256"
Sep 16 19:04:02.423297 ip-10-0-138-127 kubenswrapper[2581]: I0916 19:04:02.423166    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/a9ec2f2e-fb50-4533-b8bc-382f7c5139d5-cabundle-cert\") pod \"isvc-secondary-8f4cb6-predictor-74cf44d648-mj256\" (UID: \"a9ec2f2e-fb50-4533-b8bc-382f7c5139d5\") " pod="kserve-ci-e2e-test/isvc-secondary-8f4cb6-predictor-74cf44d648-mj256"
Sep 16 19:04:02.423297 ip-10-0-138-127 kubenswrapper[2581]: I0916 19:04:02.423258    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a9ec2f2e-fb50-4533-b8bc-382f7c5139d5-proxy-tls\") pod \"isvc-secondary-8f4cb6-predictor-74cf44d648-mj256\" (UID: \"a9ec2f2e-fb50-4533-b8bc-382f7c5139d5\") " pod="kserve-ci-e2e-test/isvc-secondary-8f4cb6-predictor-74cf44d648-mj256"
Sep 16 19:04:02.524223 ip-10-0-138-127 kubenswrapper[2581]: I0916 19:04:02.524151    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a9ec2f2e-fb50-4533-b8bc-382f7c5139d5-kserve-provision-location\") pod \"isvc-secondary-8f4cb6-predictor-74cf44d648-mj256\" (UID: \"a9ec2f2e-fb50-4533-b8bc-382f7c5139d5\") " pod="kserve-ci-e2e-test/isvc-secondary-8f4cb6-predictor-74cf44d648-mj256"
Sep 16 19:04:02.524405 ip-10-0-138-127 kubenswrapper[2581]: I0916 19:04:02.524232    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7r64w\" (UniqueName: \"kubernetes.io/projected/a9ec2f2e-fb50-4533-b8bc-382f7c5139d5-kube-api-access-7r64w\") pod \"isvc-secondary-8f4cb6-predictor-74cf44d648-mj256\" (UID: \"a9ec2f2e-fb50-4533-b8bc-382f7c5139d5\") " pod="kserve-ci-e2e-test/isvc-secondary-8f4cb6-predictor-74cf44d648-mj256"
Sep 16 19:04:02.524405 ip-10-0-138-127 kubenswrapper[2581]: I0916 19:04:02.524258    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/a9ec2f2e-fb50-4533-b8bc-382f7c5139d5-cabundle-cert\") pod \"isvc-secondary-8f4cb6-predictor-74cf44d648-mj256\" (UID: \"a9ec2f2e-fb50-4533-b8bc-382f7c5139d5\") " pod="kserve-ci-e2e-test/isvc-secondary-8f4cb6-predictor-74cf44d648-mj256"
Sep 16 19:04:02.524405 ip-10-0-138-127 kubenswrapper[2581]: I0916 19:04:02.524295    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a9ec2f2e-fb50-4533-b8bc-382f7c5139d5-proxy-tls\") pod \"isvc-secondary-8f4cb6-predictor-74cf44d648-mj256\" (UID: \"a9ec2f2e-fb50-4533-b8bc-382f7c5139d5\") " pod="kserve-ci-e2e-test/isvc-secondary-8f4cb6-predictor-74cf44d648-mj256"
Sep 16 19:04:02.524405 ip-10-0-138-127 kubenswrapper[2581]: I0916 19:04:02.524362    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a9ec2f2e-fb50-4533-b8bc-382f7c5139d5-kube-rbac-proxy-sar-config\") pod \"isvc-secondary-8f4cb6-predictor-74cf44d648-mj256\" (UID: \"a9ec2f2e-fb50-4533-b8bc-382f7c5139d5\") " pod="kserve-ci-e2e-test/isvc-secondary-8f4cb6-predictor-74cf44d648-mj256"
Sep 16 19:04:02.524668 ip-10-0-138-127 kubenswrapper[2581]: I0916 19:04:02.524644    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a9ec2f2e-fb50-4533-b8bc-382f7c5139d5-kserve-provision-location\") pod \"isvc-secondary-8f4cb6-predictor-74cf44d648-mj256\" (UID: \"a9ec2f2e-fb50-4533-b8bc-382f7c5139d5\") " pod="kserve-ci-e2e-test/isvc-secondary-8f4cb6-predictor-74cf44d648-mj256"
Sep 16 19:04:02.524966 ip-10-0-138-127 kubenswrapper[2581]: I0916 19:04:02.524948    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a9ec2f2e-fb50-4533-b8bc-382f7c5139d5-kube-rbac-proxy-sar-config\") pod \"isvc-secondary-8f4cb6-predictor-74cf44d648-mj256\" (UID: \"a9ec2f2e-fb50-4533-b8bc-382f7c5139d5\") " pod="kserve-ci-e2e-test/isvc-secondary-8f4cb6-predictor-74cf44d648-mj256"
Sep 16 19:04:02.525013 ip-10-0-138-127 kubenswrapper[2581]: I0916 19:04:02.524962    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/a9ec2f2e-fb50-4533-b8bc-382f7c5139d5-cabundle-cert\") pod \"isvc-secondary-8f4cb6-predictor-74cf44d648-mj256\" (UID: \"a9ec2f2e-fb50-4533-b8bc-382f7c5139d5\") " pod="kserve-ci-e2e-test/isvc-secondary-8f4cb6-predictor-74cf44d648-mj256"
Sep 16 19:04:02.526881 ip-10-0-138-127 kubenswrapper[2581]: I0916 19:04:02.526859    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a9ec2f2e-fb50-4533-b8bc-382f7c5139d5-proxy-tls\") pod \"isvc-secondary-8f4cb6-predictor-74cf44d648-mj256\" (UID: \"a9ec2f2e-fb50-4533-b8bc-382f7c5139d5\") " pod="kserve-ci-e2e-test/isvc-secondary-8f4cb6-predictor-74cf44d648-mj256"
Sep 16 19:04:02.535712 ip-10-0-138-127 kubenswrapper[2581]: I0916 19:04:02.535689    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7r64w\" (UniqueName: \"kubernetes.io/projected/a9ec2f2e-fb50-4533-b8bc-382f7c5139d5-kube-api-access-7r64w\") pod \"isvc-secondary-8f4cb6-predictor-74cf44d648-mj256\" (UID: \"a9ec2f2e-fb50-4533-b8bc-382f7c5139d5\") " pod="kserve-ci-e2e-test/isvc-secondary-8f4cb6-predictor-74cf44d648-mj256"
Sep 16 19:04:02.634810 ip-10-0-138-127 kubenswrapper[2581]: I0916 19:04:02.634724    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-secondary-8f4cb6-predictor-74cf44d648-mj256"
Sep 16 19:04:02.914298 ip-10-0-138-127 kubenswrapper[2581]: I0916 19:04:02.914171    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-secondary-8f4cb6-predictor-74cf44d648-mj256"]
Sep 16 19:04:02.917007 ip-10-0-138-127 kubenswrapper[2581]: W0916 19:04:02.916979    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda9ec2f2e_fb50_4533_b8bc_382f7c5139d5.slice/crio-876e611b0c7306c444014c41b0fbeedbb615e16a73dc5294e7270ceb301359ee WatchSource:0}: Error finding container 876e611b0c7306c444014c41b0fbeedbb615e16a73dc5294e7270ceb301359ee: Status 404 returned error can't find the container with id 876e611b0c7306c444014c41b0fbeedbb615e16a73dc5294e7270ceb301359ee
Sep 16 19:04:03.660974 ip-10-0-138-127 kubenswrapper[2581]: I0916 19:04:03.660939    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-secondary-8f4cb6-predictor-74cf44d648-mj256" event={"ID":"a9ec2f2e-fb50-4533-b8bc-382f7c5139d5","Type":"ContainerStarted","Data":"f8948e2d75288d8fd263949e6b258324fb68fc5529b76d9a4338285aaf1942ee"}
Sep 16 19:04:03.661501 ip-10-0-138-127 kubenswrapper[2581]: I0916 19:04:03.661001    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-secondary-8f4cb6-predictor-74cf44d648-mj256" event={"ID":"a9ec2f2e-fb50-4533-b8bc-382f7c5139d5","Type":"ContainerStarted","Data":"876e611b0c7306c444014c41b0fbeedbb615e16a73dc5294e7270ceb301359ee"}
Sep 16 19:04:07.679865 ip-10-0-138-127 kubenswrapper[2581]: I0916 19:04:07.679835    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-secondary-8f4cb6-predictor-74cf44d648-mj256_a9ec2f2e-fb50-4533-b8bc-382f7c5139d5/storage-initializer/0.log"
Sep 16 19:04:07.680250 ip-10-0-138-127 kubenswrapper[2581]: I0916 19:04:07.679874    2581 generic.go:356] "Generic (PLEG): container finished" podID="a9ec2f2e-fb50-4533-b8bc-382f7c5139d5" containerID="f8948e2d75288d8fd263949e6b258324fb68fc5529b76d9a4338285aaf1942ee" exitCode=1
Sep 16 19:04:07.680250 ip-10-0-138-127 kubenswrapper[2581]: I0916 19:04:07.679954    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-secondary-8f4cb6-predictor-74cf44d648-mj256" event={"ID":"a9ec2f2e-fb50-4533-b8bc-382f7c5139d5","Type":"ContainerDied","Data":"f8948e2d75288d8fd263949e6b258324fb68fc5529b76d9a4338285aaf1942ee"}
Sep 16 19:04:08.685755 ip-10-0-138-127 kubenswrapper[2581]: I0916 19:04:08.685725    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-secondary-8f4cb6-predictor-74cf44d648-mj256_a9ec2f2e-fb50-4533-b8bc-382f7c5139d5/storage-initializer/0.log"
Sep 16 19:04:08.686237 ip-10-0-138-127 kubenswrapper[2581]: I0916 19:04:08.685797    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-secondary-8f4cb6-predictor-74cf44d648-mj256" event={"ID":"a9ec2f2e-fb50-4533-b8bc-382f7c5139d5","Type":"ContainerStarted","Data":"3dc03675131909a2d7e6543a5e066e7914d246cc4a747eb0ef953537b098b1d3"}
Sep 16 19:04:10.472378 ip-10-0-138-127 kubenswrapper[2581]: I0916 19:04:10.472338    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-87fdd-predictor-f5488758-zhz5c" podUID="f8801259-429e-44ec-a099-9a9d4395e96f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.32:8080: connect: connection refused"
Sep 16 19:04:10.472890 ip-10-0-138-127 kubenswrapper[2581]: I0916 19:04:10.472864    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-87fdd-predictor-f5488758-zhz5c" podUID="f8801259-429e-44ec-a099-9a9d4395e96f" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 16 19:04:10.696093 ip-10-0-138-127 kubenswrapper[2581]: I0916 19:04:10.696063    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-secondary-8f4cb6-predictor-74cf44d648-mj256_a9ec2f2e-fb50-4533-b8bc-382f7c5139d5/storage-initializer/1.log"
Sep 16 19:04:10.696486 ip-10-0-138-127 kubenswrapper[2581]: I0916 19:04:10.696471    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-secondary-8f4cb6-predictor-74cf44d648-mj256_a9ec2f2e-fb50-4533-b8bc-382f7c5139d5/storage-initializer/0.log"
Sep 16 19:04:10.696547 ip-10-0-138-127 kubenswrapper[2581]: I0916 19:04:10.696506    2581 generic.go:356] "Generic (PLEG): container finished" podID="a9ec2f2e-fb50-4533-b8bc-382f7c5139d5" containerID="3dc03675131909a2d7e6543a5e066e7914d246cc4a747eb0ef953537b098b1d3" exitCode=1
Sep 16 19:04:10.696600 ip-10-0-138-127 kubenswrapper[2581]: I0916 19:04:10.696581    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-secondary-8f4cb6-predictor-74cf44d648-mj256" event={"ID":"a9ec2f2e-fb50-4533-b8bc-382f7c5139d5","Type":"ContainerDied","Data":"3dc03675131909a2d7e6543a5e066e7914d246cc4a747eb0ef953537b098b1d3"}
Sep 16 19:04:10.696664 ip-10-0-138-127 kubenswrapper[2581]: I0916 19:04:10.696654    2581 scope.go:117] "RemoveContainer" containerID="f8948e2d75288d8fd263949e6b258324fb68fc5529b76d9a4338285aaf1942ee"
Sep 16 19:04:10.697024 ip-10-0-138-127 kubenswrapper[2581]: I0916 19:04:10.697004    2581 scope.go:117] "RemoveContainer" containerID="f8948e2d75288d8fd263949e6b258324fb68fc5529b76d9a4338285aaf1942ee"
Sep 16 19:04:10.708185 ip-10-0-138-127 kubenswrapper[2581]: E0916 19:04:10.708161    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"storage-initializer\" with CrashLoopBackOff: \"back-off 10s restarting failed container=storage-initializer pod=isvc-secondary-8f4cb6-predictor-74cf44d648-mj256_kserve-ci-e2e-test(a9ec2f2e-fb50-4533-b8bc-382f7c5139d5)\"" pod="kserve-ci-e2e-test/isvc-secondary-8f4cb6-predictor-74cf44d648-mj256" podUID="a9ec2f2e-fb50-4533-b8bc-382f7c5139d5"
Sep 16 19:04:10.708474 ip-10-0-138-127 kubenswrapper[2581]: E0916 19:04:10.708444    2581 log.go:32] "RemoveContainer from runtime service failed" err="rpc error: code = Unknown desc = failed to delete container k8s_storage-initializer_isvc-secondary-8f4cb6-predictor-74cf44d648-mj256_kserve-ci-e2e-test_a9ec2f2e-fb50-4533-b8bc-382f7c5139d5_0 in pod sandbox 876e611b0c7306c444014c41b0fbeedbb615e16a73dc5294e7270ceb301359ee from index: no such id: 'f8948e2d75288d8fd263949e6b258324fb68fc5529b76d9a4338285aaf1942ee'" containerID="f8948e2d75288d8fd263949e6b258324fb68fc5529b76d9a4338285aaf1942ee"
Sep 16 19:04:10.708545 ip-10-0-138-127 kubenswrapper[2581]: I0916 19:04:10.708486    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f8948e2d75288d8fd263949e6b258324fb68fc5529b76d9a4338285aaf1942ee"} err="rpc error: code = Unknown desc = failed to delete container k8s_storage-initializer_isvc-secondary-8f4cb6-predictor-74cf44d648-mj256_kserve-ci-e2e-test_a9ec2f2e-fb50-4533-b8bc-382f7c5139d5_0 in pod sandbox 876e611b0c7306c444014c41b0fbeedbb615e16a73dc5294e7270ceb301359ee from index: no such id: 'f8948e2d75288d8fd263949e6b258324fb68fc5529b76d9a4338285aaf1942ee'"
Sep 16 19:04:11.702570 ip-10-0-138-127 kubenswrapper[2581]: I0916 19:04:11.702540    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-secondary-8f4cb6-predictor-74cf44d648-mj256_a9ec2f2e-fb50-4533-b8bc-382f7c5139d5/storage-initializer/1.log"
Sep 16 19:04:18.352614 ip-10-0-138-127 kubenswrapper[2581]: I0916 19:04:18.352582    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-secondary-8f4cb6-predictor-74cf44d648-mj256"]
Sep 16 19:04:18.482304 ip-10-0-138-127 kubenswrapper[2581]: I0916 19:04:18.482277    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-secondary-8f4cb6-predictor-74cf44d648-mj256_a9ec2f2e-fb50-4533-b8bc-382f7c5139d5/storage-initializer/1.log"
Sep 16 19:04:18.486079 ip-10-0-138-127 kubenswrapper[2581]: I0916 19:04:18.486061    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-secondary-8f4cb6-predictor-74cf44d648-mj256"
Sep 16 19:04:18.563624 ip-10-0-138-127 kubenswrapper[2581]: I0916 19:04:18.563587    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a9ec2f2e-fb50-4533-b8bc-382f7c5139d5-kube-rbac-proxy-sar-config\") pod \"a9ec2f2e-fb50-4533-b8bc-382f7c5139d5\" (UID: \"a9ec2f2e-fb50-4533-b8bc-382f7c5139d5\") "
Sep 16 19:04:18.563810 ip-10-0-138-127 kubenswrapper[2581]: I0916 19:04:18.563672    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a9ec2f2e-fb50-4533-b8bc-382f7c5139d5-proxy-tls\") pod \"a9ec2f2e-fb50-4533-b8bc-382f7c5139d5\" (UID: \"a9ec2f2e-fb50-4533-b8bc-382f7c5139d5\") "
Sep 16 19:04:18.563810 ip-10-0-138-127 kubenswrapper[2581]: I0916 19:04:18.563710    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a9ec2f2e-fb50-4533-b8bc-382f7c5139d5-kserve-provision-location\") pod \"a9ec2f2e-fb50-4533-b8bc-382f7c5139d5\" (UID: \"a9ec2f2e-fb50-4533-b8bc-382f7c5139d5\") "
Sep 16 19:04:18.563810 ip-10-0-138-127 kubenswrapper[2581]: I0916 19:04:18.563750    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/a9ec2f2e-fb50-4533-b8bc-382f7c5139d5-cabundle-cert\") pod \"a9ec2f2e-fb50-4533-b8bc-382f7c5139d5\" (UID: \"a9ec2f2e-fb50-4533-b8bc-382f7c5139d5\") "
Sep 16 19:04:18.563810 ip-10-0-138-127 kubenswrapper[2581]: I0916 19:04:18.563776    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7r64w\" (UniqueName: \"kubernetes.io/projected/a9ec2f2e-fb50-4533-b8bc-382f7c5139d5-kube-api-access-7r64w\") pod \"a9ec2f2e-fb50-4533-b8bc-382f7c5139d5\" (UID: \"a9ec2f2e-fb50-4533-b8bc-382f7c5139d5\") "
Sep 16 19:04:18.564027 ip-10-0-138-127 kubenswrapper[2581]: I0916 19:04:18.563980    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a9ec2f2e-fb50-4533-b8bc-382f7c5139d5-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "a9ec2f2e-fb50-4533-b8bc-382f7c5139d5" (UID: "a9ec2f2e-fb50-4533-b8bc-382f7c5139d5"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:04:18.564027 ip-10-0-138-127 kubenswrapper[2581]: I0916 19:04:18.564007    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a9ec2f2e-fb50-4533-b8bc-382f7c5139d5-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "a9ec2f2e-fb50-4533-b8bc-382f7c5139d5" (UID: "a9ec2f2e-fb50-4533-b8bc-382f7c5139d5"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 19:04:18.564141 ip-10-0-138-127 kubenswrapper[2581]: I0916 19:04:18.564123    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a9ec2f2e-fb50-4533-b8bc-382f7c5139d5-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "a9ec2f2e-fb50-4533-b8bc-382f7c5139d5" (UID: "a9ec2f2e-fb50-4533-b8bc-382f7c5139d5"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 19:04:18.565950 ip-10-0-138-127 kubenswrapper[2581]: I0916 19:04:18.565933    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a9ec2f2e-fb50-4533-b8bc-382f7c5139d5-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "a9ec2f2e-fb50-4533-b8bc-382f7c5139d5" (UID: "a9ec2f2e-fb50-4533-b8bc-382f7c5139d5"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 19:04:18.566066 ip-10-0-138-127 kubenswrapper[2581]: I0916 19:04:18.566050    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a9ec2f2e-fb50-4533-b8bc-382f7c5139d5-kube-api-access-7r64w" (OuterVolumeSpecName: "kube-api-access-7r64w") pod "a9ec2f2e-fb50-4533-b8bc-382f7c5139d5" (UID: "a9ec2f2e-fb50-4533-b8bc-382f7c5139d5"). InnerVolumeSpecName "kube-api-access-7r64w". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 19:04:18.664980 ip-10-0-138-127 kubenswrapper[2581]: I0916 19:04:18.664893    2581 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/a9ec2f2e-fb50-4533-b8bc-382f7c5139d5-cabundle-cert\") on node \"ip-10-0-138-127.ec2.internal\" DevicePath \"\""
Sep 16 19:04:18.664980 ip-10-0-138-127 kubenswrapper[2581]: I0916 19:04:18.664926    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7r64w\" (UniqueName: \"kubernetes.io/projected/a9ec2f2e-fb50-4533-b8bc-382f7c5139d5-kube-api-access-7r64w\") on node \"ip-10-0-138-127.ec2.internal\" DevicePath \"\""
Sep 16 19:04:18.664980 ip-10-0-138-127 kubenswrapper[2581]: I0916 19:04:18.664936    2581 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a9ec2f2e-fb50-4533-b8bc-382f7c5139d5-kube-rbac-proxy-sar-config\") on node \"ip-10-0-138-127.ec2.internal\" DevicePath \"\""
Sep 16 19:04:18.664980 ip-10-0-138-127 kubenswrapper[2581]: I0916 19:04:18.664948    2581 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a9ec2f2e-fb50-4533-b8bc-382f7c5139d5-proxy-tls\") on node \"ip-10-0-138-127.ec2.internal\" DevicePath \"\""
Sep 16 19:04:18.664980 ip-10-0-138-127 kubenswrapper[2581]: I0916 19:04:18.664957    2581 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a9ec2f2e-fb50-4533-b8bc-382f7c5139d5-kserve-provision-location\") on node \"ip-10-0-138-127.ec2.internal\" DevicePath \"\""
Sep 16 19:04:18.733835 ip-10-0-138-127 kubenswrapper[2581]: I0916 19:04:18.733805    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-secondary-8f4cb6-predictor-74cf44d648-mj256_a9ec2f2e-fb50-4533-b8bc-382f7c5139d5/storage-initializer/1.log"
Sep 16 19:04:18.734013 ip-10-0-138-127 kubenswrapper[2581]: I0916 19:04:18.733902    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-secondary-8f4cb6-predictor-74cf44d648-mj256"
Sep 16 19:04:18.734013 ip-10-0-138-127 kubenswrapper[2581]: I0916 19:04:18.733910    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-secondary-8f4cb6-predictor-74cf44d648-mj256" event={"ID":"a9ec2f2e-fb50-4533-b8bc-382f7c5139d5","Type":"ContainerDied","Data":"876e611b0c7306c444014c41b0fbeedbb615e16a73dc5294e7270ceb301359ee"}
Sep 16 19:04:18.734013 ip-10-0-138-127 kubenswrapper[2581]: I0916 19:04:18.733984    2581 scope.go:117] "RemoveContainer" containerID="3dc03675131909a2d7e6543a5e066e7914d246cc4a747eb0ef953537b098b1d3"
Sep 16 19:04:18.767499 ip-10-0-138-127 kubenswrapper[2581]: I0916 19:04:18.767465    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-secondary-8f4cb6-predictor-74cf44d648-mj256"]
Sep 16 19:04:18.773685 ip-10-0-138-127 kubenswrapper[2581]: I0916 19:04:18.773655    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-secondary-8f4cb6-predictor-74cf44d648-mj256"]
Sep 16 19:04:20.471894 ip-10-0-138-127 kubenswrapper[2581]: I0916 19:04:20.471855    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-87fdd-predictor-f5488758-zhz5c" podUID="f8801259-429e-44ec-a099-9a9d4395e96f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.32:8080: connect: connection refused"
Sep 16 19:04:20.472403 ip-10-0-138-127 kubenswrapper[2581]: I0916 19:04:20.472356    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-87fdd-predictor-f5488758-zhz5c" podUID="f8801259-429e-44ec-a099-9a9d4395e96f" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 16 19:04:20.727156 ip-10-0-138-127 kubenswrapper[2581]: I0916 19:04:20.727072    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a9ec2f2e-fb50-4533-b8bc-382f7c5139d5" path="/var/lib/kubelet/pods/a9ec2f2e-fb50-4533-b8bc-382f7c5139d5/volumes"
Sep 16 19:04:30.472403 ip-10-0-138-127 kubenswrapper[2581]: I0916 19:04:30.472369    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-logger-raw-87fdd-predictor-f5488758-zhz5c"
Sep 16 19:04:30.472882 ip-10-0-138-127 kubenswrapper[2581]: I0916 19:04:30.472467    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-logger-raw-87fdd-predictor-f5488758-zhz5c"
Sep 16 19:04:42.638045 ip-10-0-138-127 kubenswrapper[2581]: I0916 19:04:42.638010    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-logger-raw-87fdd-predictor-f5488758-zhz5c"]
Sep 16 19:04:42.638637 ip-10-0-138-127 kubenswrapper[2581]: I0916 19:04:42.638369    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-logger-raw-87fdd-predictor-f5488758-zhz5c" podUID="f8801259-429e-44ec-a099-9a9d4395e96f" containerName="kserve-container" containerID="cri-o://ed469c25248b4e759ae2e72b86e54eb49ee521231623f7525772535c7a3792a3" gracePeriod=30
Sep 16 19:04:42.638637 ip-10-0-138-127 kubenswrapper[2581]: I0916 19:04:42.638392    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-logger-raw-87fdd-predictor-f5488758-zhz5c" podUID="f8801259-429e-44ec-a099-9a9d4395e96f" containerName="agent" containerID="cri-o://cab7dbb1e8d742e6788fb0c4f11033ecfe9838f32b49f6e78c996023787e3f31" gracePeriod=30
Sep 16 19:04:42.638637 ip-10-0-138-127 kubenswrapper[2581]: I0916 19:04:42.638407    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-logger-raw-87fdd-predictor-f5488758-zhz5c" podUID="f8801259-429e-44ec-a099-9a9d4395e96f" containerName="kube-rbac-proxy" containerID="cri-o://e34a585d8fe6d34030701b5b148c47129f2ba68517395b2caf75481bbd29b435" gracePeriod=30
Sep 16 19:04:42.759396 ip-10-0-138-127 kubenswrapper[2581]: I0916 19:04:42.759369    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-scale-raw-5517d-predictor-8547668d88-xw6bk"]
Sep 16 19:04:42.759740 ip-10-0-138-127 kubenswrapper[2581]: I0916 19:04:42.759727    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a9ec2f2e-fb50-4533-b8bc-382f7c5139d5" containerName="storage-initializer"
Sep 16 19:04:42.759784 ip-10-0-138-127 kubenswrapper[2581]: I0916 19:04:42.759742    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="a9ec2f2e-fb50-4533-b8bc-382f7c5139d5" containerName="storage-initializer"
Sep 16 19:04:42.759784 ip-10-0-138-127 kubenswrapper[2581]: I0916 19:04:42.759762    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a9ec2f2e-fb50-4533-b8bc-382f7c5139d5" containerName="storage-initializer"
Sep 16 19:04:42.759784 ip-10-0-138-127 kubenswrapper[2581]: I0916 19:04:42.759768    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="a9ec2f2e-fb50-4533-b8bc-382f7c5139d5" containerName="storage-initializer"
Sep 16 19:04:42.759880 ip-10-0-138-127 kubenswrapper[2581]: I0916 19:04:42.759819    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="a9ec2f2e-fb50-4533-b8bc-382f7c5139d5" containerName="storage-initializer"
Sep 16 19:04:42.759880 ip-10-0-138-127 kubenswrapper[2581]: I0916 19:04:42.759831    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="a9ec2f2e-fb50-4533-b8bc-382f7c5139d5" containerName="storage-initializer"
Sep 16 19:04:42.762851 ip-10-0-138-127 kubenswrapper[2581]: I0916 19:04:42.762834    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-5517d-predictor-8547668d88-xw6bk"
Sep 16 19:04:42.765183 ip-10-0-138-127 kubenswrapper[2581]: I0916 19:04:42.765159    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-scale-raw-5517d-predictor-serving-cert\""
Sep 16 19:04:42.765314 ip-10-0-138-127 kubenswrapper[2581]: I0916 19:04:42.765186    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-scale-raw-5517d-kube-rbac-proxy-sar-config\""
Sep 16 19:04:42.778029 ip-10-0-138-127 kubenswrapper[2581]: I0916 19:04:42.778003    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-scale-raw-5517d-predictor-8547668d88-xw6bk"]
Sep 16 19:04:42.837447 ip-10-0-138-127 kubenswrapper[2581]: I0916 19:04:42.837411    2581 generic.go:356] "Generic (PLEG): container finished" podID="f8801259-429e-44ec-a099-9a9d4395e96f" containerID="e34a585d8fe6d34030701b5b148c47129f2ba68517395b2caf75481bbd29b435" exitCode=2
Sep 16 19:04:42.837632 ip-10-0-138-127 kubenswrapper[2581]: I0916 19:04:42.837486    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-raw-87fdd-predictor-f5488758-zhz5c" event={"ID":"f8801259-429e-44ec-a099-9a9d4395e96f","Type":"ContainerDied","Data":"e34a585d8fe6d34030701b5b148c47129f2ba68517395b2caf75481bbd29b435"}
Sep 16 19:04:42.879171 ip-10-0-138-127 kubenswrapper[2581]: I0916 19:04:42.879129    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/0fed021d-b37c-4ed9-8a77-e9258be99e13-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-scale-raw-5517d-predictor-8547668d88-xw6bk\" (UID: \"0fed021d-b37c-4ed9-8a77-e9258be99e13\") " pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-5517d-predictor-8547668d88-xw6bk"
Sep 16 19:04:42.879375 ip-10-0-138-127 kubenswrapper[2581]: I0916 19:04:42.879181    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/0fed021d-b37c-4ed9-8a77-e9258be99e13-proxy-tls\") pod \"isvc-sklearn-scale-raw-5517d-predictor-8547668d88-xw6bk\" (UID: \"0fed021d-b37c-4ed9-8a77-e9258be99e13\") " pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-5517d-predictor-8547668d88-xw6bk"
Sep 16 19:04:42.879375 ip-10-0-138-127 kubenswrapper[2581]: I0916 19:04:42.879226    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/0fed021d-b37c-4ed9-8a77-e9258be99e13-cabundle-cert\") pod \"isvc-sklearn-scale-raw-5517d-predictor-8547668d88-xw6bk\" (UID: \"0fed021d-b37c-4ed9-8a77-e9258be99e13\") " pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-5517d-predictor-8547668d88-xw6bk"
Sep 16 19:04:42.879375 ip-10-0-138-127 kubenswrapper[2581]: I0916 19:04:42.879269    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/0fed021d-b37c-4ed9-8a77-e9258be99e13-kserve-provision-location\") pod \"isvc-sklearn-scale-raw-5517d-predictor-8547668d88-xw6bk\" (UID: \"0fed021d-b37c-4ed9-8a77-e9258be99e13\") " pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-5517d-predictor-8547668d88-xw6bk"
Sep 16 19:04:42.879375 ip-10-0-138-127 kubenswrapper[2581]: I0916 19:04:42.879335    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nfz4s\" (UniqueName: \"kubernetes.io/projected/0fed021d-b37c-4ed9-8a77-e9258be99e13-kube-api-access-nfz4s\") pod \"isvc-sklearn-scale-raw-5517d-predictor-8547668d88-xw6bk\" (UID: \"0fed021d-b37c-4ed9-8a77-e9258be99e13\") " pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-5517d-predictor-8547668d88-xw6bk"
Sep 16 19:04:42.980656 ip-10-0-138-127 kubenswrapper[2581]: I0916 19:04:42.980568    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nfz4s\" (UniqueName: \"kubernetes.io/projected/0fed021d-b37c-4ed9-8a77-e9258be99e13-kube-api-access-nfz4s\") pod \"isvc-sklearn-scale-raw-5517d-predictor-8547668d88-xw6bk\" (UID: \"0fed021d-b37c-4ed9-8a77-e9258be99e13\") " pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-5517d-predictor-8547668d88-xw6bk"
Sep 16 19:04:42.980656 ip-10-0-138-127 kubenswrapper[2581]: I0916 19:04:42.980638    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/0fed021d-b37c-4ed9-8a77-e9258be99e13-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-scale-raw-5517d-predictor-8547668d88-xw6bk\" (UID: \"0fed021d-b37c-4ed9-8a77-e9258be99e13\") " pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-5517d-predictor-8547668d88-xw6bk"
Sep 16 19:04:42.980890 ip-10-0-138-127 kubenswrapper[2581]: I0916 19:04:42.980667    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/0fed021d-b37c-4ed9-8a77-e9258be99e13-proxy-tls\") pod \"isvc-sklearn-scale-raw-5517d-predictor-8547668d88-xw6bk\" (UID: \"0fed021d-b37c-4ed9-8a77-e9258be99e13\") " pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-5517d-predictor-8547668d88-xw6bk"
Sep 16 19:04:42.980890 ip-10-0-138-127 kubenswrapper[2581]: I0916 19:04:42.980688    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/0fed021d-b37c-4ed9-8a77-e9258be99e13-cabundle-cert\") pod \"isvc-sklearn-scale-raw-5517d-predictor-8547668d88-xw6bk\" (UID: \"0fed021d-b37c-4ed9-8a77-e9258be99e13\") " pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-5517d-predictor-8547668d88-xw6bk"
Sep 16 19:04:42.980890 ip-10-0-138-127 kubenswrapper[2581]: I0916 19:04:42.980716    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/0fed021d-b37c-4ed9-8a77-e9258be99e13-kserve-provision-location\") pod \"isvc-sklearn-scale-raw-5517d-predictor-8547668d88-xw6bk\" (UID: \"0fed021d-b37c-4ed9-8a77-e9258be99e13\") " pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-5517d-predictor-8547668d88-xw6bk"
Sep 16 19:04:42.981176 ip-10-0-138-127 kubenswrapper[2581]: I0916 19:04:42.981151    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/0fed021d-b37c-4ed9-8a77-e9258be99e13-kserve-provision-location\") pod \"isvc-sklearn-scale-raw-5517d-predictor-8547668d88-xw6bk\" (UID: \"0fed021d-b37c-4ed9-8a77-e9258be99e13\") " pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-5517d-predictor-8547668d88-xw6bk"
Sep 16 19:04:42.981364 ip-10-0-138-127 kubenswrapper[2581]: I0916 19:04:42.981344    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/0fed021d-b37c-4ed9-8a77-e9258be99e13-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-scale-raw-5517d-predictor-8547668d88-xw6bk\" (UID: \"0fed021d-b37c-4ed9-8a77-e9258be99e13\") " pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-5517d-predictor-8547668d88-xw6bk"
Sep 16 19:04:42.981453 ip-10-0-138-127 kubenswrapper[2581]: I0916 19:04:42.981436    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/0fed021d-b37c-4ed9-8a77-e9258be99e13-cabundle-cert\") pod \"isvc-sklearn-scale-raw-5517d-predictor-8547668d88-xw6bk\" (UID: \"0fed021d-b37c-4ed9-8a77-e9258be99e13\") " pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-5517d-predictor-8547668d88-xw6bk"
Sep 16 19:04:42.983453 ip-10-0-138-127 kubenswrapper[2581]: I0916 19:04:42.983434    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/0fed021d-b37c-4ed9-8a77-e9258be99e13-proxy-tls\") pod \"isvc-sklearn-scale-raw-5517d-predictor-8547668d88-xw6bk\" (UID: \"0fed021d-b37c-4ed9-8a77-e9258be99e13\") " pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-5517d-predictor-8547668d88-xw6bk"
Sep 16 19:04:42.989314 ip-10-0-138-127 kubenswrapper[2581]: I0916 19:04:42.989290    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nfz4s\" (UniqueName: \"kubernetes.io/projected/0fed021d-b37c-4ed9-8a77-e9258be99e13-kube-api-access-nfz4s\") pod \"isvc-sklearn-scale-raw-5517d-predictor-8547668d88-xw6bk\" (UID: \"0fed021d-b37c-4ed9-8a77-e9258be99e13\") " pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-5517d-predictor-8547668d88-xw6bk"
Sep 16 19:04:43.072411 ip-10-0-138-127 kubenswrapper[2581]: I0916 19:04:43.072362    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-5517d-predictor-8547668d88-xw6bk"
Sep 16 19:04:43.348570 ip-10-0-138-127 kubenswrapper[2581]: I0916 19:04:43.348544    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-scale-raw-5517d-predictor-8547668d88-xw6bk"]
Sep 16 19:04:43.350905 ip-10-0-138-127 kubenswrapper[2581]: W0916 19:04:43.350883    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0fed021d_b37c_4ed9_8a77_e9258be99e13.slice/crio-5ca9f2a3d15b0a9927842d5830cb0caadcc346ddd999988879ce15b3817663d9 WatchSource:0}: Error finding container 5ca9f2a3d15b0a9927842d5830cb0caadcc346ddd999988879ce15b3817663d9: Status 404 returned error can't find the container with id 5ca9f2a3d15b0a9927842d5830cb0caadcc346ddd999988879ce15b3817663d9
Sep 16 19:04:43.843214 ip-10-0-138-127 kubenswrapper[2581]: I0916 19:04:43.843162    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-5517d-predictor-8547668d88-xw6bk" event={"ID":"0fed021d-b37c-4ed9-8a77-e9258be99e13","Type":"ContainerStarted","Data":"ea36f8bbd8ad97554d1d33d6084c2424b7c1241ff1f75107cdbb66f0218b1c43"}
Sep 16 19:04:43.843716 ip-10-0-138-127 kubenswrapper[2581]: I0916 19:04:43.843245    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-5517d-predictor-8547668d88-xw6bk" event={"ID":"0fed021d-b37c-4ed9-8a77-e9258be99e13","Type":"ContainerStarted","Data":"5ca9f2a3d15b0a9927842d5830cb0caadcc346ddd999988879ce15b3817663d9"}
Sep 16 19:04:45.466105 ip-10-0-138-127 kubenswrapper[2581]: I0916 19:04:45.466065    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-87fdd-predictor-f5488758-zhz5c" podUID="f8801259-429e-44ec-a099-9a9d4395e96f" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.32:8643/healthz\": dial tcp 10.132.0.32:8643: connect: connection refused"
Sep 16 19:04:47.869394 ip-10-0-138-127 kubenswrapper[2581]: I0916 19:04:47.869351    2581 generic.go:356] "Generic (PLEG): container finished" podID="f8801259-429e-44ec-a099-9a9d4395e96f" containerID="ed469c25248b4e759ae2e72b86e54eb49ee521231623f7525772535c7a3792a3" exitCode=0
Sep 16 19:04:47.869802 ip-10-0-138-127 kubenswrapper[2581]: I0916 19:04:47.869422    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-raw-87fdd-predictor-f5488758-zhz5c" event={"ID":"f8801259-429e-44ec-a099-9a9d4395e96f","Type":"ContainerDied","Data":"ed469c25248b4e759ae2e72b86e54eb49ee521231623f7525772535c7a3792a3"}
Sep 16 19:04:47.870902 ip-10-0-138-127 kubenswrapper[2581]: I0916 19:04:47.870882    2581 generic.go:356] "Generic (PLEG): container finished" podID="0fed021d-b37c-4ed9-8a77-e9258be99e13" containerID="ea36f8bbd8ad97554d1d33d6084c2424b7c1241ff1f75107cdbb66f0218b1c43" exitCode=0
Sep 16 19:04:47.871008 ip-10-0-138-127 kubenswrapper[2581]: I0916 19:04:47.870919    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-5517d-predictor-8547668d88-xw6bk" event={"ID":"0fed021d-b37c-4ed9-8a77-e9258be99e13","Type":"ContainerDied","Data":"ea36f8bbd8ad97554d1d33d6084c2424b7c1241ff1f75107cdbb66f0218b1c43"}
Sep 16 19:04:48.877903 ip-10-0-138-127 kubenswrapper[2581]: I0916 19:04:48.877869    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-5517d-predictor-8547668d88-xw6bk" event={"ID":"0fed021d-b37c-4ed9-8a77-e9258be99e13","Type":"ContainerStarted","Data":"13f0073b09ce7f02889af4128c1aacfab74636664288f7be94ecc5f1903cfc13"}
Sep 16 19:04:48.878307 ip-10-0-138-127 kubenswrapper[2581]: I0916 19:04:48.877938    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-5517d-predictor-8547668d88-xw6bk" event={"ID":"0fed021d-b37c-4ed9-8a77-e9258be99e13","Type":"ContainerStarted","Data":"f9ddef6cfd752eea6c966c04237361600056e86f095a1005176d8e6ed38a73d7"}
Sep 16 19:04:48.878307 ip-10-0-138-127 kubenswrapper[2581]: I0916 19:04:48.878143    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-5517d-predictor-8547668d88-xw6bk"
Sep 16 19:04:48.905427 ip-10-0-138-127 kubenswrapper[2581]: I0916 19:04:48.905377    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-5517d-predictor-8547668d88-xw6bk" podStartSLOduration=2.822905873 podStartE2EDuration="6.905360495s" totalImagesPullingTime="0s" totalInitContainerRuntime="4.082454622s" isStatefulPod=false podCreationTimestamp="2026-09-16 19:04:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:04:48.904068403 +0000 UTC m=+1158.723482313" watchObservedRunningTime="2026-09-16 19:04:48.905360495 +0000 UTC m=+1158.724774473"
Sep 16 19:04:49.883581 ip-10-0-138-127 kubenswrapper[2581]: I0916 19:04:49.883536    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-5517d-predictor-8547668d88-xw6bk"
Sep 16 19:04:49.884991 ip-10-0-138-127 kubenswrapper[2581]: I0916 19:04:49.884962    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-5517d-predictor-8547668d88-xw6bk" podUID="0fed021d-b37c-4ed9-8a77-e9258be99e13" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.34:8080: connect: connection refused"
Sep 16 19:04:50.466135 ip-10-0-138-127 kubenswrapper[2581]: I0916 19:04:50.466084    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-87fdd-predictor-f5488758-zhz5c" podUID="f8801259-429e-44ec-a099-9a9d4395e96f" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.32:8643/healthz\": dial tcp 10.132.0.32:8643: connect: connection refused"
Sep 16 19:04:50.472587 ip-10-0-138-127 kubenswrapper[2581]: I0916 19:04:50.472547    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-87fdd-predictor-f5488758-zhz5c" podUID="f8801259-429e-44ec-a099-9a9d4395e96f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.32:8080: connect: connection refused"
Sep 16 19:04:50.472860 ip-10-0-138-127 kubenswrapper[2581]: I0916 19:04:50.472835    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-87fdd-predictor-f5488758-zhz5c" podUID="f8801259-429e-44ec-a099-9a9d4395e96f" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 16 19:04:50.887935 ip-10-0-138-127 kubenswrapper[2581]: I0916 19:04:50.887894    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-5517d-predictor-8547668d88-xw6bk" podUID="0fed021d-b37c-4ed9-8a77-e9258be99e13" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.34:8080: connect: connection refused"
Sep 16 19:04:55.466420 ip-10-0-138-127 kubenswrapper[2581]: I0916 19:04:55.466316    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-87fdd-predictor-f5488758-zhz5c" podUID="f8801259-429e-44ec-a099-9a9d4395e96f" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.32:8643/healthz\": dial tcp 10.132.0.32:8643: connect: connection refused"
Sep 16 19:04:55.466802 ip-10-0-138-127 kubenswrapper[2581]: I0916 19:04:55.466447    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-logger-raw-87fdd-predictor-f5488758-zhz5c"
Sep 16 19:04:55.893061 ip-10-0-138-127 kubenswrapper[2581]: I0916 19:04:55.893036    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-5517d-predictor-8547668d88-xw6bk"
Sep 16 19:04:55.893581 ip-10-0-138-127 kubenswrapper[2581]: I0916 19:04:55.893556    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-5517d-predictor-8547668d88-xw6bk" podUID="0fed021d-b37c-4ed9-8a77-e9258be99e13" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.34:8080: connect: connection refused"
Sep 16 19:05:00.466316 ip-10-0-138-127 kubenswrapper[2581]: I0916 19:05:00.466275    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-87fdd-predictor-f5488758-zhz5c" podUID="f8801259-429e-44ec-a099-9a9d4395e96f" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.32:8643/healthz\": dial tcp 10.132.0.32:8643: connect: connection refused"
Sep 16 19:05:00.473426 ip-10-0-138-127 kubenswrapper[2581]: I0916 19:05:00.473034    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-87fdd-predictor-f5488758-zhz5c" podUID="f8801259-429e-44ec-a099-9a9d4395e96f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.32:8080: connect: connection refused"
Sep 16 19:05:00.473908 ip-10-0-138-127 kubenswrapper[2581]: I0916 19:05:00.473732    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-87fdd-predictor-f5488758-zhz5c" podUID="f8801259-429e-44ec-a099-9a9d4395e96f" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 16 19:05:05.466520 ip-10-0-138-127 kubenswrapper[2581]: I0916 19:05:05.466475    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-87fdd-predictor-f5488758-zhz5c" podUID="f8801259-429e-44ec-a099-9a9d4395e96f" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.32:8643/healthz\": dial tcp 10.132.0.32:8643: connect: connection refused"
Sep 16 19:05:05.894088 ip-10-0-138-127 kubenswrapper[2581]: I0916 19:05:05.894047    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-5517d-predictor-8547668d88-xw6bk" podUID="0fed021d-b37c-4ed9-8a77-e9258be99e13" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.34:8080: connect: connection refused"
Sep 16 19:05:10.466716 ip-10-0-138-127 kubenswrapper[2581]: I0916 19:05:10.466670    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-87fdd-predictor-f5488758-zhz5c" podUID="f8801259-429e-44ec-a099-9a9d4395e96f" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.32:8643/healthz\": dial tcp 10.132.0.32:8643: connect: connection refused"
Sep 16 19:05:10.472245 ip-10-0-138-127 kubenswrapper[2581]: I0916 19:05:10.472173    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-87fdd-predictor-f5488758-zhz5c" podUID="f8801259-429e-44ec-a099-9a9d4395e96f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.32:8080: connect: connection refused"
Sep 16 19:05:10.472415 ip-10-0-138-127 kubenswrapper[2581]: I0916 19:05:10.472397    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-logger-raw-87fdd-predictor-f5488758-zhz5c"
Sep 16 19:05:10.472491 ip-10-0-138-127 kubenswrapper[2581]: I0916 19:05:10.472477    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-87fdd-predictor-f5488758-zhz5c" podUID="f8801259-429e-44ec-a099-9a9d4395e96f" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 16 19:05:10.472641 ip-10-0-138-127 kubenswrapper[2581]: I0916 19:05:10.472629    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-logger-raw-87fdd-predictor-f5488758-zhz5c"
Sep 16 19:05:12.802155 ip-10-0-138-127 kubenswrapper[2581]: I0916 19:05:12.802126    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-logger-raw-87fdd-predictor-f5488758-zhz5c"
Sep 16 19:05:12.929760 ip-10-0-138-127 kubenswrapper[2581]: I0916 19:05:12.929657    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2hlkn\" (UniqueName: \"kubernetes.io/projected/f8801259-429e-44ec-a099-9a9d4395e96f-kube-api-access-2hlkn\") pod \"f8801259-429e-44ec-a099-9a9d4395e96f\" (UID: \"f8801259-429e-44ec-a099-9a9d4395e96f\") "
Sep 16 19:05:12.929760 ip-10-0-138-127 kubenswrapper[2581]: I0916 19:05:12.929696    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f8801259-429e-44ec-a099-9a9d4395e96f-proxy-tls\") pod \"f8801259-429e-44ec-a099-9a9d4395e96f\" (UID: \"f8801259-429e-44ec-a099-9a9d4395e96f\") "
Sep 16 19:05:12.930003 ip-10-0-138-127 kubenswrapper[2581]: I0916 19:05:12.929784    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f8801259-429e-44ec-a099-9a9d4395e96f-kserve-provision-location\") pod \"f8801259-429e-44ec-a099-9a9d4395e96f\" (UID: \"f8801259-429e-44ec-a099-9a9d4395e96f\") "
Sep 16 19:05:12.930003 ip-10-0-138-127 kubenswrapper[2581]: I0916 19:05:12.929812    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/f8801259-429e-44ec-a099-9a9d4395e96f-cabundle-cert\") pod \"f8801259-429e-44ec-a099-9a9d4395e96f\" (UID: \"f8801259-429e-44ec-a099-9a9d4395e96f\") "
Sep 16 19:05:12.930118 ip-10-0-138-127 kubenswrapper[2581]: I0916 19:05:12.929998    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/f8801259-429e-44ec-a099-9a9d4395e96f-kube-rbac-proxy-sar-config\") pod \"f8801259-429e-44ec-a099-9a9d4395e96f\" (UID: \"f8801259-429e-44ec-a099-9a9d4395e96f\") "
Sep 16 19:05:12.930174 ip-10-0-138-127 kubenswrapper[2581]: I0916 19:05:12.930106    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f8801259-429e-44ec-a099-9a9d4395e96f-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "f8801259-429e-44ec-a099-9a9d4395e96f" (UID: "f8801259-429e-44ec-a099-9a9d4395e96f"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:05:12.930255 ip-10-0-138-127 kubenswrapper[2581]: I0916 19:05:12.930163    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f8801259-429e-44ec-a099-9a9d4395e96f-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "f8801259-429e-44ec-a099-9a9d4395e96f" (UID: "f8801259-429e-44ec-a099-9a9d4395e96f"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 19:05:12.930369 ip-10-0-138-127 kubenswrapper[2581]: I0916 19:05:12.930352    2581 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f8801259-429e-44ec-a099-9a9d4395e96f-kserve-provision-location\") on node \"ip-10-0-138-127.ec2.internal\" DevicePath \"\""
Sep 16 19:05:12.930411 ip-10-0-138-127 kubenswrapper[2581]: I0916 19:05:12.930373    2581 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/f8801259-429e-44ec-a099-9a9d4395e96f-cabundle-cert\") on node \"ip-10-0-138-127.ec2.internal\" DevicePath \"\""
Sep 16 19:05:12.930411 ip-10-0-138-127 kubenswrapper[2581]: I0916 19:05:12.930369    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f8801259-429e-44ec-a099-9a9d4395e96f-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "f8801259-429e-44ec-a099-9a9d4395e96f" (UID: "f8801259-429e-44ec-a099-9a9d4395e96f"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 19:05:12.932120 ip-10-0-138-127 kubenswrapper[2581]: I0916 19:05:12.932095    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f8801259-429e-44ec-a099-9a9d4395e96f-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "f8801259-429e-44ec-a099-9a9d4395e96f" (UID: "f8801259-429e-44ec-a099-9a9d4395e96f"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 19:05:12.932256 ip-10-0-138-127 kubenswrapper[2581]: I0916 19:05:12.932222    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f8801259-429e-44ec-a099-9a9d4395e96f-kube-api-access-2hlkn" (OuterVolumeSpecName: "kube-api-access-2hlkn") pod "f8801259-429e-44ec-a099-9a9d4395e96f" (UID: "f8801259-429e-44ec-a099-9a9d4395e96f"). InnerVolumeSpecName "kube-api-access-2hlkn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 19:05:12.982136 ip-10-0-138-127 kubenswrapper[2581]: I0916 19:05:12.982100    2581 generic.go:356] "Generic (PLEG): container finished" podID="f8801259-429e-44ec-a099-9a9d4395e96f" containerID="cab7dbb1e8d742e6788fb0c4f11033ecfe9838f32b49f6e78c996023787e3f31" exitCode=0
Sep 16 19:05:12.982336 ip-10-0-138-127 kubenswrapper[2581]: I0916 19:05:12.982172    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-logger-raw-87fdd-predictor-f5488758-zhz5c"
Sep 16 19:05:12.982336 ip-10-0-138-127 kubenswrapper[2581]: I0916 19:05:12.982205    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-raw-87fdd-predictor-f5488758-zhz5c" event={"ID":"f8801259-429e-44ec-a099-9a9d4395e96f","Type":"ContainerDied","Data":"cab7dbb1e8d742e6788fb0c4f11033ecfe9838f32b49f6e78c996023787e3f31"}
Sep 16 19:05:12.982336 ip-10-0-138-127 kubenswrapper[2581]: I0916 19:05:12.982276    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-raw-87fdd-predictor-f5488758-zhz5c" event={"ID":"f8801259-429e-44ec-a099-9a9d4395e96f","Type":"ContainerDied","Data":"96ac9b4f6efaec78e0d263251797b569e1960f76a1e31049106f904464a9df7d"}
Sep 16 19:05:12.982336 ip-10-0-138-127 kubenswrapper[2581]: I0916 19:05:12.982308    2581 scope.go:117] "RemoveContainer" containerID="cab7dbb1e8d742e6788fb0c4f11033ecfe9838f32b49f6e78c996023787e3f31"
Sep 16 19:05:12.991778 ip-10-0-138-127 kubenswrapper[2581]: I0916 19:05:12.991756    2581 scope.go:117] "RemoveContainer" containerID="e34a585d8fe6d34030701b5b148c47129f2ba68517395b2caf75481bbd29b435"
Sep 16 19:05:13.000166 ip-10-0-138-127 kubenswrapper[2581]: I0916 19:05:13.000147    2581 scope.go:117] "RemoveContainer" containerID="ed469c25248b4e759ae2e72b86e54eb49ee521231623f7525772535c7a3792a3"
Sep 16 19:05:13.008445 ip-10-0-138-127 kubenswrapper[2581]: I0916 19:05:13.008425    2581 scope.go:117] "RemoveContainer" containerID="4a3fed54c3387165aea5f008982bb415a51280c755383b65a2b38d5118db5fbb"
Sep 16 19:05:13.017652 ip-10-0-138-127 kubenswrapper[2581]: I0916 19:05:13.017631    2581 scope.go:117] "RemoveContainer" containerID="cab7dbb1e8d742e6788fb0c4f11033ecfe9838f32b49f6e78c996023787e3f31"
Sep 16 19:05:13.017861 ip-10-0-138-127 kubenswrapper[2581]: I0916 19:05:13.017841    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-logger-raw-87fdd-predictor-f5488758-zhz5c"]
Sep 16 19:05:13.018035 ip-10-0-138-127 kubenswrapper[2581]: E0916 19:05:13.018017    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cab7dbb1e8d742e6788fb0c4f11033ecfe9838f32b49f6e78c996023787e3f31\": container with ID starting with cab7dbb1e8d742e6788fb0c4f11033ecfe9838f32b49f6e78c996023787e3f31 not found: ID does not exist" containerID="cab7dbb1e8d742e6788fb0c4f11033ecfe9838f32b49f6e78c996023787e3f31"
Sep 16 19:05:13.018083 ip-10-0-138-127 kubenswrapper[2581]: I0916 19:05:13.018047    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cab7dbb1e8d742e6788fb0c4f11033ecfe9838f32b49f6e78c996023787e3f31"} err="failed to get container status \"cab7dbb1e8d742e6788fb0c4f11033ecfe9838f32b49f6e78c996023787e3f31\": rpc error: code = NotFound desc = could not find container \"cab7dbb1e8d742e6788fb0c4f11033ecfe9838f32b49f6e78c996023787e3f31\": container with ID starting with cab7dbb1e8d742e6788fb0c4f11033ecfe9838f32b49f6e78c996023787e3f31 not found: ID does not exist"
Sep 16 19:05:13.018083 ip-10-0-138-127 kubenswrapper[2581]: I0916 19:05:13.018067    2581 scope.go:117] "RemoveContainer" containerID="e34a585d8fe6d34030701b5b148c47129f2ba68517395b2caf75481bbd29b435"
Sep 16 19:05:13.018436 ip-10-0-138-127 kubenswrapper[2581]: E0916 19:05:13.018412    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e34a585d8fe6d34030701b5b148c47129f2ba68517395b2caf75481bbd29b435\": container with ID starting with e34a585d8fe6d34030701b5b148c47129f2ba68517395b2caf75481bbd29b435 not found: ID does not exist" containerID="e34a585d8fe6d34030701b5b148c47129f2ba68517395b2caf75481bbd29b435"
Sep 16 19:05:13.018517 ip-10-0-138-127 kubenswrapper[2581]: I0916 19:05:13.018439    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e34a585d8fe6d34030701b5b148c47129f2ba68517395b2caf75481bbd29b435"} err="failed to get container status \"e34a585d8fe6d34030701b5b148c47129f2ba68517395b2caf75481bbd29b435\": rpc error: code = NotFound desc = could not find container \"e34a585d8fe6d34030701b5b148c47129f2ba68517395b2caf75481bbd29b435\": container with ID starting with e34a585d8fe6d34030701b5b148c47129f2ba68517395b2caf75481bbd29b435 not found: ID does not exist"
Sep 16 19:05:13.018517 ip-10-0-138-127 kubenswrapper[2581]: I0916 19:05:13.018461    2581 scope.go:117] "RemoveContainer" containerID="ed469c25248b4e759ae2e72b86e54eb49ee521231623f7525772535c7a3792a3"
Sep 16 19:05:13.018766 ip-10-0-138-127 kubenswrapper[2581]: E0916 19:05:13.018748    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ed469c25248b4e759ae2e72b86e54eb49ee521231623f7525772535c7a3792a3\": container with ID starting with ed469c25248b4e759ae2e72b86e54eb49ee521231623f7525772535c7a3792a3 not found: ID does not exist" containerID="ed469c25248b4e759ae2e72b86e54eb49ee521231623f7525772535c7a3792a3"
Sep 16 19:05:13.018805 ip-10-0-138-127 kubenswrapper[2581]: I0916 19:05:13.018770    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ed469c25248b4e759ae2e72b86e54eb49ee521231623f7525772535c7a3792a3"} err="failed to get container status \"ed469c25248b4e759ae2e72b86e54eb49ee521231623f7525772535c7a3792a3\": rpc error: code = NotFound desc = could not find container \"ed469c25248b4e759ae2e72b86e54eb49ee521231623f7525772535c7a3792a3\": container with ID starting with ed469c25248b4e759ae2e72b86e54eb49ee521231623f7525772535c7a3792a3 not found: ID does not exist"
Sep 16 19:05:13.018805 ip-10-0-138-127 kubenswrapper[2581]: I0916 19:05:13.018786    2581 scope.go:117] "RemoveContainer" containerID="4a3fed54c3387165aea5f008982bb415a51280c755383b65a2b38d5118db5fbb"
Sep 16 19:05:13.019044 ip-10-0-138-127 kubenswrapper[2581]: E0916 19:05:13.019027    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4a3fed54c3387165aea5f008982bb415a51280c755383b65a2b38d5118db5fbb\": container with ID starting with 4a3fed54c3387165aea5f008982bb415a51280c755383b65a2b38d5118db5fbb not found: ID does not exist" containerID="4a3fed54c3387165aea5f008982bb415a51280c755383b65a2b38d5118db5fbb"
Sep 16 19:05:13.019088 ip-10-0-138-127 kubenswrapper[2581]: I0916 19:05:13.019048    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4a3fed54c3387165aea5f008982bb415a51280c755383b65a2b38d5118db5fbb"} err="failed to get container status \"4a3fed54c3387165aea5f008982bb415a51280c755383b65a2b38d5118db5fbb\": rpc error: code = NotFound desc = could not find container \"4a3fed54c3387165aea5f008982bb415a51280c755383b65a2b38d5118db5fbb\": container with ID starting with 4a3fed54c3387165aea5f008982bb415a51280c755383b65a2b38d5118db5fbb not found: ID does not exist"
Sep 16 19:05:13.031371 ip-10-0-138-127 kubenswrapper[2581]: I0916 19:05:13.031334    2581 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/f8801259-429e-44ec-a099-9a9d4395e96f-kube-rbac-proxy-sar-config\") on node \"ip-10-0-138-127.ec2.internal\" DevicePath \"\""
Sep 16 19:05:13.031371 ip-10-0-138-127 kubenswrapper[2581]: I0916 19:05:13.031363    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2hlkn\" (UniqueName: \"kubernetes.io/projected/f8801259-429e-44ec-a099-9a9d4395e96f-kube-api-access-2hlkn\") on node \"ip-10-0-138-127.ec2.internal\" DevicePath \"\""
Sep 16 19:05:13.031371 ip-10-0-138-127 kubenswrapper[2581]: I0916 19:05:13.031379    2581 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f8801259-429e-44ec-a099-9a9d4395e96f-proxy-tls\") on node \"ip-10-0-138-127.ec2.internal\" DevicePath \"\""
Sep 16 19:05:13.034160 ip-10-0-138-127 kubenswrapper[2581]: I0916 19:05:13.034137    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-logger-raw-87fdd-predictor-f5488758-zhz5c"]
Sep 16 19:05:14.726937 ip-10-0-138-127 kubenswrapper[2581]: I0916 19:05:14.726900    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f8801259-429e-44ec-a099-9a9d4395e96f" path="/var/lib/kubelet/pods/f8801259-429e-44ec-a099-9a9d4395e96f/volumes"
Sep 16 19:05:15.894288 ip-10-0-138-127 kubenswrapper[2581]: I0916 19:05:15.894243    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-5517d-predictor-8547668d88-xw6bk" podUID="0fed021d-b37c-4ed9-8a77-e9258be99e13" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.34:8080: connect: connection refused"
Sep 16 19:05:25.894472 ip-10-0-138-127 kubenswrapper[2581]: I0916 19:05:25.894425    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-5517d-predictor-8547668d88-xw6bk" podUID="0fed021d-b37c-4ed9-8a77-e9258be99e13" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.34:8080: connect: connection refused"
Sep 16 19:05:35.893994 ip-10-0-138-127 kubenswrapper[2581]: I0916 19:05:35.893950    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-5517d-predictor-8547668d88-xw6bk" podUID="0fed021d-b37c-4ed9-8a77-e9258be99e13" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.34:8080: connect: connection refused"
Sep 16 19:05:45.893646 ip-10-0-138-127 kubenswrapper[2581]: I0916 19:05:45.893606    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-5517d-predictor-8547668d88-xw6bk" podUID="0fed021d-b37c-4ed9-8a77-e9258be99e13" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.34:8080: connect: connection refused"
Sep 16 19:05:55.893816 ip-10-0-138-127 kubenswrapper[2581]: I0916 19:05:55.893775    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-5517d-predictor-8547668d88-xw6bk" podUID="0fed021d-b37c-4ed9-8a77-e9258be99e13" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.34:8080: connect: connection refused"
Sep 16 19:06:05.893622 ip-10-0-138-127 kubenswrapper[2581]: I0916 19:06:05.893577    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-5517d-predictor-8547668d88-xw6bk" podUID="0fed021d-b37c-4ed9-8a77-e9258be99e13" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.34:8080: connect: connection refused"
Sep 16 19:06:15.894402 ip-10-0-138-127 kubenswrapper[2581]: I0916 19:06:15.894358    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-5517d-predictor-8547668d88-xw6bk" podUID="0fed021d-b37c-4ed9-8a77-e9258be99e13" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.34:8080: connect: connection refused"
Sep 16 19:06:17.722422 ip-10-0-138-127 kubenswrapper[2581]: I0916 19:06:17.722373    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-5517d-predictor-8547668d88-xw6bk" podUID="0fed021d-b37c-4ed9-8a77-e9258be99e13" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.34:8080: connect: connection refused"
Sep 16 19:06:27.722753 ip-10-0-138-127 kubenswrapper[2581]: I0916 19:06:27.722707    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-5517d-predictor-8547668d88-xw6bk" podUID="0fed021d-b37c-4ed9-8a77-e9258be99e13" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.34:8080: connect: connection refused"
Sep 16 19:06:37.722645 ip-10-0-138-127 kubenswrapper[2581]: I0916 19:06:37.722594    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-5517d-predictor-8547668d88-xw6bk" podUID="0fed021d-b37c-4ed9-8a77-e9258be99e13" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.34:8080: connect: connection refused"
Sep 16 19:06:47.722754 ip-10-0-138-127 kubenswrapper[2581]: I0916 19:06:47.722709    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-5517d-predictor-8547668d88-xw6bk" podUID="0fed021d-b37c-4ed9-8a77-e9258be99e13" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.34:8080: connect: connection refused"
Sep 16 19:06:57.722720 ip-10-0-138-127 kubenswrapper[2581]: I0916 19:06:57.722678    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-5517d-predictor-8547668d88-xw6bk" podUID="0fed021d-b37c-4ed9-8a77-e9258be99e13" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.34:8080: connect: connection refused"
Sep 16 19:07:07.722443 ip-10-0-138-127 kubenswrapper[2581]: I0916 19:07:07.722397    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-5517d-predictor-8547668d88-xw6bk" podUID="0fed021d-b37c-4ed9-8a77-e9258be99e13" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.34:8080: connect: connection refused"
Sep 16 19:07:17.722778 ip-10-0-138-127 kubenswrapper[2581]: I0916 19:07:17.722729    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-5517d-predictor-8547668d88-xw6bk" podUID="0fed021d-b37c-4ed9-8a77-e9258be99e13" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.34:8080: connect: connection refused"
Sep 16 19:07:27.723067 ip-10-0-138-127 kubenswrapper[2581]: I0916 19:07:27.723033    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-5517d-predictor-8547668d88-xw6bk"
Sep 16 19:07:32.813926 ip-10-0-138-127 kubenswrapper[2581]: I0916 19:07:32.813889    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-scale-raw-5517d-predictor-8547668d88-xw6bk"]
Sep 16 19:07:32.814511 ip-10-0-138-127 kubenswrapper[2581]: I0916 19:07:32.814259    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-5517d-predictor-8547668d88-xw6bk" podUID="0fed021d-b37c-4ed9-8a77-e9258be99e13" containerName="kserve-container" containerID="cri-o://f9ddef6cfd752eea6c966c04237361600056e86f095a1005176d8e6ed38a73d7" gracePeriod=30
Sep 16 19:07:32.814511 ip-10-0-138-127 kubenswrapper[2581]: I0916 19:07:32.814269    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-5517d-predictor-8547668d88-xw6bk" podUID="0fed021d-b37c-4ed9-8a77-e9258be99e13" containerName="kube-rbac-proxy" containerID="cri-o://13f0073b09ce7f02889af4128c1aacfab74636664288f7be94ecc5f1903cfc13" gracePeriod=30
Sep 16 19:07:33.568869 ip-10-0-138-127 kubenswrapper[2581]: I0916 19:07:33.568836    2581 generic.go:356] "Generic (PLEG): container finished" podID="0fed021d-b37c-4ed9-8a77-e9258be99e13" containerID="13f0073b09ce7f02889af4128c1aacfab74636664288f7be94ecc5f1903cfc13" exitCode=2
Sep 16 19:07:33.569053 ip-10-0-138-127 kubenswrapper[2581]: I0916 19:07:33.568886    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-5517d-predictor-8547668d88-xw6bk" event={"ID":"0fed021d-b37c-4ed9-8a77-e9258be99e13","Type":"ContainerDied","Data":"13f0073b09ce7f02889af4128c1aacfab74636664288f7be94ecc5f1903cfc13"}
Sep 16 19:07:35.888186 ip-10-0-138-127 kubenswrapper[2581]: I0916 19:07:35.888135    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-5517d-predictor-8547668d88-xw6bk" podUID="0fed021d-b37c-4ed9-8a77-e9258be99e13" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.34:8643/healthz\": dial tcp 10.132.0.34:8643: connect: connection refused"
Sep 16 19:07:37.722795 ip-10-0-138-127 kubenswrapper[2581]: I0916 19:07:37.722745    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-5517d-predictor-8547668d88-xw6bk" podUID="0fed021d-b37c-4ed9-8a77-e9258be99e13" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.34:8080: connect: connection refused"
Sep 16 19:07:38.027354 ip-10-0-138-127 kubenswrapper[2581]: I0916 19:07:38.027307    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-v2-predictor-6bbbd567bb-88fhf"]
Sep 16 19:07:38.027691 ip-10-0-138-127 kubenswrapper[2581]: I0916 19:07:38.027677    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f8801259-429e-44ec-a099-9a9d4395e96f" containerName="kserve-container"
Sep 16 19:07:38.027741 ip-10-0-138-127 kubenswrapper[2581]: I0916 19:07:38.027693    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="f8801259-429e-44ec-a099-9a9d4395e96f" containerName="kserve-container"
Sep 16 19:07:38.027741 ip-10-0-138-127 kubenswrapper[2581]: I0916 19:07:38.027707    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f8801259-429e-44ec-a099-9a9d4395e96f" containerName="agent"
Sep 16 19:07:38.027741 ip-10-0-138-127 kubenswrapper[2581]: I0916 19:07:38.027712    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="f8801259-429e-44ec-a099-9a9d4395e96f" containerName="agent"
Sep 16 19:07:38.027741 ip-10-0-138-127 kubenswrapper[2581]: I0916 19:07:38.027732    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f8801259-429e-44ec-a099-9a9d4395e96f" containerName="storage-initializer"
Sep 16 19:07:38.027741 ip-10-0-138-127 kubenswrapper[2581]: I0916 19:07:38.027737    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="f8801259-429e-44ec-a099-9a9d4395e96f" containerName="storage-initializer"
Sep 16 19:07:38.027904 ip-10-0-138-127 kubenswrapper[2581]: I0916 19:07:38.027744    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f8801259-429e-44ec-a099-9a9d4395e96f" containerName="kube-rbac-proxy"
Sep 16 19:07:38.027904 ip-10-0-138-127 kubenswrapper[2581]: I0916 19:07:38.027749    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="f8801259-429e-44ec-a099-9a9d4395e96f" containerName="kube-rbac-proxy"
Sep 16 19:07:38.027904 ip-10-0-138-127 kubenswrapper[2581]: I0916 19:07:38.027797    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="f8801259-429e-44ec-a099-9a9d4395e96f" containerName="kube-rbac-proxy"
Sep 16 19:07:38.027904 ip-10-0-138-127 kubenswrapper[2581]: I0916 19:07:38.027805    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="f8801259-429e-44ec-a099-9a9d4395e96f" containerName="kserve-container"
Sep 16 19:07:38.027904 ip-10-0-138-127 kubenswrapper[2581]: I0916 19:07:38.027812    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="f8801259-429e-44ec-a099-9a9d4395e96f" containerName="agent"
Sep 16 19:07:38.031332 ip-10-0-138-127 kubenswrapper[2581]: I0916 19:07:38.031307    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-6bbbd567bb-88fhf"
Sep 16 19:07:38.033627 ip-10-0-138-127 kubenswrapper[2581]: I0916 19:07:38.033596    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-create-kube-rbac-proxy-sar-config\""
Sep 16 19:07:38.033627 ip-10-0-138-127 kubenswrapper[2581]: I0916 19:07:38.033658    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-create-v2-predictor-serving-cert\""
Sep 16 19:07:38.046555 ip-10-0-138-127 kubenswrapper[2581]: I0916 19:07:38.046515    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-v2-predictor-6bbbd567bb-88fhf"]
Sep 16 19:07:38.079857 ip-10-0-138-127 kubenswrapper[2581]: I0916 19:07:38.079818    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/25009467-ea1c-4c27-af68-13edd785d20b-kube-rbac-proxy-sar-config\") pod \"isvc-canary-create-v2-predictor-6bbbd567bb-88fhf\" (UID: \"25009467-ea1c-4c27-af68-13edd785d20b\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-6bbbd567bb-88fhf"
Sep 16 19:07:38.080056 ip-10-0-138-127 kubenswrapper[2581]: I0916 19:07:38.079870    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/25009467-ea1c-4c27-af68-13edd785d20b-proxy-tls\") pod \"isvc-canary-create-v2-predictor-6bbbd567bb-88fhf\" (UID: \"25009467-ea1c-4c27-af68-13edd785d20b\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-6bbbd567bb-88fhf"
Sep 16 19:07:38.080056 ip-10-0-138-127 kubenswrapper[2581]: I0916 19:07:38.079917    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9vgp2\" (UniqueName: \"kubernetes.io/projected/25009467-ea1c-4c27-af68-13edd785d20b-kube-api-access-9vgp2\") pod \"isvc-canary-create-v2-predictor-6bbbd567bb-88fhf\" (UID: \"25009467-ea1c-4c27-af68-13edd785d20b\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-6bbbd567bb-88fhf"
Sep 16 19:07:38.080056 ip-10-0-138-127 kubenswrapper[2581]: I0916 19:07:38.079955    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/25009467-ea1c-4c27-af68-13edd785d20b-kserve-provision-location\") pod \"isvc-canary-create-v2-predictor-6bbbd567bb-88fhf\" (UID: \"25009467-ea1c-4c27-af68-13edd785d20b\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-6bbbd567bb-88fhf"
Sep 16 19:07:38.080242 ip-10-0-138-127 kubenswrapper[2581]: I0916 19:07:38.080067    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/25009467-ea1c-4c27-af68-13edd785d20b-cabundle-cert\") pod \"isvc-canary-create-v2-predictor-6bbbd567bb-88fhf\" (UID: \"25009467-ea1c-4c27-af68-13edd785d20b\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-6bbbd567bb-88fhf"
Sep 16 19:07:38.121612 ip-10-0-138-127 kubenswrapper[2581]: I0916 19:07:38.121575    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-predictor-58f6457999-cll22"]
Sep 16 19:07:38.125164 ip-10-0-138-127 kubenswrapper[2581]: I0916 19:07:38.125128    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-58f6457999-cll22"
Sep 16 19:07:38.127393 ip-10-0-138-127 kubenswrapper[2581]: I0916 19:07:38.127371    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-create-predictor-serving-cert\""
Sep 16 19:07:38.139341 ip-10-0-138-127 kubenswrapper[2581]: I0916 19:07:38.139309    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-predictor-58f6457999-cll22"]
Sep 16 19:07:38.181359 ip-10-0-138-127 kubenswrapper[2581]: I0916 19:07:38.181310    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-228jt\" (UniqueName: \"kubernetes.io/projected/a16a6e56-4afc-4526-8e4d-3c87d4b9ef61-kube-api-access-228jt\") pod \"isvc-canary-create-predictor-58f6457999-cll22\" (UID: \"a16a6e56-4afc-4526-8e4d-3c87d4b9ef61\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-58f6457999-cll22"
Sep 16 19:07:38.181542 ip-10-0-138-127 kubenswrapper[2581]: I0916 19:07:38.181420    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/25009467-ea1c-4c27-af68-13edd785d20b-proxy-tls\") pod \"isvc-canary-create-v2-predictor-6bbbd567bb-88fhf\" (UID: \"25009467-ea1c-4c27-af68-13edd785d20b\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-6bbbd567bb-88fhf"
Sep 16 19:07:38.181542 ip-10-0-138-127 kubenswrapper[2581]: I0916 19:07:38.181444    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9vgp2\" (UniqueName: \"kubernetes.io/projected/25009467-ea1c-4c27-af68-13edd785d20b-kube-api-access-9vgp2\") pod \"isvc-canary-create-v2-predictor-6bbbd567bb-88fhf\" (UID: \"25009467-ea1c-4c27-af68-13edd785d20b\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-6bbbd567bb-88fhf"
Sep 16 19:07:38.181542 ip-10-0-138-127 kubenswrapper[2581]: I0916 19:07:38.181470    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/25009467-ea1c-4c27-af68-13edd785d20b-kserve-provision-location\") pod \"isvc-canary-create-v2-predictor-6bbbd567bb-88fhf\" (UID: \"25009467-ea1c-4c27-af68-13edd785d20b\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-6bbbd567bb-88fhf"
Sep 16 19:07:38.181542 ip-10-0-138-127 kubenswrapper[2581]: I0916 19:07:38.181486    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a16a6e56-4afc-4526-8e4d-3c87d4b9ef61-kserve-provision-location\") pod \"isvc-canary-create-predictor-58f6457999-cll22\" (UID: \"a16a6e56-4afc-4526-8e4d-3c87d4b9ef61\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-58f6457999-cll22"
Sep 16 19:07:38.181542 ip-10-0-138-127 kubenswrapper[2581]: I0916 19:07:38.181531    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/a16a6e56-4afc-4526-8e4d-3c87d4b9ef61-cabundle-cert\") pod \"isvc-canary-create-predictor-58f6457999-cll22\" (UID: \"a16a6e56-4afc-4526-8e4d-3c87d4b9ef61\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-58f6457999-cll22"
Sep 16 19:07:38.181789 ip-10-0-138-127 kubenswrapper[2581]: I0916 19:07:38.181564    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a16a6e56-4afc-4526-8e4d-3c87d4b9ef61-proxy-tls\") pod \"isvc-canary-create-predictor-58f6457999-cll22\" (UID: \"a16a6e56-4afc-4526-8e4d-3c87d4b9ef61\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-58f6457999-cll22"
Sep 16 19:07:38.181789 ip-10-0-138-127 kubenswrapper[2581]: I0916 19:07:38.181592    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/25009467-ea1c-4c27-af68-13edd785d20b-cabundle-cert\") pod \"isvc-canary-create-v2-predictor-6bbbd567bb-88fhf\" (UID: \"25009467-ea1c-4c27-af68-13edd785d20b\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-6bbbd567bb-88fhf"
Sep 16 19:07:38.181789 ip-10-0-138-127 kubenswrapper[2581]: I0916 19:07:38.181666    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a16a6e56-4afc-4526-8e4d-3c87d4b9ef61-kube-rbac-proxy-sar-config\") pod \"isvc-canary-create-predictor-58f6457999-cll22\" (UID: \"a16a6e56-4afc-4526-8e4d-3c87d4b9ef61\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-58f6457999-cll22"
Sep 16 19:07:38.181789 ip-10-0-138-127 kubenswrapper[2581]: I0916 19:07:38.181736    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/25009467-ea1c-4c27-af68-13edd785d20b-kube-rbac-proxy-sar-config\") pod \"isvc-canary-create-v2-predictor-6bbbd567bb-88fhf\" (UID: \"25009467-ea1c-4c27-af68-13edd785d20b\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-6bbbd567bb-88fhf"
Sep 16 19:07:38.182008 ip-10-0-138-127 kubenswrapper[2581]: I0916 19:07:38.181905    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/25009467-ea1c-4c27-af68-13edd785d20b-kserve-provision-location\") pod \"isvc-canary-create-v2-predictor-6bbbd567bb-88fhf\" (UID: \"25009467-ea1c-4c27-af68-13edd785d20b\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-6bbbd567bb-88fhf"
Sep 16 19:07:38.182321 ip-10-0-138-127 kubenswrapper[2581]: I0916 19:07:38.182299    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/25009467-ea1c-4c27-af68-13edd785d20b-cabundle-cert\") pod \"isvc-canary-create-v2-predictor-6bbbd567bb-88fhf\" (UID: \"25009467-ea1c-4c27-af68-13edd785d20b\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-6bbbd567bb-88fhf"
Sep 16 19:07:38.182321 ip-10-0-138-127 kubenswrapper[2581]: I0916 19:07:38.182312    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/25009467-ea1c-4c27-af68-13edd785d20b-kube-rbac-proxy-sar-config\") pod \"isvc-canary-create-v2-predictor-6bbbd567bb-88fhf\" (UID: \"25009467-ea1c-4c27-af68-13edd785d20b\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-6bbbd567bb-88fhf"
Sep 16 19:07:38.184172 ip-10-0-138-127 kubenswrapper[2581]: I0916 19:07:38.184152    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/25009467-ea1c-4c27-af68-13edd785d20b-proxy-tls\") pod \"isvc-canary-create-v2-predictor-6bbbd567bb-88fhf\" (UID: \"25009467-ea1c-4c27-af68-13edd785d20b\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-6bbbd567bb-88fhf"
Sep 16 19:07:38.193679 ip-10-0-138-127 kubenswrapper[2581]: I0916 19:07:38.193652    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9vgp2\" (UniqueName: \"kubernetes.io/projected/25009467-ea1c-4c27-af68-13edd785d20b-kube-api-access-9vgp2\") pod \"isvc-canary-create-v2-predictor-6bbbd567bb-88fhf\" (UID: \"25009467-ea1c-4c27-af68-13edd785d20b\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-6bbbd567bb-88fhf"
Sep 16 19:07:38.283089 ip-10-0-138-127 kubenswrapper[2581]: I0916 19:07:38.282981    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a16a6e56-4afc-4526-8e4d-3c87d4b9ef61-kserve-provision-location\") pod \"isvc-canary-create-predictor-58f6457999-cll22\" (UID: \"a16a6e56-4afc-4526-8e4d-3c87d4b9ef61\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-58f6457999-cll22"
Sep 16 19:07:38.283089 ip-10-0-138-127 kubenswrapper[2581]: I0916 19:07:38.283044    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/a16a6e56-4afc-4526-8e4d-3c87d4b9ef61-cabundle-cert\") pod \"isvc-canary-create-predictor-58f6457999-cll22\" (UID: \"a16a6e56-4afc-4526-8e4d-3c87d4b9ef61\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-58f6457999-cll22"
Sep 16 19:07:38.283089 ip-10-0-138-127 kubenswrapper[2581]: I0916 19:07:38.283076    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a16a6e56-4afc-4526-8e4d-3c87d4b9ef61-proxy-tls\") pod \"isvc-canary-create-predictor-58f6457999-cll22\" (UID: \"a16a6e56-4afc-4526-8e4d-3c87d4b9ef61\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-58f6457999-cll22"
Sep 16 19:07:38.283433 ip-10-0-138-127 kubenswrapper[2581]: I0916 19:07:38.283103    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a16a6e56-4afc-4526-8e4d-3c87d4b9ef61-kube-rbac-proxy-sar-config\") pod \"isvc-canary-create-predictor-58f6457999-cll22\" (UID: \"a16a6e56-4afc-4526-8e4d-3c87d4b9ef61\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-58f6457999-cll22"
Sep 16 19:07:38.283433 ip-10-0-138-127 kubenswrapper[2581]: I0916 19:07:38.283152    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-228jt\" (UniqueName: \"kubernetes.io/projected/a16a6e56-4afc-4526-8e4d-3c87d4b9ef61-kube-api-access-228jt\") pod \"isvc-canary-create-predictor-58f6457999-cll22\" (UID: \"a16a6e56-4afc-4526-8e4d-3c87d4b9ef61\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-58f6457999-cll22"
Sep 16 19:07:38.283551 ip-10-0-138-127 kubenswrapper[2581]: I0916 19:07:38.283506    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a16a6e56-4afc-4526-8e4d-3c87d4b9ef61-kserve-provision-location\") pod \"isvc-canary-create-predictor-58f6457999-cll22\" (UID: \"a16a6e56-4afc-4526-8e4d-3c87d4b9ef61\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-58f6457999-cll22"
Sep 16 19:07:38.283761 ip-10-0-138-127 kubenswrapper[2581]: I0916 19:07:38.283741    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/a16a6e56-4afc-4526-8e4d-3c87d4b9ef61-cabundle-cert\") pod \"isvc-canary-create-predictor-58f6457999-cll22\" (UID: \"a16a6e56-4afc-4526-8e4d-3c87d4b9ef61\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-58f6457999-cll22"
Sep 16 19:07:38.283852 ip-10-0-138-127 kubenswrapper[2581]: I0916 19:07:38.283835    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a16a6e56-4afc-4526-8e4d-3c87d4b9ef61-kube-rbac-proxy-sar-config\") pod \"isvc-canary-create-predictor-58f6457999-cll22\" (UID: \"a16a6e56-4afc-4526-8e4d-3c87d4b9ef61\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-58f6457999-cll22"
Sep 16 19:07:38.286043 ip-10-0-138-127 kubenswrapper[2581]: I0916 19:07:38.286014    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a16a6e56-4afc-4526-8e4d-3c87d4b9ef61-proxy-tls\") pod \"isvc-canary-create-predictor-58f6457999-cll22\" (UID: \"a16a6e56-4afc-4526-8e4d-3c87d4b9ef61\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-58f6457999-cll22"
Sep 16 19:07:38.292677 ip-10-0-138-127 kubenswrapper[2581]: I0916 19:07:38.292648    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-228jt\" (UniqueName: \"kubernetes.io/projected/a16a6e56-4afc-4526-8e4d-3c87d4b9ef61-kube-api-access-228jt\") pod \"isvc-canary-create-predictor-58f6457999-cll22\" (UID: \"a16a6e56-4afc-4526-8e4d-3c87d4b9ef61\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-58f6457999-cll22"
Sep 16 19:07:38.343380 ip-10-0-138-127 kubenswrapper[2581]: I0916 19:07:38.343339    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-6bbbd567bb-88fhf"
Sep 16 19:07:38.435820 ip-10-0-138-127 kubenswrapper[2581]: I0916 19:07:38.435776    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-58f6457999-cll22"
Sep 16 19:07:38.619498 ip-10-0-138-127 kubenswrapper[2581]: I0916 19:07:38.619471    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-v2-predictor-6bbbd567bb-88fhf"]
Sep 16 19:07:38.622031 ip-10-0-138-127 kubenswrapper[2581]: W0916 19:07:38.622001    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod25009467_ea1c_4c27_af68_13edd785d20b.slice/crio-eeaa688210f4ab73280d9097a7bc90344762b49e8b879deb66236dba69a4d5c5 WatchSource:0}: Error finding container eeaa688210f4ab73280d9097a7bc90344762b49e8b879deb66236dba69a4d5c5: Status 404 returned error can't find the container with id eeaa688210f4ab73280d9097a7bc90344762b49e8b879deb66236dba69a4d5c5
Sep 16 19:07:38.713982 ip-10-0-138-127 kubenswrapper[2581]: I0916 19:07:38.713955    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-predictor-58f6457999-cll22"]
Sep 16 19:07:38.716332 ip-10-0-138-127 kubenswrapper[2581]: W0916 19:07:38.716296    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda16a6e56_4afc_4526_8e4d_3c87d4b9ef61.slice/crio-163dd373155f7b15f73d6f6de2020d12759978580bd2d6c2e9f6299f1090741a WatchSource:0}: Error finding container 163dd373155f7b15f73d6f6de2020d12759978580bd2d6c2e9f6299f1090741a: Status 404 returned error can't find the container with id 163dd373155f7b15f73d6f6de2020d12759978580bd2d6c2e9f6299f1090741a
Sep 16 19:07:39.595765 ip-10-0-138-127 kubenswrapper[2581]: I0916 19:07:39.595711    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-58f6457999-cll22" event={"ID":"a16a6e56-4afc-4526-8e4d-3c87d4b9ef61","Type":"ContainerStarted","Data":"47498ef1b6dfe71ed3bb928e6cc925c998f249d6ba0d8d2276c5f25cb7d2f110"}
Sep 16 19:07:39.596315 ip-10-0-138-127 kubenswrapper[2581]: I0916 19:07:39.595804    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-58f6457999-cll22" event={"ID":"a16a6e56-4afc-4526-8e4d-3c87d4b9ef61","Type":"ContainerStarted","Data":"163dd373155f7b15f73d6f6de2020d12759978580bd2d6c2e9f6299f1090741a"}
Sep 16 19:07:39.597323 ip-10-0-138-127 kubenswrapper[2581]: I0916 19:07:39.597293    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-6bbbd567bb-88fhf" event={"ID":"25009467-ea1c-4c27-af68-13edd785d20b","Type":"ContainerStarted","Data":"b5e52398c2570593077c92264b305a9e9b7174ebc919555c5bc331e51d4ae241"}
Sep 16 19:07:39.597445 ip-10-0-138-127 kubenswrapper[2581]: I0916 19:07:39.597344    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-6bbbd567bb-88fhf" event={"ID":"25009467-ea1c-4c27-af68-13edd785d20b","Type":"ContainerStarted","Data":"eeaa688210f4ab73280d9097a7bc90344762b49e8b879deb66236dba69a4d5c5"}
Sep 16 19:07:40.889170 ip-10-0-138-127 kubenswrapper[2581]: I0916 19:07:40.889118    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-5517d-predictor-8547668d88-xw6bk" podUID="0fed021d-b37c-4ed9-8a77-e9258be99e13" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.34:8643/healthz\": dial tcp 10.132.0.34:8643: connect: connection refused"
Sep 16 19:07:42.611996 ip-10-0-138-127 kubenswrapper[2581]: I0916 19:07:42.611958    2581 generic.go:356] "Generic (PLEG): container finished" podID="a16a6e56-4afc-4526-8e4d-3c87d4b9ef61" containerID="47498ef1b6dfe71ed3bb928e6cc925c998f249d6ba0d8d2276c5f25cb7d2f110" exitCode=0
Sep 16 19:07:42.612403 ip-10-0-138-127 kubenswrapper[2581]: I0916 19:07:42.612038    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-58f6457999-cll22" event={"ID":"a16a6e56-4afc-4526-8e4d-3c87d4b9ef61","Type":"ContainerDied","Data":"47498ef1b6dfe71ed3bb928e6cc925c998f249d6ba0d8d2276c5f25cb7d2f110"}
Sep 16 19:07:43.618786 ip-10-0-138-127 kubenswrapper[2581]: I0916 19:07:43.618743    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-58f6457999-cll22" event={"ID":"a16a6e56-4afc-4526-8e4d-3c87d4b9ef61","Type":"ContainerStarted","Data":"748e23fe31f50df810da8101f35f0a14c1b8c5f949296eb1a9ed3bac2748d85e"}
Sep 16 19:07:43.619293 ip-10-0-138-127 kubenswrapper[2581]: I0916 19:07:43.618821    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-58f6457999-cll22" event={"ID":"a16a6e56-4afc-4526-8e4d-3c87d4b9ef61","Type":"ContainerStarted","Data":"adaf950f599b95f04225fb453299ec381ee8f35727706cce1145e57c65264d7a"}
Sep 16 19:07:43.619293 ip-10-0-138-127 kubenswrapper[2581]: I0916 19:07:43.619088    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-58f6457999-cll22"
Sep 16 19:07:43.620671 ip-10-0-138-127 kubenswrapper[2581]: I0916 19:07:43.620645    2581 generic.go:356] "Generic (PLEG): container finished" podID="25009467-ea1c-4c27-af68-13edd785d20b" containerID="b5e52398c2570593077c92264b305a9e9b7174ebc919555c5bc331e51d4ae241" exitCode=0
Sep 16 19:07:43.620827 ip-10-0-138-127 kubenswrapper[2581]: I0916 19:07:43.620678    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-6bbbd567bb-88fhf" event={"ID":"25009467-ea1c-4c27-af68-13edd785d20b","Type":"ContainerDied","Data":"b5e52398c2570593077c92264b305a9e9b7174ebc919555c5bc331e51d4ae241"}
Sep 16 19:07:43.639830 ip-10-0-138-127 kubenswrapper[2581]: I0916 19:07:43.639770    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-58f6457999-cll22" podStartSLOduration=1.8693614859999999 podStartE2EDuration="5.639743038s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.770381552s" isStatefulPod=false podCreationTimestamp="2026-09-16 19:07:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:07:43.638852827 +0000 UTC m=+1333.458266739" watchObservedRunningTime="2026-09-16 19:07:43.639743038 +0000 UTC m=+1333.459156947"
Sep 16 19:07:43.780157 ip-10-0-138-127 kubenswrapper[2581]: I0916 19:07:43.780118    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-5517d-predictor-8547668d88-xw6bk"
Sep 16 19:07:43.827870 ip-10-0-138-127 kubenswrapper[2581]: I0916 19:07:43.827758    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/0fed021d-b37c-4ed9-8a77-e9258be99e13-cabundle-cert\") pod \"0fed021d-b37c-4ed9-8a77-e9258be99e13\" (UID: \"0fed021d-b37c-4ed9-8a77-e9258be99e13\") "
Sep 16 19:07:43.827870 ip-10-0-138-127 kubenswrapper[2581]: I0916 19:07:43.827799    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-nfz4s\" (UniqueName: \"kubernetes.io/projected/0fed021d-b37c-4ed9-8a77-e9258be99e13-kube-api-access-nfz4s\") pod \"0fed021d-b37c-4ed9-8a77-e9258be99e13\" (UID: \"0fed021d-b37c-4ed9-8a77-e9258be99e13\") "
Sep 16 19:07:43.828099 ip-10-0-138-127 kubenswrapper[2581]: I0916 19:07:43.827931    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/0fed021d-b37c-4ed9-8a77-e9258be99e13-kserve-provision-location\") pod \"0fed021d-b37c-4ed9-8a77-e9258be99e13\" (UID: \"0fed021d-b37c-4ed9-8a77-e9258be99e13\") "
Sep 16 19:07:43.828099 ip-10-0-138-127 kubenswrapper[2581]: I0916 19:07:43.827985    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/0fed021d-b37c-4ed9-8a77-e9258be99e13-kube-rbac-proxy-sar-config\") pod \"0fed021d-b37c-4ed9-8a77-e9258be99e13\" (UID: \"0fed021d-b37c-4ed9-8a77-e9258be99e13\") "
Sep 16 19:07:43.828099 ip-10-0-138-127 kubenswrapper[2581]: I0916 19:07:43.828004    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/0fed021d-b37c-4ed9-8a77-e9258be99e13-proxy-tls\") pod \"0fed021d-b37c-4ed9-8a77-e9258be99e13\" (UID: \"0fed021d-b37c-4ed9-8a77-e9258be99e13\") "
Sep 16 19:07:43.828315 ip-10-0-138-127 kubenswrapper[2581]: I0916 19:07:43.828288    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0fed021d-b37c-4ed9-8a77-e9258be99e13-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "0fed021d-b37c-4ed9-8a77-e9258be99e13" (UID: "0fed021d-b37c-4ed9-8a77-e9258be99e13"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 19:07:43.828378 ip-10-0-138-127 kubenswrapper[2581]: I0916 19:07:43.828308    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0fed021d-b37c-4ed9-8a77-e9258be99e13-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "0fed021d-b37c-4ed9-8a77-e9258be99e13" (UID: "0fed021d-b37c-4ed9-8a77-e9258be99e13"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:07:43.828378 ip-10-0-138-127 kubenswrapper[2581]: I0916 19:07:43.828319    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0fed021d-b37c-4ed9-8a77-e9258be99e13-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "0fed021d-b37c-4ed9-8a77-e9258be99e13" (UID: "0fed021d-b37c-4ed9-8a77-e9258be99e13"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 19:07:43.828478 ip-10-0-138-127 kubenswrapper[2581]: I0916 19:07:43.828435    2581 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/0fed021d-b37c-4ed9-8a77-e9258be99e13-kserve-provision-location\") on node \"ip-10-0-138-127.ec2.internal\" DevicePath \"\""
Sep 16 19:07:43.828688 ip-10-0-138-127 kubenswrapper[2581]: I0916 19:07:43.828634    2581 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/0fed021d-b37c-4ed9-8a77-e9258be99e13-kube-rbac-proxy-sar-config\") on node \"ip-10-0-138-127.ec2.internal\" DevicePath \"\""
Sep 16 19:07:43.828688 ip-10-0-138-127 kubenswrapper[2581]: I0916 19:07:43.828664    2581 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/0fed021d-b37c-4ed9-8a77-e9258be99e13-cabundle-cert\") on node \"ip-10-0-138-127.ec2.internal\" DevicePath \"\""
Sep 16 19:07:43.830477 ip-10-0-138-127 kubenswrapper[2581]: I0916 19:07:43.830452    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0fed021d-b37c-4ed9-8a77-e9258be99e13-kube-api-access-nfz4s" (OuterVolumeSpecName: "kube-api-access-nfz4s") pod "0fed021d-b37c-4ed9-8a77-e9258be99e13" (UID: "0fed021d-b37c-4ed9-8a77-e9258be99e13"). InnerVolumeSpecName "kube-api-access-nfz4s". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 19:07:43.830477 ip-10-0-138-127 kubenswrapper[2581]: I0916 19:07:43.830460    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0fed021d-b37c-4ed9-8a77-e9258be99e13-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "0fed021d-b37c-4ed9-8a77-e9258be99e13" (UID: "0fed021d-b37c-4ed9-8a77-e9258be99e13"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 19:07:43.929912 ip-10-0-138-127 kubenswrapper[2581]: I0916 19:07:43.929861    2581 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/0fed021d-b37c-4ed9-8a77-e9258be99e13-proxy-tls\") on node \"ip-10-0-138-127.ec2.internal\" DevicePath \"\""
Sep 16 19:07:43.929912 ip-10-0-138-127 kubenswrapper[2581]: I0916 19:07:43.929896    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nfz4s\" (UniqueName: \"kubernetes.io/projected/0fed021d-b37c-4ed9-8a77-e9258be99e13-kube-api-access-nfz4s\") on node \"ip-10-0-138-127.ec2.internal\" DevicePath \"\""
Sep 16 19:07:44.627219 ip-10-0-138-127 kubenswrapper[2581]: I0916 19:07:44.627170    2581 generic.go:356] "Generic (PLEG): container finished" podID="0fed021d-b37c-4ed9-8a77-e9258be99e13" containerID="f9ddef6cfd752eea6c966c04237361600056e86f095a1005176d8e6ed38a73d7" exitCode=0
Sep 16 19:07:44.627654 ip-10-0-138-127 kubenswrapper[2581]: I0916 19:07:44.627271    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-5517d-predictor-8547668d88-xw6bk"
Sep 16 19:07:44.627654 ip-10-0-138-127 kubenswrapper[2581]: I0916 19:07:44.627273    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-5517d-predictor-8547668d88-xw6bk" event={"ID":"0fed021d-b37c-4ed9-8a77-e9258be99e13","Type":"ContainerDied","Data":"f9ddef6cfd752eea6c966c04237361600056e86f095a1005176d8e6ed38a73d7"}
Sep 16 19:07:44.627654 ip-10-0-138-127 kubenswrapper[2581]: I0916 19:07:44.627346    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-5517d-predictor-8547668d88-xw6bk" event={"ID":"0fed021d-b37c-4ed9-8a77-e9258be99e13","Type":"ContainerDied","Data":"5ca9f2a3d15b0a9927842d5830cb0caadcc346ddd999988879ce15b3817663d9"}
Sep 16 19:07:44.627654 ip-10-0-138-127 kubenswrapper[2581]: I0916 19:07:44.627360    2581 scope.go:117] "RemoveContainer" containerID="13f0073b09ce7f02889af4128c1aacfab74636664288f7be94ecc5f1903cfc13"
Sep 16 19:07:44.629888 ip-10-0-138-127 kubenswrapper[2581]: I0916 19:07:44.629860    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-6bbbd567bb-88fhf" event={"ID":"25009467-ea1c-4c27-af68-13edd785d20b","Type":"ContainerStarted","Data":"ef323aa4e12c28c4b069ec297b45cd2edd1f499114847b3b50636b19bf2dd93d"}
Sep 16 19:07:44.630189 ip-10-0-138-127 kubenswrapper[2581]: I0916 19:07:44.630173    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-58f6457999-cll22"
Sep 16 19:07:44.630303 ip-10-0-138-127 kubenswrapper[2581]: I0916 19:07:44.630236    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-6bbbd567bb-88fhf" event={"ID":"25009467-ea1c-4c27-af68-13edd785d20b","Type":"ContainerStarted","Data":"618d69e0964da844e32053e7c674fdddaffa35a0343732dd775f7f88160b7d3d"}
Sep 16 19:07:44.630412 ip-10-0-138-127 kubenswrapper[2581]: I0916 19:07:44.630399    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-6bbbd567bb-88fhf"
Sep 16 19:07:44.631495 ip-10-0-138-127 kubenswrapper[2581]: I0916 19:07:44.631464    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-58f6457999-cll22" podUID="a16a6e56-4afc-4526-8e4d-3c87d4b9ef61" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.36:8080: connect: connection refused"
Sep 16 19:07:44.637096 ip-10-0-138-127 kubenswrapper[2581]: I0916 19:07:44.637077    2581 scope.go:117] "RemoveContainer" containerID="f9ddef6cfd752eea6c966c04237361600056e86f095a1005176d8e6ed38a73d7"
Sep 16 19:07:44.647343 ip-10-0-138-127 kubenswrapper[2581]: I0916 19:07:44.647323    2581 scope.go:117] "RemoveContainer" containerID="ea36f8bbd8ad97554d1d33d6084c2424b7c1241ff1f75107cdbb66f0218b1c43"
Sep 16 19:07:44.653654 ip-10-0-138-127 kubenswrapper[2581]: I0916 19:07:44.653596    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-6bbbd567bb-88fhf" podStartSLOduration=2.639373968 podStartE2EDuration="6.653573698s" totalImagesPullingTime="0s" totalInitContainerRuntime="4.01419973s" isStatefulPod=false podCreationTimestamp="2026-09-16 19:07:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:07:44.651477383 +0000 UTC m=+1334.470891307" watchObservedRunningTime="2026-09-16 19:07:44.653573698 +0000 UTC m=+1334.472987608"
Sep 16 19:07:44.657800 ip-10-0-138-127 kubenswrapper[2581]: I0916 19:07:44.657778    2581 scope.go:117] "RemoveContainer" containerID="13f0073b09ce7f02889af4128c1aacfab74636664288f7be94ecc5f1903cfc13"
Sep 16 19:07:44.658227 ip-10-0-138-127 kubenswrapper[2581]: E0916 19:07:44.658181    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"13f0073b09ce7f02889af4128c1aacfab74636664288f7be94ecc5f1903cfc13\": container with ID starting with 13f0073b09ce7f02889af4128c1aacfab74636664288f7be94ecc5f1903cfc13 not found: ID does not exist" containerID="13f0073b09ce7f02889af4128c1aacfab74636664288f7be94ecc5f1903cfc13"
Sep 16 19:07:44.658351 ip-10-0-138-127 kubenswrapper[2581]: I0916 19:07:44.658237    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"13f0073b09ce7f02889af4128c1aacfab74636664288f7be94ecc5f1903cfc13"} err="failed to get container status \"13f0073b09ce7f02889af4128c1aacfab74636664288f7be94ecc5f1903cfc13\": rpc error: code = NotFound desc = could not find container \"13f0073b09ce7f02889af4128c1aacfab74636664288f7be94ecc5f1903cfc13\": container with ID starting with 13f0073b09ce7f02889af4128c1aacfab74636664288f7be94ecc5f1903cfc13 not found: ID does not exist"
Sep 16 19:07:44.658351 ip-10-0-138-127 kubenswrapper[2581]: I0916 19:07:44.658255    2581 scope.go:117] "RemoveContainer" containerID="f9ddef6cfd752eea6c966c04237361600056e86f095a1005176d8e6ed38a73d7"
Sep 16 19:07:44.658573 ip-10-0-138-127 kubenswrapper[2581]: E0916 19:07:44.658555    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f9ddef6cfd752eea6c966c04237361600056e86f095a1005176d8e6ed38a73d7\": container with ID starting with f9ddef6cfd752eea6c966c04237361600056e86f095a1005176d8e6ed38a73d7 not found: ID does not exist" containerID="f9ddef6cfd752eea6c966c04237361600056e86f095a1005176d8e6ed38a73d7"
Sep 16 19:07:44.658634 ip-10-0-138-127 kubenswrapper[2581]: I0916 19:07:44.658582    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f9ddef6cfd752eea6c966c04237361600056e86f095a1005176d8e6ed38a73d7"} err="failed to get container status \"f9ddef6cfd752eea6c966c04237361600056e86f095a1005176d8e6ed38a73d7\": rpc error: code = NotFound desc = could not find container \"f9ddef6cfd752eea6c966c04237361600056e86f095a1005176d8e6ed38a73d7\": container with ID starting with f9ddef6cfd752eea6c966c04237361600056e86f095a1005176d8e6ed38a73d7 not found: ID does not exist"
Sep 16 19:07:44.658634 ip-10-0-138-127 kubenswrapper[2581]: I0916 19:07:44.658605    2581 scope.go:117] "RemoveContainer" containerID="ea36f8bbd8ad97554d1d33d6084c2424b7c1241ff1f75107cdbb66f0218b1c43"
Sep 16 19:07:44.658992 ip-10-0-138-127 kubenswrapper[2581]: E0916 19:07:44.658976    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ea36f8bbd8ad97554d1d33d6084c2424b7c1241ff1f75107cdbb66f0218b1c43\": container with ID starting with ea36f8bbd8ad97554d1d33d6084c2424b7c1241ff1f75107cdbb66f0218b1c43 not found: ID does not exist" containerID="ea36f8bbd8ad97554d1d33d6084c2424b7c1241ff1f75107cdbb66f0218b1c43"
Sep 16 19:07:44.659056 ip-10-0-138-127 kubenswrapper[2581]: I0916 19:07:44.658997    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ea36f8bbd8ad97554d1d33d6084c2424b7c1241ff1f75107cdbb66f0218b1c43"} err="failed to get container status \"ea36f8bbd8ad97554d1d33d6084c2424b7c1241ff1f75107cdbb66f0218b1c43\": rpc error: code = NotFound desc = could not find container \"ea36f8bbd8ad97554d1d33d6084c2424b7c1241ff1f75107cdbb66f0218b1c43\": container with ID starting with ea36f8bbd8ad97554d1d33d6084c2424b7c1241ff1f75107cdbb66f0218b1c43 not found: ID does not exist"
Sep 16 19:07:44.664645 ip-10-0-138-127 kubenswrapper[2581]: I0916 19:07:44.664613    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-scale-raw-5517d-predictor-8547668d88-xw6bk"]
Sep 16 19:07:44.669255 ip-10-0-138-127 kubenswrapper[2581]: I0916 19:07:44.669226    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-scale-raw-5517d-predictor-8547668d88-xw6bk"]
Sep 16 19:07:44.727770 ip-10-0-138-127 kubenswrapper[2581]: I0916 19:07:44.727734    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0fed021d-b37c-4ed9-8a77-e9258be99e13" path="/var/lib/kubelet/pods/0fed021d-b37c-4ed9-8a77-e9258be99e13/volumes"
Sep 16 19:07:45.641132 ip-10-0-138-127 kubenswrapper[2581]: I0916 19:07:45.641082    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-58f6457999-cll22" podUID="a16a6e56-4afc-4526-8e4d-3c87d4b9ef61" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.36:8080: connect: connection refused"
Sep 16 19:07:45.641636 ip-10-0-138-127 kubenswrapper[2581]: I0916 19:07:45.641302    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-6bbbd567bb-88fhf"
Sep 16 19:07:45.642304 ip-10-0-138-127 kubenswrapper[2581]: I0916 19:07:45.642279    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-6bbbd567bb-88fhf" podUID="25009467-ea1c-4c27-af68-13edd785d20b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.35:8080: connect: connection refused"
Sep 16 19:07:46.645472 ip-10-0-138-127 kubenswrapper[2581]: I0916 19:07:46.645434    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-6bbbd567bb-88fhf" podUID="25009467-ea1c-4c27-af68-13edd785d20b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.35:8080: connect: connection refused"
Sep 16 19:07:50.646161 ip-10-0-138-127 kubenswrapper[2581]: I0916 19:07:50.646131    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-58f6457999-cll22"
Sep 16 19:07:50.646714 ip-10-0-138-127 kubenswrapper[2581]: I0916 19:07:50.646689    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-58f6457999-cll22" podUID="a16a6e56-4afc-4526-8e4d-3c87d4b9ef61" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.36:8080: connect: connection refused"
Sep 16 19:07:51.650545 ip-10-0-138-127 kubenswrapper[2581]: I0916 19:07:51.650509    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-6bbbd567bb-88fhf"
Sep 16 19:07:51.651005 ip-10-0-138-127 kubenswrapper[2581]: I0916 19:07:51.650981    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-6bbbd567bb-88fhf" podUID="25009467-ea1c-4c27-af68-13edd785d20b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.35:8080: connect: connection refused"
Sep 16 19:08:00.647327 ip-10-0-138-127 kubenswrapper[2581]: I0916 19:08:00.647259    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-58f6457999-cll22" podUID="a16a6e56-4afc-4526-8e4d-3c87d4b9ef61" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.36:8080: connect: connection refused"
Sep 16 19:08:01.651296 ip-10-0-138-127 kubenswrapper[2581]: I0916 19:08:01.651253    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-6bbbd567bb-88fhf" podUID="25009467-ea1c-4c27-af68-13edd785d20b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.35:8080: connect: connection refused"
Sep 16 19:08:10.646960 ip-10-0-138-127 kubenswrapper[2581]: I0916 19:08:10.646912    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-58f6457999-cll22" podUID="a16a6e56-4afc-4526-8e4d-3c87d4b9ef61" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.36:8080: connect: connection refused"
Sep 16 19:08:11.651431 ip-10-0-138-127 kubenswrapper[2581]: I0916 19:08:11.651385    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-6bbbd567bb-88fhf" podUID="25009467-ea1c-4c27-af68-13edd785d20b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.35:8080: connect: connection refused"
Sep 16 19:08:20.647144 ip-10-0-138-127 kubenswrapper[2581]: I0916 19:08:20.647104    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-58f6457999-cll22" podUID="a16a6e56-4afc-4526-8e4d-3c87d4b9ef61" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.36:8080: connect: connection refused"
Sep 16 19:08:21.651686 ip-10-0-138-127 kubenswrapper[2581]: I0916 19:08:21.651642    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-6bbbd567bb-88fhf" podUID="25009467-ea1c-4c27-af68-13edd785d20b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.35:8080: connect: connection refused"
Sep 16 19:08:30.646756 ip-10-0-138-127 kubenswrapper[2581]: I0916 19:08:30.646711    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-58f6457999-cll22" podUID="a16a6e56-4afc-4526-8e4d-3c87d4b9ef61" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.36:8080: connect: connection refused"
Sep 16 19:08:31.651796 ip-10-0-138-127 kubenswrapper[2581]: I0916 19:08:31.651753    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-6bbbd567bb-88fhf" podUID="25009467-ea1c-4c27-af68-13edd785d20b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.35:8080: connect: connection refused"
Sep 16 19:08:40.646693 ip-10-0-138-127 kubenswrapper[2581]: I0916 19:08:40.646652    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-58f6457999-cll22" podUID="a16a6e56-4afc-4526-8e4d-3c87d4b9ef61" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.36:8080: connect: connection refused"
Sep 16 19:08:41.651800 ip-10-0-138-127 kubenswrapper[2581]: I0916 19:08:41.651760    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-6bbbd567bb-88fhf" podUID="25009467-ea1c-4c27-af68-13edd785d20b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.35:8080: connect: connection refused"
Sep 16 19:08:50.647268 ip-10-0-138-127 kubenswrapper[2581]: I0916 19:08:50.647225    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-58f6457999-cll22" podUID="a16a6e56-4afc-4526-8e4d-3c87d4b9ef61" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.36:8080: connect: connection refused"
Sep 16 19:08:51.651001 ip-10-0-138-127 kubenswrapper[2581]: I0916 19:08:51.650962    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-6bbbd567bb-88fhf" podUID="25009467-ea1c-4c27-af68-13edd785d20b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.35:8080: connect: connection refused"
Sep 16 19:08:54.723055 ip-10-0-138-127 kubenswrapper[2581]: I0916 19:08:54.723004    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-58f6457999-cll22" podUID="a16a6e56-4afc-4526-8e4d-3c87d4b9ef61" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.36:8080: connect: connection refused"
Sep 16 19:09:01.651398 ip-10-0-138-127 kubenswrapper[2581]: I0916 19:09:01.651363    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-6bbbd567bb-88fhf"
Sep 16 19:09:01.793635 ip-10-0-138-127 kubenswrapper[2581]: I0916 19:09:01.793590    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-predictor-58f6457999-cll22"]
Sep 16 19:09:01.793881 ip-10-0-138-127 kubenswrapper[2581]: I0916 19:09:01.793839    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-58f6457999-cll22" podUID="a16a6e56-4afc-4526-8e4d-3c87d4b9ef61" containerName="kserve-container" containerID="cri-o://adaf950f599b95f04225fb453299ec381ee8f35727706cce1145e57c65264d7a" gracePeriod=30
Sep 16 19:09:01.793950 ip-10-0-138-127 kubenswrapper[2581]: I0916 19:09:01.793908    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-58f6457999-cll22" podUID="a16a6e56-4afc-4526-8e4d-3c87d4b9ef61" containerName="kube-rbac-proxy" containerID="cri-o://748e23fe31f50df810da8101f35f0a14c1b8c5f949296eb1a9ed3bac2748d85e" gracePeriod=30
Sep 16 19:09:01.794662 ip-10-0-138-127 kubenswrapper[2581]: I0916 19:09:01.794374    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-58f6457999-cll22"
Sep 16 19:09:01.968674 ip-10-0-138-127 kubenswrapper[2581]: I0916 19:09:01.968640    2581 generic.go:356] "Generic (PLEG): container finished" podID="a16a6e56-4afc-4526-8e4d-3c87d4b9ef61" containerID="748e23fe31f50df810da8101f35f0a14c1b8c5f949296eb1a9ed3bac2748d85e" exitCode=2
Sep 16 19:09:01.968855 ip-10-0-138-127 kubenswrapper[2581]: I0916 19:09:01.968692    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-58f6457999-cll22" event={"ID":"a16a6e56-4afc-4526-8e4d-3c87d4b9ef61","Type":"ContainerDied","Data":"748e23fe31f50df810da8101f35f0a14c1b8c5f949296eb1a9ed3bac2748d85e"}
Sep 16 19:09:05.642084 ip-10-0-138-127 kubenswrapper[2581]: I0916 19:09:05.642043    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-58f6457999-cll22" podUID="a16a6e56-4afc-4526-8e4d-3c87d4b9ef61" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.36:8643/healthz\": dial tcp 10.132.0.36:8643: connect: connection refused"
Sep 16 19:09:07.153677 ip-10-0-138-127 kubenswrapper[2581]: I0916 19:09:07.153644    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-58f6457999-cll22"
Sep 16 19:09:07.257827 ip-10-0-138-127 kubenswrapper[2581]: I0916 19:09:07.257722    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a16a6e56-4afc-4526-8e4d-3c87d4b9ef61-kube-rbac-proxy-sar-config\") pod \"a16a6e56-4afc-4526-8e4d-3c87d4b9ef61\" (UID: \"a16a6e56-4afc-4526-8e4d-3c87d4b9ef61\") "
Sep 16 19:09:07.257827 ip-10-0-138-127 kubenswrapper[2581]: I0916 19:09:07.257791    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/a16a6e56-4afc-4526-8e4d-3c87d4b9ef61-cabundle-cert\") pod \"a16a6e56-4afc-4526-8e4d-3c87d4b9ef61\" (UID: \"a16a6e56-4afc-4526-8e4d-3c87d4b9ef61\") "
Sep 16 19:09:07.258029 ip-10-0-138-127 kubenswrapper[2581]: I0916 19:09:07.257937    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a16a6e56-4afc-4526-8e4d-3c87d4b9ef61-proxy-tls\") pod \"a16a6e56-4afc-4526-8e4d-3c87d4b9ef61\" (UID: \"a16a6e56-4afc-4526-8e4d-3c87d4b9ef61\") "
Sep 16 19:09:07.258029 ip-10-0-138-127 kubenswrapper[2581]: I0916 19:09:07.257981    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-228jt\" (UniqueName: \"kubernetes.io/projected/a16a6e56-4afc-4526-8e4d-3c87d4b9ef61-kube-api-access-228jt\") pod \"a16a6e56-4afc-4526-8e4d-3c87d4b9ef61\" (UID: \"a16a6e56-4afc-4526-8e4d-3c87d4b9ef61\") "
Sep 16 19:09:07.258096 ip-10-0-138-127 kubenswrapper[2581]: I0916 19:09:07.258031    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a16a6e56-4afc-4526-8e4d-3c87d4b9ef61-kserve-provision-location\") pod \"a16a6e56-4afc-4526-8e4d-3c87d4b9ef61\" (UID: \"a16a6e56-4afc-4526-8e4d-3c87d4b9ef61\") "
Sep 16 19:09:07.258188 ip-10-0-138-127 kubenswrapper[2581]: I0916 19:09:07.258165    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a16a6e56-4afc-4526-8e4d-3c87d4b9ef61-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "a16a6e56-4afc-4526-8e4d-3c87d4b9ef61" (UID: "a16a6e56-4afc-4526-8e4d-3c87d4b9ef61"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 19:09:07.258188 ip-10-0-138-127 kubenswrapper[2581]: I0916 19:09:07.258168    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a16a6e56-4afc-4526-8e4d-3c87d4b9ef61-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "a16a6e56-4afc-4526-8e4d-3c87d4b9ef61" (UID: "a16a6e56-4afc-4526-8e4d-3c87d4b9ef61"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 19:09:07.258424 ip-10-0-138-127 kubenswrapper[2581]: I0916 19:09:07.258404    2581 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a16a6e56-4afc-4526-8e4d-3c87d4b9ef61-kube-rbac-proxy-sar-config\") on node \"ip-10-0-138-127.ec2.internal\" DevicePath \"\""
Sep 16 19:09:07.258493 ip-10-0-138-127 kubenswrapper[2581]: I0916 19:09:07.258429    2581 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/a16a6e56-4afc-4526-8e4d-3c87d4b9ef61-cabundle-cert\") on node \"ip-10-0-138-127.ec2.internal\" DevicePath \"\""
Sep 16 19:09:07.258493 ip-10-0-138-127 kubenswrapper[2581]: I0916 19:09:07.258454    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a16a6e56-4afc-4526-8e4d-3c87d4b9ef61-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "a16a6e56-4afc-4526-8e4d-3c87d4b9ef61" (UID: "a16a6e56-4afc-4526-8e4d-3c87d4b9ef61"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:09:07.260390 ip-10-0-138-127 kubenswrapper[2581]: I0916 19:09:07.260368    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a16a6e56-4afc-4526-8e4d-3c87d4b9ef61-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "a16a6e56-4afc-4526-8e4d-3c87d4b9ef61" (UID: "a16a6e56-4afc-4526-8e4d-3c87d4b9ef61"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 19:09:07.260506 ip-10-0-138-127 kubenswrapper[2581]: I0916 19:09:07.260386    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a16a6e56-4afc-4526-8e4d-3c87d4b9ef61-kube-api-access-228jt" (OuterVolumeSpecName: "kube-api-access-228jt") pod "a16a6e56-4afc-4526-8e4d-3c87d4b9ef61" (UID: "a16a6e56-4afc-4526-8e4d-3c87d4b9ef61"). InnerVolumeSpecName "kube-api-access-228jt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 19:09:07.359496 ip-10-0-138-127 kubenswrapper[2581]: I0916 19:09:07.359447    2581 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a16a6e56-4afc-4526-8e4d-3c87d4b9ef61-proxy-tls\") on node \"ip-10-0-138-127.ec2.internal\" DevicePath \"\""
Sep 16 19:09:07.359496 ip-10-0-138-127 kubenswrapper[2581]: I0916 19:09:07.359488    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-228jt\" (UniqueName: \"kubernetes.io/projected/a16a6e56-4afc-4526-8e4d-3c87d4b9ef61-kube-api-access-228jt\") on node \"ip-10-0-138-127.ec2.internal\" DevicePath \"\""
Sep 16 19:09:07.359496 ip-10-0-138-127 kubenswrapper[2581]: I0916 19:09:07.359498    2581 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a16a6e56-4afc-4526-8e4d-3c87d4b9ef61-kserve-provision-location\") on node \"ip-10-0-138-127.ec2.internal\" DevicePath \"\""
Sep 16 19:09:07.996067 ip-10-0-138-127 kubenswrapper[2581]: I0916 19:09:07.996030    2581 generic.go:356] "Generic (PLEG): container finished" podID="a16a6e56-4afc-4526-8e4d-3c87d4b9ef61" containerID="adaf950f599b95f04225fb453299ec381ee8f35727706cce1145e57c65264d7a" exitCode=0
Sep 16 19:09:07.996372 ip-10-0-138-127 kubenswrapper[2581]: I0916 19:09:07.996092    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-58f6457999-cll22"
Sep 16 19:09:07.996372 ip-10-0-138-127 kubenswrapper[2581]: I0916 19:09:07.996136    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-58f6457999-cll22" event={"ID":"a16a6e56-4afc-4526-8e4d-3c87d4b9ef61","Type":"ContainerDied","Data":"adaf950f599b95f04225fb453299ec381ee8f35727706cce1145e57c65264d7a"}
Sep 16 19:09:07.996372 ip-10-0-138-127 kubenswrapper[2581]: I0916 19:09:07.996186    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-58f6457999-cll22" event={"ID":"a16a6e56-4afc-4526-8e4d-3c87d4b9ef61","Type":"ContainerDied","Data":"163dd373155f7b15f73d6f6de2020d12759978580bd2d6c2e9f6299f1090741a"}
Sep 16 19:09:07.996372 ip-10-0-138-127 kubenswrapper[2581]: I0916 19:09:07.996231    2581 scope.go:117] "RemoveContainer" containerID="748e23fe31f50df810da8101f35f0a14c1b8c5f949296eb1a9ed3bac2748d85e"
Sep 16 19:09:08.007862 ip-10-0-138-127 kubenswrapper[2581]: I0916 19:09:08.007820    2581 scope.go:117] "RemoveContainer" containerID="adaf950f599b95f04225fb453299ec381ee8f35727706cce1145e57c65264d7a"
Sep 16 19:09:08.017532 ip-10-0-138-127 kubenswrapper[2581]: I0916 19:09:08.017424    2581 scope.go:117] "RemoveContainer" containerID="47498ef1b6dfe71ed3bb928e6cc925c998f249d6ba0d8d2276c5f25cb7d2f110"
Sep 16 19:09:08.026405 ip-10-0-138-127 kubenswrapper[2581]: I0916 19:09:08.026379    2581 scope.go:117] "RemoveContainer" containerID="748e23fe31f50df810da8101f35f0a14c1b8c5f949296eb1a9ed3bac2748d85e"
Sep 16 19:09:08.026759 ip-10-0-138-127 kubenswrapper[2581]: I0916 19:09:08.026728    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-predictor-58f6457999-cll22"]
Sep 16 19:09:08.026891 ip-10-0-138-127 kubenswrapper[2581]: E0916 19:09:08.026868    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"748e23fe31f50df810da8101f35f0a14c1b8c5f949296eb1a9ed3bac2748d85e\": container with ID starting with 748e23fe31f50df810da8101f35f0a14c1b8c5f949296eb1a9ed3bac2748d85e not found: ID does not exist" containerID="748e23fe31f50df810da8101f35f0a14c1b8c5f949296eb1a9ed3bac2748d85e"
Sep 16 19:09:08.026941 ip-10-0-138-127 kubenswrapper[2581]: I0916 19:09:08.026902    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"748e23fe31f50df810da8101f35f0a14c1b8c5f949296eb1a9ed3bac2748d85e"} err="failed to get container status \"748e23fe31f50df810da8101f35f0a14c1b8c5f949296eb1a9ed3bac2748d85e\": rpc error: code = NotFound desc = could not find container \"748e23fe31f50df810da8101f35f0a14c1b8c5f949296eb1a9ed3bac2748d85e\": container with ID starting with 748e23fe31f50df810da8101f35f0a14c1b8c5f949296eb1a9ed3bac2748d85e not found: ID does not exist"
Sep 16 19:09:08.026941 ip-10-0-138-127 kubenswrapper[2581]: I0916 19:09:08.026923    2581 scope.go:117] "RemoveContainer" containerID="adaf950f599b95f04225fb453299ec381ee8f35727706cce1145e57c65264d7a"
Sep 16 19:09:08.027260 ip-10-0-138-127 kubenswrapper[2581]: E0916 19:09:08.027244    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"adaf950f599b95f04225fb453299ec381ee8f35727706cce1145e57c65264d7a\": container with ID starting with adaf950f599b95f04225fb453299ec381ee8f35727706cce1145e57c65264d7a not found: ID does not exist" containerID="adaf950f599b95f04225fb453299ec381ee8f35727706cce1145e57c65264d7a"
Sep 16 19:09:08.027308 ip-10-0-138-127 kubenswrapper[2581]: I0916 19:09:08.027267    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"adaf950f599b95f04225fb453299ec381ee8f35727706cce1145e57c65264d7a"} err="failed to get container status \"adaf950f599b95f04225fb453299ec381ee8f35727706cce1145e57c65264d7a\": rpc error: code = NotFound desc = could not find container \"adaf950f599b95f04225fb453299ec381ee8f35727706cce1145e57c65264d7a\": container with ID starting with adaf950f599b95f04225fb453299ec381ee8f35727706cce1145e57c65264d7a not found: ID does not exist"
Sep 16 19:09:08.027308 ip-10-0-138-127 kubenswrapper[2581]: I0916 19:09:08.027284    2581 scope.go:117] "RemoveContainer" containerID="47498ef1b6dfe71ed3bb928e6cc925c998f249d6ba0d8d2276c5f25cb7d2f110"
Sep 16 19:09:08.027543 ip-10-0-138-127 kubenswrapper[2581]: E0916 19:09:08.027524    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"47498ef1b6dfe71ed3bb928e6cc925c998f249d6ba0d8d2276c5f25cb7d2f110\": container with ID starting with 47498ef1b6dfe71ed3bb928e6cc925c998f249d6ba0d8d2276c5f25cb7d2f110 not found: ID does not exist" containerID="47498ef1b6dfe71ed3bb928e6cc925c998f249d6ba0d8d2276c5f25cb7d2f110"
Sep 16 19:09:08.027595 ip-10-0-138-127 kubenswrapper[2581]: I0916 19:09:08.027548    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"47498ef1b6dfe71ed3bb928e6cc925c998f249d6ba0d8d2276c5f25cb7d2f110"} err="failed to get container status \"47498ef1b6dfe71ed3bb928e6cc925c998f249d6ba0d8d2276c5f25cb7d2f110\": rpc error: code = NotFound desc = could not find container \"47498ef1b6dfe71ed3bb928e6cc925c998f249d6ba0d8d2276c5f25cb7d2f110\": container with ID starting with 47498ef1b6dfe71ed3bb928e6cc925c998f249d6ba0d8d2276c5f25cb7d2f110 not found: ID does not exist"
Sep 16 19:09:08.031041 ip-10-0-138-127 kubenswrapper[2581]: I0916 19:09:08.031008    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-predictor-58f6457999-cll22"]
Sep 16 19:09:08.170949 ip-10-0-138-127 kubenswrapper[2581]: I0916 19:09:08.170905    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-v2-predictor-6bbbd567bb-88fhf"]
Sep 16 19:09:08.171382 ip-10-0-138-127 kubenswrapper[2581]: I0916 19:09:08.171276    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-6bbbd567bb-88fhf" podUID="25009467-ea1c-4c27-af68-13edd785d20b" containerName="kserve-container" containerID="cri-o://618d69e0964da844e32053e7c674fdddaffa35a0343732dd775f7f88160b7d3d" gracePeriod=30
Sep 16 19:09:08.171434 ip-10-0-138-127 kubenswrapper[2581]: I0916 19:09:08.171361    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-6bbbd567bb-88fhf" podUID="25009467-ea1c-4c27-af68-13edd785d20b" containerName="kube-rbac-proxy" containerID="cri-o://ef323aa4e12c28c4b069ec297b45cd2edd1f499114847b3b50636b19bf2dd93d" gracePeriod=30
Sep 16 19:09:08.727348 ip-10-0-138-127 kubenswrapper[2581]: I0916 19:09:08.727258    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a16a6e56-4afc-4526-8e4d-3c87d4b9ef61" path="/var/lib/kubelet/pods/a16a6e56-4afc-4526-8e4d-3c87d4b9ef61/volumes"
Sep 16 19:09:09.003164 ip-10-0-138-127 kubenswrapper[2581]: I0916 19:09:09.003077    2581 generic.go:356] "Generic (PLEG): container finished" podID="25009467-ea1c-4c27-af68-13edd785d20b" containerID="ef323aa4e12c28c4b069ec297b45cd2edd1f499114847b3b50636b19bf2dd93d" exitCode=2
Sep 16 19:09:09.003333 ip-10-0-138-127 kubenswrapper[2581]: I0916 19:09:09.003155    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-6bbbd567bb-88fhf" event={"ID":"25009467-ea1c-4c27-af68-13edd785d20b","Type":"ContainerDied","Data":"ef323aa4e12c28c4b069ec297b45cd2edd1f499114847b3b50636b19bf2dd93d"}
Sep 16 19:09:11.646216 ip-10-0-138-127 kubenswrapper[2581]: I0916 19:09:11.646164    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-6bbbd567bb-88fhf" podUID="25009467-ea1c-4c27-af68-13edd785d20b" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.35:8643/healthz\": dial tcp 10.132.0.35:8643: connect: connection refused"
Sep 16 19:09:11.651702 ip-10-0-138-127 kubenswrapper[2581]: I0916 19:09:11.651667    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-6bbbd567bb-88fhf" podUID="25009467-ea1c-4c27-af68-13edd785d20b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.35:8080: connect: connection refused"
Sep 16 19:09:13.330789 ip-10-0-138-127 kubenswrapper[2581]: I0916 19:09:13.330749    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-predictor-5c9955746c-8fpkr"]
Sep 16 19:09:13.331243 ip-10-0-138-127 kubenswrapper[2581]: I0916 19:09:13.331089    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a16a6e56-4afc-4526-8e4d-3c87d4b9ef61" containerName="storage-initializer"
Sep 16 19:09:13.331243 ip-10-0-138-127 kubenswrapper[2581]: I0916 19:09:13.331101    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="a16a6e56-4afc-4526-8e4d-3c87d4b9ef61" containerName="storage-initializer"
Sep 16 19:09:13.331243 ip-10-0-138-127 kubenswrapper[2581]: I0916 19:09:13.331110    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a16a6e56-4afc-4526-8e4d-3c87d4b9ef61" containerName="kserve-container"
Sep 16 19:09:13.331243 ip-10-0-138-127 kubenswrapper[2581]: I0916 19:09:13.331115    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="a16a6e56-4afc-4526-8e4d-3c87d4b9ef61" containerName="kserve-container"
Sep 16 19:09:13.331243 ip-10-0-138-127 kubenswrapper[2581]: I0916 19:09:13.331123    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a16a6e56-4afc-4526-8e4d-3c87d4b9ef61" containerName="kube-rbac-proxy"
Sep 16 19:09:13.331243 ip-10-0-138-127 kubenswrapper[2581]: I0916 19:09:13.331128    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="a16a6e56-4afc-4526-8e4d-3c87d4b9ef61" containerName="kube-rbac-proxy"
Sep 16 19:09:13.331243 ip-10-0-138-127 kubenswrapper[2581]: I0916 19:09:13.331142    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0fed021d-b37c-4ed9-8a77-e9258be99e13" containerName="storage-initializer"
Sep 16 19:09:13.331243 ip-10-0-138-127 kubenswrapper[2581]: I0916 19:09:13.331147    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="0fed021d-b37c-4ed9-8a77-e9258be99e13" containerName="storage-initializer"
Sep 16 19:09:13.331243 ip-10-0-138-127 kubenswrapper[2581]: I0916 19:09:13.331153    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0fed021d-b37c-4ed9-8a77-e9258be99e13" containerName="kserve-container"
Sep 16 19:09:13.331243 ip-10-0-138-127 kubenswrapper[2581]: I0916 19:09:13.331159    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="0fed021d-b37c-4ed9-8a77-e9258be99e13" containerName="kserve-container"
Sep 16 19:09:13.331243 ip-10-0-138-127 kubenswrapper[2581]: I0916 19:09:13.331165    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0fed021d-b37c-4ed9-8a77-e9258be99e13" containerName="kube-rbac-proxy"
Sep 16 19:09:13.331243 ip-10-0-138-127 kubenswrapper[2581]: I0916 19:09:13.331169    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="0fed021d-b37c-4ed9-8a77-e9258be99e13" containerName="kube-rbac-proxy"
Sep 16 19:09:13.331243 ip-10-0-138-127 kubenswrapper[2581]: I0916 19:09:13.331243    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="a16a6e56-4afc-4526-8e4d-3c87d4b9ef61" containerName="kserve-container"
Sep 16 19:09:13.331243 ip-10-0-138-127 kubenswrapper[2581]: I0916 19:09:13.331252    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="0fed021d-b37c-4ed9-8a77-e9258be99e13" containerName="kserve-container"
Sep 16 19:09:13.331714 ip-10-0-138-127 kubenswrapper[2581]: I0916 19:09:13.331259    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="0fed021d-b37c-4ed9-8a77-e9258be99e13" containerName="kube-rbac-proxy"
Sep 16 19:09:13.331714 ip-10-0-138-127 kubenswrapper[2581]: I0916 19:09:13.331266    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="a16a6e56-4afc-4526-8e4d-3c87d4b9ef61" containerName="kube-rbac-proxy"
Sep 16 19:09:13.334669 ip-10-0-138-127 kubenswrapper[2581]: I0916 19:09:13.334647    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-5c9955746c-8fpkr"
Sep 16 19:09:13.336861 ip-10-0-138-127 kubenswrapper[2581]: I0916 19:09:13.336819    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-promote-kube-rbac-proxy-sar-config\""
Sep 16 19:09:13.337016 ip-10-0-138-127 kubenswrapper[2581]: I0916 19:09:13.336973    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-promote-predictor-serving-cert\""
Sep 16 19:09:13.348011 ip-10-0-138-127 kubenswrapper[2581]: I0916 19:09:13.347976    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-predictor-5c9955746c-8fpkr"]
Sep 16 19:09:13.513450 ip-10-0-138-127 kubenswrapper[2581]: I0916 19:09:13.513387    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/626e5804-1f71-42af-abc8-e8934619263b-kube-rbac-proxy-sar-config\") pod \"isvc-canary-promote-predictor-5c9955746c-8fpkr\" (UID: \"626e5804-1f71-42af-abc8-e8934619263b\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-5c9955746c-8fpkr"
Sep 16 19:09:13.513670 ip-10-0-138-127 kubenswrapper[2581]: I0916 19:09:13.513531    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/626e5804-1f71-42af-abc8-e8934619263b-cabundle-cert\") pod \"isvc-canary-promote-predictor-5c9955746c-8fpkr\" (UID: \"626e5804-1f71-42af-abc8-e8934619263b\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-5c9955746c-8fpkr"
Sep 16 19:09:13.513670 ip-10-0-138-127 kubenswrapper[2581]: I0916 19:09:13.513611    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/626e5804-1f71-42af-abc8-e8934619263b-kserve-provision-location\") pod \"isvc-canary-promote-predictor-5c9955746c-8fpkr\" (UID: \"626e5804-1f71-42af-abc8-e8934619263b\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-5c9955746c-8fpkr"
Sep 16 19:09:13.513670 ip-10-0-138-127 kubenswrapper[2581]: I0916 19:09:13.513646    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/626e5804-1f71-42af-abc8-e8934619263b-proxy-tls\") pod \"isvc-canary-promote-predictor-5c9955746c-8fpkr\" (UID: \"626e5804-1f71-42af-abc8-e8934619263b\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-5c9955746c-8fpkr"
Sep 16 19:09:13.513846 ip-10-0-138-127 kubenswrapper[2581]: I0916 19:09:13.513698    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7qfzv\" (UniqueName: \"kubernetes.io/projected/626e5804-1f71-42af-abc8-e8934619263b-kube-api-access-7qfzv\") pod \"isvc-canary-promote-predictor-5c9955746c-8fpkr\" (UID: \"626e5804-1f71-42af-abc8-e8934619263b\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-5c9955746c-8fpkr"
Sep 16 19:09:13.614848 ip-10-0-138-127 kubenswrapper[2581]: I0916 19:09:13.614810    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/626e5804-1f71-42af-abc8-e8934619263b-kserve-provision-location\") pod \"isvc-canary-promote-predictor-5c9955746c-8fpkr\" (UID: \"626e5804-1f71-42af-abc8-e8934619263b\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-5c9955746c-8fpkr"
Sep 16 19:09:13.614848 ip-10-0-138-127 kubenswrapper[2581]: I0916 19:09:13.614856    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/626e5804-1f71-42af-abc8-e8934619263b-proxy-tls\") pod \"isvc-canary-promote-predictor-5c9955746c-8fpkr\" (UID: \"626e5804-1f71-42af-abc8-e8934619263b\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-5c9955746c-8fpkr"
Sep 16 19:09:13.615102 ip-10-0-138-127 kubenswrapper[2581]: I0916 19:09:13.614885    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7qfzv\" (UniqueName: \"kubernetes.io/projected/626e5804-1f71-42af-abc8-e8934619263b-kube-api-access-7qfzv\") pod \"isvc-canary-promote-predictor-5c9955746c-8fpkr\" (UID: \"626e5804-1f71-42af-abc8-e8934619263b\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-5c9955746c-8fpkr"
Sep 16 19:09:13.615102 ip-10-0-138-127 kubenswrapper[2581]: I0916 19:09:13.614965    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/626e5804-1f71-42af-abc8-e8934619263b-kube-rbac-proxy-sar-config\") pod \"isvc-canary-promote-predictor-5c9955746c-8fpkr\" (UID: \"626e5804-1f71-42af-abc8-e8934619263b\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-5c9955746c-8fpkr"
Sep 16 19:09:13.615102 ip-10-0-138-127 kubenswrapper[2581]: I0916 19:09:13.615048    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/626e5804-1f71-42af-abc8-e8934619263b-cabundle-cert\") pod \"isvc-canary-promote-predictor-5c9955746c-8fpkr\" (UID: \"626e5804-1f71-42af-abc8-e8934619263b\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-5c9955746c-8fpkr"
Sep 16 19:09:13.615550 ip-10-0-138-127 kubenswrapper[2581]: I0916 19:09:13.615319    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/626e5804-1f71-42af-abc8-e8934619263b-kserve-provision-location\") pod \"isvc-canary-promote-predictor-5c9955746c-8fpkr\" (UID: \"626e5804-1f71-42af-abc8-e8934619263b\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-5c9955746c-8fpkr"
Sep 16 19:09:13.615811 ip-10-0-138-127 kubenswrapper[2581]: I0916 19:09:13.615769    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/626e5804-1f71-42af-abc8-e8934619263b-cabundle-cert\") pod \"isvc-canary-promote-predictor-5c9955746c-8fpkr\" (UID: \"626e5804-1f71-42af-abc8-e8934619263b\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-5c9955746c-8fpkr"
Sep 16 19:09:13.615927 ip-10-0-138-127 kubenswrapper[2581]: I0916 19:09:13.615797    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/626e5804-1f71-42af-abc8-e8934619263b-kube-rbac-proxy-sar-config\") pod \"isvc-canary-promote-predictor-5c9955746c-8fpkr\" (UID: \"626e5804-1f71-42af-abc8-e8934619263b\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-5c9955746c-8fpkr"
Sep 16 19:09:13.617891 ip-10-0-138-127 kubenswrapper[2581]: I0916 19:09:13.617865    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/626e5804-1f71-42af-abc8-e8934619263b-proxy-tls\") pod \"isvc-canary-promote-predictor-5c9955746c-8fpkr\" (UID: \"626e5804-1f71-42af-abc8-e8934619263b\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-5c9955746c-8fpkr"
Sep 16 19:09:13.624188 ip-10-0-138-127 kubenswrapper[2581]: I0916 19:09:13.624157    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7qfzv\" (UniqueName: \"kubernetes.io/projected/626e5804-1f71-42af-abc8-e8934619263b-kube-api-access-7qfzv\") pod \"isvc-canary-promote-predictor-5c9955746c-8fpkr\" (UID: \"626e5804-1f71-42af-abc8-e8934619263b\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-5c9955746c-8fpkr"
Sep 16 19:09:13.641524 ip-10-0-138-127 kubenswrapper[2581]: I0916 19:09:13.641491    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-6bbbd567bb-88fhf"
Sep 16 19:09:13.645235 ip-10-0-138-127 kubenswrapper[2581]: I0916 19:09:13.645204    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-5c9955746c-8fpkr"
Sep 16 19:09:13.816510 ip-10-0-138-127 kubenswrapper[2581]: I0916 19:09:13.816452    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/25009467-ea1c-4c27-af68-13edd785d20b-cabundle-cert\") pod \"25009467-ea1c-4c27-af68-13edd785d20b\" (UID: \"25009467-ea1c-4c27-af68-13edd785d20b\") "
Sep 16 19:09:13.816510 ip-10-0-138-127 kubenswrapper[2581]: I0916 19:09:13.816511    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/25009467-ea1c-4c27-af68-13edd785d20b-proxy-tls\") pod \"25009467-ea1c-4c27-af68-13edd785d20b\" (UID: \"25009467-ea1c-4c27-af68-13edd785d20b\") "
Sep 16 19:09:13.816731 ip-10-0-138-127 kubenswrapper[2581]: I0916 19:09:13.816527    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9vgp2\" (UniqueName: \"kubernetes.io/projected/25009467-ea1c-4c27-af68-13edd785d20b-kube-api-access-9vgp2\") pod \"25009467-ea1c-4c27-af68-13edd785d20b\" (UID: \"25009467-ea1c-4c27-af68-13edd785d20b\") "
Sep 16 19:09:13.816731 ip-10-0-138-127 kubenswrapper[2581]: I0916 19:09:13.816581    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/25009467-ea1c-4c27-af68-13edd785d20b-kube-rbac-proxy-sar-config\") pod \"25009467-ea1c-4c27-af68-13edd785d20b\" (UID: \"25009467-ea1c-4c27-af68-13edd785d20b\") "
Sep 16 19:09:13.816731 ip-10-0-138-127 kubenswrapper[2581]: I0916 19:09:13.816611    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/25009467-ea1c-4c27-af68-13edd785d20b-kserve-provision-location\") pod \"25009467-ea1c-4c27-af68-13edd785d20b\" (UID: \"25009467-ea1c-4c27-af68-13edd785d20b\") "
Sep 16 19:09:13.816965 ip-10-0-138-127 kubenswrapper[2581]: I0916 19:09:13.816938    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/25009467-ea1c-4c27-af68-13edd785d20b-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "25009467-ea1c-4c27-af68-13edd785d20b" (UID: "25009467-ea1c-4c27-af68-13edd785d20b"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 19:09:13.817005 ip-10-0-138-127 kubenswrapper[2581]: I0916 19:09:13.816955    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/25009467-ea1c-4c27-af68-13edd785d20b-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "25009467-ea1c-4c27-af68-13edd785d20b" (UID: "25009467-ea1c-4c27-af68-13edd785d20b"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 19:09:13.817005 ip-10-0-138-127 kubenswrapper[2581]: I0916 19:09:13.816976    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/25009467-ea1c-4c27-af68-13edd785d20b-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "25009467-ea1c-4c27-af68-13edd785d20b" (UID: "25009467-ea1c-4c27-af68-13edd785d20b"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:09:13.818977 ip-10-0-138-127 kubenswrapper[2581]: I0916 19:09:13.818955    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/25009467-ea1c-4c27-af68-13edd785d20b-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "25009467-ea1c-4c27-af68-13edd785d20b" (UID: "25009467-ea1c-4c27-af68-13edd785d20b"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 19:09:13.819044 ip-10-0-138-127 kubenswrapper[2581]: I0916 19:09:13.818974    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/25009467-ea1c-4c27-af68-13edd785d20b-kube-api-access-9vgp2" (OuterVolumeSpecName: "kube-api-access-9vgp2") pod "25009467-ea1c-4c27-af68-13edd785d20b" (UID: "25009467-ea1c-4c27-af68-13edd785d20b"). InnerVolumeSpecName "kube-api-access-9vgp2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 19:09:13.918262 ip-10-0-138-127 kubenswrapper[2581]: I0916 19:09:13.918141    2581 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/25009467-ea1c-4c27-af68-13edd785d20b-kube-rbac-proxy-sar-config\") on node \"ip-10-0-138-127.ec2.internal\" DevicePath \"\""
Sep 16 19:09:13.918262 ip-10-0-138-127 kubenswrapper[2581]: I0916 19:09:13.918178    2581 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/25009467-ea1c-4c27-af68-13edd785d20b-kserve-provision-location\") on node \"ip-10-0-138-127.ec2.internal\" DevicePath \"\""
Sep 16 19:09:13.918262 ip-10-0-138-127 kubenswrapper[2581]: I0916 19:09:13.918188    2581 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/25009467-ea1c-4c27-af68-13edd785d20b-cabundle-cert\") on node \"ip-10-0-138-127.ec2.internal\" DevicePath \"\""
Sep 16 19:09:13.918262 ip-10-0-138-127 kubenswrapper[2581]: I0916 19:09:13.918227    2581 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/25009467-ea1c-4c27-af68-13edd785d20b-proxy-tls\") on node \"ip-10-0-138-127.ec2.internal\" DevicePath \"\""
Sep 16 19:09:13.918262 ip-10-0-138-127 kubenswrapper[2581]: I0916 19:09:13.918238    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9vgp2\" (UniqueName: \"kubernetes.io/projected/25009467-ea1c-4c27-af68-13edd785d20b-kube-api-access-9vgp2\") on node \"ip-10-0-138-127.ec2.internal\" DevicePath \"\""
Sep 16 19:09:13.942654 ip-10-0-138-127 kubenswrapper[2581]: I0916 19:09:13.942627    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-predictor-5c9955746c-8fpkr"]
Sep 16 19:09:13.945011 ip-10-0-138-127 kubenswrapper[2581]: W0916 19:09:13.944981    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod626e5804_1f71_42af_abc8_e8934619263b.slice/crio-8eb77f7510c29a4d493c39e6c18beb99d1dd6b8dcd9dd699300f2ce6322f3db9 WatchSource:0}: Error finding container 8eb77f7510c29a4d493c39e6c18beb99d1dd6b8dcd9dd699300f2ce6322f3db9: Status 404 returned error can't find the container with id 8eb77f7510c29a4d493c39e6c18beb99d1dd6b8dcd9dd699300f2ce6322f3db9
Sep 16 19:09:14.025505 ip-10-0-138-127 kubenswrapper[2581]: I0916 19:09:14.025461    2581 generic.go:356] "Generic (PLEG): container finished" podID="25009467-ea1c-4c27-af68-13edd785d20b" containerID="618d69e0964da844e32053e7c674fdddaffa35a0343732dd775f7f88160b7d3d" exitCode=0
Sep 16 19:09:14.025709 ip-10-0-138-127 kubenswrapper[2581]: I0916 19:09:14.025579    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-6bbbd567bb-88fhf"
Sep 16 19:09:14.025709 ip-10-0-138-127 kubenswrapper[2581]: I0916 19:09:14.025595    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-6bbbd567bb-88fhf" event={"ID":"25009467-ea1c-4c27-af68-13edd785d20b","Type":"ContainerDied","Data":"618d69e0964da844e32053e7c674fdddaffa35a0343732dd775f7f88160b7d3d"}
Sep 16 19:09:14.025709 ip-10-0-138-127 kubenswrapper[2581]: I0916 19:09:14.025677    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-6bbbd567bb-88fhf" event={"ID":"25009467-ea1c-4c27-af68-13edd785d20b","Type":"ContainerDied","Data":"eeaa688210f4ab73280d9097a7bc90344762b49e8b879deb66236dba69a4d5c5"}
Sep 16 19:09:14.025861 ip-10-0-138-127 kubenswrapper[2581]: I0916 19:09:14.025717    2581 scope.go:117] "RemoveContainer" containerID="ef323aa4e12c28c4b069ec297b45cd2edd1f499114847b3b50636b19bf2dd93d"
Sep 16 19:09:14.027688 ip-10-0-138-127 kubenswrapper[2581]: I0916 19:09:14.027649    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-5c9955746c-8fpkr" event={"ID":"626e5804-1f71-42af-abc8-e8934619263b","Type":"ContainerStarted","Data":"38399c055bd2d8e6758bc31e074947807fc61c7b763c48151fa154010eb8c297"}
Sep 16 19:09:14.027851 ip-10-0-138-127 kubenswrapper[2581]: I0916 19:09:14.027722    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-5c9955746c-8fpkr" event={"ID":"626e5804-1f71-42af-abc8-e8934619263b","Type":"ContainerStarted","Data":"8eb77f7510c29a4d493c39e6c18beb99d1dd6b8dcd9dd699300f2ce6322f3db9"}
Sep 16 19:09:14.036728 ip-10-0-138-127 kubenswrapper[2581]: I0916 19:09:14.036696    2581 scope.go:117] "RemoveContainer" containerID="618d69e0964da844e32053e7c674fdddaffa35a0343732dd775f7f88160b7d3d"
Sep 16 19:09:14.052027 ip-10-0-138-127 kubenswrapper[2581]: I0916 19:09:14.051969    2581 scope.go:117] "RemoveContainer" containerID="b5e52398c2570593077c92264b305a9e9b7174ebc919555c5bc331e51d4ae241"
Sep 16 19:09:14.062361 ip-10-0-138-127 kubenswrapper[2581]: I0916 19:09:14.062316    2581 scope.go:117] "RemoveContainer" containerID="ef323aa4e12c28c4b069ec297b45cd2edd1f499114847b3b50636b19bf2dd93d"
Sep 16 19:09:14.063069 ip-10-0-138-127 kubenswrapper[2581]: E0916 19:09:14.063021    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ef323aa4e12c28c4b069ec297b45cd2edd1f499114847b3b50636b19bf2dd93d\": container with ID starting with ef323aa4e12c28c4b069ec297b45cd2edd1f499114847b3b50636b19bf2dd93d not found: ID does not exist" containerID="ef323aa4e12c28c4b069ec297b45cd2edd1f499114847b3b50636b19bf2dd93d"
Sep 16 19:09:14.063171 ip-10-0-138-127 kubenswrapper[2581]: I0916 19:09:14.063095    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ef323aa4e12c28c4b069ec297b45cd2edd1f499114847b3b50636b19bf2dd93d"} err="failed to get container status \"ef323aa4e12c28c4b069ec297b45cd2edd1f499114847b3b50636b19bf2dd93d\": rpc error: code = NotFound desc = could not find container \"ef323aa4e12c28c4b069ec297b45cd2edd1f499114847b3b50636b19bf2dd93d\": container with ID starting with ef323aa4e12c28c4b069ec297b45cd2edd1f499114847b3b50636b19bf2dd93d not found: ID does not exist"
Sep 16 19:09:14.063171 ip-10-0-138-127 kubenswrapper[2581]: I0916 19:09:14.063139    2581 scope.go:117] "RemoveContainer" containerID="618d69e0964da844e32053e7c674fdddaffa35a0343732dd775f7f88160b7d3d"
Sep 16 19:09:14.063613 ip-10-0-138-127 kubenswrapper[2581]: E0916 19:09:14.063593    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"618d69e0964da844e32053e7c674fdddaffa35a0343732dd775f7f88160b7d3d\": container with ID starting with 618d69e0964da844e32053e7c674fdddaffa35a0343732dd775f7f88160b7d3d not found: ID does not exist" containerID="618d69e0964da844e32053e7c674fdddaffa35a0343732dd775f7f88160b7d3d"
Sep 16 19:09:14.063690 ip-10-0-138-127 kubenswrapper[2581]: I0916 19:09:14.063623    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"618d69e0964da844e32053e7c674fdddaffa35a0343732dd775f7f88160b7d3d"} err="failed to get container status \"618d69e0964da844e32053e7c674fdddaffa35a0343732dd775f7f88160b7d3d\": rpc error: code = NotFound desc = could not find container \"618d69e0964da844e32053e7c674fdddaffa35a0343732dd775f7f88160b7d3d\": container with ID starting with 618d69e0964da844e32053e7c674fdddaffa35a0343732dd775f7f88160b7d3d not found: ID does not exist"
Sep 16 19:09:14.063690 ip-10-0-138-127 kubenswrapper[2581]: I0916 19:09:14.063647    2581 scope.go:117] "RemoveContainer" containerID="b5e52398c2570593077c92264b305a9e9b7174ebc919555c5bc331e51d4ae241"
Sep 16 19:09:14.063922 ip-10-0-138-127 kubenswrapper[2581]: E0916 19:09:14.063905    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b5e52398c2570593077c92264b305a9e9b7174ebc919555c5bc331e51d4ae241\": container with ID starting with b5e52398c2570593077c92264b305a9e9b7174ebc919555c5bc331e51d4ae241 not found: ID does not exist" containerID="b5e52398c2570593077c92264b305a9e9b7174ebc919555c5bc331e51d4ae241"
Sep 16 19:09:14.063970 ip-10-0-138-127 kubenswrapper[2581]: I0916 19:09:14.063928    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b5e52398c2570593077c92264b305a9e9b7174ebc919555c5bc331e51d4ae241"} err="failed to get container status \"b5e52398c2570593077c92264b305a9e9b7174ebc919555c5bc331e51d4ae241\": rpc error: code = NotFound desc = could not find container \"b5e52398c2570593077c92264b305a9e9b7174ebc919555c5bc331e51d4ae241\": container with ID starting with b5e52398c2570593077c92264b305a9e9b7174ebc919555c5bc331e51d4ae241 not found: ID does not exist"
Sep 16 19:09:14.077752 ip-10-0-138-127 kubenswrapper[2581]: I0916 19:09:14.077702    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-v2-predictor-6bbbd567bb-88fhf"]
Sep 16 19:09:14.081888 ip-10-0-138-127 kubenswrapper[2581]: I0916 19:09:14.081852    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-v2-predictor-6bbbd567bb-88fhf"]
Sep 16 19:09:14.728974 ip-10-0-138-127 kubenswrapper[2581]: I0916 19:09:14.728888    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="25009467-ea1c-4c27-af68-13edd785d20b" path="/var/lib/kubelet/pods/25009467-ea1c-4c27-af68-13edd785d20b/volumes"
Sep 16 19:09:18.048714 ip-10-0-138-127 kubenswrapper[2581]: I0916 19:09:18.048679    2581 generic.go:356] "Generic (PLEG): container finished" podID="626e5804-1f71-42af-abc8-e8934619263b" containerID="38399c055bd2d8e6758bc31e074947807fc61c7b763c48151fa154010eb8c297" exitCode=0
Sep 16 19:09:18.049105 ip-10-0-138-127 kubenswrapper[2581]: I0916 19:09:18.048762    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-5c9955746c-8fpkr" event={"ID":"626e5804-1f71-42af-abc8-e8934619263b","Type":"ContainerDied","Data":"38399c055bd2d8e6758bc31e074947807fc61c7b763c48151fa154010eb8c297"}
Sep 16 19:09:19.055701 ip-10-0-138-127 kubenswrapper[2581]: I0916 19:09:19.055664    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-5c9955746c-8fpkr" event={"ID":"626e5804-1f71-42af-abc8-e8934619263b","Type":"ContainerStarted","Data":"5cfc094dc80fe3a064f9e9e4f16fe9760273f5ca1013d2bba0ed8704fec1b38e"}
Sep 16 19:09:19.056253 ip-10-0-138-127 kubenswrapper[2581]: I0916 19:09:19.055732    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-5c9955746c-8fpkr" event={"ID":"626e5804-1f71-42af-abc8-e8934619263b","Type":"ContainerStarted","Data":"f402585b61cce48c8e333e16a5e531a74e8778e1ea9b142a0c9d5102d07b537f"}
Sep 16 19:09:19.056253 ip-10-0-138-127 kubenswrapper[2581]: I0916 19:09:19.055984    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-5c9955746c-8fpkr"
Sep 16 19:09:19.080128 ip-10-0-138-127 kubenswrapper[2581]: I0916 19:09:19.080072    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-5c9955746c-8fpkr" podStartSLOduration=2.11496314 podStartE2EDuration="6.080054956s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.965091816s" isStatefulPod=false podCreationTimestamp="2026-09-16 19:09:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:09:19.078793507 +0000 UTC m=+1428.898207419" watchObservedRunningTime="2026-09-16 19:09:19.080054956 +0000 UTC m=+1428.899468865"
Sep 16 19:09:20.060282 ip-10-0-138-127 kubenswrapper[2581]: I0916 19:09:20.060237    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-5c9955746c-8fpkr"
Sep 16 19:09:20.061588 ip-10-0-138-127 kubenswrapper[2581]: I0916 19:09:20.061556    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-5c9955746c-8fpkr" podUID="626e5804-1f71-42af-abc8-e8934619263b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.37:8080: connect: connection refused"
Sep 16 19:09:21.064640 ip-10-0-138-127 kubenswrapper[2581]: I0916 19:09:21.064599    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-5c9955746c-8fpkr" podUID="626e5804-1f71-42af-abc8-e8934619263b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.37:8080: connect: connection refused"
Sep 16 19:09:26.069820 ip-10-0-138-127 kubenswrapper[2581]: I0916 19:09:26.069739    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-5c9955746c-8fpkr"
Sep 16 19:09:26.070432 ip-10-0-138-127 kubenswrapper[2581]: I0916 19:09:26.070404    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-5c9955746c-8fpkr" podUID="626e5804-1f71-42af-abc8-e8934619263b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.37:8080: connect: connection refused"
Sep 16 19:09:36.071280 ip-10-0-138-127 kubenswrapper[2581]: I0916 19:09:36.071237    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-5c9955746c-8fpkr" podUID="626e5804-1f71-42af-abc8-e8934619263b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.37:8080: connect: connection refused"
Sep 16 19:09:46.071094 ip-10-0-138-127 kubenswrapper[2581]: I0916 19:09:46.071048    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-5c9955746c-8fpkr" podUID="626e5804-1f71-42af-abc8-e8934619263b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.37:8080: connect: connection refused"
Sep 16 19:09:56.070817 ip-10-0-138-127 kubenswrapper[2581]: I0916 19:09:56.070773    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-5c9955746c-8fpkr" podUID="626e5804-1f71-42af-abc8-e8934619263b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.37:8080: connect: connection refused"
Sep 16 19:10:06.070347 ip-10-0-138-127 kubenswrapper[2581]: I0916 19:10:06.070306    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-5c9955746c-8fpkr" podUID="626e5804-1f71-42af-abc8-e8934619263b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.37:8080: connect: connection refused"
Sep 16 19:10:16.070629 ip-10-0-138-127 kubenswrapper[2581]: I0916 19:10:16.070582    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-5c9955746c-8fpkr" podUID="626e5804-1f71-42af-abc8-e8934619263b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.37:8080: connect: connection refused"
Sep 16 19:10:26.070664 ip-10-0-138-127 kubenswrapper[2581]: I0916 19:10:26.070627    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-5c9955746c-8fpkr" podUID="626e5804-1f71-42af-abc8-e8934619263b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.37:8080: connect: connection refused"
Sep 16 19:10:36.071460 ip-10-0-138-127 kubenswrapper[2581]: I0916 19:10:36.071427    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-5c9955746c-8fpkr"
Sep 16 19:10:43.419235 ip-10-0-138-127 kubenswrapper[2581]: I0916 19:10:43.419173    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-predictor-5c9955746c-8fpkr"]
Sep 16 19:10:43.419680 ip-10-0-138-127 kubenswrapper[2581]: I0916 19:10:43.419435    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-5c9955746c-8fpkr" podUID="626e5804-1f71-42af-abc8-e8934619263b" containerName="kserve-container" containerID="cri-o://f402585b61cce48c8e333e16a5e531a74e8778e1ea9b142a0c9d5102d07b537f" gracePeriod=30
Sep 16 19:10:43.419680 ip-10-0-138-127 kubenswrapper[2581]: I0916 19:10:43.419500    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-5c9955746c-8fpkr" podUID="626e5804-1f71-42af-abc8-e8934619263b" containerName="kube-rbac-proxy" containerID="cri-o://5cfc094dc80fe3a064f9e9e4f16fe9760273f5ca1013d2bba0ed8704fec1b38e" gracePeriod=30
Sep 16 19:10:43.486679 ip-10-0-138-127 kubenswrapper[2581]: I0916 19:10:43.486639    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-5ccc95dd4-6mcqd"]
Sep 16 19:10:43.487035 ip-10-0-138-127 kubenswrapper[2581]: I0916 19:10:43.487018    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="25009467-ea1c-4c27-af68-13edd785d20b" containerName="storage-initializer"
Sep 16 19:10:43.487116 ip-10-0-138-127 kubenswrapper[2581]: I0916 19:10:43.487038    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="25009467-ea1c-4c27-af68-13edd785d20b" containerName="storage-initializer"
Sep 16 19:10:43.487116 ip-10-0-138-127 kubenswrapper[2581]: I0916 19:10:43.487066    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="25009467-ea1c-4c27-af68-13edd785d20b" containerName="kserve-container"
Sep 16 19:10:43.487116 ip-10-0-138-127 kubenswrapper[2581]: I0916 19:10:43.487077    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="25009467-ea1c-4c27-af68-13edd785d20b" containerName="kserve-container"
Sep 16 19:10:43.487116 ip-10-0-138-127 kubenswrapper[2581]: I0916 19:10:43.487105    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="25009467-ea1c-4c27-af68-13edd785d20b" containerName="kube-rbac-proxy"
Sep 16 19:10:43.487116 ip-10-0-138-127 kubenswrapper[2581]: I0916 19:10:43.487114    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="25009467-ea1c-4c27-af68-13edd785d20b" containerName="kube-rbac-proxy"
Sep 16 19:10:43.487413 ip-10-0-138-127 kubenswrapper[2581]: I0916 19:10:43.487240    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="25009467-ea1c-4c27-af68-13edd785d20b" containerName="kube-rbac-proxy"
Sep 16 19:10:43.487413 ip-10-0-138-127 kubenswrapper[2581]: I0916 19:10:43.487256    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="25009467-ea1c-4c27-af68-13edd785d20b" containerName="kserve-container"
Sep 16 19:10:43.490483 ip-10-0-138-127 kubenswrapper[2581]: I0916 19:10:43.490463    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-5ccc95dd4-6mcqd"
Sep 16 19:10:43.492567 ip-10-0-138-127 kubenswrapper[2581]: I0916 19:10:43.492548    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-promote-v2-predictor-serving-cert\""
Sep 16 19:10:43.502507 ip-10-0-138-127 kubenswrapper[2581]: I0916 19:10:43.502485    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-5ccc95dd4-6mcqd"]
Sep 16 19:10:43.566044 ip-10-0-138-127 kubenswrapper[2581]: I0916 19:10:43.565998    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/761d83a0-a534-43c7-960e-7111a20b1806-kube-rbac-proxy-sar-config\") pod \"isvc-canary-promote-v2-predictor-5ccc95dd4-6mcqd\" (UID: \"761d83a0-a534-43c7-960e-7111a20b1806\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-5ccc95dd4-6mcqd"
Sep 16 19:10:43.566044 ip-10-0-138-127 kubenswrapper[2581]: I0916 19:10:43.566054    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bjcv2\" (UniqueName: \"kubernetes.io/projected/761d83a0-a534-43c7-960e-7111a20b1806-kube-api-access-bjcv2\") pod \"isvc-canary-promote-v2-predictor-5ccc95dd4-6mcqd\" (UID: \"761d83a0-a534-43c7-960e-7111a20b1806\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-5ccc95dd4-6mcqd"
Sep 16 19:10:43.566276 ip-10-0-138-127 kubenswrapper[2581]: I0916 19:10:43.566082    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/761d83a0-a534-43c7-960e-7111a20b1806-cabundle-cert\") pod \"isvc-canary-promote-v2-predictor-5ccc95dd4-6mcqd\" (UID: \"761d83a0-a534-43c7-960e-7111a20b1806\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-5ccc95dd4-6mcqd"
Sep 16 19:10:43.566276 ip-10-0-138-127 kubenswrapper[2581]: I0916 19:10:43.566144    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/761d83a0-a534-43c7-960e-7111a20b1806-proxy-tls\") pod \"isvc-canary-promote-v2-predictor-5ccc95dd4-6mcqd\" (UID: \"761d83a0-a534-43c7-960e-7111a20b1806\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-5ccc95dd4-6mcqd"
Sep 16 19:10:43.566276 ip-10-0-138-127 kubenswrapper[2581]: I0916 19:10:43.566228    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/761d83a0-a534-43c7-960e-7111a20b1806-kserve-provision-location\") pod \"isvc-canary-promote-v2-predictor-5ccc95dd4-6mcqd\" (UID: \"761d83a0-a534-43c7-960e-7111a20b1806\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-5ccc95dd4-6mcqd"
Sep 16 19:10:43.667402 ip-10-0-138-127 kubenswrapper[2581]: I0916 19:10:43.667368    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/761d83a0-a534-43c7-960e-7111a20b1806-kube-rbac-proxy-sar-config\") pod \"isvc-canary-promote-v2-predictor-5ccc95dd4-6mcqd\" (UID: \"761d83a0-a534-43c7-960e-7111a20b1806\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-5ccc95dd4-6mcqd"
Sep 16 19:10:43.667402 ip-10-0-138-127 kubenswrapper[2581]: I0916 19:10:43.667410    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bjcv2\" (UniqueName: \"kubernetes.io/projected/761d83a0-a534-43c7-960e-7111a20b1806-kube-api-access-bjcv2\") pod \"isvc-canary-promote-v2-predictor-5ccc95dd4-6mcqd\" (UID: \"761d83a0-a534-43c7-960e-7111a20b1806\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-5ccc95dd4-6mcqd"
Sep 16 19:10:43.667651 ip-10-0-138-127 kubenswrapper[2581]: I0916 19:10:43.667426    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/761d83a0-a534-43c7-960e-7111a20b1806-cabundle-cert\") pod \"isvc-canary-promote-v2-predictor-5ccc95dd4-6mcqd\" (UID: \"761d83a0-a534-43c7-960e-7111a20b1806\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-5ccc95dd4-6mcqd"
Sep 16 19:10:43.667651 ip-10-0-138-127 kubenswrapper[2581]: I0916 19:10:43.667556    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/761d83a0-a534-43c7-960e-7111a20b1806-proxy-tls\") pod \"isvc-canary-promote-v2-predictor-5ccc95dd4-6mcqd\" (UID: \"761d83a0-a534-43c7-960e-7111a20b1806\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-5ccc95dd4-6mcqd"
Sep 16 19:10:43.667651 ip-10-0-138-127 kubenswrapper[2581]: I0916 19:10:43.667627    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/761d83a0-a534-43c7-960e-7111a20b1806-kserve-provision-location\") pod \"isvc-canary-promote-v2-predictor-5ccc95dd4-6mcqd\" (UID: \"761d83a0-a534-43c7-960e-7111a20b1806\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-5ccc95dd4-6mcqd"
Sep 16 19:10:43.668024 ip-10-0-138-127 kubenswrapper[2581]: I0916 19:10:43.668007    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/761d83a0-a534-43c7-960e-7111a20b1806-kserve-provision-location\") pod \"isvc-canary-promote-v2-predictor-5ccc95dd4-6mcqd\" (UID: \"761d83a0-a534-43c7-960e-7111a20b1806\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-5ccc95dd4-6mcqd"
Sep 16 19:10:43.668091 ip-10-0-138-127 kubenswrapper[2581]: I0916 19:10:43.668072    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/761d83a0-a534-43c7-960e-7111a20b1806-kube-rbac-proxy-sar-config\") pod \"isvc-canary-promote-v2-predictor-5ccc95dd4-6mcqd\" (UID: \"761d83a0-a534-43c7-960e-7111a20b1806\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-5ccc95dd4-6mcqd"
Sep 16 19:10:43.668171 ip-10-0-138-127 kubenswrapper[2581]: I0916 19:10:43.668154    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/761d83a0-a534-43c7-960e-7111a20b1806-cabundle-cert\") pod \"isvc-canary-promote-v2-predictor-5ccc95dd4-6mcqd\" (UID: \"761d83a0-a534-43c7-960e-7111a20b1806\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-5ccc95dd4-6mcqd"
Sep 16 19:10:43.670331 ip-10-0-138-127 kubenswrapper[2581]: I0916 19:10:43.670279    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/761d83a0-a534-43c7-960e-7111a20b1806-proxy-tls\") pod \"isvc-canary-promote-v2-predictor-5ccc95dd4-6mcqd\" (UID: \"761d83a0-a534-43c7-960e-7111a20b1806\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-5ccc95dd4-6mcqd"
Sep 16 19:10:43.677072 ip-10-0-138-127 kubenswrapper[2581]: I0916 19:10:43.677051    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bjcv2\" (UniqueName: \"kubernetes.io/projected/761d83a0-a534-43c7-960e-7111a20b1806-kube-api-access-bjcv2\") pod \"isvc-canary-promote-v2-predictor-5ccc95dd4-6mcqd\" (UID: \"761d83a0-a534-43c7-960e-7111a20b1806\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-5ccc95dd4-6mcqd"
Sep 16 19:10:43.800162 ip-10-0-138-127 kubenswrapper[2581]: I0916 19:10:43.800125    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-5ccc95dd4-6mcqd"
Sep 16 19:10:44.077459 ip-10-0-138-127 kubenswrapper[2581]: I0916 19:10:44.077428    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-5ccc95dd4-6mcqd"]
Sep 16 19:10:44.081039 ip-10-0-138-127 kubenswrapper[2581]: W0916 19:10:44.081014    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod761d83a0_a534_43c7_960e_7111a20b1806.slice/crio-7f7f88ae0ccea892f838e442c0652f2ef098b2e871e2815a33efad130844f096 WatchSource:0}: Error finding container 7f7f88ae0ccea892f838e442c0652f2ef098b2e871e2815a33efad130844f096: Status 404 returned error can't find the container with id 7f7f88ae0ccea892f838e442c0652f2ef098b2e871e2815a33efad130844f096
Sep 16 19:10:44.418073 ip-10-0-138-127 kubenswrapper[2581]: I0916 19:10:44.417976    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-5ccc95dd4-6mcqd" event={"ID":"761d83a0-a534-43c7-960e-7111a20b1806","Type":"ContainerStarted","Data":"d596997cb91a047e2bef4117e3852aebbe169734ddd5ed498726caed993bc4de"}
Sep 16 19:10:44.418073 ip-10-0-138-127 kubenswrapper[2581]: I0916 19:10:44.418052    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-5ccc95dd4-6mcqd" event={"ID":"761d83a0-a534-43c7-960e-7111a20b1806","Type":"ContainerStarted","Data":"7f7f88ae0ccea892f838e442c0652f2ef098b2e871e2815a33efad130844f096"}
Sep 16 19:10:44.420448 ip-10-0-138-127 kubenswrapper[2581]: I0916 19:10:44.420425    2581 generic.go:356] "Generic (PLEG): container finished" podID="626e5804-1f71-42af-abc8-e8934619263b" containerID="5cfc094dc80fe3a064f9e9e4f16fe9760273f5ca1013d2bba0ed8704fec1b38e" exitCode=2
Sep 16 19:10:44.420763 ip-10-0-138-127 kubenswrapper[2581]: I0916 19:10:44.420462    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-5c9955746c-8fpkr" event={"ID":"626e5804-1f71-42af-abc8-e8934619263b","Type":"ContainerDied","Data":"5cfc094dc80fe3a064f9e9e4f16fe9760273f5ca1013d2bba0ed8704fec1b38e"}
Sep 16 19:10:46.065363 ip-10-0-138-127 kubenswrapper[2581]: I0916 19:10:46.065321    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-5c9955746c-8fpkr" podUID="626e5804-1f71-42af-abc8-e8934619263b" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.37:8643/healthz\": dial tcp 10.132.0.37:8643: connect: connection refused"
Sep 16 19:10:46.070666 ip-10-0-138-127 kubenswrapper[2581]: I0916 19:10:46.070634    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-5c9955746c-8fpkr" podUID="626e5804-1f71-42af-abc8-e8934619263b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.37:8080: connect: connection refused"
Sep 16 19:10:48.437476 ip-10-0-138-127 kubenswrapper[2581]: I0916 19:10:48.437444    2581 generic.go:356] "Generic (PLEG): container finished" podID="761d83a0-a534-43c7-960e-7111a20b1806" containerID="d596997cb91a047e2bef4117e3852aebbe169734ddd5ed498726caed993bc4de" exitCode=0
Sep 16 19:10:48.437952 ip-10-0-138-127 kubenswrapper[2581]: I0916 19:10:48.437508    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-5ccc95dd4-6mcqd" event={"ID":"761d83a0-a534-43c7-960e-7111a20b1806","Type":"ContainerDied","Data":"d596997cb91a047e2bef4117e3852aebbe169734ddd5ed498726caed993bc4de"}
Sep 16 19:10:48.440083 ip-10-0-138-127 kubenswrapper[2581]: I0916 19:10:48.440062    2581 generic.go:356] "Generic (PLEG): container finished" podID="626e5804-1f71-42af-abc8-e8934619263b" containerID="f402585b61cce48c8e333e16a5e531a74e8778e1ea9b142a0c9d5102d07b537f" exitCode=0
Sep 16 19:10:48.440215 ip-10-0-138-127 kubenswrapper[2581]: I0916 19:10:48.440119    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-5c9955746c-8fpkr" event={"ID":"626e5804-1f71-42af-abc8-e8934619263b","Type":"ContainerDied","Data":"f402585b61cce48c8e333e16a5e531a74e8778e1ea9b142a0c9d5102d07b537f"}
Sep 16 19:10:48.490444 ip-10-0-138-127 kubenswrapper[2581]: I0916 19:10:48.490419    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-5c9955746c-8fpkr"
Sep 16 19:10:48.615124 ip-10-0-138-127 kubenswrapper[2581]: I0916 19:10:48.615079    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/626e5804-1f71-42af-abc8-e8934619263b-cabundle-cert\") pod \"626e5804-1f71-42af-abc8-e8934619263b\" (UID: \"626e5804-1f71-42af-abc8-e8934619263b\") "
Sep 16 19:10:48.615315 ip-10-0-138-127 kubenswrapper[2581]: I0916 19:10:48.615131    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7qfzv\" (UniqueName: \"kubernetes.io/projected/626e5804-1f71-42af-abc8-e8934619263b-kube-api-access-7qfzv\") pod \"626e5804-1f71-42af-abc8-e8934619263b\" (UID: \"626e5804-1f71-42af-abc8-e8934619263b\") "
Sep 16 19:10:48.615315 ip-10-0-138-127 kubenswrapper[2581]: I0916 19:10:48.615176    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/626e5804-1f71-42af-abc8-e8934619263b-kube-rbac-proxy-sar-config\") pod \"626e5804-1f71-42af-abc8-e8934619263b\" (UID: \"626e5804-1f71-42af-abc8-e8934619263b\") "
Sep 16 19:10:48.615315 ip-10-0-138-127 kubenswrapper[2581]: I0916 19:10:48.615246    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/626e5804-1f71-42af-abc8-e8934619263b-proxy-tls\") pod \"626e5804-1f71-42af-abc8-e8934619263b\" (UID: \"626e5804-1f71-42af-abc8-e8934619263b\") "
Sep 16 19:10:48.615501 ip-10-0-138-127 kubenswrapper[2581]: I0916 19:10:48.615324    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/626e5804-1f71-42af-abc8-e8934619263b-kserve-provision-location\") pod \"626e5804-1f71-42af-abc8-e8934619263b\" (UID: \"626e5804-1f71-42af-abc8-e8934619263b\") "
Sep 16 19:10:48.615552 ip-10-0-138-127 kubenswrapper[2581]: I0916 19:10:48.615530    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/626e5804-1f71-42af-abc8-e8934619263b-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "626e5804-1f71-42af-abc8-e8934619263b" (UID: "626e5804-1f71-42af-abc8-e8934619263b"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 19:10:48.615705 ip-10-0-138-127 kubenswrapper[2581]: I0916 19:10:48.615577    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/626e5804-1f71-42af-abc8-e8934619263b-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "626e5804-1f71-42af-abc8-e8934619263b" (UID: "626e5804-1f71-42af-abc8-e8934619263b"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 19:10:48.615765 ip-10-0-138-127 kubenswrapper[2581]: I0916 19:10:48.615715    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/626e5804-1f71-42af-abc8-e8934619263b-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "626e5804-1f71-42af-abc8-e8934619263b" (UID: "626e5804-1f71-42af-abc8-e8934619263b"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:10:48.617392 ip-10-0-138-127 kubenswrapper[2581]: I0916 19:10:48.617375    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/626e5804-1f71-42af-abc8-e8934619263b-kube-api-access-7qfzv" (OuterVolumeSpecName: "kube-api-access-7qfzv") pod "626e5804-1f71-42af-abc8-e8934619263b" (UID: "626e5804-1f71-42af-abc8-e8934619263b"). InnerVolumeSpecName "kube-api-access-7qfzv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 19:10:48.617441 ip-10-0-138-127 kubenswrapper[2581]: I0916 19:10:48.617428    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/626e5804-1f71-42af-abc8-e8934619263b-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "626e5804-1f71-42af-abc8-e8934619263b" (UID: "626e5804-1f71-42af-abc8-e8934619263b"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 19:10:48.716326 ip-10-0-138-127 kubenswrapper[2581]: I0916 19:10:48.716282    2581 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/626e5804-1f71-42af-abc8-e8934619263b-kube-rbac-proxy-sar-config\") on node \"ip-10-0-138-127.ec2.internal\" DevicePath \"\""
Sep 16 19:10:48.716326 ip-10-0-138-127 kubenswrapper[2581]: I0916 19:10:48.716321    2581 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/626e5804-1f71-42af-abc8-e8934619263b-proxy-tls\") on node \"ip-10-0-138-127.ec2.internal\" DevicePath \"\""
Sep 16 19:10:48.716326 ip-10-0-138-127 kubenswrapper[2581]: I0916 19:10:48.716331    2581 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/626e5804-1f71-42af-abc8-e8934619263b-kserve-provision-location\") on node \"ip-10-0-138-127.ec2.internal\" DevicePath \"\""
Sep 16 19:10:48.716577 ip-10-0-138-127 kubenswrapper[2581]: I0916 19:10:48.716340    2581 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/626e5804-1f71-42af-abc8-e8934619263b-cabundle-cert\") on node \"ip-10-0-138-127.ec2.internal\" DevicePath \"\""
Sep 16 19:10:48.716577 ip-10-0-138-127 kubenswrapper[2581]: I0916 19:10:48.716349    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7qfzv\" (UniqueName: \"kubernetes.io/projected/626e5804-1f71-42af-abc8-e8934619263b-kube-api-access-7qfzv\") on node \"ip-10-0-138-127.ec2.internal\" DevicePath \"\""
Sep 16 19:10:49.445957 ip-10-0-138-127 kubenswrapper[2581]: I0916 19:10:49.445921    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-5ccc95dd4-6mcqd" event={"ID":"761d83a0-a534-43c7-960e-7111a20b1806","Type":"ContainerStarted","Data":"4b80e9271b1e8dd7636ba3d22e4a458eaedf3f0b7093863894b5a837c1cc9e8c"}
Sep 16 19:10:49.446408 ip-10-0-138-127 kubenswrapper[2581]: I0916 19:10:49.445985    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-5ccc95dd4-6mcqd" event={"ID":"761d83a0-a534-43c7-960e-7111a20b1806","Type":"ContainerStarted","Data":"3e835e65be64d623f1a114f315e69ad5688b5381d33f7d9805083fc35703fc1e"}
Sep 16 19:10:49.446408 ip-10-0-138-127 kubenswrapper[2581]: I0916 19:10:49.446173    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-5ccc95dd4-6mcqd"
Sep 16 19:10:49.447835 ip-10-0-138-127 kubenswrapper[2581]: I0916 19:10:49.447813    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-5c9955746c-8fpkr" event={"ID":"626e5804-1f71-42af-abc8-e8934619263b","Type":"ContainerDied","Data":"8eb77f7510c29a4d493c39e6c18beb99d1dd6b8dcd9dd699300f2ce6322f3db9"}
Sep 16 19:10:49.447835 ip-10-0-138-127 kubenswrapper[2581]: I0916 19:10:49.447829    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-5c9955746c-8fpkr"
Sep 16 19:10:49.447972 ip-10-0-138-127 kubenswrapper[2581]: I0916 19:10:49.447868    2581 scope.go:117] "RemoveContainer" containerID="5cfc094dc80fe3a064f9e9e4f16fe9760273f5ca1013d2bba0ed8704fec1b38e"
Sep 16 19:10:49.456135 ip-10-0-138-127 kubenswrapper[2581]: I0916 19:10:49.456117    2581 scope.go:117] "RemoveContainer" containerID="f402585b61cce48c8e333e16a5e531a74e8778e1ea9b142a0c9d5102d07b537f"
Sep 16 19:10:49.464302 ip-10-0-138-127 kubenswrapper[2581]: I0916 19:10:49.464277    2581 scope.go:117] "RemoveContainer" containerID="38399c055bd2d8e6758bc31e074947807fc61c7b763c48151fa154010eb8c297"
Sep 16 19:10:49.479346 ip-10-0-138-127 kubenswrapper[2581]: I0916 19:10:49.479293    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-5ccc95dd4-6mcqd" podStartSLOduration=2.791949589 podStartE2EDuration="6.47927687s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.687327281s" isStatefulPod=false podCreationTimestamp="2026-09-16 19:10:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:10:49.478123334 +0000 UTC m=+1519.297537256" watchObservedRunningTime="2026-09-16 19:10:49.47927687 +0000 UTC m=+1519.298690779"
Sep 16 19:10:49.492290 ip-10-0-138-127 kubenswrapper[2581]: I0916 19:10:49.492262    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-predictor-5c9955746c-8fpkr"]
Sep 16 19:10:49.498784 ip-10-0-138-127 kubenswrapper[2581]: I0916 19:10:49.498759    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-predictor-5c9955746c-8fpkr"]
Sep 16 19:10:50.452925 ip-10-0-138-127 kubenswrapper[2581]: I0916 19:10:50.452893    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-5ccc95dd4-6mcqd"
Sep 16 19:10:50.454467 ip-10-0-138-127 kubenswrapper[2581]: I0916 19:10:50.454440    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-5ccc95dd4-6mcqd" podUID="761d83a0-a534-43c7-960e-7111a20b1806" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.38:8080: connect: connection refused"
Sep 16 19:10:50.728446 ip-10-0-138-127 kubenswrapper[2581]: I0916 19:10:50.728360    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="626e5804-1f71-42af-abc8-e8934619263b" path="/var/lib/kubelet/pods/626e5804-1f71-42af-abc8-e8934619263b/volumes"
Sep 16 19:10:51.457471 ip-10-0-138-127 kubenswrapper[2581]: I0916 19:10:51.457427    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-5ccc95dd4-6mcqd" podUID="761d83a0-a534-43c7-960e-7111a20b1806" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.38:8080: connect: connection refused"
Sep 16 19:10:56.462332 ip-10-0-138-127 kubenswrapper[2581]: I0916 19:10:56.462305    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-5ccc95dd4-6mcqd"
Sep 16 19:10:56.462824 ip-10-0-138-127 kubenswrapper[2581]: I0916 19:10:56.462795    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-5ccc95dd4-6mcqd" podUID="761d83a0-a534-43c7-960e-7111a20b1806" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.38:8080: connect: connection refused"
Sep 16 19:11:06.462858 ip-10-0-138-127 kubenswrapper[2581]: I0916 19:11:06.462809    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-5ccc95dd4-6mcqd" podUID="761d83a0-a534-43c7-960e-7111a20b1806" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.38:8080: connect: connection refused"
Sep 16 19:11:16.463115 ip-10-0-138-127 kubenswrapper[2581]: I0916 19:11:16.463065    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-5ccc95dd4-6mcqd" podUID="761d83a0-a534-43c7-960e-7111a20b1806" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.38:8080: connect: connection refused"
Sep 16 19:11:26.463038 ip-10-0-138-127 kubenswrapper[2581]: I0916 19:11:26.462985    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-5ccc95dd4-6mcqd" podUID="761d83a0-a534-43c7-960e-7111a20b1806" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.38:8080: connect: connection refused"
Sep 16 19:11:36.463636 ip-10-0-138-127 kubenswrapper[2581]: I0916 19:11:36.463589    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-5ccc95dd4-6mcqd" podUID="761d83a0-a534-43c7-960e-7111a20b1806" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.38:8080: connect: connection refused"
Sep 16 19:11:46.462813 ip-10-0-138-127 kubenswrapper[2581]: I0916 19:11:46.462773    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-5ccc95dd4-6mcqd" podUID="761d83a0-a534-43c7-960e-7111a20b1806" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.38:8080: connect: connection refused"
Sep 16 19:11:56.462927 ip-10-0-138-127 kubenswrapper[2581]: I0916 19:11:56.462876    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-5ccc95dd4-6mcqd" podUID="761d83a0-a534-43c7-960e-7111a20b1806" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.38:8080: connect: connection refused"
Sep 16 19:12:06.464026 ip-10-0-138-127 kubenswrapper[2581]: I0916 19:12:06.463995    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-5ccc95dd4-6mcqd"
Sep 16 19:12:13.717219 ip-10-0-138-127 kubenswrapper[2581]: I0916 19:12:13.717158    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-5ccc95dd4-6mcqd"]
Sep 16 19:12:13.717674 ip-10-0-138-127 kubenswrapper[2581]: I0916 19:12:13.717399    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-5ccc95dd4-6mcqd" podUID="761d83a0-a534-43c7-960e-7111a20b1806" containerName="kserve-container" containerID="cri-o://3e835e65be64d623f1a114f315e69ad5688b5381d33f7d9805083fc35703fc1e" gracePeriod=30
Sep 16 19:12:13.717674 ip-10-0-138-127 kubenswrapper[2581]: I0916 19:12:13.717486    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-5ccc95dd4-6mcqd" podUID="761d83a0-a534-43c7-960e-7111a20b1806" containerName="kube-rbac-proxy" containerID="cri-o://4b80e9271b1e8dd7636ba3d22e4a458eaedf3f0b7093863894b5a837c1cc9e8c" gracePeriod=30
Sep 16 19:12:14.798790 ip-10-0-138-127 kubenswrapper[2581]: I0916 19:12:14.798761    2581 generic.go:356] "Generic (PLEG): container finished" podID="761d83a0-a534-43c7-960e-7111a20b1806" containerID="4b80e9271b1e8dd7636ba3d22e4a458eaedf3f0b7093863894b5a837c1cc9e8c" exitCode=2
Sep 16 19:12:14.799174 ip-10-0-138-127 kubenswrapper[2581]: I0916 19:12:14.798823    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-5ccc95dd4-6mcqd" event={"ID":"761d83a0-a534-43c7-960e-7111a20b1806","Type":"ContainerDied","Data":"4b80e9271b1e8dd7636ba3d22e4a458eaedf3f0b7093863894b5a837c1cc9e8c"}
Sep 16 19:12:16.457917 ip-10-0-138-127 kubenswrapper[2581]: I0916 19:12:16.457871    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-5ccc95dd4-6mcqd" podUID="761d83a0-a534-43c7-960e-7111a20b1806" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.38:8643/healthz\": dial tcp 10.132.0.38:8643: connect: connection refused"
Sep 16 19:12:16.463373 ip-10-0-138-127 kubenswrapper[2581]: I0916 19:12:16.463336    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-5ccc95dd4-6mcqd" podUID="761d83a0-a534-43c7-960e-7111a20b1806" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.38:8080: connect: connection refused"
Sep 16 19:12:18.817389 ip-10-0-138-127 kubenswrapper[2581]: I0916 19:12:18.817358    2581 generic.go:356] "Generic (PLEG): container finished" podID="761d83a0-a534-43c7-960e-7111a20b1806" containerID="3e835e65be64d623f1a114f315e69ad5688b5381d33f7d9805083fc35703fc1e" exitCode=0
Sep 16 19:12:18.817731 ip-10-0-138-127 kubenswrapper[2581]: I0916 19:12:18.817426    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-5ccc95dd4-6mcqd" event={"ID":"761d83a0-a534-43c7-960e-7111a20b1806","Type":"ContainerDied","Data":"3e835e65be64d623f1a114f315e69ad5688b5381d33f7d9805083fc35703fc1e"}
Sep 16 19:12:18.867936 ip-10-0-138-127 kubenswrapper[2581]: I0916 19:12:18.867911    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-5ccc95dd4-6mcqd"
Sep 16 19:12:18.896244 ip-10-0-138-127 kubenswrapper[2581]: I0916 19:12:18.896218    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/761d83a0-a534-43c7-960e-7111a20b1806-cabundle-cert\") pod \"761d83a0-a534-43c7-960e-7111a20b1806\" (UID: \"761d83a0-a534-43c7-960e-7111a20b1806\") "
Sep 16 19:12:18.896393 ip-10-0-138-127 kubenswrapper[2581]: I0916 19:12:18.896257    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-bjcv2\" (UniqueName: \"kubernetes.io/projected/761d83a0-a534-43c7-960e-7111a20b1806-kube-api-access-bjcv2\") pod \"761d83a0-a534-43c7-960e-7111a20b1806\" (UID: \"761d83a0-a534-43c7-960e-7111a20b1806\") "
Sep 16 19:12:18.896393 ip-10-0-138-127 kubenswrapper[2581]: I0916 19:12:18.896308    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/761d83a0-a534-43c7-960e-7111a20b1806-proxy-tls\") pod \"761d83a0-a534-43c7-960e-7111a20b1806\" (UID: \"761d83a0-a534-43c7-960e-7111a20b1806\") "
Sep 16 19:12:18.896393 ip-10-0-138-127 kubenswrapper[2581]: I0916 19:12:18.896358    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/761d83a0-a534-43c7-960e-7111a20b1806-kserve-provision-location\") pod \"761d83a0-a534-43c7-960e-7111a20b1806\" (UID: \"761d83a0-a534-43c7-960e-7111a20b1806\") "
Sep 16 19:12:18.896596 ip-10-0-138-127 kubenswrapper[2581]: I0916 19:12:18.896417    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/761d83a0-a534-43c7-960e-7111a20b1806-kube-rbac-proxy-sar-config\") pod \"761d83a0-a534-43c7-960e-7111a20b1806\" (UID: \"761d83a0-a534-43c7-960e-7111a20b1806\") "
Sep 16 19:12:18.896689 ip-10-0-138-127 kubenswrapper[2581]: I0916 19:12:18.896664    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/761d83a0-a534-43c7-960e-7111a20b1806-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "761d83a0-a534-43c7-960e-7111a20b1806" (UID: "761d83a0-a534-43c7-960e-7111a20b1806"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 19:12:18.896900 ip-10-0-138-127 kubenswrapper[2581]: I0916 19:12:18.896866    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/761d83a0-a534-43c7-960e-7111a20b1806-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "761d83a0-a534-43c7-960e-7111a20b1806" (UID: "761d83a0-a534-43c7-960e-7111a20b1806"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:12:18.896900 ip-10-0-138-127 kubenswrapper[2581]: I0916 19:12:18.896892    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/761d83a0-a534-43c7-960e-7111a20b1806-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "761d83a0-a534-43c7-960e-7111a20b1806" (UID: "761d83a0-a534-43c7-960e-7111a20b1806"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 19:12:18.899070 ip-10-0-138-127 kubenswrapper[2581]: I0916 19:12:18.899044    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/761d83a0-a534-43c7-960e-7111a20b1806-kube-api-access-bjcv2" (OuterVolumeSpecName: "kube-api-access-bjcv2") pod "761d83a0-a534-43c7-960e-7111a20b1806" (UID: "761d83a0-a534-43c7-960e-7111a20b1806"). InnerVolumeSpecName "kube-api-access-bjcv2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 19:12:18.899181 ip-10-0-138-127 kubenswrapper[2581]: I0916 19:12:18.899076    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/761d83a0-a534-43c7-960e-7111a20b1806-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "761d83a0-a534-43c7-960e-7111a20b1806" (UID: "761d83a0-a534-43c7-960e-7111a20b1806"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 19:12:18.997103 ip-10-0-138-127 kubenswrapper[2581]: I0916 19:12:18.997065    2581 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/761d83a0-a534-43c7-960e-7111a20b1806-proxy-tls\") on node \"ip-10-0-138-127.ec2.internal\" DevicePath \"\""
Sep 16 19:12:18.997103 ip-10-0-138-127 kubenswrapper[2581]: I0916 19:12:18.997095    2581 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/761d83a0-a534-43c7-960e-7111a20b1806-kserve-provision-location\") on node \"ip-10-0-138-127.ec2.internal\" DevicePath \"\""
Sep 16 19:12:18.997103 ip-10-0-138-127 kubenswrapper[2581]: I0916 19:12:18.997109    2581 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/761d83a0-a534-43c7-960e-7111a20b1806-kube-rbac-proxy-sar-config\") on node \"ip-10-0-138-127.ec2.internal\" DevicePath \"\""
Sep 16 19:12:18.997379 ip-10-0-138-127 kubenswrapper[2581]: I0916 19:12:18.997121    2581 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/761d83a0-a534-43c7-960e-7111a20b1806-cabundle-cert\") on node \"ip-10-0-138-127.ec2.internal\" DevicePath \"\""
Sep 16 19:12:18.997379 ip-10-0-138-127 kubenswrapper[2581]: I0916 19:12:18.997133    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bjcv2\" (UniqueName: \"kubernetes.io/projected/761d83a0-a534-43c7-960e-7111a20b1806-kube-api-access-bjcv2\") on node \"ip-10-0-138-127.ec2.internal\" DevicePath \"\""
Sep 16 19:12:19.824012 ip-10-0-138-127 kubenswrapper[2581]: I0916 19:12:19.823981    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-5ccc95dd4-6mcqd"
Sep 16 19:12:19.824505 ip-10-0-138-127 kubenswrapper[2581]: I0916 19:12:19.823971    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-5ccc95dd4-6mcqd" event={"ID":"761d83a0-a534-43c7-960e-7111a20b1806","Type":"ContainerDied","Data":"7f7f88ae0ccea892f838e442c0652f2ef098b2e871e2815a33efad130844f096"}
Sep 16 19:12:19.824505 ip-10-0-138-127 kubenswrapper[2581]: I0916 19:12:19.824140    2581 scope.go:117] "RemoveContainer" containerID="4b80e9271b1e8dd7636ba3d22e4a458eaedf3f0b7093863894b5a837c1cc9e8c"
Sep 16 19:12:19.833100 ip-10-0-138-127 kubenswrapper[2581]: I0916 19:12:19.833083    2581 scope.go:117] "RemoveContainer" containerID="3e835e65be64d623f1a114f315e69ad5688b5381d33f7d9805083fc35703fc1e"
Sep 16 19:12:19.842739 ip-10-0-138-127 kubenswrapper[2581]: I0916 19:12:19.842707    2581 scope.go:117] "RemoveContainer" containerID="d596997cb91a047e2bef4117e3852aebbe169734ddd5ed498726caed993bc4de"
Sep 16 19:12:19.846872 ip-10-0-138-127 kubenswrapper[2581]: I0916 19:12:19.846848    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-5ccc95dd4-6mcqd"]
Sep 16 19:12:19.852604 ip-10-0-138-127 kubenswrapper[2581]: I0916 19:12:19.852584    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-5ccc95dd4-6mcqd"]
Sep 16 19:12:20.727151 ip-10-0-138-127 kubenswrapper[2581]: I0916 19:12:20.727113    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="761d83a0-a534-43c7-960e-7111a20b1806" path="/var/lib/kubelet/pods/761d83a0-a534-43c7-960e-7111a20b1806/volumes"
Sep 16 19:12:48.569671 ip-10-0-138-127 kubenswrapper[2581]: I0916 19:12:48.569627    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-7fd84f4ff-27rf6"]
Sep 16 19:12:48.570312 ip-10-0-138-127 kubenswrapper[2581]: I0916 19:12:48.570121    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="626e5804-1f71-42af-abc8-e8934619263b" containerName="storage-initializer"
Sep 16 19:12:48.575210 ip-10-0-138-127 kubenswrapper[2581]: I0916 19:12:48.570141    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="626e5804-1f71-42af-abc8-e8934619263b" containerName="storage-initializer"
Sep 16 19:12:48.575210 ip-10-0-138-127 kubenswrapper[2581]: I0916 19:12:48.570562    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="626e5804-1f71-42af-abc8-e8934619263b" containerName="kserve-container"
Sep 16 19:12:48.575210 ip-10-0-138-127 kubenswrapper[2581]: I0916 19:12:48.570575    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="626e5804-1f71-42af-abc8-e8934619263b" containerName="kserve-container"
Sep 16 19:12:48.575501 ip-10-0-138-127 kubenswrapper[2581]: I0916 19:12:48.570609    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="761d83a0-a534-43c7-960e-7111a20b1806" containerName="kube-rbac-proxy"
Sep 16 19:12:48.575501 ip-10-0-138-127 kubenswrapper[2581]: I0916 19:12:48.575451    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="761d83a0-a534-43c7-960e-7111a20b1806" containerName="kube-rbac-proxy"
Sep 16 19:12:48.575501 ip-10-0-138-127 kubenswrapper[2581]: I0916 19:12:48.575493    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="761d83a0-a534-43c7-960e-7111a20b1806" containerName="kserve-container"
Sep 16 19:12:48.575615 ip-10-0-138-127 kubenswrapper[2581]: I0916 19:12:48.575510    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="761d83a0-a534-43c7-960e-7111a20b1806" containerName="kserve-container"
Sep 16 19:12:48.575686 ip-10-0-138-127 kubenswrapper[2581]: I0916 19:12:48.575674    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="761d83a0-a534-43c7-960e-7111a20b1806" containerName="storage-initializer"
Sep 16 19:12:48.575686 ip-10-0-138-127 kubenswrapper[2581]: I0916 19:12:48.575685    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="761d83a0-a534-43c7-960e-7111a20b1806" containerName="storage-initializer"
Sep 16 19:12:48.575762 ip-10-0-138-127 kubenswrapper[2581]: I0916 19:12:48.575696    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="626e5804-1f71-42af-abc8-e8934619263b" containerName="kube-rbac-proxy"
Sep 16 19:12:48.575762 ip-10-0-138-127 kubenswrapper[2581]: I0916 19:12:48.575703    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="626e5804-1f71-42af-abc8-e8934619263b" containerName="kube-rbac-proxy"
Sep 16 19:12:48.575829 ip-10-0-138-127 kubenswrapper[2581]: I0916 19:12:48.575788    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="626e5804-1f71-42af-abc8-e8934619263b" containerName="kube-rbac-proxy"
Sep 16 19:12:48.575829 ip-10-0-138-127 kubenswrapper[2581]: I0916 19:12:48.575801    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="761d83a0-a534-43c7-960e-7111a20b1806" containerName="kserve-container"
Sep 16 19:12:48.575829 ip-10-0-138-127 kubenswrapper[2581]: I0916 19:12:48.575806    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="761d83a0-a534-43c7-960e-7111a20b1806" containerName="kube-rbac-proxy"
Sep 16 19:12:48.575829 ip-10-0-138-127 kubenswrapper[2581]: I0916 19:12:48.575812    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="626e5804-1f71-42af-abc8-e8934619263b" containerName="kserve-container"
Sep 16 19:12:48.580698 ip-10-0-138-127 kubenswrapper[2581]: I0916 19:12:48.580677    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-7fd84f4ff-27rf6"
Sep 16 19:12:48.584021 ip-10-0-138-127 kubenswrapper[2581]: I0916 19:12:48.583992    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Sep 16 19:12:48.584163 ip-10-0-138-127 kubenswrapper[2581]: I0916 19:12:48.584083    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Sep 16 19:12:48.584266 ip-10-0-138-127 kubenswrapper[2581]: I0916 19:12:48.584184    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-logger-raw-cipn-kube-rbac-proxy-sar-config\""
Sep 16 19:12:48.584266 ip-10-0-138-127 kubenswrapper[2581]: I0916 19:12:48.584245    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Sep 16 19:12:48.584266 ip-10-0-138-127 kubenswrapper[2581]: I0916 19:12:48.584261    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-c8lqg\""
Sep 16 19:12:48.584439 ip-10-0-138-127 kubenswrapper[2581]: I0916 19:12:48.584356    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Sep 16 19:12:48.584513 ip-10-0-138-127 kubenswrapper[2581]: I0916 19:12:48.584498    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-logger-raw-cipn-predictor-serving-cert\""
Sep 16 19:12:48.586947 ip-10-0-138-127 kubenswrapper[2581]: I0916 19:12:48.586924    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-7fd84f4ff-27rf6"]
Sep 16 19:12:48.754930 ip-10-0-138-127 kubenswrapper[2581]: I0916 19:12:48.754898    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/54d699c3-2780-4189-a57d-0fe9c44b801c-kserve-provision-location\") pod \"isvc-logger-raw-cipn-predictor-7fd84f4ff-27rf6\" (UID: \"54d699c3-2780-4189-a57d-0fe9c44b801c\") " pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-7fd84f4ff-27rf6"
Sep 16 19:12:48.755111 ip-10-0-138-127 kubenswrapper[2581]: I0916 19:12:48.754944    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/54d699c3-2780-4189-a57d-0fe9c44b801c-proxy-tls\") pod \"isvc-logger-raw-cipn-predictor-7fd84f4ff-27rf6\" (UID: \"54d699c3-2780-4189-a57d-0fe9c44b801c\") " pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-7fd84f4ff-27rf6"
Sep 16 19:12:48.755111 ip-10-0-138-127 kubenswrapper[2581]: I0916 19:12:48.754971    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/54d699c3-2780-4189-a57d-0fe9c44b801c-cabundle-cert\") pod \"isvc-logger-raw-cipn-predictor-7fd84f4ff-27rf6\" (UID: \"54d699c3-2780-4189-a57d-0fe9c44b801c\") " pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-7fd84f4ff-27rf6"
Sep 16 19:12:48.755111 ip-10-0-138-127 kubenswrapper[2581]: I0916 19:12:48.755077    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v5ctf\" (UniqueName: \"kubernetes.io/projected/54d699c3-2780-4189-a57d-0fe9c44b801c-kube-api-access-v5ctf\") pod \"isvc-logger-raw-cipn-predictor-7fd84f4ff-27rf6\" (UID: \"54d699c3-2780-4189-a57d-0fe9c44b801c\") " pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-7fd84f4ff-27rf6"
Sep 16 19:12:48.755242 ip-10-0-138-127 kubenswrapper[2581]: I0916 19:12:48.755113    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/54d699c3-2780-4189-a57d-0fe9c44b801c-kube-rbac-proxy-sar-config\") pod \"isvc-logger-raw-cipn-predictor-7fd84f4ff-27rf6\" (UID: \"54d699c3-2780-4189-a57d-0fe9c44b801c\") " pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-7fd84f4ff-27rf6"
Sep 16 19:12:48.856333 ip-10-0-138-127 kubenswrapper[2581]: I0916 19:12:48.856242    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v5ctf\" (UniqueName: \"kubernetes.io/projected/54d699c3-2780-4189-a57d-0fe9c44b801c-kube-api-access-v5ctf\") pod \"isvc-logger-raw-cipn-predictor-7fd84f4ff-27rf6\" (UID: \"54d699c3-2780-4189-a57d-0fe9c44b801c\") " pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-7fd84f4ff-27rf6"
Sep 16 19:12:48.856333 ip-10-0-138-127 kubenswrapper[2581]: I0916 19:12:48.856287    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/54d699c3-2780-4189-a57d-0fe9c44b801c-kube-rbac-proxy-sar-config\") pod \"isvc-logger-raw-cipn-predictor-7fd84f4ff-27rf6\" (UID: \"54d699c3-2780-4189-a57d-0fe9c44b801c\") " pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-7fd84f4ff-27rf6"
Sep 16 19:12:48.856560 ip-10-0-138-127 kubenswrapper[2581]: I0916 19:12:48.856351    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/54d699c3-2780-4189-a57d-0fe9c44b801c-kserve-provision-location\") pod \"isvc-logger-raw-cipn-predictor-7fd84f4ff-27rf6\" (UID: \"54d699c3-2780-4189-a57d-0fe9c44b801c\") " pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-7fd84f4ff-27rf6"
Sep 16 19:12:48.856560 ip-10-0-138-127 kubenswrapper[2581]: I0916 19:12:48.856387    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/54d699c3-2780-4189-a57d-0fe9c44b801c-proxy-tls\") pod \"isvc-logger-raw-cipn-predictor-7fd84f4ff-27rf6\" (UID: \"54d699c3-2780-4189-a57d-0fe9c44b801c\") " pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-7fd84f4ff-27rf6"
Sep 16 19:12:48.856560 ip-10-0-138-127 kubenswrapper[2581]: I0916 19:12:48.856425    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/54d699c3-2780-4189-a57d-0fe9c44b801c-cabundle-cert\") pod \"isvc-logger-raw-cipn-predictor-7fd84f4ff-27rf6\" (UID: \"54d699c3-2780-4189-a57d-0fe9c44b801c\") " pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-7fd84f4ff-27rf6"
Sep 16 19:12:48.856841 ip-10-0-138-127 kubenswrapper[2581]: I0916 19:12:48.856815    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/54d699c3-2780-4189-a57d-0fe9c44b801c-kserve-provision-location\") pod \"isvc-logger-raw-cipn-predictor-7fd84f4ff-27rf6\" (UID: \"54d699c3-2780-4189-a57d-0fe9c44b801c\") " pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-7fd84f4ff-27rf6"
Sep 16 19:12:48.857100 ip-10-0-138-127 kubenswrapper[2581]: I0916 19:12:48.857081    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/54d699c3-2780-4189-a57d-0fe9c44b801c-kube-rbac-proxy-sar-config\") pod \"isvc-logger-raw-cipn-predictor-7fd84f4ff-27rf6\" (UID: \"54d699c3-2780-4189-a57d-0fe9c44b801c\") " pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-7fd84f4ff-27rf6"
Sep 16 19:12:48.857147 ip-10-0-138-127 kubenswrapper[2581]: I0916 19:12:48.857106    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/54d699c3-2780-4189-a57d-0fe9c44b801c-cabundle-cert\") pod \"isvc-logger-raw-cipn-predictor-7fd84f4ff-27rf6\" (UID: \"54d699c3-2780-4189-a57d-0fe9c44b801c\") " pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-7fd84f4ff-27rf6"
Sep 16 19:12:48.859067 ip-10-0-138-127 kubenswrapper[2581]: I0916 19:12:48.859029    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/54d699c3-2780-4189-a57d-0fe9c44b801c-proxy-tls\") pod \"isvc-logger-raw-cipn-predictor-7fd84f4ff-27rf6\" (UID: \"54d699c3-2780-4189-a57d-0fe9c44b801c\") " pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-7fd84f4ff-27rf6"
Sep 16 19:12:48.865917 ip-10-0-138-127 kubenswrapper[2581]: I0916 19:12:48.865896    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-v5ctf\" (UniqueName: \"kubernetes.io/projected/54d699c3-2780-4189-a57d-0fe9c44b801c-kube-api-access-v5ctf\") pod \"isvc-logger-raw-cipn-predictor-7fd84f4ff-27rf6\" (UID: \"54d699c3-2780-4189-a57d-0fe9c44b801c\") " pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-7fd84f4ff-27rf6"
Sep 16 19:12:48.891805 ip-10-0-138-127 kubenswrapper[2581]: I0916 19:12:48.891770    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-7fd84f4ff-27rf6"
Sep 16 19:12:49.171497 ip-10-0-138-127 kubenswrapper[2581]: I0916 19:12:49.171474    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-7fd84f4ff-27rf6"]
Sep 16 19:12:49.173699 ip-10-0-138-127 kubenswrapper[2581]: W0916 19:12:49.173676    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod54d699c3_2780_4189_a57d_0fe9c44b801c.slice/crio-62ad43645a487199b1eaad4c5a98ed27e6889b8723369965bf961d07a6228633 WatchSource:0}: Error finding container 62ad43645a487199b1eaad4c5a98ed27e6889b8723369965bf961d07a6228633: Status 404 returned error can't find the container with id 62ad43645a487199b1eaad4c5a98ed27e6889b8723369965bf961d07a6228633
Sep 16 19:12:49.954109 ip-10-0-138-127 kubenswrapper[2581]: I0916 19:12:49.954074    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-7fd84f4ff-27rf6" event={"ID":"54d699c3-2780-4189-a57d-0fe9c44b801c","Type":"ContainerStarted","Data":"cb108b88271476f75e2f032bb4a2f056f7ffd2f72ea292ceaf4d1c05e2dc15f1"}
Sep 16 19:12:49.954548 ip-10-0-138-127 kubenswrapper[2581]: I0916 19:12:49.954140    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-7fd84f4ff-27rf6" event={"ID":"54d699c3-2780-4189-a57d-0fe9c44b801c","Type":"ContainerStarted","Data":"62ad43645a487199b1eaad4c5a98ed27e6889b8723369965bf961d07a6228633"}
Sep 16 19:12:53.972958 ip-10-0-138-127 kubenswrapper[2581]: I0916 19:12:53.972917    2581 generic.go:356] "Generic (PLEG): container finished" podID="54d699c3-2780-4189-a57d-0fe9c44b801c" containerID="cb108b88271476f75e2f032bb4a2f056f7ffd2f72ea292ceaf4d1c05e2dc15f1" exitCode=0
Sep 16 19:12:53.973504 ip-10-0-138-127 kubenswrapper[2581]: I0916 19:12:53.972996    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-7fd84f4ff-27rf6" event={"ID":"54d699c3-2780-4189-a57d-0fe9c44b801c","Type":"ContainerDied","Data":"cb108b88271476f75e2f032bb4a2f056f7ffd2f72ea292ceaf4d1c05e2dc15f1"}
Sep 16 19:12:54.980091 ip-10-0-138-127 kubenswrapper[2581]: I0916 19:12:54.980054    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-7fd84f4ff-27rf6" event={"ID":"54d699c3-2780-4189-a57d-0fe9c44b801c","Type":"ContainerStarted","Data":"f90ded570dc95019176343fe35eb034f42d167f0788de99ae9ce73a1a5da5b41"}
Sep 16 19:12:54.980545 ip-10-0-138-127 kubenswrapper[2581]: I0916 19:12:54.980130    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-7fd84f4ff-27rf6" event={"ID":"54d699c3-2780-4189-a57d-0fe9c44b801c","Type":"ContainerStarted","Data":"e98c28b2762af10364084a327f959b31849d8c72f8beb5a2a0d04c3956c99ad4"}
Sep 16 19:12:54.980545 ip-10-0-138-127 kubenswrapper[2581]: I0916 19:12:54.980157    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-7fd84f4ff-27rf6" event={"ID":"54d699c3-2780-4189-a57d-0fe9c44b801c","Type":"ContainerStarted","Data":"14a5ff17ec44d8b511b0f993d8c32df930ece8f4217db2ff8bdb8329a72b2f34"}
Sep 16 19:12:54.980545 ip-10-0-138-127 kubenswrapper[2581]: I0916 19:12:54.980496    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-7fd84f4ff-27rf6"
Sep 16 19:12:54.980690 ip-10-0-138-127 kubenswrapper[2581]: I0916 19:12:54.980561    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-7fd84f4ff-27rf6"
Sep 16 19:12:54.980690 ip-10-0-138-127 kubenswrapper[2581]: I0916 19:12:54.980587    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-7fd84f4ff-27rf6"
Sep 16 19:12:54.981905 ip-10-0-138-127 kubenswrapper[2581]: I0916 19:12:54.981872    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-7fd84f4ff-27rf6" podUID="54d699c3-2780-4189-a57d-0fe9c44b801c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.39:8080: connect: connection refused"
Sep 16 19:12:54.982691 ip-10-0-138-127 kubenswrapper[2581]: I0916 19:12:54.982664    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-7fd84f4ff-27rf6" podUID="54d699c3-2780-4189-a57d-0fe9c44b801c" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 16 19:12:55.012046 ip-10-0-138-127 kubenswrapper[2581]: I0916 19:12:55.011983    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-7fd84f4ff-27rf6" podStartSLOduration=3.12424608 podStartE2EDuration="7.011967054s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.887720974s" isStatefulPod=false podCreationTimestamp="2026-09-16 19:12:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:12:55.009552407 +0000 UTC m=+1644.828966315" watchObservedRunningTime="2026-09-16 19:12:55.011967054 +0000 UTC m=+1644.831380962"
Sep 16 19:12:55.984542 ip-10-0-138-127 kubenswrapper[2581]: I0916 19:12:55.984497    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-7fd84f4ff-27rf6" podUID="54d699c3-2780-4189-a57d-0fe9c44b801c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.39:8080: connect: connection refused"
Sep 16 19:12:55.985017 ip-10-0-138-127 kubenswrapper[2581]: I0916 19:12:55.984988    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-7fd84f4ff-27rf6" podUID="54d699c3-2780-4189-a57d-0fe9c44b801c" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 16 19:13:00.988301 ip-10-0-138-127 kubenswrapper[2581]: I0916 19:13:00.988273    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-7fd84f4ff-27rf6"
Sep 16 19:13:00.989013 ip-10-0-138-127 kubenswrapper[2581]: I0916 19:13:00.988983    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-7fd84f4ff-27rf6" podUID="54d699c3-2780-4189-a57d-0fe9c44b801c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.39:8080: connect: connection refused"
Sep 16 19:13:00.989272 ip-10-0-138-127 kubenswrapper[2581]: I0916 19:13:00.989245    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-7fd84f4ff-27rf6" podUID="54d699c3-2780-4189-a57d-0fe9c44b801c" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 16 19:13:10.989658 ip-10-0-138-127 kubenswrapper[2581]: I0916 19:13:10.989613    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-7fd84f4ff-27rf6" podUID="54d699c3-2780-4189-a57d-0fe9c44b801c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.39:8080: connect: connection refused"
Sep 16 19:13:10.990133 ip-10-0-138-127 kubenswrapper[2581]: I0916 19:13:10.990070    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-7fd84f4ff-27rf6" podUID="54d699c3-2780-4189-a57d-0fe9c44b801c" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 16 19:13:20.988998 ip-10-0-138-127 kubenswrapper[2581]: I0916 19:13:20.988942    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-7fd84f4ff-27rf6" podUID="54d699c3-2780-4189-a57d-0fe9c44b801c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.39:8080: connect: connection refused"
Sep 16 19:13:20.989466 ip-10-0-138-127 kubenswrapper[2581]: I0916 19:13:20.989227    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-7fd84f4ff-27rf6" podUID="54d699c3-2780-4189-a57d-0fe9c44b801c" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 16 19:13:30.988914 ip-10-0-138-127 kubenswrapper[2581]: I0916 19:13:30.988866    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-7fd84f4ff-27rf6" podUID="54d699c3-2780-4189-a57d-0fe9c44b801c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.39:8080: connect: connection refused"
Sep 16 19:13:30.989490 ip-10-0-138-127 kubenswrapper[2581]: I0916 19:13:30.989243    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-7fd84f4ff-27rf6" podUID="54d699c3-2780-4189-a57d-0fe9c44b801c" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 16 19:13:40.989672 ip-10-0-138-127 kubenswrapper[2581]: I0916 19:13:40.989628    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-7fd84f4ff-27rf6" podUID="54d699c3-2780-4189-a57d-0fe9c44b801c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.39:8080: connect: connection refused"
Sep 16 19:13:40.990147 ip-10-0-138-127 kubenswrapper[2581]: I0916 19:13:40.990115    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-7fd84f4ff-27rf6" podUID="54d699c3-2780-4189-a57d-0fe9c44b801c" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 16 19:13:50.988905 ip-10-0-138-127 kubenswrapper[2581]: I0916 19:13:50.988857    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-7fd84f4ff-27rf6" podUID="54d699c3-2780-4189-a57d-0fe9c44b801c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.39:8080: connect: connection refused"
Sep 16 19:13:50.989425 ip-10-0-138-127 kubenswrapper[2581]: I0916 19:13:50.989327    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-7fd84f4ff-27rf6" podUID="54d699c3-2780-4189-a57d-0fe9c44b801c" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 16 19:14:00.989429 ip-10-0-138-127 kubenswrapper[2581]: I0916 19:14:00.989326    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-7fd84f4ff-27rf6" podUID="54d699c3-2780-4189-a57d-0fe9c44b801c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.39:8080: connect: connection refused"
Sep 16 19:14:00.989906 ip-10-0-138-127 kubenswrapper[2581]: I0916 19:14:00.989866    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-7fd84f4ff-27rf6" podUID="54d699c3-2780-4189-a57d-0fe9c44b801c" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 16 19:14:10.989358 ip-10-0-138-127 kubenswrapper[2581]: I0916 19:14:10.989323    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-7fd84f4ff-27rf6"
Sep 16 19:14:10.989966 ip-10-0-138-127 kubenswrapper[2581]: I0916 19:14:10.989751    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-7fd84f4ff-27rf6"
Sep 16 19:14:23.799236 ip-10-0-138-127 kubenswrapper[2581]: I0916 19:14:23.799183    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-7fd84f4ff-27rf6"]
Sep 16 19:14:23.799754 ip-10-0-138-127 kubenswrapper[2581]: I0916 19:14:23.799480    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-7fd84f4ff-27rf6" podUID="54d699c3-2780-4189-a57d-0fe9c44b801c" containerName="kserve-container" containerID="cri-o://14a5ff17ec44d8b511b0f993d8c32df930ece8f4217db2ff8bdb8329a72b2f34" gracePeriod=30
Sep 16 19:14:23.799754 ip-10-0-138-127 kubenswrapper[2581]: I0916 19:14:23.799512    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-7fd84f4ff-27rf6" podUID="54d699c3-2780-4189-a57d-0fe9c44b801c" containerName="agent" containerID="cri-o://f90ded570dc95019176343fe35eb034f42d167f0788de99ae9ce73a1a5da5b41" gracePeriod=30
Sep 16 19:14:23.799754 ip-10-0-138-127 kubenswrapper[2581]: I0916 19:14:23.799538    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-7fd84f4ff-27rf6" podUID="54d699c3-2780-4189-a57d-0fe9c44b801c" containerName="kube-rbac-proxy" containerID="cri-o://e98c28b2762af10364084a327f959b31849d8c72f8beb5a2a0d04c3956c99ad4" gracePeriod=30
Sep 16 19:14:24.347943 ip-10-0-138-127 kubenswrapper[2581]: I0916 19:14:24.347913    2581 generic.go:356] "Generic (PLEG): container finished" podID="54d699c3-2780-4189-a57d-0fe9c44b801c" containerID="e98c28b2762af10364084a327f959b31849d8c72f8beb5a2a0d04c3956c99ad4" exitCode=2
Sep 16 19:14:24.348108 ip-10-0-138-127 kubenswrapper[2581]: I0916 19:14:24.347950    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-7fd84f4ff-27rf6" event={"ID":"54d699c3-2780-4189-a57d-0fe9c44b801c","Type":"ContainerDied","Data":"e98c28b2762af10364084a327f959b31849d8c72f8beb5a2a0d04c3956c99ad4"}
Sep 16 19:14:25.255446 ip-10-0-138-127 kubenswrapper[2581]: I0916 19:14:25.255415    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-jlscm/must-gather-gmkvj"]
Sep 16 19:14:25.258814 ip-10-0-138-127 kubenswrapper[2581]: I0916 19:14:25.258798    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-jlscm/must-gather-gmkvj"
Sep 16 19:14:25.261069 ip-10-0-138-127 kubenswrapper[2581]: I0916 19:14:25.261047    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-jlscm\"/\"openshift-service-ca.crt\""
Sep 16 19:14:25.261232 ip-10-0-138-127 kubenswrapper[2581]: I0916 19:14:25.261135    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-jlscm\"/\"kube-root-ca.crt\""
Sep 16 19:14:25.261327 ip-10-0-138-127 kubenswrapper[2581]: I0916 19:14:25.261305    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-jlscm\"/\"default-dockercfg-w4rk5\""
Sep 16 19:14:25.267411 ip-10-0-138-127 kubenswrapper[2581]: I0916 19:14:25.267387    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-jlscm/must-gather-gmkvj"]
Sep 16 19:14:25.368427 ip-10-0-138-127 kubenswrapper[2581]: I0916 19:14:25.368393    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/79a80f32-e7e8-4d05-8ea6-6a1a279afcda-must-gather-output\") pod \"must-gather-gmkvj\" (UID: \"79a80f32-e7e8-4d05-8ea6-6a1a279afcda\") " pod="openshift-must-gather-jlscm/must-gather-gmkvj"
Sep 16 19:14:25.368643 ip-10-0-138-127 kubenswrapper[2581]: I0916 19:14:25.368529    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z7x2r\" (UniqueName: \"kubernetes.io/projected/79a80f32-e7e8-4d05-8ea6-6a1a279afcda-kube-api-access-z7x2r\") pod \"must-gather-gmkvj\" (UID: \"79a80f32-e7e8-4d05-8ea6-6a1a279afcda\") " pod="openshift-must-gather-jlscm/must-gather-gmkvj"
Sep 16 19:14:25.469849 ip-10-0-138-127 kubenswrapper[2581]: I0916 19:14:25.469814    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z7x2r\" (UniqueName: \"kubernetes.io/projected/79a80f32-e7e8-4d05-8ea6-6a1a279afcda-kube-api-access-z7x2r\") pod \"must-gather-gmkvj\" (UID: \"79a80f32-e7e8-4d05-8ea6-6a1a279afcda\") " pod="openshift-must-gather-jlscm/must-gather-gmkvj"
Sep 16 19:14:25.469849 ip-10-0-138-127 kubenswrapper[2581]: I0916 19:14:25.469861    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/79a80f32-e7e8-4d05-8ea6-6a1a279afcda-must-gather-output\") pod \"must-gather-gmkvj\" (UID: \"79a80f32-e7e8-4d05-8ea6-6a1a279afcda\") " pod="openshift-must-gather-jlscm/must-gather-gmkvj"
Sep 16 19:14:25.470167 ip-10-0-138-127 kubenswrapper[2581]: I0916 19:14:25.470152    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/79a80f32-e7e8-4d05-8ea6-6a1a279afcda-must-gather-output\") pod \"must-gather-gmkvj\" (UID: \"79a80f32-e7e8-4d05-8ea6-6a1a279afcda\") " pod="openshift-must-gather-jlscm/must-gather-gmkvj"
Sep 16 19:14:25.478589 ip-10-0-138-127 kubenswrapper[2581]: I0916 19:14:25.478563    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-z7x2r\" (UniqueName: \"kubernetes.io/projected/79a80f32-e7e8-4d05-8ea6-6a1a279afcda-kube-api-access-z7x2r\") pod \"must-gather-gmkvj\" (UID: \"79a80f32-e7e8-4d05-8ea6-6a1a279afcda\") " pod="openshift-must-gather-jlscm/must-gather-gmkvj"
Sep 16 19:14:25.576848 ip-10-0-138-127 kubenswrapper[2581]: I0916 19:14:25.576812    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-jlscm/must-gather-gmkvj"
Sep 16 19:14:25.847061 ip-10-0-138-127 kubenswrapper[2581]: I0916 19:14:25.847037    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-jlscm/must-gather-gmkvj"]
Sep 16 19:14:25.849627 ip-10-0-138-127 kubenswrapper[2581]: W0916 19:14:25.849598    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod79a80f32_e7e8_4d05_8ea6_6a1a279afcda.slice/crio-c6cd2240c827a27a7cde5ed970879a646ad1102aae68aff1d43232dfbc37cb22 WatchSource:0}: Error finding container c6cd2240c827a27a7cde5ed970879a646ad1102aae68aff1d43232dfbc37cb22: Status 404 returned error can't find the container with id c6cd2240c827a27a7cde5ed970879a646ad1102aae68aff1d43232dfbc37cb22
Sep 16 19:14:25.851435 ip-10-0-138-127 kubenswrapper[2581]: I0916 19:14:25.851417    2581 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 16 19:14:25.985238 ip-10-0-138-127 kubenswrapper[2581]: I0916 19:14:25.985169    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-7fd84f4ff-27rf6" podUID="54d699c3-2780-4189-a57d-0fe9c44b801c" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.39:8643/healthz\": dial tcp 10.132.0.39:8643: connect: connection refused"
Sep 16 19:14:26.357608 ip-10-0-138-127 kubenswrapper[2581]: I0916 19:14:26.357572    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-jlscm/must-gather-gmkvj" event={"ID":"79a80f32-e7e8-4d05-8ea6-6a1a279afcda","Type":"ContainerStarted","Data":"c6cd2240c827a27a7cde5ed970879a646ad1102aae68aff1d43232dfbc37cb22"}
Sep 16 19:14:30.386793 ip-10-0-138-127 kubenswrapper[2581]: I0916 19:14:30.386757    2581 generic.go:356] "Generic (PLEG): container finished" podID="54d699c3-2780-4189-a57d-0fe9c44b801c" containerID="14a5ff17ec44d8b511b0f993d8c32df930ece8f4217db2ff8bdb8329a72b2f34" exitCode=0
Sep 16 19:14:30.387217 ip-10-0-138-127 kubenswrapper[2581]: I0916 19:14:30.386829    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-7fd84f4ff-27rf6" event={"ID":"54d699c3-2780-4189-a57d-0fe9c44b801c","Type":"ContainerDied","Data":"14a5ff17ec44d8b511b0f993d8c32df930ece8f4217db2ff8bdb8329a72b2f34"}
Sep 16 19:14:30.985583 ip-10-0-138-127 kubenswrapper[2581]: I0916 19:14:30.985539    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-7fd84f4ff-27rf6" podUID="54d699c3-2780-4189-a57d-0fe9c44b801c" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.39:8643/healthz\": dial tcp 10.132.0.39:8643: connect: connection refused"
Sep 16 19:14:30.989416 ip-10-0-138-127 kubenswrapper[2581]: I0916 19:14:30.989376    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-7fd84f4ff-27rf6" podUID="54d699c3-2780-4189-a57d-0fe9c44b801c" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 16 19:14:30.990556 ip-10-0-138-127 kubenswrapper[2581]: I0916 19:14:30.990527    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-7fd84f4ff-27rf6" podUID="54d699c3-2780-4189-a57d-0fe9c44b801c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.39:8080: connect: connection refused"
Sep 16 19:14:32.397695 ip-10-0-138-127 kubenswrapper[2581]: I0916 19:14:32.397651    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-jlscm/must-gather-gmkvj" event={"ID":"79a80f32-e7e8-4d05-8ea6-6a1a279afcda","Type":"ContainerStarted","Data":"8d9e7249fb785889b410ac1d4012e9d17d7fde364c0561c6ccc3aac29172165b"}
Sep 16 19:14:32.398079 ip-10-0-138-127 kubenswrapper[2581]: I0916 19:14:32.397703    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-jlscm/must-gather-gmkvj" event={"ID":"79a80f32-e7e8-4d05-8ea6-6a1a279afcda","Type":"ContainerStarted","Data":"0c9b1d6b3919aaed9214b0e625e017b430dd61ee6b84044a69ab29e5df7b2c68"}
Sep 16 19:14:32.417468 ip-10-0-138-127 kubenswrapper[2581]: I0916 19:14:32.417414    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-jlscm/must-gather-gmkvj" podStartSLOduration=1.249167802 podStartE2EDuration="7.417396318s" totalImagesPullingTime="6.168228516s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 19:14:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:14:32.415512064 +0000 UTC m=+1742.234925971" watchObservedRunningTime="2026-09-16 19:14:32.417396318 +0000 UTC m=+1742.236810226"
Sep 16 19:14:35.984973 ip-10-0-138-127 kubenswrapper[2581]: I0916 19:14:35.984921    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-7fd84f4ff-27rf6" podUID="54d699c3-2780-4189-a57d-0fe9c44b801c" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.39:8643/healthz\": dial tcp 10.132.0.39:8643: connect: connection refused"
Sep 16 19:14:35.985673 ip-10-0-138-127 kubenswrapper[2581]: I0916 19:14:35.985649    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-7fd84f4ff-27rf6"
Sep 16 19:14:40.984639 ip-10-0-138-127 kubenswrapper[2581]: I0916 19:14:40.984594    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-7fd84f4ff-27rf6" podUID="54d699c3-2780-4189-a57d-0fe9c44b801c" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.39:8643/healthz\": dial tcp 10.132.0.39:8643: connect: connection refused"
Sep 16 19:14:40.989734 ip-10-0-138-127 kubenswrapper[2581]: I0916 19:14:40.989697    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-7fd84f4ff-27rf6" podUID="54d699c3-2780-4189-a57d-0fe9c44b801c" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 16 19:14:40.989889 ip-10-0-138-127 kubenswrapper[2581]: I0916 19:14:40.989867    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-7fd84f4ff-27rf6" podUID="54d699c3-2780-4189-a57d-0fe9c44b801c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.39:8080: connect: connection refused"
Sep 16 19:14:41.465050 ip-10-0-138-127 kubenswrapper[2581]: I0916 19:14:41.465016    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-7fd84f4ff-27rf6_54d699c3-2780-4189-a57d-0fe9c44b801c/kserve-container/0.log"
Sep 16 19:14:41.495007 ip-10-0-138-127 kubenswrapper[2581]: I0916 19:14:41.494972    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-7fd84f4ff-27rf6_54d699c3-2780-4189-a57d-0fe9c44b801c/kube-rbac-proxy/0.log"
Sep 16 19:14:41.507276 ip-10-0-138-127 kubenswrapper[2581]: I0916 19:14:41.507248    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-7fd84f4ff-27rf6_54d699c3-2780-4189-a57d-0fe9c44b801c/agent/0.log"
Sep 16 19:14:41.519463 ip-10-0-138-127 kubenswrapper[2581]: I0916 19:14:41.519435    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-7fd84f4ff-27rf6_54d699c3-2780-4189-a57d-0fe9c44b801c/storage-initializer/0.log"
Sep 16 19:14:42.375266 ip-10-0-138-127 kubenswrapper[2581]: I0916 19:14:42.375234    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-7fd84f4ff-27rf6_54d699c3-2780-4189-a57d-0fe9c44b801c/kserve-container/0.log"
Sep 16 19:14:42.391761 ip-10-0-138-127 kubenswrapper[2581]: I0916 19:14:42.391728    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-7fd84f4ff-27rf6_54d699c3-2780-4189-a57d-0fe9c44b801c/kube-rbac-proxy/0.log"
Sep 16 19:14:42.404182 ip-10-0-138-127 kubenswrapper[2581]: I0916 19:14:42.404153    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-7fd84f4ff-27rf6_54d699c3-2780-4189-a57d-0fe9c44b801c/agent/0.log"
Sep 16 19:14:42.416183 ip-10-0-138-127 kubenswrapper[2581]: I0916 19:14:42.416160    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-7fd84f4ff-27rf6_54d699c3-2780-4189-a57d-0fe9c44b801c/storage-initializer/0.log"
Sep 16 19:14:43.374060 ip-10-0-138-127 kubenswrapper[2581]: I0916 19:14:43.374030    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-7fd84f4ff-27rf6_54d699c3-2780-4189-a57d-0fe9c44b801c/kserve-container/0.log"
Sep 16 19:14:43.393039 ip-10-0-138-127 kubenswrapper[2581]: I0916 19:14:43.393011    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-7fd84f4ff-27rf6_54d699c3-2780-4189-a57d-0fe9c44b801c/kube-rbac-proxy/0.log"
Sep 16 19:14:43.407771 ip-10-0-138-127 kubenswrapper[2581]: I0916 19:14:43.407735    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-7fd84f4ff-27rf6_54d699c3-2780-4189-a57d-0fe9c44b801c/agent/0.log"
Sep 16 19:14:43.421061 ip-10-0-138-127 kubenswrapper[2581]: I0916 19:14:43.421039    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-7fd84f4ff-27rf6_54d699c3-2780-4189-a57d-0fe9c44b801c/storage-initializer/0.log"
Sep 16 19:14:44.343650 ip-10-0-138-127 kubenswrapper[2581]: I0916 19:14:44.343608    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-7fd84f4ff-27rf6_54d699c3-2780-4189-a57d-0fe9c44b801c/kserve-container/0.log"
Sep 16 19:14:44.373082 ip-10-0-138-127 kubenswrapper[2581]: I0916 19:14:44.373039    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-7fd84f4ff-27rf6_54d699c3-2780-4189-a57d-0fe9c44b801c/kube-rbac-proxy/0.log"
Sep 16 19:14:44.405635 ip-10-0-138-127 kubenswrapper[2581]: I0916 19:14:44.405600    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-7fd84f4ff-27rf6_54d699c3-2780-4189-a57d-0fe9c44b801c/agent/0.log"
Sep 16 19:14:44.427058 ip-10-0-138-127 kubenswrapper[2581]: I0916 19:14:44.427029    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-7fd84f4ff-27rf6_54d699c3-2780-4189-a57d-0fe9c44b801c/storage-initializer/0.log"
Sep 16 19:14:45.399039 ip-10-0-138-127 kubenswrapper[2581]: I0916 19:14:45.399011    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-7fd84f4ff-27rf6_54d699c3-2780-4189-a57d-0fe9c44b801c/kserve-container/0.log"
Sep 16 19:14:45.421402 ip-10-0-138-127 kubenswrapper[2581]: I0916 19:14:45.421373    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-7fd84f4ff-27rf6_54d699c3-2780-4189-a57d-0fe9c44b801c/kube-rbac-proxy/0.log"
Sep 16 19:14:45.445113 ip-10-0-138-127 kubenswrapper[2581]: I0916 19:14:45.445081    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-7fd84f4ff-27rf6_54d699c3-2780-4189-a57d-0fe9c44b801c/agent/0.log"
Sep 16 19:14:45.479044 ip-10-0-138-127 kubenswrapper[2581]: I0916 19:14:45.479007    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-7fd84f4ff-27rf6_54d699c3-2780-4189-a57d-0fe9c44b801c/storage-initializer/0.log"
Sep 16 19:14:45.985208 ip-10-0-138-127 kubenswrapper[2581]: I0916 19:14:45.985150    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-7fd84f4ff-27rf6" podUID="54d699c3-2780-4189-a57d-0fe9c44b801c" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.39:8643/healthz\": dial tcp 10.132.0.39:8643: connect: connection refused"
Sep 16 19:14:46.489674 ip-10-0-138-127 kubenswrapper[2581]: I0916 19:14:46.489638    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-7fd84f4ff-27rf6_54d699c3-2780-4189-a57d-0fe9c44b801c/kserve-container/0.log"
Sep 16 19:14:46.507111 ip-10-0-138-127 kubenswrapper[2581]: I0916 19:14:46.507084    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-7fd84f4ff-27rf6_54d699c3-2780-4189-a57d-0fe9c44b801c/kube-rbac-proxy/0.log"
Sep 16 19:14:46.523080 ip-10-0-138-127 kubenswrapper[2581]: I0916 19:14:46.523052    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-7fd84f4ff-27rf6_54d699c3-2780-4189-a57d-0fe9c44b801c/agent/0.log"
Sep 16 19:14:46.540431 ip-10-0-138-127 kubenswrapper[2581]: I0916 19:14:46.540401    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-7fd84f4ff-27rf6_54d699c3-2780-4189-a57d-0fe9c44b801c/storage-initializer/0.log"
Sep 16 19:14:47.483889 ip-10-0-138-127 kubenswrapper[2581]: I0916 19:14:47.483852    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-7fd84f4ff-27rf6_54d699c3-2780-4189-a57d-0fe9c44b801c/kserve-container/0.log"
Sep 16 19:14:47.526725 ip-10-0-138-127 kubenswrapper[2581]: I0916 19:14:47.526697    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-7fd84f4ff-27rf6_54d699c3-2780-4189-a57d-0fe9c44b801c/kube-rbac-proxy/0.log"
Sep 16 19:14:47.552887 ip-10-0-138-127 kubenswrapper[2581]: I0916 19:14:47.552857    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-7fd84f4ff-27rf6_54d699c3-2780-4189-a57d-0fe9c44b801c/agent/0.log"
Sep 16 19:14:47.578306 ip-10-0-138-127 kubenswrapper[2581]: I0916 19:14:47.578266    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-7fd84f4ff-27rf6_54d699c3-2780-4189-a57d-0fe9c44b801c/storage-initializer/0.log"
Sep 16 19:14:48.482649 ip-10-0-138-127 kubenswrapper[2581]: I0916 19:14:48.482618    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-7fd84f4ff-27rf6_54d699c3-2780-4189-a57d-0fe9c44b801c/kserve-container/0.log"
Sep 16 19:14:48.499439 ip-10-0-138-127 kubenswrapper[2581]: I0916 19:14:48.499405    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-7fd84f4ff-27rf6_54d699c3-2780-4189-a57d-0fe9c44b801c/kube-rbac-proxy/0.log"
Sep 16 19:14:48.514214 ip-10-0-138-127 kubenswrapper[2581]: I0916 19:14:48.514161    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-7fd84f4ff-27rf6_54d699c3-2780-4189-a57d-0fe9c44b801c/agent/0.log"
Sep 16 19:14:48.528720 ip-10-0-138-127 kubenswrapper[2581]: I0916 19:14:48.528699    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-7fd84f4ff-27rf6_54d699c3-2780-4189-a57d-0fe9c44b801c/storage-initializer/0.log"
Sep 16 19:14:49.388723 ip-10-0-138-127 kubenswrapper[2581]: I0916 19:14:49.388674    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-7fd84f4ff-27rf6_54d699c3-2780-4189-a57d-0fe9c44b801c/kserve-container/0.log"
Sep 16 19:14:49.403378 ip-10-0-138-127 kubenswrapper[2581]: I0916 19:14:49.403352    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-7fd84f4ff-27rf6_54d699c3-2780-4189-a57d-0fe9c44b801c/kube-rbac-proxy/0.log"
Sep 16 19:14:49.415890 ip-10-0-138-127 kubenswrapper[2581]: I0916 19:14:49.415856    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-7fd84f4ff-27rf6_54d699c3-2780-4189-a57d-0fe9c44b801c/agent/0.log"
Sep 16 19:14:49.428151 ip-10-0-138-127 kubenswrapper[2581]: I0916 19:14:49.428128    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-7fd84f4ff-27rf6_54d699c3-2780-4189-a57d-0fe9c44b801c/storage-initializer/0.log"
Sep 16 19:14:50.331319 ip-10-0-138-127 kubenswrapper[2581]: I0916 19:14:50.331287    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-7fd84f4ff-27rf6_54d699c3-2780-4189-a57d-0fe9c44b801c/kserve-container/0.log"
Sep 16 19:14:50.347391 ip-10-0-138-127 kubenswrapper[2581]: I0916 19:14:50.347364    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-7fd84f4ff-27rf6_54d699c3-2780-4189-a57d-0fe9c44b801c/kube-rbac-proxy/0.log"
Sep 16 19:14:50.360584 ip-10-0-138-127 kubenswrapper[2581]: I0916 19:14:50.360541    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-7fd84f4ff-27rf6_54d699c3-2780-4189-a57d-0fe9c44b801c/agent/0.log"
Sep 16 19:14:50.374565 ip-10-0-138-127 kubenswrapper[2581]: I0916 19:14:50.374545    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-7fd84f4ff-27rf6_54d699c3-2780-4189-a57d-0fe9c44b801c/storage-initializer/0.log"
Sep 16 19:14:50.984629 ip-10-0-138-127 kubenswrapper[2581]: I0916 19:14:50.984587    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-7fd84f4ff-27rf6" podUID="54d699c3-2780-4189-a57d-0fe9c44b801c" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.39:8643/healthz\": dial tcp 10.132.0.39:8643: connect: connection refused"
Sep 16 19:14:50.989431 ip-10-0-138-127 kubenswrapper[2581]: I0916 19:14:50.989403    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-7fd84f4ff-27rf6" podUID="54d699c3-2780-4189-a57d-0fe9c44b801c" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 16 19:14:50.989555 ip-10-0-138-127 kubenswrapper[2581]: I0916 19:14:50.989525    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-7fd84f4ff-27rf6"
Sep 16 19:14:50.989891 ip-10-0-138-127 kubenswrapper[2581]: I0916 19:14:50.989865    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-7fd84f4ff-27rf6" podUID="54d699c3-2780-4189-a57d-0fe9c44b801c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.39:8080: connect: connection refused"
Sep 16 19:14:50.990005 ip-10-0-138-127 kubenswrapper[2581]: I0916 19:14:50.989990    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-7fd84f4ff-27rf6"
Sep 16 19:14:51.263648 ip-10-0-138-127 kubenswrapper[2581]: I0916 19:14:51.263572    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-7fd84f4ff-27rf6_54d699c3-2780-4189-a57d-0fe9c44b801c/kserve-container/0.log"
Sep 16 19:14:51.284294 ip-10-0-138-127 kubenswrapper[2581]: I0916 19:14:51.284256    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-7fd84f4ff-27rf6_54d699c3-2780-4189-a57d-0fe9c44b801c/kube-rbac-proxy/0.log"
Sep 16 19:14:51.297961 ip-10-0-138-127 kubenswrapper[2581]: I0916 19:14:51.297932    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-7fd84f4ff-27rf6_54d699c3-2780-4189-a57d-0fe9c44b801c/agent/0.log"
Sep 16 19:14:51.314429 ip-10-0-138-127 kubenswrapper[2581]: I0916 19:14:51.314395    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-7fd84f4ff-27rf6_54d699c3-2780-4189-a57d-0fe9c44b801c/storage-initializer/0.log"
Sep 16 19:14:52.285741 ip-10-0-138-127 kubenswrapper[2581]: I0916 19:14:52.285707    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-7fd84f4ff-27rf6_54d699c3-2780-4189-a57d-0fe9c44b801c/kserve-container/0.log"
Sep 16 19:14:52.301404 ip-10-0-138-127 kubenswrapper[2581]: I0916 19:14:52.301369    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-7fd84f4ff-27rf6_54d699c3-2780-4189-a57d-0fe9c44b801c/kube-rbac-proxy/0.log"
Sep 16 19:14:52.314300 ip-10-0-138-127 kubenswrapper[2581]: I0916 19:14:52.314276    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-7fd84f4ff-27rf6_54d699c3-2780-4189-a57d-0fe9c44b801c/agent/0.log"
Sep 16 19:14:52.329079 ip-10-0-138-127 kubenswrapper[2581]: I0916 19:14:52.329049    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-7fd84f4ff-27rf6_54d699c3-2780-4189-a57d-0fe9c44b801c/storage-initializer/0.log"
Sep 16 19:14:53.273136 ip-10-0-138-127 kubenswrapper[2581]: I0916 19:14:53.273104    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-7fd84f4ff-27rf6_54d699c3-2780-4189-a57d-0fe9c44b801c/kserve-container/0.log"
Sep 16 19:14:53.289387 ip-10-0-138-127 kubenswrapper[2581]: I0916 19:14:53.289345    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-7fd84f4ff-27rf6_54d699c3-2780-4189-a57d-0fe9c44b801c/kube-rbac-proxy/0.log"
Sep 16 19:14:53.304552 ip-10-0-138-127 kubenswrapper[2581]: I0916 19:14:53.304530    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-7fd84f4ff-27rf6_54d699c3-2780-4189-a57d-0fe9c44b801c/agent/0.log"
Sep 16 19:14:53.325900 ip-10-0-138-127 kubenswrapper[2581]: I0916 19:14:53.325820    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-7fd84f4ff-27rf6_54d699c3-2780-4189-a57d-0fe9c44b801c/storage-initializer/0.log"
Sep 16 19:14:54.198511 ip-10-0-138-127 kubenswrapper[2581]: I0916 19:14:54.198419    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-7fd84f4ff-27rf6_54d699c3-2780-4189-a57d-0fe9c44b801c/kserve-container/0.log"
Sep 16 19:14:54.224416 ip-10-0-138-127 kubenswrapper[2581]: I0916 19:14:54.224374    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-7fd84f4ff-27rf6_54d699c3-2780-4189-a57d-0fe9c44b801c/kube-rbac-proxy/0.log"
Sep 16 19:14:54.245360 ip-10-0-138-127 kubenswrapper[2581]: I0916 19:14:54.245330    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-7fd84f4ff-27rf6_54d699c3-2780-4189-a57d-0fe9c44b801c/agent/0.log"
Sep 16 19:14:54.262955 ip-10-0-138-127 kubenswrapper[2581]: I0916 19:14:54.262926    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-7fd84f4ff-27rf6_54d699c3-2780-4189-a57d-0fe9c44b801c/storage-initializer/0.log"
Sep 16 19:14:54.474445 ip-10-0-138-127 kubenswrapper[2581]: I0916 19:14:54.474416    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-7fd84f4ff-27rf6"
Sep 16 19:14:54.505572 ip-10-0-138-127 kubenswrapper[2581]: I0916 19:14:54.505521    2581 generic.go:356] "Generic (PLEG): container finished" podID="54d699c3-2780-4189-a57d-0fe9c44b801c" containerID="f90ded570dc95019176343fe35eb034f42d167f0788de99ae9ce73a1a5da5b41" exitCode=0
Sep 16 19:14:54.505748 ip-10-0-138-127 kubenswrapper[2581]: I0916 19:14:54.505586    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-7fd84f4ff-27rf6"
Sep 16 19:14:54.505748 ip-10-0-138-127 kubenswrapper[2581]: I0916 19:14:54.505584    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-7fd84f4ff-27rf6" event={"ID":"54d699c3-2780-4189-a57d-0fe9c44b801c","Type":"ContainerDied","Data":"f90ded570dc95019176343fe35eb034f42d167f0788de99ae9ce73a1a5da5b41"}
Sep 16 19:14:54.505748 ip-10-0-138-127 kubenswrapper[2581]: I0916 19:14:54.505697    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-7fd84f4ff-27rf6" event={"ID":"54d699c3-2780-4189-a57d-0fe9c44b801c","Type":"ContainerDied","Data":"62ad43645a487199b1eaad4c5a98ed27e6889b8723369965bf961d07a6228633"}
Sep 16 19:14:54.505940 ip-10-0-138-127 kubenswrapper[2581]: I0916 19:14:54.505744    2581 scope.go:117] "RemoveContainer" containerID="f90ded570dc95019176343fe35eb034f42d167f0788de99ae9ce73a1a5da5b41"
Sep 16 19:14:54.515571 ip-10-0-138-127 kubenswrapper[2581]: I0916 19:14:54.515552    2581 scope.go:117] "RemoveContainer" containerID="e98c28b2762af10364084a327f959b31849d8c72f8beb5a2a0d04c3956c99ad4"
Sep 16 19:14:54.524407 ip-10-0-138-127 kubenswrapper[2581]: I0916 19:14:54.524383    2581 scope.go:117] "RemoveContainer" containerID="14a5ff17ec44d8b511b0f993d8c32df930ece8f4217db2ff8bdb8329a72b2f34"
Sep 16 19:14:54.533670 ip-10-0-138-127 kubenswrapper[2581]: I0916 19:14:54.533647    2581 scope.go:117] "RemoveContainer" containerID="cb108b88271476f75e2f032bb4a2f056f7ffd2f72ea292ceaf4d1c05e2dc15f1"
Sep 16 19:14:54.541318 ip-10-0-138-127 kubenswrapper[2581]: I0916 19:14:54.541300    2581 scope.go:117] "RemoveContainer" containerID="f90ded570dc95019176343fe35eb034f42d167f0788de99ae9ce73a1a5da5b41"
Sep 16 19:14:54.541456 ip-10-0-138-127 kubenswrapper[2581]: I0916 19:14:54.541431    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/54d699c3-2780-4189-a57d-0fe9c44b801c-cabundle-cert\") pod \"54d699c3-2780-4189-a57d-0fe9c44b801c\" (UID: \"54d699c3-2780-4189-a57d-0fe9c44b801c\") "
Sep 16 19:14:54.541581 ip-10-0-138-127 kubenswrapper[2581]: I0916 19:14:54.541546    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/54d699c3-2780-4189-a57d-0fe9c44b801c-kserve-provision-location\") pod \"54d699c3-2780-4189-a57d-0fe9c44b801c\" (UID: \"54d699c3-2780-4189-a57d-0fe9c44b801c\") "
Sep 16 19:14:54.541659 ip-10-0-138-127 kubenswrapper[2581]: E0916 19:14:54.541632    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f90ded570dc95019176343fe35eb034f42d167f0788de99ae9ce73a1a5da5b41\": container with ID starting with f90ded570dc95019176343fe35eb034f42d167f0788de99ae9ce73a1a5da5b41 not found: ID does not exist" containerID="f90ded570dc95019176343fe35eb034f42d167f0788de99ae9ce73a1a5da5b41"
Sep 16 19:14:54.541727 ip-10-0-138-127 kubenswrapper[2581]: I0916 19:14:54.541673    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f90ded570dc95019176343fe35eb034f42d167f0788de99ae9ce73a1a5da5b41"} err="failed to get container status \"f90ded570dc95019176343fe35eb034f42d167f0788de99ae9ce73a1a5da5b41\": rpc error: code = NotFound desc = could not find container \"f90ded570dc95019176343fe35eb034f42d167f0788de99ae9ce73a1a5da5b41\": container with ID starting with f90ded570dc95019176343fe35eb034f42d167f0788de99ae9ce73a1a5da5b41 not found: ID does not exist"
Sep 16 19:14:54.541727 ip-10-0-138-127 kubenswrapper[2581]: I0916 19:14:54.541699    2581 scope.go:117] "RemoveContainer" containerID="e98c28b2762af10364084a327f959b31849d8c72f8beb5a2a0d04c3956c99ad4"
Sep 16 19:14:54.541846 ip-10-0-138-127 kubenswrapper[2581]: I0916 19:14:54.541638    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/54d699c3-2780-4189-a57d-0fe9c44b801c-proxy-tls\") pod \"54d699c3-2780-4189-a57d-0fe9c44b801c\" (UID: \"54d699c3-2780-4189-a57d-0fe9c44b801c\") "
Sep 16 19:14:54.541846 ip-10-0-138-127 kubenswrapper[2581]: I0916 19:14:54.541779    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/54d699c3-2780-4189-a57d-0fe9c44b801c-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "54d699c3-2780-4189-a57d-0fe9c44b801c" (UID: "54d699c3-2780-4189-a57d-0fe9c44b801c"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 19:14:54.541846 ip-10-0-138-127 kubenswrapper[2581]: I0916 19:14:54.541786    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/54d699c3-2780-4189-a57d-0fe9c44b801c-kube-rbac-proxy-sar-config\") pod \"54d699c3-2780-4189-a57d-0fe9c44b801c\" (UID: \"54d699c3-2780-4189-a57d-0fe9c44b801c\") "
Sep 16 19:14:54.541846 ip-10-0-138-127 kubenswrapper[2581]: I0916 19:14:54.541828    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-v5ctf\" (UniqueName: \"kubernetes.io/projected/54d699c3-2780-4189-a57d-0fe9c44b801c-kube-api-access-v5ctf\") pod \"54d699c3-2780-4189-a57d-0fe9c44b801c\" (UID: \"54d699c3-2780-4189-a57d-0fe9c44b801c\") "
Sep 16 19:14:54.542064 ip-10-0-138-127 kubenswrapper[2581]: I0916 19:14:54.541876    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/54d699c3-2780-4189-a57d-0fe9c44b801c-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "54d699c3-2780-4189-a57d-0fe9c44b801c" (UID: "54d699c3-2780-4189-a57d-0fe9c44b801c"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:14:54.542120 ip-10-0-138-127 kubenswrapper[2581]: I0916 19:14:54.542064    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/54d699c3-2780-4189-a57d-0fe9c44b801c-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "54d699c3-2780-4189-a57d-0fe9c44b801c" (UID: "54d699c3-2780-4189-a57d-0fe9c44b801c"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 19:14:54.542218 ip-10-0-138-127 kubenswrapper[2581]: I0916 19:14:54.542179    2581 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/54d699c3-2780-4189-a57d-0fe9c44b801c-kube-rbac-proxy-sar-config\") on node \"ip-10-0-138-127.ec2.internal\" DevicePath \"\""
Sep 16 19:14:54.542218 ip-10-0-138-127 kubenswrapper[2581]: I0916 19:14:54.542220    2581 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/54d699c3-2780-4189-a57d-0fe9c44b801c-cabundle-cert\") on node \"ip-10-0-138-127.ec2.internal\" DevicePath \"\""
Sep 16 19:14:54.542218 ip-10-0-138-127 kubenswrapper[2581]: I0916 19:14:54.542236    2581 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/54d699c3-2780-4189-a57d-0fe9c44b801c-kserve-provision-location\") on node \"ip-10-0-138-127.ec2.internal\" DevicePath \"\""
Sep 16 19:14:54.542218 ip-10-0-138-127 kubenswrapper[2581]: E0916 19:14:54.542233    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e98c28b2762af10364084a327f959b31849d8c72f8beb5a2a0d04c3956c99ad4\": container with ID starting with e98c28b2762af10364084a327f959b31849d8c72f8beb5a2a0d04c3956c99ad4 not found: ID does not exist" containerID="e98c28b2762af10364084a327f959b31849d8c72f8beb5a2a0d04c3956c99ad4"
Sep 16 19:14:54.542592 ip-10-0-138-127 kubenswrapper[2581]: I0916 19:14:54.542268    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e98c28b2762af10364084a327f959b31849d8c72f8beb5a2a0d04c3956c99ad4"} err="failed to get container status \"e98c28b2762af10364084a327f959b31849d8c72f8beb5a2a0d04c3956c99ad4\": rpc error: code = NotFound desc = could not find container \"e98c28b2762af10364084a327f959b31849d8c72f8beb5a2a0d04c3956c99ad4\": container with ID starting with e98c28b2762af10364084a327f959b31849d8c72f8beb5a2a0d04c3956c99ad4 not found: ID does not exist"
Sep 16 19:14:54.542592 ip-10-0-138-127 kubenswrapper[2581]: I0916 19:14:54.542292    2581 scope.go:117] "RemoveContainer" containerID="14a5ff17ec44d8b511b0f993d8c32df930ece8f4217db2ff8bdb8329a72b2f34"
Sep 16 19:14:54.542592 ip-10-0-138-127 kubenswrapper[2581]: E0916 19:14:54.542553    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"14a5ff17ec44d8b511b0f993d8c32df930ece8f4217db2ff8bdb8329a72b2f34\": container with ID starting with 14a5ff17ec44d8b511b0f993d8c32df930ece8f4217db2ff8bdb8329a72b2f34 not found: ID does not exist" containerID="14a5ff17ec44d8b511b0f993d8c32df930ece8f4217db2ff8bdb8329a72b2f34"
Sep 16 19:14:54.542592 ip-10-0-138-127 kubenswrapper[2581]: I0916 19:14:54.542577    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"14a5ff17ec44d8b511b0f993d8c32df930ece8f4217db2ff8bdb8329a72b2f34"} err="failed to get container status \"14a5ff17ec44d8b511b0f993d8c32df930ece8f4217db2ff8bdb8329a72b2f34\": rpc error: code = NotFound desc = could not find container \"14a5ff17ec44d8b511b0f993d8c32df930ece8f4217db2ff8bdb8329a72b2f34\": container with ID starting with 14a5ff17ec44d8b511b0f993d8c32df930ece8f4217db2ff8bdb8329a72b2f34 not found: ID does not exist"
Sep 16 19:14:54.542592 ip-10-0-138-127 kubenswrapper[2581]: I0916 19:14:54.542595    2581 scope.go:117] "RemoveContainer" containerID="cb108b88271476f75e2f032bb4a2f056f7ffd2f72ea292ceaf4d1c05e2dc15f1"
Sep 16 19:14:54.542863 ip-10-0-138-127 kubenswrapper[2581]: E0916 19:14:54.542853    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cb108b88271476f75e2f032bb4a2f056f7ffd2f72ea292ceaf4d1c05e2dc15f1\": container with ID starting with cb108b88271476f75e2f032bb4a2f056f7ffd2f72ea292ceaf4d1c05e2dc15f1 not found: ID does not exist" containerID="cb108b88271476f75e2f032bb4a2f056f7ffd2f72ea292ceaf4d1c05e2dc15f1"
Sep 16 19:14:54.542926 ip-10-0-138-127 kubenswrapper[2581]: I0916 19:14:54.542869    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cb108b88271476f75e2f032bb4a2f056f7ffd2f72ea292ceaf4d1c05e2dc15f1"} err="failed to get container status \"cb108b88271476f75e2f032bb4a2f056f7ffd2f72ea292ceaf4d1c05e2dc15f1\": rpc error: code = NotFound desc = could not find container \"cb108b88271476f75e2f032bb4a2f056f7ffd2f72ea292ceaf4d1c05e2dc15f1\": container with ID starting with cb108b88271476f75e2f032bb4a2f056f7ffd2f72ea292ceaf4d1c05e2dc15f1 not found: ID does not exist"
Sep 16 19:14:54.543965 ip-10-0-138-127 kubenswrapper[2581]: I0916 19:14:54.543946    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/54d699c3-2780-4189-a57d-0fe9c44b801c-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "54d699c3-2780-4189-a57d-0fe9c44b801c" (UID: "54d699c3-2780-4189-a57d-0fe9c44b801c"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 19:14:54.544170 ip-10-0-138-127 kubenswrapper[2581]: I0916 19:14:54.544149    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/54d699c3-2780-4189-a57d-0fe9c44b801c-kube-api-access-v5ctf" (OuterVolumeSpecName: "kube-api-access-v5ctf") pod "54d699c3-2780-4189-a57d-0fe9c44b801c" (UID: "54d699c3-2780-4189-a57d-0fe9c44b801c"). InnerVolumeSpecName "kube-api-access-v5ctf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 19:14:54.643212 ip-10-0-138-127 kubenswrapper[2581]: I0916 19:14:54.643157    2581 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/54d699c3-2780-4189-a57d-0fe9c44b801c-proxy-tls\") on node \"ip-10-0-138-127.ec2.internal\" DevicePath \"\""
Sep 16 19:14:54.643212 ip-10-0-138-127 kubenswrapper[2581]: I0916 19:14:54.643188    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v5ctf\" (UniqueName: \"kubernetes.io/projected/54d699c3-2780-4189-a57d-0fe9c44b801c-kube-api-access-v5ctf\") on node \"ip-10-0-138-127.ec2.internal\" DevicePath \"\""
Sep 16 19:14:54.824452 ip-10-0-138-127 kubenswrapper[2581]: I0916 19:14:54.824412    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-7fd84f4ff-27rf6"]
Sep 16 19:14:54.829062 ip-10-0-138-127 kubenswrapper[2581]: I0916 19:14:54.829032    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-7fd84f4ff-27rf6"]
Sep 16 19:14:56.516723 ip-10-0-138-127 kubenswrapper[2581]: I0916 19:14:56.516689    2581 generic.go:356] "Generic (PLEG): container finished" podID="79a80f32-e7e8-4d05-8ea6-6a1a279afcda" containerID="0c9b1d6b3919aaed9214b0e625e017b430dd61ee6b84044a69ab29e5df7b2c68" exitCode=0
Sep 16 19:14:56.517142 ip-10-0-138-127 kubenswrapper[2581]: I0916 19:14:56.516760    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-jlscm/must-gather-gmkvj" event={"ID":"79a80f32-e7e8-4d05-8ea6-6a1a279afcda","Type":"ContainerDied","Data":"0c9b1d6b3919aaed9214b0e625e017b430dd61ee6b84044a69ab29e5df7b2c68"}
Sep 16 19:14:56.517142 ip-10-0-138-127 kubenswrapper[2581]: I0916 19:14:56.517097    2581 scope.go:117] "RemoveContainer" containerID="0c9b1d6b3919aaed9214b0e625e017b430dd61ee6b84044a69ab29e5df7b2c68"
Sep 16 19:14:56.728099 ip-10-0-138-127 kubenswrapper[2581]: I0916 19:14:56.728066    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="54d699c3-2780-4189-a57d-0fe9c44b801c" path="/var/lib/kubelet/pods/54d699c3-2780-4189-a57d-0fe9c44b801c/volumes"
Sep 16 19:14:56.822748 ip-10-0-138-127 kubenswrapper[2581]: I0916 19:14:56.822716    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-must-gather-jlscm_must-gather-gmkvj_79a80f32-e7e8-4d05-8ea6-6a1a279afcda/gather/0.log"
Sep 16 19:14:57.406270 ip-10-0-138-127 kubenswrapper[2581]: I0916 19:14:57.406237    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-ng22b/must-gather-s96ls"]
Sep 16 19:14:57.406570 ip-10-0-138-127 kubenswrapper[2581]: I0916 19:14:57.406551    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="54d699c3-2780-4189-a57d-0fe9c44b801c" containerName="storage-initializer"
Sep 16 19:14:57.406570 ip-10-0-138-127 kubenswrapper[2581]: I0916 19:14:57.406568    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="54d699c3-2780-4189-a57d-0fe9c44b801c" containerName="storage-initializer"
Sep 16 19:14:57.406736 ip-10-0-138-127 kubenswrapper[2581]: I0916 19:14:57.406587    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="54d699c3-2780-4189-a57d-0fe9c44b801c" containerName="agent"
Sep 16 19:14:57.406736 ip-10-0-138-127 kubenswrapper[2581]: I0916 19:14:57.406593    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="54d699c3-2780-4189-a57d-0fe9c44b801c" containerName="agent"
Sep 16 19:14:57.406736 ip-10-0-138-127 kubenswrapper[2581]: I0916 19:14:57.406603    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="54d699c3-2780-4189-a57d-0fe9c44b801c" containerName="kube-rbac-proxy"
Sep 16 19:14:57.406736 ip-10-0-138-127 kubenswrapper[2581]: I0916 19:14:57.406609    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="54d699c3-2780-4189-a57d-0fe9c44b801c" containerName="kube-rbac-proxy"
Sep 16 19:14:57.406736 ip-10-0-138-127 kubenswrapper[2581]: I0916 19:14:57.406615    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="54d699c3-2780-4189-a57d-0fe9c44b801c" containerName="kserve-container"
Sep 16 19:14:57.406736 ip-10-0-138-127 kubenswrapper[2581]: I0916 19:14:57.406620    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="54d699c3-2780-4189-a57d-0fe9c44b801c" containerName="kserve-container"
Sep 16 19:14:57.406736 ip-10-0-138-127 kubenswrapper[2581]: I0916 19:14:57.406671    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="54d699c3-2780-4189-a57d-0fe9c44b801c" containerName="kube-rbac-proxy"
Sep 16 19:14:57.406736 ip-10-0-138-127 kubenswrapper[2581]: I0916 19:14:57.406678    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="54d699c3-2780-4189-a57d-0fe9c44b801c" containerName="kserve-container"
Sep 16 19:14:57.406736 ip-10-0-138-127 kubenswrapper[2581]: I0916 19:14:57.406683    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="54d699c3-2780-4189-a57d-0fe9c44b801c" containerName="agent"
Sep 16 19:14:57.410714 ip-10-0-138-127 kubenswrapper[2581]: I0916 19:14:57.410698    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-ng22b/must-gather-s96ls"
Sep 16 19:14:57.413423 ip-10-0-138-127 kubenswrapper[2581]: I0916 19:14:57.413363    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-ng22b\"/\"kube-root-ca.crt\""
Sep 16 19:14:57.413423 ip-10-0-138-127 kubenswrapper[2581]: I0916 19:14:57.413392    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-ng22b\"/\"default-dockercfg-zthnt\""
Sep 16 19:14:57.413596 ip-10-0-138-127 kubenswrapper[2581]: I0916 19:14:57.413541    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-ng22b\"/\"openshift-service-ca.crt\""
Sep 16 19:14:57.419176 ip-10-0-138-127 kubenswrapper[2581]: I0916 19:14:57.419155    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-ng22b/must-gather-s96ls"]
Sep 16 19:14:57.467114 ip-10-0-138-127 kubenswrapper[2581]: I0916 19:14:57.467072    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/34831f4c-3444-474b-bc28-28cff8cbac8f-must-gather-output\") pod \"must-gather-s96ls\" (UID: \"34831f4c-3444-474b-bc28-28cff8cbac8f\") " pod="openshift-must-gather-ng22b/must-gather-s96ls"
Sep 16 19:14:57.467319 ip-10-0-138-127 kubenswrapper[2581]: I0916 19:14:57.467134    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vlb9v\" (UniqueName: \"kubernetes.io/projected/34831f4c-3444-474b-bc28-28cff8cbac8f-kube-api-access-vlb9v\") pod \"must-gather-s96ls\" (UID: \"34831f4c-3444-474b-bc28-28cff8cbac8f\") " pod="openshift-must-gather-ng22b/must-gather-s96ls"
Sep 16 19:14:57.567959 ip-10-0-138-127 kubenswrapper[2581]: I0916 19:14:57.567921    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vlb9v\" (UniqueName: \"kubernetes.io/projected/34831f4c-3444-474b-bc28-28cff8cbac8f-kube-api-access-vlb9v\") pod \"must-gather-s96ls\" (UID: \"34831f4c-3444-474b-bc28-28cff8cbac8f\") " pod="openshift-must-gather-ng22b/must-gather-s96ls"
Sep 16 19:14:57.568382 ip-10-0-138-127 kubenswrapper[2581]: I0916 19:14:57.567976    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/34831f4c-3444-474b-bc28-28cff8cbac8f-must-gather-output\") pod \"must-gather-s96ls\" (UID: \"34831f4c-3444-474b-bc28-28cff8cbac8f\") " pod="openshift-must-gather-ng22b/must-gather-s96ls"
Sep 16 19:14:57.568382 ip-10-0-138-127 kubenswrapper[2581]: I0916 19:14:57.568281    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/34831f4c-3444-474b-bc28-28cff8cbac8f-must-gather-output\") pod \"must-gather-s96ls\" (UID: \"34831f4c-3444-474b-bc28-28cff8cbac8f\") " pod="openshift-must-gather-ng22b/must-gather-s96ls"
Sep 16 19:14:57.577945 ip-10-0-138-127 kubenswrapper[2581]: I0916 19:14:57.577926    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vlb9v\" (UniqueName: \"kubernetes.io/projected/34831f4c-3444-474b-bc28-28cff8cbac8f-kube-api-access-vlb9v\") pod \"must-gather-s96ls\" (UID: \"34831f4c-3444-474b-bc28-28cff8cbac8f\") " pod="openshift-must-gather-ng22b/must-gather-s96ls"
Sep 16 19:14:57.720750 ip-10-0-138-127 kubenswrapper[2581]: I0916 19:14:57.720660    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-ng22b/must-gather-s96ls"
Sep 16 19:14:57.993803 ip-10-0-138-127 kubenswrapper[2581]: I0916 19:14:57.993776    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-ng22b/must-gather-s96ls"]
Sep 16 19:14:57.996328 ip-10-0-138-127 kubenswrapper[2581]: W0916 19:14:57.996300    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod34831f4c_3444_474b_bc28_28cff8cbac8f.slice/crio-2033ef6dce047999f749732f809068d184805d607b733bb483f2332a8230f711 WatchSource:0}: Error finding container 2033ef6dce047999f749732f809068d184805d607b733bb483f2332a8230f711: Status 404 returned error can't find the container with id 2033ef6dce047999f749732f809068d184805d607b733bb483f2332a8230f711
Sep 16 19:14:58.526825 ip-10-0-138-127 kubenswrapper[2581]: I0916 19:14:58.526785    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-ng22b/must-gather-s96ls" event={"ID":"34831f4c-3444-474b-bc28-28cff8cbac8f","Type":"ContainerStarted","Data":"2033ef6dce047999f749732f809068d184805d607b733bb483f2332a8230f711"}
Sep 16 19:14:59.533944 ip-10-0-138-127 kubenswrapper[2581]: I0916 19:14:59.533906    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-ng22b/must-gather-s96ls" event={"ID":"34831f4c-3444-474b-bc28-28cff8cbac8f","Type":"ContainerStarted","Data":"d8ba9ce39751de85fa90d0f096dd9c991f8880c24bc9e06c51a3e9e95d7cc184"}
Sep 16 19:14:59.534366 ip-10-0-138-127 kubenswrapper[2581]: I0916 19:14:59.533953    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-ng22b/must-gather-s96ls" event={"ID":"34831f4c-3444-474b-bc28-28cff8cbac8f","Type":"ContainerStarted","Data":"66c898b8f8b00051f41b884ba07ad9859e4ff98e2b44b5f5a56ca6b679ce6cd0"}
Sep 16 19:14:59.553382 ip-10-0-138-127 kubenswrapper[2581]: I0916 19:14:59.553323    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-ng22b/must-gather-s96ls" podStartSLOduration=1.789121351 podStartE2EDuration="2.553305792s" totalImagesPullingTime="764.184441ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 19:14:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:14:59.552529881 +0000 UTC m=+1769.371943791" watchObservedRunningTime="2026-09-16 19:14:59.553305792 +0000 UTC m=+1769.372719701"
Sep 16 19:15:00.333963 ip-10-0-138-127 kubenswrapper[2581]: I0916 19:15:00.333925    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-c78nz_f09babc9-6295-4a0a-be9f-fa0175b0de51/global-pull-secret-syncer/0.log"
Sep 16 19:15:00.557297 ip-10-0-138-127 kubenswrapper[2581]: I0916 19:15:00.557259    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-vttdl_b31b8735-005e-4387-852c-bc67df0e77cb/konnectivity-agent/0.log"
Sep 16 19:15:00.614825 ip-10-0-138-127 kubenswrapper[2581]: I0916 19:15:00.614741    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-138-127.ec2.internal_37df0dc0824bc6d3c2498649ba68632f/haproxy/0.log"
Sep 16 19:15:02.259324 ip-10-0-138-127 kubenswrapper[2581]: I0916 19:15:02.259270    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-must-gather-jlscm/must-gather-gmkvj"]
Sep 16 19:15:02.259841 ip-10-0-138-127 kubenswrapper[2581]: I0916 19:15:02.259614    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-must-gather-jlscm/must-gather-gmkvj" podUID="79a80f32-e7e8-4d05-8ea6-6a1a279afcda" containerName="copy" containerID="cri-o://8d9e7249fb785889b410ac1d4012e9d17d7fde364c0561c6ccc3aac29172165b" gracePeriod=2
Sep 16 19:15:02.262325 ip-10-0-138-127 kubenswrapper[2581]: E0916 19:15:02.262292    2581 status_manager.go:1018] "Failed to get status for pod" err="pods \"must-gather-gmkvj\" is forbidden: User \"system:node:ip-10-0-138-127.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"openshift-must-gather-jlscm\": no relationship found between node 'ip-10-0-138-127.ec2.internal' and this object" podUID="79a80f32-e7e8-4d05-8ea6-6a1a279afcda" pod="openshift-must-gather-jlscm/must-gather-gmkvj"
Sep 16 19:15:02.265742 ip-10-0-138-127 kubenswrapper[2581]: I0916 19:15:02.265719    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-must-gather-jlscm/must-gather-gmkvj"]
Sep 16 19:15:02.568150 ip-10-0-138-127 kubenswrapper[2581]: I0916 19:15:02.562439    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-must-gather-jlscm_must-gather-gmkvj_79a80f32-e7e8-4d05-8ea6-6a1a279afcda/copy/0.log"
Sep 16 19:15:02.568150 ip-10-0-138-127 kubenswrapper[2581]: I0916 19:15:02.562892    2581 generic.go:356] "Generic (PLEG): container finished" podID="79a80f32-e7e8-4d05-8ea6-6a1a279afcda" containerID="8d9e7249fb785889b410ac1d4012e9d17d7fde364c0561c6ccc3aac29172165b" exitCode=143
Sep 16 19:15:02.636763 ip-10-0-138-127 kubenswrapper[2581]: I0916 19:15:02.636724    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-must-gather-jlscm_must-gather-gmkvj_79a80f32-e7e8-4d05-8ea6-6a1a279afcda/copy/0.log"
Sep 16 19:15:02.643374 ip-10-0-138-127 kubenswrapper[2581]: I0916 19:15:02.643340    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-jlscm/must-gather-gmkvj"
Sep 16 19:15:02.645556 ip-10-0-138-127 kubenswrapper[2581]: E0916 19:15:02.645495    2581 status_manager.go:1018] "Failed to get status for pod" err="pods \"must-gather-gmkvj\" is forbidden: User \"system:node:ip-10-0-138-127.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"openshift-must-gather-jlscm\": no relationship found between node 'ip-10-0-138-127.ec2.internal' and this object" podUID="79a80f32-e7e8-4d05-8ea6-6a1a279afcda" pod="openshift-must-gather-jlscm/must-gather-gmkvj"
Sep 16 19:15:02.728107 ip-10-0-138-127 kubenswrapper[2581]: I0916 19:15:02.728080    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/79a80f32-e7e8-4d05-8ea6-6a1a279afcda-must-gather-output\") pod \"79a80f32-e7e8-4d05-8ea6-6a1a279afcda\" (UID: \"79a80f32-e7e8-4d05-8ea6-6a1a279afcda\") "
Sep 16 19:15:02.731745 ip-10-0-138-127 kubenswrapper[2581]: I0916 19:15:02.731558    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/79a80f32-e7e8-4d05-8ea6-6a1a279afcda-must-gather-output" (OuterVolumeSpecName: "must-gather-output") pod "79a80f32-e7e8-4d05-8ea6-6a1a279afcda" (UID: "79a80f32-e7e8-4d05-8ea6-6a1a279afcda"). InnerVolumeSpecName "must-gather-output". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:15:02.731964 ip-10-0-138-127 kubenswrapper[2581]: I0916 19:15:02.731704    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-z7x2r\" (UniqueName: \"kubernetes.io/projected/79a80f32-e7e8-4d05-8ea6-6a1a279afcda-kube-api-access-z7x2r\") pod \"79a80f32-e7e8-4d05-8ea6-6a1a279afcda\" (UID: \"79a80f32-e7e8-4d05-8ea6-6a1a279afcda\") "
Sep 16 19:15:02.732854 ip-10-0-138-127 kubenswrapper[2581]: I0916 19:15:02.732821    2581 reconciler_common.go:299] "Volume detached for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/79a80f32-e7e8-4d05-8ea6-6a1a279afcda-must-gather-output\") on node \"ip-10-0-138-127.ec2.internal\" DevicePath \"\""
Sep 16 19:15:02.735481 ip-10-0-138-127 kubenswrapper[2581]: I0916 19:15:02.735112    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/79a80f32-e7e8-4d05-8ea6-6a1a279afcda-kube-api-access-z7x2r" (OuterVolumeSpecName: "kube-api-access-z7x2r") pod "79a80f32-e7e8-4d05-8ea6-6a1a279afcda" (UID: "79a80f32-e7e8-4d05-8ea6-6a1a279afcda"). InnerVolumeSpecName "kube-api-access-z7x2r". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 19:15:02.834945 ip-10-0-138-127 kubenswrapper[2581]: I0916 19:15:02.834060    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z7x2r\" (UniqueName: \"kubernetes.io/projected/79a80f32-e7e8-4d05-8ea6-6a1a279afcda-kube-api-access-z7x2r\") on node \"ip-10-0-138-127.ec2.internal\" DevicePath \"\""
Sep 16 19:15:03.571059 ip-10-0-138-127 kubenswrapper[2581]: I0916 19:15:03.571024    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-must-gather-jlscm_must-gather-gmkvj_79a80f32-e7e8-4d05-8ea6-6a1a279afcda/copy/0.log"
Sep 16 19:15:03.571725 ip-10-0-138-127 kubenswrapper[2581]: I0916 19:15:03.571565    2581 scope.go:117] "RemoveContainer" containerID="8d9e7249fb785889b410ac1d4012e9d17d7fde364c0561c6ccc3aac29172165b"
Sep 16 19:15:03.571816 ip-10-0-138-127 kubenswrapper[2581]: I0916 19:15:03.571731    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-jlscm/must-gather-gmkvj"
Sep 16 19:15:03.594506 ip-10-0-138-127 kubenswrapper[2581]: I0916 19:15:03.594432    2581 scope.go:117] "RemoveContainer" containerID="0c9b1d6b3919aaed9214b0e625e017b430dd61ee6b84044a69ab29e5df7b2c68"
Sep 16 19:15:04.249788 ip-10-0-138-127 kubenswrapper[2581]: I0916 19:15:04.249724    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_f3f31704-4b1f-4604-aac3-6cb89d0f642e/alertmanager/0.log"
Sep 16 19:15:04.303657 ip-10-0-138-127 kubenswrapper[2581]: I0916 19:15:04.303622    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_f3f31704-4b1f-4604-aac3-6cb89d0f642e/config-reloader/0.log"
Sep 16 19:15:04.404926 ip-10-0-138-127 kubenswrapper[2581]: I0916 19:15:04.404754    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_f3f31704-4b1f-4604-aac3-6cb89d0f642e/kube-rbac-proxy-web/0.log"
Sep 16 19:15:04.470697 ip-10-0-138-127 kubenswrapper[2581]: I0916 19:15:04.470635    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_f3f31704-4b1f-4604-aac3-6cb89d0f642e/kube-rbac-proxy/0.log"
Sep 16 19:15:04.522290 ip-10-0-138-127 kubenswrapper[2581]: I0916 19:15:04.522259    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_f3f31704-4b1f-4604-aac3-6cb89d0f642e/kube-rbac-proxy-metric/0.log"
Sep 16 19:15:04.583411 ip-10-0-138-127 kubenswrapper[2581]: I0916 19:15:04.583371    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_f3f31704-4b1f-4604-aac3-6cb89d0f642e/prom-label-proxy/0.log"
Sep 16 19:15:04.647257 ip-10-0-138-127 kubenswrapper[2581]: I0916 19:15:04.636975    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_f3f31704-4b1f-4604-aac3-6cb89d0f642e/init-config-reloader/0.log"
Sep 16 19:15:04.731302 ip-10-0-138-127 kubenswrapper[2581]: I0916 19:15:04.731263    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="79a80f32-e7e8-4d05-8ea6-6a1a279afcda" path="/var/lib/kubelet/pods/79a80f32-e7e8-4d05-8ea6-6a1a279afcda/volumes"
Sep 16 19:15:04.909383 ip-10-0-138-127 kubenswrapper[2581]: I0916 19:15:04.909295    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_metrics-server-5c9d8b5484-7hpps_df30013a-07d7-491e-b50a-4c208c6ce24d/metrics-server/0.log"
Sep 16 19:15:04.954481 ip-10-0-138-127 kubenswrapper[2581]: I0916 19:15:04.954448    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_monitoring-plugin-5d85778b9d-h492n_511bd723-e9cb-4806-b954-bf03f6b23d9d/monitoring-plugin/0.log"
Sep 16 19:15:05.126268 ip-10-0-138-127 kubenswrapper[2581]: I0916 19:15:05.126169    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-hsp8v_b5732a85-64c1-4cf8-b80a-c06f8d0a4b3b/node-exporter/0.log"
Sep 16 19:15:05.168235 ip-10-0-138-127 kubenswrapper[2581]: I0916 19:15:05.168188    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-hsp8v_b5732a85-64c1-4cf8-b80a-c06f8d0a4b3b/kube-rbac-proxy/0.log"
Sep 16 19:15:05.206894 ip-10-0-138-127 kubenswrapper[2581]: I0916 19:15:05.206863    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-hsp8v_b5732a85-64c1-4cf8-b80a-c06f8d0a4b3b/init-textfile/0.log"
Sep 16 19:15:05.345701 ip-10-0-138-127 kubenswrapper[2581]: I0916 19:15:05.345664    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-67dcf8c979-4hdsw_dc53a5ab-d957-42f4-9eb7-a3c38c3ca302/kube-rbac-proxy-main/0.log"
Sep 16 19:15:05.385905 ip-10-0-138-127 kubenswrapper[2581]: I0916 19:15:05.385823    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-67dcf8c979-4hdsw_dc53a5ab-d957-42f4-9eb7-a3c38c3ca302/kube-rbac-proxy-self/0.log"
Sep 16 19:15:05.425312 ip-10-0-138-127 kubenswrapper[2581]: I0916 19:15:05.425282    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-67dcf8c979-4hdsw_dc53a5ab-d957-42f4-9eb7-a3c38c3ca302/openshift-state-metrics/0.log"
Sep 16 19:15:05.471260 ip-10-0-138-127 kubenswrapper[2581]: I0916 19:15:05.471229    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_4629d3f4-54ea-4bdf-9b75-650bbc2299d6/prometheus/0.log"
Sep 16 19:15:05.500716 ip-10-0-138-127 kubenswrapper[2581]: I0916 19:15:05.500662    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_4629d3f4-54ea-4bdf-9b75-650bbc2299d6/config-reloader/0.log"
Sep 16 19:15:05.529752 ip-10-0-138-127 kubenswrapper[2581]: I0916 19:15:05.529722    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_4629d3f4-54ea-4bdf-9b75-650bbc2299d6/thanos-sidecar/0.log"
Sep 16 19:15:05.561216 ip-10-0-138-127 kubenswrapper[2581]: I0916 19:15:05.561169    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_4629d3f4-54ea-4bdf-9b75-650bbc2299d6/kube-rbac-proxy-web/0.log"
Sep 16 19:15:05.603787 ip-10-0-138-127 kubenswrapper[2581]: I0916 19:15:05.603698    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_4629d3f4-54ea-4bdf-9b75-650bbc2299d6/kube-rbac-proxy/0.log"
Sep 16 19:15:05.637318 ip-10-0-138-127 kubenswrapper[2581]: I0916 19:15:05.637231    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_4629d3f4-54ea-4bdf-9b75-650bbc2299d6/kube-rbac-proxy-thanos/0.log"
Sep 16 19:15:05.669140 ip-10-0-138-127 kubenswrapper[2581]: I0916 19:15:05.669104    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_4629d3f4-54ea-4bdf-9b75-650bbc2299d6/init-config-reloader/0.log"
Sep 16 19:15:05.809498 ip-10-0-138-127 kubenswrapper[2581]: I0916 19:15:05.809464    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-6f6cf8bc4f-phh6w_2f040fb2-a2fb-432b-ae79-d71cb4c69847/telemeter-client/0.log"
Sep 16 19:15:05.841284 ip-10-0-138-127 kubenswrapper[2581]: I0916 19:15:05.841246    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-6f6cf8bc4f-phh6w_2f040fb2-a2fb-432b-ae79-d71cb4c69847/reload/0.log"
Sep 16 19:15:05.873154 ip-10-0-138-127 kubenswrapper[2581]: I0916 19:15:05.873127    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-6f6cf8bc4f-phh6w_2f040fb2-a2fb-432b-ae79-d71cb4c69847/kube-rbac-proxy/0.log"
Sep 16 19:15:08.455908 ip-10-0-138-127 kubenswrapper[2581]: I0916 19:15:08.455872    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_downloads-76f99584f4-67c8w_ddcab39a-92a7-4106-927a-8160c999373c/download-server/0.log"
Sep 16 19:15:08.769579 ip-10-0-138-127 kubenswrapper[2581]: I0916 19:15:08.769547    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-ng22b/perf-node-gather-daemonset-zjfqr"]
Sep 16 19:15:08.769908 ip-10-0-138-127 kubenswrapper[2581]: I0916 19:15:08.769894    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="79a80f32-e7e8-4d05-8ea6-6a1a279afcda" containerName="gather"
Sep 16 19:15:08.769962 ip-10-0-138-127 kubenswrapper[2581]: I0916 19:15:08.769911    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="79a80f32-e7e8-4d05-8ea6-6a1a279afcda" containerName="gather"
Sep 16 19:15:08.769962 ip-10-0-138-127 kubenswrapper[2581]: I0916 19:15:08.769930    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="79a80f32-e7e8-4d05-8ea6-6a1a279afcda" containerName="copy"
Sep 16 19:15:08.769962 ip-10-0-138-127 kubenswrapper[2581]: I0916 19:15:08.769935    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="79a80f32-e7e8-4d05-8ea6-6a1a279afcda" containerName="copy"
Sep 16 19:15:08.770055 ip-10-0-138-127 kubenswrapper[2581]: I0916 19:15:08.769990    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="79a80f32-e7e8-4d05-8ea6-6a1a279afcda" containerName="copy"
Sep 16 19:15:08.770055 ip-10-0-138-127 kubenswrapper[2581]: I0916 19:15:08.770003    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="79a80f32-e7e8-4d05-8ea6-6a1a279afcda" containerName="gather"
Sep 16 19:15:08.774894 ip-10-0-138-127 kubenswrapper[2581]: I0916 19:15:08.774855    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-ng22b/perf-node-gather-daemonset-zjfqr"
Sep 16 19:15:08.787004 ip-10-0-138-127 kubenswrapper[2581]: I0916 19:15:08.786970    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-ng22b/perf-node-gather-daemonset-zjfqr"]
Sep 16 19:15:08.895049 ip-10-0-138-127 kubenswrapper[2581]: I0916 19:15:08.895013    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/b5d3d720-14e1-4612-a173-5b92a80474d3-sys\") pod \"perf-node-gather-daemonset-zjfqr\" (UID: \"b5d3d720-14e1-4612-a173-5b92a80474d3\") " pod="openshift-must-gather-ng22b/perf-node-gather-daemonset-zjfqr"
Sep 16 19:15:08.895049 ip-10-0-138-127 kubenswrapper[2581]: I0916 19:15:08.895054    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/b5d3d720-14e1-4612-a173-5b92a80474d3-podres\") pod \"perf-node-gather-daemonset-zjfqr\" (UID: \"b5d3d720-14e1-4612-a173-5b92a80474d3\") " pod="openshift-must-gather-ng22b/perf-node-gather-daemonset-zjfqr"
Sep 16 19:15:08.895336 ip-10-0-138-127 kubenswrapper[2581]: I0916 19:15:08.895100    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8l7hr\" (UniqueName: \"kubernetes.io/projected/b5d3d720-14e1-4612-a173-5b92a80474d3-kube-api-access-8l7hr\") pod \"perf-node-gather-daemonset-zjfqr\" (UID: \"b5d3d720-14e1-4612-a173-5b92a80474d3\") " pod="openshift-must-gather-ng22b/perf-node-gather-daemonset-zjfqr"
Sep 16 19:15:08.895336 ip-10-0-138-127 kubenswrapper[2581]: I0916 19:15:08.895149    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/b5d3d720-14e1-4612-a173-5b92a80474d3-lib-modules\") pod \"perf-node-gather-daemonset-zjfqr\" (UID: \"b5d3d720-14e1-4612-a173-5b92a80474d3\") " pod="openshift-must-gather-ng22b/perf-node-gather-daemonset-zjfqr"
Sep 16 19:15:08.895336 ip-10-0-138-127 kubenswrapper[2581]: I0916 19:15:08.895240    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/b5d3d720-14e1-4612-a173-5b92a80474d3-proc\") pod \"perf-node-gather-daemonset-zjfqr\" (UID: \"b5d3d720-14e1-4612-a173-5b92a80474d3\") " pod="openshift-must-gather-ng22b/perf-node-gather-daemonset-zjfqr"
Sep 16 19:15:08.996519 ip-10-0-138-127 kubenswrapper[2581]: I0916 19:15:08.996484    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/b5d3d720-14e1-4612-a173-5b92a80474d3-proc\") pod \"perf-node-gather-daemonset-zjfqr\" (UID: \"b5d3d720-14e1-4612-a173-5b92a80474d3\") " pod="openshift-must-gather-ng22b/perf-node-gather-daemonset-zjfqr"
Sep 16 19:15:08.996715 ip-10-0-138-127 kubenswrapper[2581]: I0916 19:15:08.996649    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/b5d3d720-14e1-4612-a173-5b92a80474d3-proc\") pod \"perf-node-gather-daemonset-zjfqr\" (UID: \"b5d3d720-14e1-4612-a173-5b92a80474d3\") " pod="openshift-must-gather-ng22b/perf-node-gather-daemonset-zjfqr"
Sep 16 19:15:08.996715 ip-10-0-138-127 kubenswrapper[2581]: I0916 19:15:08.996668    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/b5d3d720-14e1-4612-a173-5b92a80474d3-sys\") pod \"perf-node-gather-daemonset-zjfqr\" (UID: \"b5d3d720-14e1-4612-a173-5b92a80474d3\") " pod="openshift-must-gather-ng22b/perf-node-gather-daemonset-zjfqr"
Sep 16 19:15:08.996715 ip-10-0-138-127 kubenswrapper[2581]: I0916 19:15:08.996703    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/b5d3d720-14e1-4612-a173-5b92a80474d3-podres\") pod \"perf-node-gather-daemonset-zjfqr\" (UID: \"b5d3d720-14e1-4612-a173-5b92a80474d3\") " pod="openshift-must-gather-ng22b/perf-node-gather-daemonset-zjfqr"
Sep 16 19:15:08.996886 ip-10-0-138-127 kubenswrapper[2581]: I0916 19:15:08.996753    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8l7hr\" (UniqueName: \"kubernetes.io/projected/b5d3d720-14e1-4612-a173-5b92a80474d3-kube-api-access-8l7hr\") pod \"perf-node-gather-daemonset-zjfqr\" (UID: \"b5d3d720-14e1-4612-a173-5b92a80474d3\") " pod="openshift-must-gather-ng22b/perf-node-gather-daemonset-zjfqr"
Sep 16 19:15:08.996886 ip-10-0-138-127 kubenswrapper[2581]: I0916 19:15:08.996766    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/b5d3d720-14e1-4612-a173-5b92a80474d3-sys\") pod \"perf-node-gather-daemonset-zjfqr\" (UID: \"b5d3d720-14e1-4612-a173-5b92a80474d3\") " pod="openshift-must-gather-ng22b/perf-node-gather-daemonset-zjfqr"
Sep 16 19:15:08.996886 ip-10-0-138-127 kubenswrapper[2581]: I0916 19:15:08.996784    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/b5d3d720-14e1-4612-a173-5b92a80474d3-lib-modules\") pod \"perf-node-gather-daemonset-zjfqr\" (UID: \"b5d3d720-14e1-4612-a173-5b92a80474d3\") " pod="openshift-must-gather-ng22b/perf-node-gather-daemonset-zjfqr"
Sep 16 19:15:08.996886 ip-10-0-138-127 kubenswrapper[2581]: I0916 19:15:08.996866    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/b5d3d720-14e1-4612-a173-5b92a80474d3-podres\") pod \"perf-node-gather-daemonset-zjfqr\" (UID: \"b5d3d720-14e1-4612-a173-5b92a80474d3\") " pod="openshift-must-gather-ng22b/perf-node-gather-daemonset-zjfqr"
Sep 16 19:15:08.997061 ip-10-0-138-127 kubenswrapper[2581]: I0916 19:15:08.996931    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/b5d3d720-14e1-4612-a173-5b92a80474d3-lib-modules\") pod \"perf-node-gather-daemonset-zjfqr\" (UID: \"b5d3d720-14e1-4612-a173-5b92a80474d3\") " pod="openshift-must-gather-ng22b/perf-node-gather-daemonset-zjfqr"
Sep 16 19:15:09.006447 ip-10-0-138-127 kubenswrapper[2581]: I0916 19:15:09.006419    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8l7hr\" (UniqueName: \"kubernetes.io/projected/b5d3d720-14e1-4612-a173-5b92a80474d3-kube-api-access-8l7hr\") pod \"perf-node-gather-daemonset-zjfqr\" (UID: \"b5d3d720-14e1-4612-a173-5b92a80474d3\") " pod="openshift-must-gather-ng22b/perf-node-gather-daemonset-zjfqr"
Sep 16 19:15:09.088253 ip-10-0-138-127 kubenswrapper[2581]: I0916 19:15:09.088147    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-ng22b/perf-node-gather-daemonset-zjfqr"
Sep 16 19:15:09.381434 ip-10-0-138-127 kubenswrapper[2581]: I0916 19:15:09.381402    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-ng22b/perf-node-gather-daemonset-zjfqr"]
Sep 16 19:15:09.382654 ip-10-0-138-127 kubenswrapper[2581]: W0916 19:15:09.382628    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-podb5d3d720_14e1_4612_a173_5b92a80474d3.slice/crio-5dcd83189438aa9d957aa541b4fcce71d7b04a16df61eadebca687df1c87acba WatchSource:0}: Error finding container 5dcd83189438aa9d957aa541b4fcce71d7b04a16df61eadebca687df1c87acba: Status 404 returned error can't find the container with id 5dcd83189438aa9d957aa541b4fcce71d7b04a16df61eadebca687df1c87acba
Sep 16 19:15:09.616240 ip-10-0-138-127 kubenswrapper[2581]: I0916 19:15:09.616189    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-ng22b/perf-node-gather-daemonset-zjfqr" event={"ID":"b5d3d720-14e1-4612-a173-5b92a80474d3","Type":"ContainerStarted","Data":"11bee28c316cca053d6ddf989b9750528394d245097b735d60736c21dc2e4a53"}
Sep 16 19:15:09.616723 ip-10-0-138-127 kubenswrapper[2581]: I0916 19:15:09.616272    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-ng22b/perf-node-gather-daemonset-zjfqr" event={"ID":"b5d3d720-14e1-4612-a173-5b92a80474d3","Type":"ContainerStarted","Data":"5dcd83189438aa9d957aa541b4fcce71d7b04a16df61eadebca687df1c87acba"}
Sep 16 19:15:09.616723 ip-10-0-138-127 kubenswrapper[2581]: I0916 19:15:09.616325    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-ng22b/perf-node-gather-daemonset-zjfqr"
Sep 16 19:15:09.654216 ip-10-0-138-127 kubenswrapper[2581]: I0916 19:15:09.654079    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-ng22b/perf-node-gather-daemonset-zjfqr" podStartSLOduration=1.654059131 podStartE2EDuration="1.654059131s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-16 19:15:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:15:09.651079326 +0000 UTC m=+1779.470493235" watchObservedRunningTime="2026-09-16 19:15:09.654059131 +0000 UTC m=+1779.473473041"
Sep 16 19:15:10.151724 ip-10-0-138-127 kubenswrapper[2581]: I0916 19:15:10.151694    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-qsxrs_61521525-706f-428c-80f6-2cdfcb30fdc4/dns/0.log"
Sep 16 19:15:10.180825 ip-10-0-138-127 kubenswrapper[2581]: I0916 19:15:10.180797    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-qsxrs_61521525-706f-428c-80f6-2cdfcb30fdc4/kube-rbac-proxy/0.log"
Sep 16 19:15:10.206741 ip-10-0-138-127 kubenswrapper[2581]: I0916 19:15:10.206710    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-lrclz_eace7050-d66d-4057-9041-62afa5df8ae7/dns-node-resolver/0.log"
Sep 16 19:15:10.785832 ip-10-0-138-127 kubenswrapper[2581]: I0916 19:15:10.785769    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-mvhf8_99b45db8-0fbb-445b-be31-49fba3f220ec/node-ca/0.log"
Sep 16 19:15:11.989819 ip-10-0-138-127 kubenswrapper[2581]: I0916 19:15:11.989784    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-z69gl_60f08a67-da8a-40c3-a0b9-93d7668c7da1/serve-healthcheck-canary/0.log"
Sep 16 19:15:12.476018 ip-10-0-138-127 kubenswrapper[2581]: I0916 19:15:12.475975    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-c2dhr_6e068f1b-74ad-44e1-aee0-5307a9eae059/kube-rbac-proxy/0.log"
Sep 16 19:15:12.505813 ip-10-0-138-127 kubenswrapper[2581]: I0916 19:15:12.505780    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-c2dhr_6e068f1b-74ad-44e1-aee0-5307a9eae059/exporter/0.log"
Sep 16 19:15:12.537559 ip-10-0-138-127 kubenswrapper[2581]: I0916 19:15:12.537526    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-c2dhr_6e068f1b-74ad-44e1-aee0-5307a9eae059/extractor/0.log"
Sep 16 19:15:15.322013 ip-10-0-138-127 kubenswrapper[2581]: I0916 19:15:15.321981    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_s3-init-f6mqj_c9b793f9-ac61-43c7-a947-06ca75687644/s3-init/0.log"
Sep 16 19:15:15.347181 ip-10-0-138-127 kubenswrapper[2581]: I0916 19:15:15.347147    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_s3-init-f6mqj_c9b793f9-ac61-43c7-a947-06ca75687644/download-hf-model/0.log"
Sep 16 19:15:15.632760 ip-10-0-138-127 kubenswrapper[2581]: I0916 19:15:15.632681    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-ng22b/perf-node-gather-daemonset-zjfqr"
Sep 16 19:15:20.125747 ip-10-0-138-127 kubenswrapper[2581]: I0916 19:15:20.125712    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-8b4d697c5-7s7hs_39b80a07-c65f-4e50-88e9-a5269e8c6d66/migrator/0.log"
Sep 16 19:15:20.158265 ip-10-0-138-127 kubenswrapper[2581]: I0916 19:15:20.158144    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-8b4d697c5-7s7hs_39b80a07-c65f-4e50-88e9-a5269e8c6d66/graceful-termination/0.log"
Sep 16 19:15:20.558747 ip-10-0-138-127 kubenswrapper[2581]: I0916 19:15:20.558711    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator-operator_kube-storage-version-migrator-operator-6c8f84f897-dd6d5_f3b6e193-3e4d-4dea-9909-b9eb2040ce3d/kube-storage-version-migrator-operator/1.log"
Sep 16 19:15:20.559430 ip-10-0-138-127 kubenswrapper[2581]: I0916 19:15:20.559401    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator-operator_kube-storage-version-migrator-operator-6c8f84f897-dd6d5_f3b6e193-3e4d-4dea-9909-b9eb2040ce3d/kube-storage-version-migrator-operator/0.log"
Sep 16 19:15:21.578312 ip-10-0-138-127 kubenswrapper[2581]: I0916 19:15:21.578280    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-fzt7c_5acb5489-33eb-4c23-b35d-484caffde098/kube-multus-additional-cni-plugins/0.log"
Sep 16 19:15:21.604273 ip-10-0-138-127 kubenswrapper[2581]: I0916 19:15:21.604229    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-fzt7c_5acb5489-33eb-4c23-b35d-484caffde098/egress-router-binary-copy/0.log"
Sep 16 19:15:21.632169 ip-10-0-138-127 kubenswrapper[2581]: I0916 19:15:21.632140    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-fzt7c_5acb5489-33eb-4c23-b35d-484caffde098/cni-plugins/0.log"
Sep 16 19:15:21.662060 ip-10-0-138-127 kubenswrapper[2581]: I0916 19:15:21.662028    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-fzt7c_5acb5489-33eb-4c23-b35d-484caffde098/bond-cni-plugin/0.log"
Sep 16 19:15:21.688726 ip-10-0-138-127 kubenswrapper[2581]: I0916 19:15:21.688701    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-fzt7c_5acb5489-33eb-4c23-b35d-484caffde098/routeoverride-cni/0.log"
Sep 16 19:15:21.714880 ip-10-0-138-127 kubenswrapper[2581]: I0916 19:15:21.714854    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-fzt7c_5acb5489-33eb-4c23-b35d-484caffde098/whereabouts-cni-bincopy/0.log"
Sep 16 19:15:21.745226 ip-10-0-138-127 kubenswrapper[2581]: I0916 19:15:21.745182    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-fzt7c_5acb5489-33eb-4c23-b35d-484caffde098/whereabouts-cni/0.log"
Sep 16 19:15:22.265754 ip-10-0-138-127 kubenswrapper[2581]: I0916 19:15:22.265724    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-vxjzx_bf1c06c8-20b0-4816-a092-02e87cd69a7f/kube-multus/0.log"
Sep 16 19:15:22.376787 ip-10-0-138-127 kubenswrapper[2581]: I0916 19:15:22.376759    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-c2kp2_eaf5897a-76a0-4bf3-aa4e-59aa6c193ddb/network-metrics-daemon/0.log"
Sep 16 19:15:22.406754 ip-10-0-138-127 kubenswrapper[2581]: I0916 19:15:22.406721    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-c2kp2_eaf5897a-76a0-4bf3-aa4e-59aa6c193ddb/kube-rbac-proxy/0.log"
Sep 16 19:15:23.353601 ip-10-0-138-127 kubenswrapper[2581]: I0916 19:15:23.353548    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-csmcl_99e923fa-5e08-487f-ada3-1692ef20ca11/ovn-controller/0.log"
Sep 16 19:15:23.382684 ip-10-0-138-127 kubenswrapper[2581]: I0916 19:15:23.382651    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-csmcl_99e923fa-5e08-487f-ada3-1692ef20ca11/ovn-acl-logging/0.log"
Sep 16 19:15:23.419831 ip-10-0-138-127 kubenswrapper[2581]: I0916 19:15:23.419805    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-csmcl_99e923fa-5e08-487f-ada3-1692ef20ca11/kube-rbac-proxy-node/0.log"
Sep 16 19:15:23.447992 ip-10-0-138-127 kubenswrapper[2581]: I0916 19:15:23.447962    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-csmcl_99e923fa-5e08-487f-ada3-1692ef20ca11/kube-rbac-proxy-ovn-metrics/0.log"
Sep 16 19:15:23.476108 ip-10-0-138-127 kubenswrapper[2581]: I0916 19:15:23.476076    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-csmcl_99e923fa-5e08-487f-ada3-1692ef20ca11/northd/0.log"
Sep 16 19:15:23.508063 ip-10-0-138-127 kubenswrapper[2581]: I0916 19:15:23.508036    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-csmcl_99e923fa-5e08-487f-ada3-1692ef20ca11/nbdb/0.log"
Sep 16 19:15:23.537389 ip-10-0-138-127 kubenswrapper[2581]: I0916 19:15:23.537362    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-csmcl_99e923fa-5e08-487f-ada3-1692ef20ca11/sbdb/0.log"
Sep 16 19:15:23.599649 ip-10-0-138-127 kubenswrapper[2581]: I0916 19:15:23.599618    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-csmcl_99e923fa-5e08-487f-ada3-1692ef20ca11/ovnkube-controller/0.log"
Sep 16 19:15:25.214871 ip-10-0-138-127 kubenswrapper[2581]: I0916 19:15:25.214832    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-diagnostics_network-check-target-bz5jk_4775e8c4-09ba-43db-98b7-073aeb452ce8/network-check-target-container/0.log"